2023-10-11T16:01:44.259 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-11T16:01:44.260 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} --name choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi --archive /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971 --verbose --owner scheduled_choffman@teuthology -- /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/orig.config.yaml 2023-10-11T16:01:44.319 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-11T16:01:44.404 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971 branch: wip-choffman-62793 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} email: choffman@redhat.com first_in_suite: false job_id: '7421971' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-choffman-62793 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 05a43c8fb3988914c879a50970f9054a098e0612 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 05a43c8fb3988914c879a50970f9054a098e0612 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-testing-62793 sha1: 3f7a9d091654926539d989dd4c83dfee517abc9e owner: scheduled_choffman@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5030 sha1: 05a43c8fb3988914c879a50970f9054a098e0612 sleep_before_teardown: 0 suite: fs:workload suite_branch: wip-testing-62793 suite_path: /home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa suite_relpath: qa suite_repo: https://github.com/chrisphoffman/ceph.git suite_sha1: 3f7a9d091654926539d989dd4c83dfee517abc9e syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-11_15:53:20 tube: smithi user: choffman verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-11T16:01:44.405 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa; will attempt to use it 2023-10-11T16:01:44.406 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa/tasks 2023-10-11T16:01:44.406 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-11T16:01:44.408 INFO:teuthology.task.internal:Checking packages... 2023-10-11T16:01:44.433 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '05a43c8fb3988914c879a50970f9054a098e0612' 2023-10-11T16:01:44.433 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-11T16:01:44.434 INFO:teuthology.packaging:ref: None 2023-10-11T16:01:44.434 INFO:teuthology.packaging:tag: None 2023-10-11T16:01:44.434 INFO:teuthology.packaging:branch: wip-choffman-62793 2023-10-11T16:01:44.435 INFO:teuthology.packaging:sha1: 05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:01:44.435 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-choffman-62793 2023-10-11T16:01:44.582 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3908.g05a43c8f 2023-10-11T16:01:44.582 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-11T16:01:44.589 INFO:teuthology.task.internal:no buildpackages task found 2023-10-11T16:01:44.589 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-11T16:01:44.626 INFO:teuthology.task.internal:Saving configuration 2023-10-11T16:01:44.655 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-11T16:01:44.661 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T16:01:44.688 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 15:56:09.277490', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T16:01:44.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 15:56:09.279473', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T16:01:44.734 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 15:56:09.281207', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T16:01:44.735 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-11T16:01:44.740 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-10-11T16:01:44.741 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-10-11T16:01:44.741 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-10-11T16:01:44.741 INFO:teuthology.run_tasks:Running task console_log... 2023-10-11T16:01:44.885 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f15ea3dbca0>, signals=[15]) 2023-10-11T16:01:44.886 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-11T16:01:44.892 INFO:teuthology.task.internal:Opening connections... 2023-10-11T16:01:44.892 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:01:44.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:01:44.960 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2023-10-11T16:01:44.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:01:45.029 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-10-11T16:01:45.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:01:45.099 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-11T16:01:45.105 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2023-10-11T16:01:45.127 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2023-10-11T16:01:45.127 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2023-10-11T16:01:45.183 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="Red Hat Enterprise Linux" 2023-10-11T16:01:45.184 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8.6 (Ootpa)" 2023-10-11T16:01:45.184 INFO:teuthology.orchestra.run.smithi111.stdout:ID="rhel" 2023-10-11T16:01:45.184 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="fedora" 2023-10-11T16:01:45.185 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8.6" 2023-10-11T16:01:45.185 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8" 2023-10-11T16:01:45.185 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-11T16:01:45.185 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2023-10-11T16:01:45.185 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-11T16:01:45.186 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://www.redhat.com/" 2023-10-11T16:01:45.186 INFO:teuthology.orchestra.run.smithi111.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-11T16:01:45.186 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-11T16:01:45.186 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:01:45.186 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-11T16:01:45.187 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-11T16:01:45.187 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-11T16:01:45.187 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-11T16:01:45.188 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2023-10-11T16:01:45.215 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2023-10-11T16:01:45.236 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2023-10-11T16:01:45.236 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2023-10-11T16:01:45.295 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="Red Hat Enterprise Linux" 2023-10-11T16:01:45.296 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="8.6 (Ootpa)" 2023-10-11T16:01:45.296 INFO:teuthology.orchestra.run.smithi130.stdout:ID="rhel" 2023-10-11T16:01:45.296 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="fedora" 2023-10-11T16:01:45.296 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="8.6" 2023-10-11T16:01:45.296 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el8" 2023-10-11T16:01:45.297 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-11T16:01:45.297 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2023-10-11T16:01:45.297 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-11T16:01:45.297 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://www.redhat.com/" 2023-10-11T16:01:45.298 INFO:teuthology.orchestra.run.smithi130.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-11T16:01:45.298 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-11T16:01:45.298 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:01:45.298 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-11T16:01:45.298 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-11T16:01:45.299 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-11T16:01:45.299 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-11T16:01:45.299 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2023-10-11T16:01:45.323 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-10-11T16:01:45.344 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-10-11T16:01:45.344 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-10-11T16:01:45.403 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Red Hat Enterprise Linux" 2023-10-11T16:01:45.403 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8.6 (Ootpa)" 2023-10-11T16:01:45.403 INFO:teuthology.orchestra.run.smithi150.stdout:ID="rhel" 2023-10-11T16:01:45.403 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="fedora" 2023-10-11T16:01:45.404 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8.6" 2023-10-11T16:01:45.404 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2023-10-11T16:01:45.404 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-11T16:01:45.404 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-10-11T16:01:45.404 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-11T16:01:45.405 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.redhat.com/" 2023-10-11T16:01:45.405 INFO:teuthology.orchestra.run.smithi150.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-11T16:01:45.405 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-11T16:01:45.405 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:01:45.406 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-11T16:01:45.406 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-11T16:01:45.406 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-11T16:01:45.406 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-11T16:01:45.407 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-10-11T16:01:45.431 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-11T16:01:45.441 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-11T16:01:45.460 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-11T16:01:45.460 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2023-10-11T16:01:45.463 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2023-10-11T16:01:45.465 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-10-11T16:01:45.484 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-11T16:01:45.490 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-11T16:01:45.490 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2023-10-11T16:01:45.520 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2023-10-11T16:01:45.524 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-10-11T16:01:45.551 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-11T16:01:45.628 INFO:teuthology.run_tasks:Running task kernel... 2023-10-11T16:01:45.642 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-11T16:01:45.643 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-10-11T16:01:45.643 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-10-11T16:01:45.644 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-11T16:01:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:01:45.663 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-10-11T16:01:45.822 DEBUG:teuthology.task.kernel:sha1 for client.0 is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-11T16:01:45.823 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-11T16:01:45.823 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-10-11T16:01:45.842 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-11T16:01:45.843 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-11T16:01:45.843 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-11T16:01:45.844 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum upgrade -y linux-firmware 2023-10-11T16:01:47.122 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T16:01:47.123 INFO:teuthology.orchestra.run.smithi111.stdout:Unable to read consumer identity 2023-10-11T16:01:48.783 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-10-11T16:01:50.067 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-10-11T16:01:53.533 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 621 kB/s | 24 kB 00:00 2023-10-11T16:01:54.324 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T16:01:54.325 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2023-10-11T16:01:54.326 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T16:01:54.454 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.0... 2023-10-11T16:01:54.455 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-11T16:01:54.683 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/8/flavors/default/x86_64/ 2023-10-11T16:01:54.683 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/8/flavors/default/x86_64/ --input-file=- 2023-10-11T16:01:54.686 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi111.front.sepia.ceph.com to complete... 2023-10-11T16:02:27.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11 16:02:27 URL:https://3.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/8/flavors/default/x86_64/kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm [383161052/383161052] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-11T16:02:27.457 INFO:teuthology.orchestra.run.smithi111.stderr:FINISHED --2023-10-11 16:02:27-- 2023-10-11T16:02:27.457 INFO:teuthology.orchestra.run.smithi111.stderr:Total wall clock time: 33s 2023-10-11T16:02:27.457 INFO:teuthology.orchestra.run.smithi111.stderr:Downloaded: 1 files, 365M in 32s (11.3 MB/s) 2023-10-11T16:02:27.525 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.0... 2023-10-11T16:02:27.526 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-11T16:02:29.677 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying... ######################################## 2023-10-11T16:02:29.742 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing... ######################################## 2023-10-11T16:02:29.854 INFO:teuthology.orchestra.run.smithi111.stdout:Updating / installing... 2023-10-11T16:03:08.053 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-6.6.0_rc5_g47672119149b-1 ######################################## 2023-10-11T16:03:11.203 INFO:teuthology.orchestra.run.smithi111.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-10-11T16:03:37.413 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-11T16:03:37.414 DEBUG:teuthology.orchestra.run.smithi111:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-11T16:03:37.441 INFO:teuthology.orchestra.run.smithi111.stdout:/boot/System.map-6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.441 INFO:teuthology.orchestra.run.smithi111.stdout:/boot/config-6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.442 INFO:teuthology.orchestra.run.smithi111.stdout:/boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.442 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.442 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel 2023-10-11T16:03:37.442 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch 2023-10-11T16:03:37.442 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86 2023-10-11T16:03:37.443 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto 2023-10-11T16:03:37.443 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-11T16:03:37.443 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-11T16:03:37.443 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-11T16:03:37.443 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-11T16:03:37.444 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-11T16:03:37.444 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-11T16:03:37.444 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-11T16:03:37.444 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-11T16:03:37.444 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-11T16:03:37.445 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-11T16:03:37.445 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-11T16:03:37.445 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-11T16:03:37.445 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-11T16:03:37.445 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-11T16:03:37.446 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-11T16:03:37.446 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-11T16:03:37.446 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-11T16:03:37.446 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-11T16:03:37.446 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-11T16:03:37.447 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-11T16:03:37.447 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel 2023-10-11T16:03:37.447 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu 2023-10-11T16:03:37.447 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce 2023-10-11T16:03:37.447 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-11T16:03:37.448 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm 2023-10-11T16:03:37.448 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-11T16:03:37.448 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-11T16:03:37.448 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-11T16:03:37.448 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto 2023-10-11T16:03:37.448 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-11T16:03:37.449 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-11T16:03:37.449 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-11T16:03:37.449 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx 2023-10-11T16:03:37.449 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-11T16:03:37.449 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-11T16:03:37.450 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-11T16:03:37.450 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-11T16:03:37.450 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-11T16:03:37.450 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-11T16:03:37.450 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-11T16:03:37.451 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-11T16:03:37.451 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-11T16:03:37.451 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-11T16:03:37.451 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-11T16:03:37.451 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-11T16:03:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-11T16:03:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-11T16:03:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-11T16:03:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-11T16:03:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-11T16:03:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-11T16:03:37.453 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-11T16:03:37.453 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-11T16:03:37.453 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-11T16:03:37.453 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-11T16:03:37.453 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-11T16:03:37.454 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-11T16:03:37.454 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-11T16:03:37.454 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-11T16:03:37.454 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-11T16:03:37.454 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-11T16:03:37.455 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-11T16:03:37.455 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-11T16:03:37.455 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-11T16:03:37.455 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-11T16:03:37.455 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-11T16:03:37.456 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-11T16:03:37.456 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-11T16:03:37.456 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-11T16:03:37.456 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-11T16:03:37.456 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-11T16:03:37.457 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-11T16:03:37.457 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-11T16:03:37.457 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-11T16:03:37.457 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-11T16:03:37.457 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-11T16:03:37.458 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-11T16:03:37.458 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-11T16:03:37.458 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers 2023-10-11T16:03:37.458 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi 2023-10-11T16:03:37.458 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-11T16:03:37.458 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-11T16:03:37.459 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/custom_method.ko 2023-10-11T16:03:37.459 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-11T16:03:37.459 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-11T16:03:37.459 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-11T16:03:37.459 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-11T16:03:37.459 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-11T16:03:37.460 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata 2023-10-11T16:03:37.460 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-11T16:03:37.460 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-11T16:03:37.460 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ata_generic.ko 2023-10-11T16:03:37.460 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-11T16:03:37.461 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-11T16:03:37.461 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-11T16:03:37.461 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-11T16:03:37.461 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-11T16:03:37.461 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-11T16:03:37.462 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-11T16:03:37.462 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-11T16:03:37.462 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-11T16:03:37.462 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-11T16:03:37.462 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-11T16:03:37.482 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-11T16:03:37.482 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-11T16:03:37.483 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-11T16:03:37.483 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-11T16:03:37.483 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-11T16:03:37.483 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-11T16:03:37.483 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-11T16:03:37.484 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-11T16:03:37.484 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-11T16:03:37.484 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-11T16:03:37.484 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-11T16:03:37.484 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-11T16:03:37.485 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-11T16:03:37.485 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-11T16:03:37.485 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-11T16:03:37.485 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-11T16:03:37.485 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-11T16:03:37.486 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-11T16:03:37.486 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-11T16:03:37.486 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-11T16:03:37.486 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-11T16:03:37.486 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-11T16:03:37.487 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-11T16:03:37.487 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-11T16:03:37.487 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sis.ko 2023-10-11T16:03:37.487 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-11T16:03:37.487 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-11T16:03:37.487 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-11T16:03:37.488 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-11T16:03:37.488 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-11T16:03:37.488 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-11T16:03:37.488 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-11T16:03:37.488 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-11T16:03:37.489 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-11T16:03:37.489 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-11T16:03:37.489 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-11T16:03:37.489 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-11T16:03:37.489 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-11T16:03:37.489 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-11T16:03:37.490 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-11T16:03:37.490 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-11T16:03:37.490 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-11T16:03:37.490 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay 2023-10-11T16:03:37.490 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-11T16:03:37.490 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base 2023-10-11T16:03:37.491 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap 2023-10-11T16:03:37.491 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-11T16:03:37.491 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-11T16:03:37.491 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma 2023-10-11T16:03:37.491 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-11T16:03:37.492 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block 2023-10-11T16:03:37.492 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe 2023-10-11T16:03:37.492 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-11T16:03:37.492 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/brd.ko 2023-10-11T16:03:37.492 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd 2023-10-11T16:03:37.492 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-11T16:03:37.493 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-11T16:03:37.493 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/loop.ko 2023-10-11T16:03:37.493 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx 2023-10-11T16:03:37.493 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-11T16:03:37.493 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-11T16:03:37.494 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk 2023-10-11T16:03:37.494 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-11T16:03:37.494 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-11T16:03:37.494 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-11T16:03:37.494 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/virtio_blk.ko 2023-10-11T16:03:37.495 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback 2023-10-11T16:03:37.495 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-11T16:03:37.495 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkfront.ko 2023-10-11T16:03:37.495 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram 2023-10-11T16:03:37.495 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-11T16:03:37.496 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char 2023-10-11T16:03:37.496 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-11T16:03:37.496 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random 2023-10-11T16:03:37.496 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-11T16:03:37.496 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-11T16:03:37.496 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-11T16:03:37.497 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-11T16:03:37.497 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-11T16:03:37.497 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi 2023-10-11T16:03:37.497 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-11T16:03:37.497 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-11T16:03:37.497 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-11T16:03:37.498 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-11T16:03:37.498 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-11T16:03:37.498 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-11T16:03:37.498 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-11T16:03:37.498 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave 2023-10-11T16:03:37.499 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-11T16:03:37.499 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-11T16:03:37.499 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-11T16:03:37.499 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm 2023-10-11T16:03:37.499 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm.ko 2023-10-11T16:03:37.500 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-11T16:03:37.500 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-11T16:03:37.500 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-11T16:03:37.501 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-11T16:03:37.501 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-11T16:03:37.501 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/uv_mmtimer.ko 2023-10-11T16:03:37.501 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/virtio_console.ko 2023-10-11T16:03:37.502 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq 2023-10-11T16:03:37.502 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-11T16:03:37.502 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-11T16:03:37.502 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-11T16:03:37.502 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-11T16:03:37.503 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-11T16:03:37.503 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-11T16:03:37.503 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto 2023-10-11T16:03:37.503 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp 2023-10-11T16:03:37.503 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-11T16:03:37.504 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-11T16:03:37.504 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel 2023-10-11T16:03:37.504 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat 2023-10-11T16:03:37.504 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-11T16:03:37.504 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-11T16:03:37.504 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-11T16:03:37.505 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-11T16:03:37.505 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-11T16:03:37.505 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-11T16:03:37.505 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio 2023-10-11T16:03:37.505 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-11T16:03:37.506 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca 2023-10-11T16:03:37.506 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-11T16:03:37.506 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq 2023-10-11T16:03:37.506 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-11T16:03:37.506 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma 2023-10-11T16:03:37.507 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw 2023-10-11T16:03:37.507 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-11T16:03:37.507 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat 2023-10-11T16:03:37.507 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-11T16:03:37.507 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac 2023-10-11T16:03:37.508 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-11T16:03:37.508 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-11T16:03:37.508 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-11T16:03:37.508 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-11T16:03:37.508 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-11T16:03:37.508 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-11T16:03:37.509 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-11T16:03:37.509 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-11T16:03:37.509 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-11T16:03:37.509 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-11T16:03:37.509 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-11T16:03:37.510 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-11T16:03:37.510 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-11T16:03:37.510 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware 2023-10-11T16:03:37.510 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/edd.ko 2023-10-11T16:03:37.510 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-11T16:03:37.510 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio 2023-10-11T16:03:37.511 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-11T16:03:37.511 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-11T16:03:37.511 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid 2023-10-11T16:03:37.511 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-11T16:03:37.511 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-11T16:03:37.512 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-11T16:03:37.512 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-11T16:03:37.512 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-11T16:03:37.512 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-11T16:03:37.512 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-11T16:03:37.512 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-11T16:03:37.513 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-11T16:03:37.513 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-11T16:03:37.513 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-11T16:03:37.513 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-11T16:03:37.513 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-11T16:03:37.514 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-11T16:03:37.514 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-11T16:03:37.514 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-11T16:03:37.514 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-11T16:03:37.514 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-11T16:03:37.515 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-11T16:03:37.515 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-11T16:03:37.515 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-11T16:03:37.515 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-11T16:03:37.515 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-11T16:03:37.516 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-11T16:03:37.516 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-11T16:03:37.516 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-11T16:03:37.516 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-11T16:03:37.516 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-11T16:03:37.516 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-11T16:03:37.517 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-11T16:03:37.517 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-11T16:03:37.517 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-11T16:03:37.517 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-11T16:03:37.517 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-11T16:03:37.518 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-11T16:03:37.518 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-11T16:03:37.518 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-11T16:03:37.518 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-11T16:03:37.518 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-11T16:03:37.519 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-11T16:03:37.519 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-11T16:03:37.519 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-11T16:03:37.519 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-11T16:03:37.519 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-11T16:03:37.519 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-11T16:03:37.520 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-11T16:03:37.520 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-11T16:03:37.520 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-11T16:03:37.520 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-11T16:03:37.520 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-11T16:03:37.520 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-11T16:03:37.521 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-11T16:03:37.521 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-11T16:03:37.521 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-11T16:03:37.521 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-11T16:03:37.521 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-11T16:03:37.521 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-11T16:03:37.522 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-11T16:03:37.522 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-11T16:03:37.522 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-11T16:03:37.522 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-11T16:03:37.522 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-11T16:03:37.523 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-11T16:03:37.523 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-11T16:03:37.523 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv 2023-10-11T16:03:37.523 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-11T16:03:37.523 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-11T16:03:37.523 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-11T16:03:37.524 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon 2023-10-11T16:03:37.524 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-11T16:03:37.524 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-11T16:03:37.524 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-11T16:03:37.524 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-11T16:03:37.525 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-11T16:03:37.525 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-11T16:03:37.525 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-11T16:03:37.525 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-11T16:03:37.525 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-11T16:03:37.525 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-11T16:03:37.526 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-11T16:03:37.526 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-11T16:03:37.526 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-11T16:03:37.526 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-11T16:03:37.526 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-11T16:03:37.527 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-11T16:03:37.527 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-11T16:03:37.527 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-11T16:03:37.527 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-11T16:03:37.527 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-11T16:03:37.528 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-11T16:03:37.528 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-11T16:03:37.528 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-11T16:03:37.528 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-11T16:03:37.528 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-11T16:03:37.528 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-11T16:03:37.529 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-11T16:03:37.529 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-11T16:03:37.529 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-11T16:03:37.529 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-11T16:03:37.529 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-11T16:03:37.530 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-11T16:03:37.530 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-11T16:03:37.530 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-11T16:03:37.530 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-11T16:03:37.530 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-11T16:03:37.530 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-11T16:03:37.531 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-11T16:03:37.531 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-11T16:03:37.531 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-11T16:03:37.531 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-11T16:03:37.531 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-11T16:03:37.531 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-11T16:03:37.532 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-11T16:03:37.532 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-11T16:03:37.532 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-11T16:03:37.532 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-11T16:03:37.532 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-11T16:03:37.533 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-11T16:03:37.533 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-11T16:03:37.533 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-11T16:03:37.533 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-11T16:03:37.533 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-11T16:03:37.533 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-11T16:03:37.534 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-11T16:03:37.534 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-11T16:03:37.534 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-11T16:03:37.534 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-11T16:03:37.534 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-11T16:03:37.535 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-11T16:03:37.535 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-11T16:03:37.535 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-11T16:03:37.535 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-11T16:03:37.535 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-11T16:03:37.535 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-11T16:03:37.536 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-11T16:03:37.536 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-11T16:03:37.536 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-11T16:03:37.536 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-11T16:03:37.536 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-11T16:03:37.537 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-11T16:03:37.537 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-11T16:03:37.537 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-11T16:03:37.537 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-11T16:03:37.537 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-11T16:03:37.538 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-11T16:03:37.538 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-11T16:03:37.538 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-11T16:03:37.538 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-11T16:03:37.538 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-11T16:03:37.539 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-11T16:03:37.539 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-11T16:03:37.539 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-11T16:03:37.539 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-11T16:03:37.539 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-11T16:03:37.540 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-11T16:03:37.540 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-11T16:03:37.540 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-11T16:03:37.540 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus 2023-10-11T16:03:37.540 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-11T16:03:37.540 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-11T16:03:37.541 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-11T16:03:37.541 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-11T16:03:37.541 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-11T16:03:37.541 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-11T16:03:37.541 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-11T16:03:37.541 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-11T16:03:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-11T16:03:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-11T16:03:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-11T16:03:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-11T16:03:37.542 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-11T16:03:37.543 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-11T16:03:37.543 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-11T16:03:37.543 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-11T16:03:37.543 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-11T16:03:37.543 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-11T16:03:37.543 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-11T16:03:37.544 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-11T16:03:37.544 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-11T16:03:37.544 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-11T16:03:37.544 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-11T16:03:37.544 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-11T16:03:37.544 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-11T16:03:37.545 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-11T16:03:37.545 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-11T16:03:37.545 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-11T16:03:37.545 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-11T16:03:37.545 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-11T16:03:37.546 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-11T16:03:37.546 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-11T16:03:37.546 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-11T16:03:37.546 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-11T16:03:37.546 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-11T16:03:37.546 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-11T16:03:37.547 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-11T16:03:37.547 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-11T16:03:37.547 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-11T16:03:37.547 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-11T16:03:37.547 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-11T16:03:37.548 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c 2023-10-11T16:03:37.548 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos 2023-10-11T16:03:37.548 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-11T16:03:37.548 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-11T16:03:37.548 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses 2023-10-11T16:03:37.549 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-11T16:03:37.549 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-11T16:03:37.549 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-11T16:03:37.549 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-11T16:03:37.549 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-11T16:03:37.550 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-11T16:03:37.550 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-11T16:03:37.550 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-11T16:03:37.550 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-11T16:03:37.550 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-11T16:03:37.550 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-11T16:03:37.551 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-11T16:03:37.551 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-11T16:03:37.551 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-11T16:03:37.551 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-11T16:03:37.551 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-11T16:03:37.552 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-11T16:03:37.552 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-11T16:03:37.552 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-11T16:03:37.552 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-11T16:03:37.552 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-11T16:03:37.552 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-11T16:03:37.553 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-11T16:03:37.553 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-dev.ko 2023-10-11T16:03:37.553 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-11T16:03:37.553 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-11T16:03:37.553 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-11T16:03:37.554 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-11T16:03:37.554 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband 2023-10-11T16:03:37.554 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core 2023-10-11T16:03:37.554 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-11T16:03:37.554 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-11T16:03:37.555 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-11T16:03:37.555 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-11T16:03:37.555 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-11T16:03:37.555 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-11T16:03:37.555 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-11T16:03:37.555 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw 2023-10-11T16:03:37.556 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4 2023-10-11T16:03:37.556 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-11T16:03:37.556 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4 2023-10-11T16:03:37.556 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-11T16:03:37.556 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5 2023-10-11T16:03:37.557 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-11T16:03:37.557 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca 2023-10-11T16:03:37.557 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-11T16:03:37.557 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp 2023-10-11T16:03:37.557 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib 2023-10-11T16:03:37.558 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-11T16:03:37.558 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser 2023-10-11T16:03:37.558 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-11T16:03:37.558 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert 2023-10-11T16:03:37.558 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-11T16:03:37.558 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp 2023-10-11T16:03:37.559 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-11T16:03:37.559 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt 2023-10-11T16:03:37.559 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-11T16:03:37.559 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input 2023-10-11T16:03:37.559 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport 2023-10-11T16:03:37.560 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-11T16:03:37.560 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-11T16:03:37.560 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-11T16:03:37.560 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-11T16:03:37.560 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard 2023-10-11T16:03:37.560 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-11T16:03:37.561 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse 2023-10-11T16:03:37.561 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-11T16:03:37.561 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-11T16:03:37.561 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-11T16:03:37.561 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-11T16:03:37.562 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-11T16:03:37.562 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-11T16:03:37.562 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-11T16:03:37.562 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-11T16:03:37.562 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4 2023-10-11T16:03:37.563 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-11T16:03:37.563 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio 2023-10-11T16:03:37.563 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-11T16:03:37.563 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-11T16:03:37.563 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-11T16:03:37.563 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-11T16:03:37.564 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-11T16:03:37.564 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu 2023-10-11T16:03:37.564 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd 2023-10-11T16:03:37.564 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-11T16:03:37.564 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds 2023-10-11T16:03:37.564 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-11T16:03:37.565 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-11T16:03:37.565 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-11T16:03:37.565 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-11T16:03:37.565 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger 2023-10-11T16:03:37.565 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-11T16:03:37.566 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-11T16:03:37.566 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-11T16:03:37.566 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-11T16:03:37.566 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-11T16:03:37.566 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-11T16:03:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-11T16:03:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-11T16:03:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md 2023-10-11T16:03:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache 2023-10-11T16:03:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-11T16:03:37.567 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-11T16:03:37.568 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-11T16:03:37.568 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-11T16:03:37.568 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-11T16:03:37.568 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-11T16:03:37.568 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-11T16:03:37.569 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-11T16:03:37.569 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-11T16:03:37.569 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-11T16:03:37.569 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-11T16:03:37.569 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-11T16:03:37.569 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-11T16:03:37.570 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-11T16:03:37.570 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-11T16:03:37.570 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-11T16:03:37.570 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-11T16:03:37.570 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-11T16:03:37.571 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-11T16:03:37.571 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data 2023-10-11T16:03:37.571 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-11T16:03:37.571 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-11T16:03:37.571 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-11T16:03:37.571 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-11T16:03:37.572 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-11T16:03:37.572 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message 2023-10-11T16:03:37.572 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion 2023-10-11T16:03:37.572 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-11T16:03:37.572 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-11T16:03:37.573 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-11T16:03:37.573 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-11T16:03:37.573 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-11T16:03:37.573 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-11T16:03:37.573 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-11T16:03:37.573 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd 2023-10-11T16:03:37.574 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-11T16:03:37.574 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-11T16:03:37.574 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mfd-core.ko 2023-10-11T16:03:37.574 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-11T16:03:37.574 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-11T16:03:37.575 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-11T16:03:37.575 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-11T16:03:37.575 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc 2023-10-11T16:03:37.575 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl 2023-10-11T16:03:37.575 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-11T16:03:37.575 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-11T16:03:37.576 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-11T16:03:37.576 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-11T16:03:37.576 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710 2023-10-11T16:03:37.576 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-11T16:03:37.576 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom 2023-10-11T16:03:37.576 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-11T16:03:37.577 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-11T16:03:37.577 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-11T16:03:37.577 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-11T16:03:37.577 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-11T16:03:37.577 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-11T16:03:37.578 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-11T16:03:37.578 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-11T16:03:37.578 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d 2023-10-11T16:03:37.578 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-11T16:03:37.578 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-11T16:03:37.579 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei 2023-10-11T16:03:37.579 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-11T16:03:37.579 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-11T16:03:37.579 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-11T16:03:37.579 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru 2023-10-11T16:03:37.579 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-11T16:03:37.580 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp 2023-10-11T16:03:37.580 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-11T16:03:37.580 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-11T16:03:37.580 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-11T16:03:37.580 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-11T16:03:37.581 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-11T16:03:37.581 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-11T16:03:37.581 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-11T16:03:37.581 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci 2023-10-11T16:03:37.581 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-11T16:03:37.581 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net 2023-10-11T16:03:37.582 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding 2023-10-11T16:03:37.582 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-11T16:03:37.582 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa 2023-10-11T16:03:37.582 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53 2023-10-11T16:03:37.582 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-11T16:03:37.582 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-11T16:03:37.583 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-11T16:03:37.583 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-11T16:03:37.583 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-11T16:03:37.583 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-11T16:03:37.583 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-11T16:03:37.584 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet 2023-10-11T16:03:37.584 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com 2023-10-11T16:03:37.584 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-11T16:03:37.584 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-11T16:03:37.584 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-11T16:03:37.584 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-11T16:03:37.585 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390 2023-10-11T16:03:37.585 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-11T16:03:37.585 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-11T16:03:37.585 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-11T16:03:37.585 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-11T16:03:37.585 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec 2023-10-11T16:03:37.586 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-11T16:03:37.586 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere 2023-10-11T16:03:37.586 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-11T16:03:37.586 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon 2023-10-11T16:03:37.586 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-11T16:03:37.587 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera 2023-10-11T16:03:37.587 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-11T16:03:37.587 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd 2023-10-11T16:03:37.587 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-11T16:03:37.587 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-11T16:03:37.587 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-11T16:03:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros 2023-10-11T16:03:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx 2023-10-11T16:03:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-11T16:03:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-11T16:03:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-11T16:03:37.589 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-11T16:03:37.589 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-11T16:03:37.589 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-11T16:03:37.589 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-11T16:03:37.589 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-11T16:03:37.590 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom 2023-10-11T16:03:37.590 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-11T16:03:37.590 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-11T16:03:37.590 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-11T16:03:37.590 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-11T16:03:37.591 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-11T16:03:37.591 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-11T16:03:37.591 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-11T16:03:37.591 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-11T16:03:37.591 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade 2023-10-11T16:03:37.591 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna 2023-10-11T16:03:37.592 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-11T16:03:37.592 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio 2023-10-11T16:03:37.592 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-11T16:03:37.592 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-11T16:03:37.592 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-11T16:03:37.592 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-11T16:03:37.593 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-11T16:03:37.593 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-11T16:03:37.593 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-11T16:03:37.593 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-11T16:03:37.593 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-11T16:03:37.593 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-11T16:03:37.594 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco 2023-10-11T16:03:37.594 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic 2023-10-11T16:03:37.594 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-11T16:03:37.594 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec 2023-10-11T16:03:37.594 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip 2023-10-11T16:03:37.595 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-11T16:03:37.595 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-11T16:03:37.595 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-11T16:03:37.595 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-11T16:03:37.595 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-11T16:03:37.596 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-11T16:03:37.596 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink 2023-10-11T16:03:37.596 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-11T16:03:37.596 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-11T16:03:37.596 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-11T16:03:37.596 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex 2023-10-11T16:03:37.597 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet 2023-10-11T16:03:37.597 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-11T16:03:37.597 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-11T16:03:37.597 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-11T16:03:37.597 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel 2023-10-11T16:03:37.597 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-11T16:03:37.598 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000 2023-10-11T16:03:37.598 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-11T16:03:37.598 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-11T16:03:37.598 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-11T16:03:37.598 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-11T16:03:37.599 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-11T16:03:37.599 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e 2023-10-11T16:03:37.599 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-11T16:03:37.599 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf 2023-10-11T16:03:37.599 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-11T16:03:37.600 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb 2023-10-11T16:03:37.600 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-11T16:03:37.600 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-11T16:03:37.600 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-11T16:03:37.600 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-11T16:03:37.600 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-11T16:03:37.601 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-11T16:03:37.601 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-11T16:03:37.601 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-11T16:03:37.601 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell 2023-10-11T16:03:37.601 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-11T16:03:37.601 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-11T16:03:37.602 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-11T16:03:37.602 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox 2023-10-11T16:03:37.602 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-11T16:03:37.602 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-11T16:03:37.602 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-11T16:03:37.603 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-11T16:03:37.603 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-11T16:03:37.603 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-11T16:03:37.603 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel 2023-10-11T16:03:37.603 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-11T16:03:37.603 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom 2023-10-11T16:03:37.604 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-11T16:03:37.604 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-11T16:03:37.604 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi 2023-10-11T16:03:37.604 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-11T16:03:37.604 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-11T16:03:37.605 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion 2023-10-11T16:03:37.605 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-11T16:03:37.605 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia 2023-10-11T16:03:37.605 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-11T16:03:37.605 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines 2023-10-11T16:03:37.606 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-11T16:03:37.606 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-11T16:03:37.606 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic 2023-10-11T16:03:37.606 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-11T16:03:37.606 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-11T16:03:37.607 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-11T16:03:37.607 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-11T16:03:37.607 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-11T16:03:37.607 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc 2023-10-11T16:03:37.607 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-11T16:03:37.608 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek 2023-10-11T16:03:37.608 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-11T16:03:37.608 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-11T16:03:37.608 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-11T16:03:37.608 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-11T16:03:37.609 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker 2023-10-11T16:03:37.609 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-11T16:03:37.609 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc 2023-10-11T16:03:37.609 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-11T16:03:37.609 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan 2023-10-11T16:03:37.609 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-11T16:03:37.610 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis 2023-10-11T16:03:37.610 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-11T16:03:37.610 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-11T16:03:37.610 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc 2023-10-11T16:03:37.610 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-11T16:03:37.611 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-11T16:03:37.611 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-11T16:03:37.611 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-11T16:03:37.611 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro 2023-10-11T16:03:37.611 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-11T16:03:37.612 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-11T16:03:37.612 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-11T16:03:37.612 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun 2023-10-11T16:03:37.612 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-11T16:03:37.612 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-11T16:03:37.612 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-11T16:03:37.613 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-11T16:03:37.613 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti 2023-10-11T16:03:37.613 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-11T16:03:37.613 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti 2023-10-11T16:03:37.613 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-11T16:03:37.614 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via 2023-10-11T16:03:37.614 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-11T16:03:37.614 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-11T16:03:37.614 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet 2023-10-11T16:03:37.614 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-11T16:03:37.615 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-11T16:03:37.615 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom 2023-10-11T16:03:37.615 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-11T16:03:37.615 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-11T16:03:37.615 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv 2023-10-11T16:03:37.615 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-11T16:03:37.616 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154 2023-10-11T16:03:37.616 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-11T16:03:37.616 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-11T16:03:37.616 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan 2023-10-11T16:03:37.616 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-11T16:03:37.616 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-11T16:03:37.617 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-11T16:03:37.617 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio 2023-10-11T16:03:37.617 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-11T16:03:37.617 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-11T16:03:37.617 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-11T16:03:37.618 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-11T16:03:37.618 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-11T16:03:37.618 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/net_failover.ko 2023-10-11T16:03:37.618 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-11T16:03:37.618 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-11T16:03:37.619 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs 2023-10-11T16:03:37.619 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-11T16:03:37.619 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-11T16:03:37.619 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy 2023-10-11T16:03:37.619 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-11T16:03:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-11T16:03:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-11T16:03:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-11T16:03:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-11T16:03:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-11T16:03:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-11T16:03:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-11T16:03:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/dp83640.ko 2023-10-11T16:03:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-11T16:03:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-11T16:03:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-11T16:03:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-11T16:03:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-11T16:03:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-11T16:03:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-11T16:03:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-11T16:03:37.623 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-11T16:03:37.623 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-11T16:03:37.623 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-11T16:03:37.623 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-11T16:03:37.623 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp 2023-10-11T16:03:37.624 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-11T16:03:37.624 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-11T16:03:37.624 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-11T16:03:37.624 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-11T16:03:37.624 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-11T16:03:37.624 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-11T16:03:37.625 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-11T16:03:37.625 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-11T16:03:37.625 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-11T16:03:37.625 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip 2023-10-11T16:03:37.625 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slhc.ko 2023-10-11T16:03:37.626 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-11T16:03:37.626 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-11T16:03:37.626 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-11T16:03:37.626 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team 2023-10-11T16:03:37.626 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-11T16:03:37.627 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-11T16:03:37.627 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-11T16:03:37.627 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-11T16:03:37.627 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-11T16:03:37.627 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-11T16:03:37.627 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tun.ko 2023-10-11T16:03:37.628 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-11T16:03:37.628 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/virtio_net.ko 2023-10-11T16:03:37.628 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3 2023-10-11T16:03:37.628 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-11T16:03:37.628 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan 2023-10-11T16:03:37.629 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-11T16:03:37.629 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback 2023-10-11T16:03:37.629 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-11T16:03:37.629 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netfront.ko 2023-10-11T16:03:37.629 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb 2023-10-11T16:03:37.630 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-11T16:03:37.630 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme 2023-10-11T16:03:37.630 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host 2023-10-11T16:03:37.630 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-11T16:03:37.630 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-11T16:03:37.631 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport 2023-10-11T16:03:37.631 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-11T16:03:37.631 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-11T16:03:37.631 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-11T16:03:37.631 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-11T16:03:37.631 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci 2023-10-11T16:03:37.632 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug 2023-10-11T16:03:37.632 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-11T16:03:37.632 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie 2023-10-11T16:03:37.632 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-11T16:03:37.632 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-11T16:03:37.633 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia 2023-10-11T16:03:37.633 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-11T16:03:37.633 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-11T16:03:37.633 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-11T16:03:37.633 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl 2023-10-11T16:03:37.634 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel 2023-10-11T16:03:37.634 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-11T16:03:37.634 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform 2023-10-11T16:03:37.634 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86 2023-10-11T16:03:37.634 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-11T16:03:37.635 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-11T16:03:37.635 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-11T16:03:37.635 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-11T16:03:37.635 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-11T16:03:37.635 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-11T16:03:37.635 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-11T16:03:37.636 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-11T16:03:37.636 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-11T16:03:37.636 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-11T16:03:37.636 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-11T16:03:37.636 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-11T16:03:37.637 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel 2023-10-11T16:03:37.637 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-11T16:03:37.637 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-11T16:03:37.637 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-11T16:03:37.637 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-11T16:03:37.638 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-11T16:03:37.638 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-11T16:03:37.638 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-11T16:03:37.638 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-11T16:03:37.638 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-11T16:03:37.638 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-11T16:03:37.639 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-11T16:03:37.639 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-11T16:03:37.639 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap 2023-10-11T16:03:37.639 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-11T16:03:37.639 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-11T16:03:37.640 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps 2023-10-11T16:03:37.640 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients 2023-10-11T16:03:37.640 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-11T16:03:37.640 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-11T16:03:37.640 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-11T16:03:37.641 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-11T16:03:37.641 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp 2023-10-11T16:03:37.641 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-11T16:03:37.641 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-11T16:03:37.641 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc 2023-10-11T16:03:37.642 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-11T16:03:37.642 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-11T16:03:37.642 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-11T16:03:37.642 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-11T16:03:37.642 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-11T16:03:37.642 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-11T16:03:37.643 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-11T16:03:37.643 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-11T16:03:37.643 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-11T16:03:37.643 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-11T16:03:37.643 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-11T16:03:37.644 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-11T16:03:37.644 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-11T16:03:37.644 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-11T16:03:37.644 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-11T16:03:37.644 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-11T16:03:37.644 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-11T16:03:37.645 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-11T16:03:37.645 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-11T16:03:37.645 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-11T16:03:37.645 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-11T16:03:37.645 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-11T16:03:37.646 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-11T16:03:37.646 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-11T16:03:37.646 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-11T16:03:37.646 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-11T16:03:37.646 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-11T16:03:37.646 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-11T16:03:37.647 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-11T16:03:37.647 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-11T16:03:37.647 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-11T16:03:37.647 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi 2023-10-11T16:03:37.647 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-11T16:03:37.648 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-11T16:03:37.648 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-11T16:03:37.648 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-11T16:03:37.648 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-11T16:03:37.648 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid 2023-10-11T16:03:37.649 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-11T16:03:37.649 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-11T16:03:37.649 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx 2023-10-11T16:03:37.649 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-11T16:03:37.649 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-11T16:03:37.649 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-11T16:03:37.650 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr 2023-10-11T16:03:37.650 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-11T16:03:37.650 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-11T16:03:37.650 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi 2023-10-11T16:03:37.650 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-11T16:03:37.651 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa 2023-10-11T16:03:37.651 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-11T16:03:37.651 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc 2023-10-11T16:03:37.651 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-11T16:03:37.651 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i 2023-10-11T16:03:37.652 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-11T16:03:37.652 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-11T16:03:37.652 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor 2023-10-11T16:03:37.652 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-11T16:03:37.652 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi 2023-10-11T16:03:37.653 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-11T16:03:37.653 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-11T16:03:37.653 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-11T16:03:37.653 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-11T16:03:37.653 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-11T16:03:37.653 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-11T16:03:37.654 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler 2023-10-11T16:03:37.654 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-11T16:03:37.654 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-11T16:03:37.654 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-11T16:03:37.654 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-11T16:03:37.655 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-11T16:03:37.655 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r 2023-10-11T16:03:37.655 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-11T16:03:37.655 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-11T16:03:37.655 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe 2023-10-11T16:03:37.655 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-11T16:03:37.656 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-11T16:03:37.656 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic 2023-10-11T16:03:37.656 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-11T16:03:37.656 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-11T16:03:37.656 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-11T16:03:37.657 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-11T16:03:37.657 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-11T16:03:37.657 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-11T16:03:37.657 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-11T16:03:37.657 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci 2023-10-11T16:03:37.658 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-11T16:03:37.658 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-11T16:03:37.658 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-11T16:03:37.658 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc 2023-10-11T16:03:37.658 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-11T16:03:37.658 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-11T16:03:37.659 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-11T16:03:37.659 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas 2023-10-11T16:03:37.661 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-11T16:03:37.662 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc 2023-10-11T16:03:37.662 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-11T16:03:37.662 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid 2023-10-11T16:03:37.662 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-11T16:03:37.662 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-11T16:03:37.663 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-11T16:03:37.663 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-11T16:03:37.663 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas 2023-10-11T16:03:37.663 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-11T16:03:37.663 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas 2023-10-11T16:03:37.663 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-11T16:03:37.664 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-11T16:03:37.664 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001 2023-10-11T16:03:37.664 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-11T16:03:37.664 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-11T16:03:37.664 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-11T16:03:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx 2023-10-11T16:03:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-11T16:03:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-11T16:03:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx 2023-10-11T16:03:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-11T16:03:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-11T16:03:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-11T16:03:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-11T16:03:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-11T16:03:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-11T16:03:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-11T16:03:37.667 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-11T16:03:37.667 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-11T16:03:37.667 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-11T16:03:37.667 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-11T16:03:37.667 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2 2023-10-11T16:03:37.668 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-11T16:03:37.668 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-11T16:03:37.668 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-11T16:03:37.668 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-11T16:03:37.668 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb 2023-10-11T16:03:37.668 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-11T16:03:37.669 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target 2023-10-11T16:03:37.669 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi 2023-10-11T16:03:37.669 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-11T16:03:37.669 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback 2023-10-11T16:03:37.669 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-11T16:03:37.670 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-11T16:03:37.670 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-11T16:03:37.670 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-11T16:03:37.670 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-11T16:03:37.670 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-11T16:03:37.671 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc 2023-10-11T16:03:37.671 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-11T16:03:37.671 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal 2023-10-11T16:03:37.671 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel 2023-10-11T16:03:37.671 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-11T16:03:37.671 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-11T16:03:37.672 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-11T16:03:37.672 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-11T16:03:37.672 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-11T16:03:37.672 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-11T16:03:37.672 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-11T16:03:37.673 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-11T16:03:37.673 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-11T16:03:37.673 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-11T16:03:37.673 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-11T16:03:37.673 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-11T16:03:37.673 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-11T16:03:37.674 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-11T16:03:37.674 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-11T16:03:37.674 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-11T16:03:37.674 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty 2023-10-11T16:03:37.674 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless 2023-10-11T16:03:37.675 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-11T16:03:37.675 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-11T16:03:37.675 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-11T16:03:37.675 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-11T16:03:37.675 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial 2023-10-11T16:03:37.676 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250 2023-10-11T16:03:37.676 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-11T16:03:37.676 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-11T16:03:37.676 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm 2023-10-11T16:03:37.676 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-11T16:03:37.676 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs 2023-10-11T16:03:37.677 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core 2023-10-11T16:03:37.677 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-11T16:03:37.677 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host 2023-10-11T16:03:37.677 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-11T16:03:37.677 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio 2023-10-11T16:03:37.678 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-11T16:03:37.678 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-11T16:03:37.678 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-11T16:03:37.678 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-11T16:03:37.678 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-11T16:03:37.679 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb 2023-10-11T16:03:37.679 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class 2023-10-11T16:03:37.679 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-11T16:03:37.679 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-11T16:03:37.679 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-11T16:03:37.679 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-11T16:03:37.680 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host 2023-10-11T16:03:37.680 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-11T16:03:37.680 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image 2023-10-11T16:03:37.680 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-11T16:03:37.680 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-11T16:03:37.681 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc 2023-10-11T16:03:37.681 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-11T16:03:37.681 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-11T16:03:37.681 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-11T16:03:37.681 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-11T16:03:37.682 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-11T16:03:37.682 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-11T16:03:37.682 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-11T16:03:37.682 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-11T16:03:37.682 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-11T16:03:37.682 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-11T16:03:37.683 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga 2023-10-11T16:03:37.683 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-11T16:03:37.683 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-11T16:03:37.683 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-11T16:03:37.683 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-11T16:03:37.684 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-11T16:03:37.684 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-11T16:03:37.684 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-11T16:03:37.684 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy 2023-10-11T16:03:37.684 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-11T16:03:37.684 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial 2023-10-11T16:03:37.685 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-11T16:03:37.685 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-11T16:03:37.685 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-11T16:03:37.685 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-11T16:03:37.685 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-11T16:03:37.686 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-11T16:03:37.686 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-11T16:03:37.686 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-11T16:03:37.686 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-11T16:03:37.686 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-11T16:03:37.687 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-11T16:03:37.687 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-11T16:03:37.687 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-11T16:03:37.687 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-11T16:03:37.687 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-11T16:03:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-11T16:03:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-11T16:03:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-11T16:03:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-11T16:03:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-11T16:03:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-11T16:03:37.689 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-11T16:03:37.689 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-11T16:03:37.689 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-11T16:03:37.689 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-11T16:03:37.689 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-11T16:03:37.690 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-11T16:03:37.690 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-11T16:03:37.690 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-11T16:03:37.690 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-11T16:03:37.690 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-11T16:03:37.690 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-11T16:03:37.691 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-11T16:03:37.691 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-11T16:03:37.691 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-11T16:03:37.691 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-11T16:03:37.691 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-11T16:03:37.692 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-11T16:03:37.692 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-11T16:03:37.692 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-11T16:03:37.692 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-11T16:03:37.692 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-11T16:03:37.692 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-11T16:03:37.693 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-11T16:03:37.693 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-11T16:03:37.693 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage 2023-10-11T16:03:37.693 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-11T16:03:37.693 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-11T16:03:37.694 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-11T16:03:37.694 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-11T16:03:37.694 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-11T16:03:37.694 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-11T16:03:37.694 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-11T16:03:37.694 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-11T16:03:37.695 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-11T16:03:37.695 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-11T16:03:37.695 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-11T16:03:37.695 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-11T16:03:37.695 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-11T16:03:37.696 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-11T16:03:37.696 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-11T16:03:37.696 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip 2023-10-11T16:03:37.696 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-11T16:03:37.696 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-11T16:03:37.697 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-11T16:03:37.697 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio 2023-10-11T16:03:37.697 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci 2023-10-11T16:03:37.697 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-11T16:03:37.697 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-11T16:03:37.697 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-11T16:03:37.698 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-11T16:03:37.698 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost 2023-10-11T16:03:37.698 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-11T16:03:37.698 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-11T16:03:37.698 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-11T16:03:37.699 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-11T16:03:37.699 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video 2023-10-11T16:03:37.699 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight 2023-10-11T16:03:37.699 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-11T16:03:37.699 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-11T16:03:37.700 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-11T16:03:37.700 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio 2023-10-11T16:03:37.700 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio.ko 2023-10-11T16:03:37.700 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-11T16:03:37.700 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-11T16:03:37.700 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci.ko 2023-10-11T16:03:37.701 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-11T16:03:37.701 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-11T16:03:37.701 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_ring.ko 2023-10-11T16:03:37.701 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1 2023-10-11T16:03:37.701 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters 2023-10-11T16:03:37.702 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-11T16:03:37.702 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-11T16:03:37.702 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves 2023-10-11T16:03:37.702 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-11T16:03:37.703 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-11T16:03:37.703 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-11T16:03:37.703 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-11T16:03:37.703 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-11T16:03:37.703 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-11T16:03:37.703 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-11T16:03:37.704 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-11T16:03:37.704 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-11T16:03:37.704 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-11T16:03:37.704 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-11T16:03:37.704 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-11T16:03:37.705 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog 2023-10-11T16:03:37.705 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-11T16:03:37.705 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-11T16:03:37.705 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-11T16:03:37.705 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-11T16:03:37.706 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-11T16:03:37.706 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-11T16:03:37.706 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-11T16:03:37.706 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-11T16:03:37.706 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-11T16:03:37.706 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-11T16:03:37.707 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-11T16:03:37.707 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-11T16:03:37.707 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-11T16:03:37.707 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-11T16:03:37.707 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-11T16:03:37.708 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-11T16:03:37.708 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-11T16:03:37.708 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-11T16:03:37.708 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-11T16:03:37.708 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-11T16:03:37.709 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-11T16:03:37.709 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-11T16:03:37.709 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-11T16:03:37.709 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-11T16:03:37.709 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-11T16:03:37.709 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-11T16:03:37.710 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen 2023-10-11T16:03:37.710 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-11T16:03:37.710 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-11T16:03:37.710 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-11T16:03:37.710 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-11T16:03:37.711 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback 2023-10-11T16:03:37.711 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-11T16:03:37.711 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-11T16:03:37.711 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs 2023-10-11T16:03:37.711 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-11T16:03:37.712 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs 2023-10-11T16:03:37.712 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p 2023-10-11T16:03:37.712 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-11T16:03:37.712 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs 2023-10-11T16:03:37.712 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-11T16:03:37.712 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs 2023-10-11T16:03:37.713 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-11T16:03:37.713 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-11T16:03:37.713 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs 2023-10-11T16:03:37.713 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-11T16:03:37.713 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles 2023-10-11T16:03:37.714 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-11T16:03:37.714 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph 2023-10-11T16:03:37.714 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-11T16:03:37.714 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda 2023-10-11T16:03:37.714 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-11T16:03:37.715 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs 2023-10-11T16:03:37.715 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-11T16:03:37.715 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm 2023-10-11T16:03:37.715 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-11T16:03:37.715 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs 2023-10-11T16:03:37.716 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-11T16:03:37.716 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs 2023-10-11T16:03:37.716 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-11T16:03:37.716 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat 2023-10-11T16:03:37.716 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/fat.ko 2023-10-11T16:03:37.717 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-11T16:03:37.717 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/vfat.ko 2023-10-11T16:03:37.717 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache 2023-10-11T16:03:37.717 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-11T16:03:37.717 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse 2023-10-11T16:03:37.717 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-11T16:03:37.718 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/fuse.ko 2023-10-11T16:03:37.718 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2 2023-10-11T16:03:37.718 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-11T16:03:37.718 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs 2023-10-11T16:03:37.718 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-11T16:03:37.719 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus 2023-10-11T16:03:37.719 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-11T16:03:37.719 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs 2023-10-11T16:03:37.719 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-11T16:03:37.719 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs 2023-10-11T16:03:37.720 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-11T16:03:37.720 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd 2023-10-11T16:03:37.720 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-11T16:03:37.720 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix 2023-10-11T16:03:37.720 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-11T16:03:37.720 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs 2023-10-11T16:03:37.721 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-11T16:03:37.721 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs 2023-10-11T16:03:37.721 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout 2023-10-11T16:03:37.721 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-11T16:03:37.721 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout 2023-10-11T16:03:37.722 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-11T16:03:37.722 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout 2023-10-11T16:03:37.722 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-11T16:03:37.722 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-11T16:03:37.722 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-11T16:03:37.723 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-11T16:03:37.723 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common 2023-10-11T16:03:37.723 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-11T16:03:37.723 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-11T16:03:37.723 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd 2023-10-11T16:03:37.724 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-11T16:03:37.724 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2 2023-10-11T16:03:37.724 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-11T16:03:37.724 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls 2023-10-11T16:03:37.724 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-11T16:03:37.725 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-11T16:03:37.725 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-11T16:03:37.725 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-11T16:03:37.725 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-11T16:03:37.725 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-11T16:03:37.725 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-11T16:03:37.726 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-11T16:03:37.726 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-11T16:03:37.726 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-11T16:03:37.726 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-11T16:03:37.726 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-11T16:03:37.727 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-11T16:03:37.727 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-11T16:03:37.727 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-11T16:03:37.727 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-11T16:03:37.727 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-11T16:03:37.727 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-11T16:03:37.728 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-11T16:03:37.728 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-11T16:03:37.728 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-11T16:03:37.728 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-11T16:03:37.728 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-11T16:03:37.729 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-11T16:03:37.729 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-11T16:03:37.729 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-11T16:03:37.729 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-11T16:03:37.729 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-11T16:03:37.730 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-11T16:03:37.730 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-11T16:03:37.730 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-11T16:03:37.730 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-11T16:03:37.730 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-11T16:03:37.731 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-11T16:03:37.731 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-11T16:03:37.731 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-11T16:03:37.731 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-11T16:03:37.731 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-11T16:03:37.732 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-11T16:03:37.732 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-11T16:03:37.732 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-11T16:03:37.732 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-11T16:03:37.732 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-11T16:03:37.732 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-11T16:03:37.733 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-11T16:03:37.733 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-11T16:03:37.733 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-11T16:03:37.733 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-11T16:03:37.733 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-11T16:03:37.734 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-11T16:03:37.734 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2 2023-10-11T16:03:37.734 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster 2023-10-11T16:03:37.734 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-11T16:03:37.734 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm 2023-10-11T16:03:37.735 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-11T16:03:37.735 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs 2023-10-11T16:03:37.735 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-11T16:03:37.735 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-11T16:03:37.735 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-11T16:03:37.736 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-11T16:03:37.736 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-11T16:03:37.736 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs 2023-10-11T16:03:37.736 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-11T16:03:37.736 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore 2023-10-11T16:03:37.736 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-11T16:03:37.737 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs 2023-10-11T16:03:37.737 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-11T16:03:37.737 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs 2023-10-11T16:03:37.737 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-11T16:03:37.737 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb 2023-10-11T16:03:37.738 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client 2023-10-11T16:03:37.738 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-11T16:03:37.738 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common 2023-10-11T16:03:37.738 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-11T16:03:37.738 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-11T16:03:37.739 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs 2023-10-11T16:03:37.739 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-11T16:03:37.739 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv 2023-10-11T16:03:37.739 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-11T16:03:37.739 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf 2023-10-11T16:03:37.740 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-11T16:03:37.740 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs 2023-10-11T16:03:37.740 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-11T16:03:37.740 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs 2023-10-11T16:03:37.740 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-11T16:03:37.741 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel 2023-10-11T16:03:37.741 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu 2023-10-11T16:03:37.741 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu/rcutorture.ko 2023-10-11T16:03:37.741 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-11T16:03:37.741 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace 2023-10-11T16:03:37.741 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-11T16:03:37.742 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib 2023-10-11T16:03:37.742 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/asn1_encoder.ko 2023-10-11T16:03:37.742 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-11T16:03:37.742 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-11T16:03:37.742 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto 2023-10-11T16:03:37.743 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-11T16:03:37.743 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-11T16:03:37.743 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-11T16:03:37.743 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-11T16:03:37.743 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4 2023-10-11T16:03:37.744 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-11T16:03:37.744 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-11T16:03:37.744 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math 2023-10-11T16:03:37.744 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-11T16:03:37.744 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6 2023-10-11T16:03:37.744 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-11T16:03:37.745 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon 2023-10-11T16:03:37.745 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-11T16:03:37.745 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-11T16:03:37.745 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-11T16:03:37.745 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-11T16:03:37.749 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd 2023-10-11T16:03:37.749 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd/zstd_compress.ko 2023-10-11T16:03:37.749 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm 2023-10-11T16:03:37.749 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-11T16:03:37.749 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net 2023-10-11T16:03:37.750 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan 2023-10-11T16:03:37.750 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-11T16:03:37.750 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-11T16:03:37.750 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-11T16:03:37.750 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-11T16:03:37.751 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-11T16:03:37.751 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-11T16:03:37.751 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-11T16:03:37.751 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-11T16:03:37.751 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802 2023-10-11T16:03:37.752 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-11T16:03:37.752 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-11T16:03:37.752 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-11T16:03:37.752 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-11T16:03:37.752 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-11T16:03:37.753 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q 2023-10-11T16:03:37.753 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-11T16:03:37.753 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p 2023-10-11T16:03:37.753 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-11T16:03:37.753 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-11T16:03:37.753 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-11T16:03:37.754 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-11T16:03:37.754 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv 2023-10-11T16:03:37.754 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-11T16:03:37.754 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge 2023-10-11T16:03:37.754 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-11T16:03:37.755 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-11T16:03:37.755 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter 2023-10-11T16:03:37.755 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-11T16:03:37.755 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-11T16:03:37.755 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-11T16:03:37.756 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-11T16:03:37.756 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-11T16:03:37.756 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-11T16:03:37.756 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-11T16:03:37.756 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-11T16:03:37.756 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-11T16:03:37.757 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-11T16:03:37.757 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-11T16:03:37.757 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-11T16:03:37.757 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-11T16:03:37.757 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-11T16:03:37.758 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-11T16:03:37.758 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-11T16:03:37.758 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-11T16:03:37.758 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-11T16:03:37.758 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-11T16:03:37.759 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-11T16:03:37.759 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-11T16:03:37.759 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-11T16:03:37.759 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-11T16:03:37.759 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-11T16:03:37.759 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph 2023-10-11T16:03:37.760 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-11T16:03:37.760 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core 2023-10-11T16:03:37.760 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/failover.ko 2023-10-11T16:03:37.760 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-11T16:03:37.760 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp 2023-10-11T16:03:37.761 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-11T16:03:37.761 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-11T16:03:37.761 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-11T16:03:37.761 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-11T16:03:37.761 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver 2023-10-11T16:03:37.762 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-11T16:03:37.762 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa 2023-10-11T16:03:37.762 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-11T16:03:37.762 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-11T16:03:37.762 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-11T16:03:37.762 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-11T16:03:37.763 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-11T16:03:37.763 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154 2023-10-11T16:03:37.763 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan 2023-10-11T16:03:37.763 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-11T16:03:37.763 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-11T16:03:37.764 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-11T16:03:37.764 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4 2023-10-11T16:03:37.764 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-11T16:03:37.764 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-11T16:03:37.764 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-11T16:03:37.765 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-11T16:03:37.765 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-11T16:03:37.765 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-11T16:03:37.765 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-11T16:03:37.765 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-11T16:03:37.765 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-11T16:03:37.766 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-11T16:03:37.766 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter 2023-10-11T16:03:37.766 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-11T16:03:37.766 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-11T16:03:37.766 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-11T16:03:37.767 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-11T16:03:37.767 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-11T16:03:37.767 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-11T16:03:37.767 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-11T16:03:37.767 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-11T16:03:37.768 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-11T16:03:37.768 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-11T16:03:37.768 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-11T16:03:37.768 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-11T16:03:37.768 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-11T16:03:37.769 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-11T16:03:37.769 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-11T16:03:37.769 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-11T16:03:37.769 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-11T16:03:37.769 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-11T16:03:37.769 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-11T16:03:37.770 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-11T16:03:37.770 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-11T16:03:37.770 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-11T16:03:37.770 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-11T16:03:37.770 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-11T16:03:37.771 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-11T16:03:37.771 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-11T16:03:37.771 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-11T16:03:37.771 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-11T16:03:37.771 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-11T16:03:37.772 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-11T16:03:37.772 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-11T16:03:37.772 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-11T16:03:37.772 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-11T16:03:37.772 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-11T16:03:37.773 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-11T16:03:37.773 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-11T16:03:37.773 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-11T16:03:37.773 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-11T16:03:37.773 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6 2023-10-11T16:03:37.773 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-11T16:03:37.774 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-11T16:03:37.774 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-11T16:03:37.774 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-11T16:03:37.774 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-11T16:03:37.774 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-11T16:03:37.775 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-11T16:03:37.775 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter 2023-10-11T16:03:37.775 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-11T16:03:37.775 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-11T16:03:37.775 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-11T16:03:37.776 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-11T16:03:37.776 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-11T16:03:37.776 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-11T16:03:37.776 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-11T16:03:37.776 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-11T16:03:37.777 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-11T16:03:37.777 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-11T16:03:37.777 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-11T16:03:37.777 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-11T16:03:37.777 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-11T16:03:37.778 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-11T16:03:37.778 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-11T16:03:37.778 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-11T16:03:37.778 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-11T16:03:37.778 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-11T16:03:37.778 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-11T16:03:37.779 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-11T16:03:37.779 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-11T16:03:37.779 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-11T16:03:37.779 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-11T16:03:37.779 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-11T16:03:37.780 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-11T16:03:37.780 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-11T16:03:37.780 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-11T16:03:37.780 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key 2023-10-11T16:03:37.780 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-11T16:03:37.781 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp 2023-10-11T16:03:37.781 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-11T16:03:37.781 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-11T16:03:37.781 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-11T16:03:37.781 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-11T16:03:37.782 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-11T16:03:37.782 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-11T16:03:37.782 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-11T16:03:37.782 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc 2023-10-11T16:03:37.782 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-11T16:03:37.782 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154 2023-10-11T16:03:37.783 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-11T16:03:37.783 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls 2023-10-11T16:03:37.783 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-11T16:03:37.783 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter 2023-10-11T16:03:37.783 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset 2023-10-11T16:03:37.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-11T16:03:37.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-11T16:03:37.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-11T16:03:37.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-11T16:03:37.784 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-11T16:03:37.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-11T16:03:37.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-11T16:03:37.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-11T16:03:37.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-11T16:03:37.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-11T16:03:37.785 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-11T16:03:37.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-11T16:03:37.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-11T16:03:37.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-11T16:03:37.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-11T16:03:37.786 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-11T16:03:37.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs 2023-10-11T16:03:37.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-11T16:03:37.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-11T16:03:37.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-11T16:03:37.787 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-11T16:03:37.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-11T16:03:37.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-11T16:03:37.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-11T16:03:37.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-11T16:03:37.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-11T16:03:37.788 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-11T16:03:37.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-11T16:03:37.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-11T16:03:37.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-11T16:03:37.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-11T16:03:37.789 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-11T16:03:37.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-11T16:03:37.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-11T16:03:37.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-11T16:03:37.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-11T16:03:37.790 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-11T16:03:37.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-11T16:03:37.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-11T16:03:37.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-11T16:03:37.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-11T16:03:37.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-11T16:03:37.791 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-11T16:03:37.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-11T16:03:37.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-11T16:03:37.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-11T16:03:37.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-11T16:03:37.792 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-11T16:03:37.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-11T16:03:37.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-11T16:03:37.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-11T16:03:37.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-11T16:03:37.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-11T16:03:37.793 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-11T16:03:37.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-11T16:03:37.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-11T16:03:37.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-11T16:03:37.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-11T16:03:37.794 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-11T16:03:37.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-11T16:03:37.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-11T16:03:37.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-11T16:03:37.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-11T16:03:37.795 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-11T16:03:37.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-11T16:03:37.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-11T16:03:37.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-11T16:03:37.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-11T16:03:37.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-11T16:03:37.796 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-11T16:03:37.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-11T16:03:37.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-11T16:03:37.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-11T16:03:37.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-11T16:03:37.797 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-11T16:03:37.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_quota.ko 2023-10-11T16:03:37.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-11T16:03:37.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-11T16:03:37.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-11T16:03:37.798 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-11T16:03:37.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tunnel.ko 2023-10-11T16:03:37.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-11T16:03:37.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-11T16:03:37.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-11T16:03:37.799 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-11T16:03:37.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-11T16:03:37.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-11T16:03:37.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-11T16:03:37.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-11T16:03:37.800 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-11T16:03:37.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-11T16:03:37.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-11T16:03:37.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-11T16:03:37.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-11T16:03:37.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-11T16:03:37.801 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-11T16:03:37.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-11T16:03:37.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-11T16:03:37.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-11T16:03:37.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-11T16:03:37.802 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-11T16:03:37.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-11T16:03:37.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-11T16:03:37.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-11T16:03:37.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-11T16:03:37.803 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-11T16:03:37.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-11T16:03:37.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-11T16:03:37.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-11T16:03:37.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-11T16:03:37.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-11T16:03:37.804 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-11T16:03:37.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-11T16:03:37.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-11T16:03:37.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-11T16:03:37.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-11T16:03:37.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-11T16:03:37.805 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-11T16:03:37.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-11T16:03:37.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-11T16:03:37.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-11T16:03:37.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-11T16:03:37.806 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-11T16:03:37.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-11T16:03:37.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-11T16:03:37.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-11T16:03:37.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-11T16:03:37.807 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-11T16:03:37.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-11T16:03:37.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-11T16:03:37.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-11T16:03:37.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-11T16:03:37.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-11T16:03:37.808 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-11T16:03:37.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-11T16:03:37.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-11T16:03:37.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-11T16:03:37.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-11T16:03:37.809 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-11T16:03:37.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-11T16:03:37.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-11T16:03:37.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-11T16:03:37.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-11T16:03:37.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-11T16:03:37.810 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-11T16:03:37.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-11T16:03:37.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-11T16:03:37.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-11T16:03:37.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-11T16:03:37.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-11T16:03:37.811 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-11T16:03:37.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-11T16:03:37.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink 2023-10-11T16:03:37.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-11T16:03:37.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh 2023-10-11T16:03:37.812 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-11T16:03:37.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch 2023-10-11T16:03:37.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-11T16:03:37.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-11T16:03:37.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-11T16:03:37.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-11T16:03:37.813 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet 2023-10-11T16:03:37.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-11T16:03:37.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds 2023-10-11T16:03:37.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-11T16:03:37.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-11T16:03:37.814 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-11T16:03:37.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched 2023-10-11T16:03:37.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-11T16:03:37.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-11T16:03:37.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-11T16:03:37.815 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-11T16:03:37.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-11T16:03:37.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-11T16:03:37.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-11T16:03:37.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-11T16:03:37.816 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-11T16:03:37.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-11T16:03:37.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-11T16:03:37.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-11T16:03:37.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-11T16:03:37.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-11T16:03:37.817 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-11T16:03:37.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-11T16:03:37.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-11T16:03:37.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-11T16:03:37.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-11T16:03:37.818 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-11T16:03:37.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-11T16:03:37.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-11T16:03:37.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-11T16:03:37.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-11T16:03:37.819 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-11T16:03:37.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-11T16:03:37.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-11T16:03:37.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-11T16:03:37.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-11T16:03:37.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-11T16:03:37.820 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-11T16:03:37.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-11T16:03:37.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-11T16:03:37.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-11T16:03:37.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-11T16:03:37.821 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-11T16:03:37.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-11T16:03:37.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-11T16:03:37.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-11T16:03:37.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-11T16:03:37.822 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-11T16:03:37.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-11T16:03:37.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-11T16:03:37.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-11T16:03:37.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp 2023-10-11T16:03:37.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-11T16:03:37.823 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-11T16:03:37.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc 2023-10-11T16:03:37.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss 2023-10-11T16:03:37.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-11T16:03:37.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-11T16:03:37.824 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-11T16:03:37.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma 2023-10-11T16:03:37.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-11T16:03:37.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc 2023-10-11T16:03:37.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-11T16:03:37.825 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-11T16:03:37.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix 2023-10-11T16:03:37.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-11T16:03:37.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock 2023-10-11T16:03:37.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-11T16:03:37.826 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-11T16:03:37.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-11T16:03:37.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-11T16:03:37.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-11T16:03:37.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm 2023-10-11T16:03:37.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-11T16:03:37.827 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security 2023-10-11T16:03:37.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys 2023-10-11T16:03:37.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys 2023-10-11T16:03:37.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-11T16:03:37.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys 2023-10-11T16:03:37.828 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-11T16:03:37.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt 2023-10-11T16:03:37.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib 2023-10-11T16:03:37.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-11T16:03:37.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-11T16:03:37.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-11T16:03:37.829 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-11T16:03:37.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-11T16:03:37.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-11T16:03:37.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-11T16:03:37.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-11T16:03:37.830 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-11T16:03:37.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-11T16:03:37.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-11T16:03:37.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-11T16:03:37.831 INFO:teuthology.orchestra.run.smithi111.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-11T16:03:37.832 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.832 DEBUG:teuthology.orchestra.run.smithi111:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-11T16:03:37.860 INFO:teuthology.orchestra.run.smithi111.stdout:postinstall scriptlet (using /bin/sh): 2023-10-11T16:03:37.860 INFO:teuthology.orchestra.run.smithi111.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g47672119149b -a -r /boot/System.map-6.6.0-rc5-g47672119149b ]; then 2023-10-11T16:03:37.860 INFO:teuthology.orchestra.run.smithi111.stdout:cp /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm 2023-10-11T16:03:37.860 INFO:teuthology.orchestra.run.smithi111.stdout:cp /boot/System.map-6.6.0-rc5-g47672119149b /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-11T16:03:37.861 INFO:teuthology.orchestra.run.smithi111.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/System.map-6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.861 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/installkernel 6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-11T16:03:37.861 INFO:teuthology.orchestra.run.smithi111.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-11T16:03:37.861 INFO:teuthology.orchestra.run.smithi111.stdout:fi 2023-10-11T16:03:37.861 INFO:teuthology.orchestra.run.smithi111.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-11T16:03:37.861 INFO:teuthology.orchestra.run.smithi111.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-11T16:03:37.862 INFO:teuthology.orchestra.run.smithi111.stdout:new-kernel-pkg --remove 6.6.0-rc5-g47672119149b --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g47672119149b.img 2023-10-11T16:03:37.862 INFO:teuthology.orchestra.run.smithi111.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-11T16:03:37.862 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-install remove 6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.862 INFO:teuthology.orchestra.run.smithi111.stdout:fi 2023-10-11T16:03:37.862 INFO:teuthology.orchestra.run.smithi111.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-11T16:03:37.863 INFO:teuthology.orchestra.run.smithi111.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-11T16:03:37.863 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.863 INFO:teuthology.orchestra.run.smithi111.stdout:fi 2023-10-11T16:03:37.864 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2023-10-11T16:03:37.945 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2023-10-11T16:03:37.945 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2023-10-11T16:03:37.945 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.02 2023-10-11T16:03:37.945 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 123.el8 2023-10-11T16:03:37.946 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2023-10-11T16:03:37.946 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-10-11T16:03:37.946 INFO:teuthology.orchestra.run.smithi111.stdout:Group : System Environment/Base 2023-10-11T16:03:37.946 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 9265450 2023-10-11T16:03:37.946 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2023-10-11T16:03:37.946 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-10-11T16:03:37.947 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-10-11T16:03:37.947 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-10-11T16:03:37.947 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-10-11T16:03:37.947 INFO:teuthology.orchestra.run.smithi111.stdout:Relocations : (not relocatable) 2023-10-11T16:03:37.947 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : Red Hat, Inc. 2023-10-11T16:03:37.947 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : Red Hat, Inc. 2023-10-11T16:03:37.948 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-11T16:03:37.948 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2023-10-11T16:03:37.948 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2023-10-11T16:03:37.948 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:03:37.948 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-11T16:03:37.948 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-11T16:03:37.949 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-11T16:03:37.949 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2023-10-11T16:03:37.949 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:03:37.949 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2023-10-11T16:03:37.950 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-11T16:03:37.950 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 6.6.0-rc5-g47672119149b 2023-10-11T16:03:37.951 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-11T16:03:39.187 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2023-10-11T16:03:40.978 INFO:teuthology.orchestra.run.smithi111.stderr:done 2023-10-11T16:03:40.979 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2023-10-11T16:03:40.996 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.Nbj4D2fhtz 2023-10-11T16:03:40.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Nbj4D2fhtz 2023-10-11T16:03:41.070 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.Nbj4D2fhtz 2023-10-11T16:03:41.227 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.Nbj4D2fhtz is 6KB 2023-10-11T16:03:41.276 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.Nbj4D2fhtz 2023-10-11T16:03:41.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2023-10-11T16:03:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-10-11T16:03:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-10-11T16:03:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.6.0-rc5-g47672119149b.conf 2023-10-11T16:03:41.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.6.0-rc5-g47672119149b 2023-10-11T16:03:41.552 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2023-10-11T16:04:11.555 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-11T16:04:11.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T16:04:11.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:04:46.075 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-10-11T16:04:55.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T16:04:55.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:04:58.138 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-10-11T16:05:10.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T16:05:10.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:05:28.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-10-11T16:05:43.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T16:05:43.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:05:46.650 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2023-10-11T16:06:04.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T16:06:04.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:06:05.037 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-10-11T16:06:05.459 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T16:06:05.459 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-g47672119149b"... 2023-10-11T16:06:05.460 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2023-10-11T16:06:05.508 INFO:teuthology.orchestra.run.smithi111.stdout:6.6.0-rc5-g47672119149b 2023-10-11T16:06:05.508 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-11T16:06:05.508 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-11T16:06:05.509 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-11T16:06:06.509 DEBUG:teuthology.parallel:result is None 2023-10-11T16:06:06.510 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-11T16:06:06.517 INFO:teuthology.task.internal:Creating test directory... 2023-10-11T16:06:06.518 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-11T16:06:06.523 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-11T16:06:06.526 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-11T16:06:06.565 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-11T16:06:06.573 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-11T16:06:06.616 INFO:teuthology.task.internal:Creating archive directory... 2023-10-11T16:06:06.617 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-11T16:06:06.622 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-11T16:06:06.624 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-11T16:06:06.684 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-11T16:06:06.695 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-11T16:06:06.696 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-11T16:06:06.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:06:06.714 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-11T16:06:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:06:06.734 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-11T16:06:06.739 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-11T16:06:06.757 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-11T16:06:06.833 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T16:06:06.850 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T16:06:06.854 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T16:06:06.865 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T16:06:06.882 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T16:06:06.892 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T16:06:06.895 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-11T16:06:06.907 INFO:teuthology.task.internal:Configuring sudo... 2023-10-11T16:06:06.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-11T16:06:06.912 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-11T16:06:06.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-11T16:06:06.976 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-11T16:06:06.986 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-11T16:06:06.987 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-11T16:06:06.992 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-11T16:06:06.994 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-11T16:06:07.030 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T16:06:07.143 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T16:06:07.212 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T16:06:07.285 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T16:06:07.368 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:06:07.368 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-11T16:06:07.437 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T16:06:07.482 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T16:06:07.535 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T16:06:07.621 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T16:06:07.663 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:06:07.663 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-11T16:06:07.733 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T16:06:07.782 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T16:06:07.833 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T16:06:07.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T16:06:07.966 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:06:07.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-11T16:06:08.036 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2023-10-11T16:06:08.041 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2023-10-11T16:06:08.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-10-11T16:06:08.106 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T16:06:08.141 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T16:06:08.157 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T16:06:08.632 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-11T16:06:08.642 INFO:teuthology.task.internal:Starting timer... 2023-10-11T16:06:08.642 INFO:teuthology.run_tasks:Running task pcp... 2023-10-11T16:06:08.697 INFO:teuthology.run_tasks:Running task selinux... 2023-10-11T16:06:08.746 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-11T16:06:08.747 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2023-10-11T16:06:08.864 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2023-10-11T16:06:08.868 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2023-10-11T16:06:08.940 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2023-10-11T16:06:08.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2023-10-11T16:06:09.008 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2023-10-11T16:06:09.009 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-11T16:06:09.010 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2023-10-11T16:06:09.064 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2023-10-11T16:06:09.064 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2023-10-11T16:06:09.097 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2023-10-11T16:06:09.097 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2023-10-11T16:06:09.124 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2023-10-11T16:06:09.124 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi130.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2023-10-11T16:06:09.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T16:06:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:06:09.200 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T16:06:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:06:09.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T16:06:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:06:09.256 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-11T16:06:09.256 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2023-10-11T16:06:09.332 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2023-10-11T16:06:09.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2023-10-11T16:06:09.386 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-11T16:06:09.397 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-11T16:06:09.403 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-11T16:06:09.534 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-11T16:06:09.551 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-11T16:06:09.553 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com,smithi130.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-11T16:18:37.113 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi130.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2023-10-11T16:18:37.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T16:18:37.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:18:37.207 DEBUG:teuthology.orchestra.run.smithi111:> true 2023-10-11T16:18:37.279 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2023-10-11T16:18:37.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2023-10-11T16:18:37.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:18:37.348 DEBUG:teuthology.orchestra.run.smithi130:> true 2023-10-11T16:18:37.420 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2023-10-11T16:18:37.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-11T16:18:37.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T16:18:37.489 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-10-11T16:18:37.556 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-10-11T16:18:37.557 INFO:teuthology.run_tasks:Running task clock... 2023-10-11T16:18:37.567 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-11T16:18:37.567 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-11T16:18:37.568 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T16:18:37.573 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-11T16:18:37.573 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T16:18:37.575 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-11T16:18:37.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T16:18:37.608 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-11T16:18:37.624 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-11T16:18:37.629 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-11T16:18:37.642 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-11T16:18:37.645 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-11T16:18:37.676 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-11T16:18:37.707 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2023-10-11T16:18:37.717 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2023-10-11T16:18:37.719 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2023-10-11T16:18:37.728 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2023-10-11T16:18:37.734 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-11T16:18:37.743 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-11T16:18:37.748 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-11T16:18:37.757 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-11T16:18:37.767 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2023-10-11T16:18:37.796 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2023-10-11T16:18:37.822 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2023-10-11T16:18:37.824 INFO:teuthology.orchestra.run.smithi150.stderr:bash: ntpq: command not found 2023-10-11T16:18:37.825 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T16:18:37.825 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2023-10-11T16:18:37.825 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.825 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.826 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.826 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.827 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T16:18:37.828 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-10-11T16:18:37.828 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.828 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.828 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.828 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.830 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-11T16:18:37.864 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-11T16:18:37.971 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2023-10-11T16:18:37.978 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T16:18:37.978 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2023-10-11T16:18:37.978 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.978 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.979 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.979 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:18:37.980 INFO:teuthology.run_tasks:Running task pexec... 2023-10-11T16:18:37.991 INFO:teuthology.task.pexec:Executing custom commands... 2023-10-11T16:18:37.991 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:18:37.992 DEBUG:teuthology.orchestra.run.smithi111:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-11T16:18:37.992 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi130.front.sepia.ceph.com 2023-10-11T16:18:37.992 DEBUG:teuthology.orchestra.run.smithi130:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-11T16:18:37.993 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi150.front.sepia.ceph.com 2023-10-11T16:18:37.993 DEBUG:teuthology.orchestra.run.smithi150:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-11T16:18:38.364 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2023-10-11T16:18:38.365 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-10-11T16:18:38.470 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T16:18:39.774 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:04:22 ago on Wed 11 Oct 2023 04:14:17 PM UTC. 2023-10-11T16:18:39.802 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:21 ago on Wed 11 Oct 2023 04:14:18 PM UTC. 2023-10-11T16:18:40.083 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Oct 2023 04:16:53 PM UTC. 2023-10-11T16:18:41.658 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-11T16:18:41.659 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-11T16:18:41.659 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-10-11T16:18:41.660 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-11T16:18:41.660 INFO:teuthology.orchestra.run.smithi150.stdout:Resetting modules: 2023-10-11T16:18:41.660 INFO:teuthology.orchestra.run.smithi150.stdout: container-tools 2023-10-11T16:18:41.660 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:18:41.661 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-11T16:18:41.661 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-10-11T16:18:41.661 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:18:41.675 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2023-10-11T16:18:41.676 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-10-11T16:18:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2023-10-11T16:18:41.677 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-10-11T16:18:41.677 INFO:teuthology.orchestra.run.smithi130.stdout:Resetting modules: 2023-10-11T16:18:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools 2023-10-11T16:18:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:18:41.677 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2023-10-11T16:18:41.678 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-10-11T16:18:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:18:41.924 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T16:18:41.926 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T16:18:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2023-10-11T16:18:41.926 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T16:18:41.926 INFO:teuthology.orchestra.run.smithi111.stdout:Resetting modules: 2023-10-11T16:18:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools 2023-10-11T16:18:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:18:41.927 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T16:18:41.927 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2023-10-11T16:18:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:18:42.625 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:24 ago on Wed 11 Oct 2023 04:14:18 PM UTC. 2023-10-11T16:18:42.630 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:04:25 ago on Wed 11 Oct 2023 04:14:17 PM UTC. 2023-10-11T16:18:43.164 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:49 ago on Wed 11 Oct 2023 04:16:53 PM UTC. 2023-10-11T16:18:48.148 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-11T16:18:48.621 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-10-11T16:18:48.981 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T16:18:49.571 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T16:18:49.986 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2023-10-11T16:18:50.125 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:04:32 ago on Wed 11 Oct 2023 04:14:17 PM UTC. 2023-10-11T16:18:50.455 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2023-10-11T16:18:51.073 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Oct 2023 04:16:53 PM UTC. 2023-10-11T16:18:51.856 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-11T16:18:51.944 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:33 ago on Wed 11 Oct 2023 04:14:18 PM UTC. 2023-10-11T16:18:51.948 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================================ 2023-10-11T16:18:51.949 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-11T16:18:51.949 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================================ 2023-10-11T16:18:51.949 INFO:teuthology.orchestra.run.smithi150.stdout:Installing group/module packages: 2023-10-11T16:18:51.949 INFO:teuthology.orchestra.run.smithi150.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-10-11T16:18:51.949 INFO:teuthology.orchestra.run.smithi150.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-10-11T16:18:51.950 INFO:teuthology.orchestra.run.smithi150.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-10-11T16:18:51.950 INFO:teuthology.orchestra.run.smithi150.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-10-11T16:18:51.950 INFO:teuthology.orchestra.run.smithi150.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-10-11T16:18:51.950 INFO:teuthology.orchestra.run.smithi150.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-10-11T16:18:51.951 INFO:teuthology.orchestra.run.smithi150.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-10-11T16:18:51.951 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-11T16:18:51.951 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-10-11T16:18:51.952 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-10-11T16:18:51.952 INFO:teuthology.orchestra.run.smithi150.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-10-11T16:18:51.952 INFO:teuthology.orchestra.run.smithi150.stdout:Installing module profiles: 2023-10-11T16:18:51.953 INFO:teuthology.orchestra.run.smithi150.stdout: container-tools/common 2023-10-11T16:18:51.953 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:18:51.953 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling module streams: 2023-10-11T16:18:51.954 INFO:teuthology.orchestra.run.smithi150.stdout: container-tools rhel8 2023-10-11T16:18:51.955 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:18:51.955 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-11T16:18:51.956 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================================ 2023-10-11T16:18:51.956 INFO:teuthology.orchestra.run.smithi150.stdout:Install 10 Packages 2023-10-11T16:18:51.956 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:18:51.957 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 18 M 2023-10-11T16:18:51.958 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 65 M 2023-10-11T16:18:51.958 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-11T16:18:52.167 INFO:teuthology.orchestra.run.smithi150.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 189 kB/s | 41 kB 00:00 2023-10-11T16:18:52.192 INFO:teuthology.orchestra.run.smithi150.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 103 kB/s | 25 kB 00:00 2023-10-11T16:18:52.226 INFO:teuthology.orchestra.run.smithi150.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 342 kB/s | 94 kB 00:00 2023-10-11T16:18:52.385 INFO:teuthology.orchestra.run.smithi150.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 227 kB/s | 49 kB 00:00 2023-10-11T16:18:52.418 INFO:teuthology.orchestra.run.smithi150.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.3 MB/s | 524 kB 00:00 2023-10-11T16:18:52.494 INFO:teuthology.orchestra.run.smithi150.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.3 MB/s | 149 kB 00:00 2023-10-11T16:18:52.586 INFO:teuthology.orchestra.run.smithi150.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-10-11T16:18:52.611 INFO:teuthology.orchestra.run.smithi150.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.1 MB/s | 213 kB 00:00 2023-10-11T16:18:52.736 INFO:teuthology.orchestra.run.smithi150.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 29 MB/s | 7.0 MB 00:00 2023-10-11T16:18:52.786 INFO:teuthology.orchestra.run.smithi150.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 11 MB/s | 2.2 MB 00:00 2023-10-11T16:18:52.787 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-11T16:18:52.787 INFO:teuthology.orchestra.run.smithi150.stdout:Total 22 MB/s | 18 MB 00:00 2023-10-11T16:18:52.868 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T16:18:52.967 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================================ 2023-10-11T16:18:52.968 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T16:18:52.968 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================================ 2023-10-11T16:18:52.968 INFO:teuthology.orchestra.run.smithi111.stdout:Installing group/module packages: 2023-10-11T16:18:52.969 INFO:teuthology.orchestra.run.smithi111.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-10-11T16:18:52.969 INFO:teuthology.orchestra.run.smithi111.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-10-11T16:18:52.969 INFO:teuthology.orchestra.run.smithi111.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-10-11T16:18:52.970 INFO:teuthology.orchestra.run.smithi111.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-10-11T16:18:52.970 INFO:teuthology.orchestra.run.smithi111.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-10-11T16:18:52.970 INFO:teuthology.orchestra.run.smithi111.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-10-11T16:18:52.971 INFO:teuthology.orchestra.run.smithi111.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-10-11T16:18:52.971 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T16:18:52.972 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-10-11T16:18:52.972 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-10-11T16:18:52.972 INFO:teuthology.orchestra.run.smithi111.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-10-11T16:18:52.972 INFO:teuthology.orchestra.run.smithi111.stdout:Installing module profiles: 2023-10-11T16:18:52.972 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools/common 2023-10-11T16:18:52.973 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:18:52.973 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling module streams: 2023-10-11T16:18:52.973 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools rhel8 2023-10-11T16:18:52.973 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:18:52.973 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T16:18:52.973 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================================ 2023-10-11T16:18:52.974 INFO:teuthology.orchestra.run.smithi111.stdout:Install 10 Packages 2023-10-11T16:18:52.974 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:18:52.975 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 18 M 2023-10-11T16:18:52.975 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 65 M 2023-10-11T16:18:52.975 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T16:18:52.985 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-11T16:18:53.004 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-11T16:18:53.004 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-11T16:18:53.107 INFO:teuthology.orchestra.run.smithi111.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 686 kB/s | 94 kB 00:00 2023-10-11T16:18:53.166 INFO:teuthology.orchestra.run.smithi111.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 209 kB/s | 41 kB 00:00 2023-10-11T16:18:53.186 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-11T16:18:53.189 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-11T16:18:53.191 INFO:teuthology.orchestra.run.smithi111.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 112 kB/s | 25 kB 00:00 2023-10-11T16:18:53.241 INFO:teuthology.orchestra.run.smithi111.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 369 kB/s | 49 kB 00:00 2023-10-11T16:18:53.283 INFO:teuthology.orchestra.run.smithi111.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 4.4 MB/s | 524 kB 00:00 2023-10-11T16:18:53.400 INFO:teuthology.orchestra.run.smithi111.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 941 kB/s | 149 kB 00:00 2023-10-11T16:18:53.425 INFO:teuthology.orchestra.run.smithi111.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.5 MB/s | 213 kB 00:00 2023-10-11T16:18:53.559 INFO:teuthology.orchestra.run.smithi111.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 16 MB/s | 2.2 MB 00:00 2023-10-11T16:18:53.652 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2023-10-11T16:18:53.684 INFO:teuthology.orchestra.run.smithi111.stdout:(9/10): buildah-1.26.2-1.module+el8.6.0+16771+2 16 MB/s | 8.0 MB 00:00 2023-10-11T16:18:53.708 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-11T16:18:53.744 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================================ 2023-10-11T16:18:53.745 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2023-10-11T16:18:53.745 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================================ 2023-10-11T16:18:53.745 INFO:teuthology.orchestra.run.smithi130.stdout:Installing group/module packages: 2023-10-11T16:18:53.745 INFO:teuthology.orchestra.run.smithi130.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-10-11T16:18:53.746 INFO:teuthology.orchestra.run.smithi130.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-10-11T16:18:53.746 INFO:teuthology.orchestra.run.smithi130.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-10-11T16:18:53.746 INFO:teuthology.orchestra.run.smithi130.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-10-11T16:18:53.746 INFO:teuthology.orchestra.run.smithi130.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-10-11T16:18:53.746 INFO:teuthology.orchestra.run.smithi130.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-10-11T16:18:53.747 INFO:teuthology.orchestra.run.smithi130.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-10-11T16:18:53.747 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2023-10-11T16:18:53.747 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-10-11T16:18:53.747 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-10-11T16:18:53.748 INFO:teuthology.orchestra.run.smithi130.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-10-11T16:18:53.748 INFO:teuthology.orchestra.run.smithi130.stdout:Installing module profiles: 2023-10-11T16:18:53.748 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools/common 2023-10-11T16:18:53.748 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:18:53.748 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling module streams: 2023-10-11T16:18:53.749 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools rhel8 2023-10-11T16:18:53.749 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:18:53.749 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2023-10-11T16:18:53.749 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================================ 2023-10-11T16:18:53.749 INFO:teuthology.orchestra.run.smithi130.stdout:Install 10 Packages 2023-10-11T16:18:53.750 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:18:53.750 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 18 M 2023-10-11T16:18:53.751 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 65 M 2023-10-11T16:18:53.751 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2023-10-11T16:18:53.784 INFO:teuthology.orchestra.run.smithi111.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 18 MB/s | 7.0 MB 00:00 2023-10-11T16:18:53.785 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T16:18:53.785 INFO:teuthology.orchestra.run.smithi111.stdout:Total 22 MB/s | 18 MB 00:00 2023-10-11T16:18:53.824 INFO:teuthology.orchestra.run.smithi130.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-10-11T16:18:53.849 INFO:teuthology.orchestra.run.smithi130.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 244 kB/s | 25 kB 00:00 2023-10-11T16:18:53.866 INFO:teuthology.orchestra.run.smithi130.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 343 kB/s | 41 kB 00:00 2023-10-11T16:18:53.872 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-10-11T16:18:53.891 INFO:teuthology.orchestra.run.smithi130.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 736 kB/s | 49 kB 00:00 2023-10-11T16:18:53.925 INFO:teuthology.orchestra.run.smithi130.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.8 MB/s | 524 kB 00:00 2023-10-11T16:18:53.950 INFO:teuthology.orchestra.run.smithi130.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-10-11T16:18:53.976 INFO:teuthology.orchestra.run.smithi130.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-10-11T16:18:54.037 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-11T16:18:54.068 INFO:teuthology.orchestra.run.smithi130.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-10-11T16:18:54.123 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T16:18:54.146 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T16:18:54.147 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T16:18:54.176 INFO:teuthology.orchestra.run.smithi130.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 31 MB/s | 7.0 MB 00:00 2023-10-11T16:18:54.222 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-10-11T16:18:54.260 INFO:teuthology.orchestra.run.smithi130.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 20 MB/s | 8.0 MB 00:00 2023-10-11T16:18:54.261 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2023-10-11T16:18:54.261 INFO:teuthology.orchestra.run.smithi130.stdout:Total 35 MB/s | 18 MB 00:00 2023-10-11T16:18:54.417 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-10-11T16:18:54.461 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2023-10-11T16:18:54.480 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2023-10-11T16:18:54.480 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2023-10-11T16:18:54.663 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2023-10-11T16:18:54.667 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2023-10-11T16:18:54.723 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T16:18:54.728 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T16:18:54.801 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-10-11T16:18:55.219 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2023-10-11T16:18:55.297 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T16:18:55.392 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-10-11T16:18:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-10-11T16:18:55.552 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-10-11T16:18:55.600 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-11T16:18:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-11T16:18:55.795 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-10-11T16:18:55.872 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-10-11T16:18:55.980 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-10-11T16:18:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-10-11T16:18:56.337 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-10-11T16:18:56.417 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-10-11T16:18:56.512 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-10-11T16:18:56.640 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-10-11T16:18:56.784 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-10-11T16:18:56.971 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-11T16:18:57.100 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-10-11T16:18:57.242 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-11T16:18:57.242 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-10-11T16:18:57.242 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-11T16:18:57.242 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-10-11T16:18:57.243 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-10-11T16:18:57.243 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-10-11T16:18:57.243 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-10-11T16:18:57.243 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-10-11T16:18:57.243 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-10-11T16:18:57.244 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-10-11T16:18:57.363 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-10-11T16:18:57.957 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-10-11T16:18:58.172 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-10-11T16:18:58.283 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-10-11T16:18:58.332 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-10-11T16:18:58.477 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-10-11T16:18:58.477 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:04:40 ago on Wed 11 Oct 2023 04:14:17 PM UTC. 2023-10-11T16:18:58.501 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-11T16:18:58.532 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-10-11T16:18:58.725 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-10-11T16:18:58.768 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-11T16:18:58.768 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-10-11T16:18:58.768 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-11T16:18:58.768 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-10-11T16:18:58.769 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-10-11T16:18:58.769 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-10-11T16:18:58.769 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-10-11T16:18:58.769 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-10-11T16:18:58.769 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-10-11T16:18:58.770 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-10-11T16:18:58.917 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-11T16:18:59.305 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-11T16:18:59.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-10-11T16:18:59.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-11T16:18:59.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-10-11T16:18:59.306 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-10-11T16:18:59.306 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-10-11T16:18:59.306 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-10-11T16:18:59.306 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-10-11T16:18:59.307 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-10-11T16:18:59.307 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-10-11T16:18:59.926 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-10-11T16:18:59.926 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:41 ago on Wed 11 Oct 2023 04:14:18 PM UTC. 2023-10-11T16:19:00.754 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-10-11T16:19:00.755 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:07 ago on Wed 11 Oct 2023 04:16:53 PM UTC. 2023-10-11T16:19:05.090 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-10-11T16:19:05.300 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:19:05.301 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-11T16:19:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-10-11T16:19:05.301 INFO:teuthology.orchestra.run.smithi150.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-10-11T16:19:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-11T16:19:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-11T16:19:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:05.302 INFO:teuthology.orchestra.run.smithi150.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-10-11T16:19:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: yajl-2.1.0-10.el8.x86_64 2023-10-11T16:19:05.303 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:19:05.303 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-11T16:19:05.479 DEBUG:teuthology.parallel:result is None 2023-10-11T16:19:08.067 INFO:teuthology.orchestra.run.smithi130.stdout:Installed products updated. 2023-10-11T16:19:08.114 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T16:19:08.265 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:19:08.265 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2023-10-11T16:19:08.265 INFO:teuthology.orchestra.run.smithi130.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:08.266 INFO:teuthology.orchestra.run.smithi130.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-10-11T16:19:08.266 INFO:teuthology.orchestra.run.smithi130.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:08.266 INFO:teuthology.orchestra.run.smithi130.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-10-11T16:19:08.266 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-11T16:19:08.266 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-11T16:19:08.267 INFO:teuthology.orchestra.run.smithi130.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:08.267 INFO:teuthology.orchestra.run.smithi130.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:08.267 INFO:teuthology.orchestra.run.smithi130.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-10-11T16:19:08.268 INFO:teuthology.orchestra.run.smithi130.stdout: yajl-2.1.0-10.el8.x86_64 2023-10-11T16:19:08.268 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:19:08.268 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2023-10-11T16:19:08.321 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:19:08.321 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T16:19:08.321 INFO:teuthology.orchestra.run.smithi111.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:08.322 INFO:teuthology.orchestra.run.smithi111.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-10-11T16:19:08.322 INFO:teuthology.orchestra.run.smithi111.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:08.322 INFO:teuthology.orchestra.run.smithi111.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-10-11T16:19:08.323 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-11T16:19:08.323 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-11T16:19:08.323 INFO:teuthology.orchestra.run.smithi111.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:08.323 INFO:teuthology.orchestra.run.smithi111.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-10-11T16:19:08.324 INFO:teuthology.orchestra.run.smithi111.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-10-11T16:19:08.324 INFO:teuthology.orchestra.run.smithi111.stdout: yajl-2.1.0-10.el8.x86_64 2023-10-11T16:19:08.324 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:19:08.324 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T16:19:08.424 DEBUG:teuthology.parallel:result is None 2023-10-11T16:19:08.523 DEBUG:teuthology.parallel:result is None 2023-10-11T16:19:08.523 INFO:teuthology.run_tasks:Running task install... 2023-10-11T16:19:08.533 DEBUG:teuthology.task.install:project ceph 2023-10-11T16:19:08.534 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612'}} 2023-10-11T16:19:08.534 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612'} 2023-10-11T16:19:08.534 INFO:teuthology.task.install:Using flavor: default 2023-10-11T16:19:08.540 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-11T16:19:08.541 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-11T16:19:08.541 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'tag': None, 'wait_for_package': False} 2023-10-11T16:19:08.542 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:19:08.544 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'tag': None, 'wait_for_package': False} 2023-10-11T16:19:08.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:19:08.546 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'tag': None, 'wait_for_package': False} 2023-10-11T16:19:08.546 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:19:08.691 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/ 2023-10-11T16:19:08.692 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3908.g05a43c8f 2023-10-11T16:19:08.764 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/ 2023-10-11T16:19:08.764 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3908.g05a43c8f 2023-10-11T16:19:08.774 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/ 2023-10-11T16:19:08.774 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3908.g05a43c8f 2023-10-11T16:19:08.821 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-11T16:19:08.822 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:19:08.822 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-11T16:19:08.875 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-11T16:19:08.876 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:19:08.876 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-11T16:19:08.883 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-11T16:19:08.883 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:19:08.883 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-11T16:19:08.893 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-11T16:19:08.893 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/05a43c8fb3988914c879a50970f9054a098e0612/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-11T16:19:08.903 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-11T16:19:08.903 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/05a43c8fb3988914c879a50970f9054a098e0612/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-11T16:19:08.910 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-11T16:19:08.911 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/05a43c8fb3988914c879a50970f9054a098e0612/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-11T16:19:08.973 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-11T16:19:08.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-11T16:19:09.002 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-11T16:19:09.049 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-11T16:19:09.054 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-11T16:19:09.099 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-11T16:19:09.116 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2023-10-11T16:19:09.117 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2023-10-11T16:19:09.119 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2023-10-11T16:19:09.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2023-10-11T16:19:09.167 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2023-10-11T16:19:09.170 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2023-10-11T16:19:09.499 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2023-10-11T16:19:09.511 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-10-11T16:19:09.616 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T16:19:10.209 INFO:teuthology.orchestra.run.smithi130.stdout:44 files removed 2023-10-11T16:19:10.222 INFO:teuthology.orchestra.run.smithi150.stdout:44 files removed 2023-10-11T16:19:10.263 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-11T16:19:10.273 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-11T16:19:10.355 INFO:teuthology.orchestra.run.smithi111.stdout:44 files removed 2023-10-11T16:19:10.410 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-11T16:19:10.618 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2023-10-11T16:19:10.619 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-10-11T16:19:10.859 INFO:teuthology.orchestra.run.smithi111.stdout:Updating Subscription Management repositories. 2023-10-11T16:19:11.718 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 222 kB/s | 80 kB 00:00 2023-10-11T16:19:11.752 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 208 kB/s | 80 kB 00:00 2023-10-11T16:19:12.129 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 25 kB/s | 9.4 kB 00:00 2023-10-11T16:19:12.146 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-10-11T16:19:12.413 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 5.7 kB/s | 1.5 kB 00:00 2023-10-11T16:19:12.430 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 212 kB/s | 80 kB 00:00 2023-10-11T16:19:12.541 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 4.0 kB/s | 1.5 kB 00:00 2023-10-11T16:19:12.556 INFO:teuthology.orchestra.run.smithi150.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-10-11T16:19:12.709 INFO:teuthology.orchestra.run.smithi130.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-10-11T16:19:12.755 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 32 kB/s | 9.4 kB 00:00 2023-10-11T16:19:13.106 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2023-10-11T16:19:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-10-11T16:19:14.200 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 16 MB 00:01 2023-10-11T16:19:14.708 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-10-11T16:19:14.835 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 16 MB 00:02 2023-10-11T16:19:17.707 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 384 kB/s | 24 kB 00:00 2023-10-11T16:19:18.342 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 576 kB/s | 24 kB 00:00 2023-10-11T16:19:18.783 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 432 kB/s | 24 kB 00:00 2023-10-11T16:19:18.959 INFO:teuthology.orchestra.run.smithi150.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 MB/s | 47 MB 00:01 2023-10-11T16:19:19.543 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 MB/s | 53 MB 00:01 2023-10-11T16:19:20.160 INFO:teuthology.orchestra.run.smithi111.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 MB/s | 47 MB 00:01 2023-10-11T16:19:27.202 INFO:teuthology.orchestra.run.smithi150.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 MB/s | 53 MB 00:01 2023-10-11T16:19:27.744 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 MB/s | 47 MB 00:00 2023-10-11T16:19:29.578 INFO:teuthology.orchestra.run.smithi111.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 MB/s | 53 MB 00:01 2023-10-11T16:19:38.425 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-11T16:19:38.426 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-11T16:19:38.440 INFO:teuthology.orchestra.run.smithi150.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-11T16:19:38.441 INFO:teuthology.orchestra.run.smithi150.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-11T16:19:38.442 INFO:teuthology.orchestra.run.smithi150.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-11T16:19:38.444 INFO:teuthology.orchestra.run.smithi150.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-11T16:19:38.446 INFO:teuthology.orchestra.run.smithi150.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-11T16:19:38.449 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-11T16:19:38.450 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-11T16:19:38.580 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-10-11T16:19:38.592 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================================================== 2023-10-11T16:19:38.592 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-10-11T16:19:38.592 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================================================== 2023-10-11T16:19:38.592 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-10-11T16:19:38.593 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-11T16:19:38.593 INFO:teuthology.orchestra.run.smithi150.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-11T16:19:38.593 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.4 k 2023-10-11T16:19:38.593 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 5.3 M 2023-10-11T16:19:38.593 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 848 k 2023-10-11T16:19:38.593 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 166 k 2023-10-11T16:19:38.594 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 1.5 M 2023-10-11T16:19:38.594 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 127 k 2023-10-11T16:19:38.594 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 1.7 M 2023-10-11T16:19:38.594 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 7.4 M 2023-10-11T16:19:38.594 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 47 k 2023-10-11T16:19:38.595 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 13 M 2023-10-11T16:19:38.595 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 41 M 2023-10-11T16:19:38.595 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 265 k 2023-10-11T16:19:38.595 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 83 k 2023-10-11T16:19:38.595 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 258 k 2023-10-11T16:19:38.596 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 18 k 2023-10-11T16:19:38.596 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-11T16:19:38.596 INFO:teuthology.orchestra.run.smithi150.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-11T16:19:38.596 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-11T16:19:38.596 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 31 k 2023-10-11T16:19:38.597 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 770 k 2023-10-11T16:19:38.597 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 139 k 2023-10-11T16:19:38.597 INFO:teuthology.orchestra.run.smithi150.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-11T16:19:38.597 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-11T16:19:38.597 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-11T16:19:38.598 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-11T16:19:38.598 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-11T16:19:38.598 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-11T16:19:38.598 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 198 k 2023-10-11T16:19:38.598 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 371 k 2023-10-11T16:19:38.599 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 364 k 2023-10-11T16:19:38.599 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 106 k 2023-10-11T16:19:38.599 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 94 k 2023-10-11T16:19:38.599 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.4 M 2023-10-11T16:19:38.599 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 179 k 2023-10-11T16:19:38.600 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-11T16:19:38.600 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2023-10-11T16:19:38.600 INFO:teuthology.orchestra.run.smithi150.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-11T16:19:38.600 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.7 M 2023-10-11T16:19:38.600 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.6 M 2023-10-11T16:19:38.601 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-10-11T16:19:38.601 INFO:teuthology.orchestra.run.smithi150.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-11T16:19:38.601 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 22 M 2023-10-11T16:19:38.601 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 23 k 2023-10-11T16:19:38.601 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 2.3 M 2023-10-11T16:19:38.601 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 240 k 2023-10-11T16:19:38.602 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 4.6 M 2023-10-11T16:19:38.602 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 18 M 2023-10-11T16:19:38.602 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 15 k 2023-10-11T16:19:38.602 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 24 k 2023-10-11T16:19:38.602 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-11T16:19:38.603 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-11T16:19:38.603 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-11T16:19:38.603 INFO:teuthology.orchestra.run.smithi150.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-11T16:19:38.603 INFO:teuthology.orchestra.run.smithi150.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-11T16:19:38.603 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 137 k 2023-10-11T16:19:38.604 INFO:teuthology.orchestra.run.smithi150.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-11T16:19:38.604 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-11T16:19:38.604 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-11T16:19:38.604 INFO:teuthology.orchestra.run.smithi150.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-11T16:19:38.604 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-11T16:19:38.605 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-11T16:19:38.605 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 473 k 2023-10-11T16:19:38.605 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-11T16:19:38.605 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.5 M 2023-10-11T16:19:38.605 INFO:teuthology.orchestra.run.smithi150.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-11T16:19:38.606 INFO:teuthology.orchestra.run.smithi150.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-11T16:19:38.606 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-11T16:19:38.606 INFO:teuthology.orchestra.run.smithi150.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-11T16:19:38.606 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-11T16:19:38.606 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-11T16:19:38.607 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-11T16:19:38.607 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-11T16:19:38.607 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-11T16:19:38.607 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-11T16:19:38.607 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-11T16:19:38.607 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-11T16:19:38.608 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-11T16:19:38.608 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-11T16:19:38.608 INFO:teuthology.orchestra.run.smithi150.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-11T16:19:38.608 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-11T16:19:38.608 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 45 k 2023-10-11T16:19:38.609 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 119 k 2023-10-11T16:19:38.609 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-11T16:19:38.609 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-11T16:19:38.609 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-11T16:19:38.609 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-11T16:19:38.609 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-11T16:19:38.610 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-11T16:19:38.610 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-11T16:19:38.610 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-11T16:19:38.610 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-11T16:19:38.610 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-11T16:19:38.611 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-11T16:19:38.611 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T16:19:38.611 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-11T16:19:38.611 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-11T16:19:38.611 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-11T16:19:38.612 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-11T16:19:38.612 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-11T16:19:38.612 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-11T16:19:38.612 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-11T16:19:38.612 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-11T16:19:38.612 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-11T16:19:38.613 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-11T16:19:38.613 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-11T16:19:38.613 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-11T16:19:38.613 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-11T16:19:38.613 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-11T16:19:38.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-11T16:19:38.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-11T16:19:38.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-11T16:19:38.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-11T16:19:38.614 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T16:19:38.615 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-11T16:19:38.615 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-11T16:19:38.615 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-11T16:19:38.615 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-11T16:19:38.615 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-11T16:19:38.616 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-11T16:19:38.616 INFO:teuthology.orchestra.run.smithi150.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-11T16:19:38.616 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-11T16:19:38.616 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-11T16:19:38.616 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-11T16:19:38.617 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-11T16:19:38.617 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-11T16:19:38.617 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-11T16:19:38.617 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-11T16:19:38.617 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-11T16:19:38.618 INFO:teuthology.orchestra.run.smithi150.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-11T16:19:38.618 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-11T16:19:38.618 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-11T16:19:38.618 INFO:teuthology.orchestra.run.smithi150.stdout:Enabling module streams: 2023-10-11T16:19:38.618 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql 10 2023-10-11T16:19:38.619 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:19:38.619 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-10-11T16:19:38.619 INFO:teuthology.orchestra.run.smithi150.stdout:=========================================================================================================================== 2023-10-11T16:19:38.619 INFO:teuthology.orchestra.run.smithi150.stdout:Install 128 Packages 2023-10-11T16:19:38.619 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 3 Packages 2023-10-11T16:19:38.619 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:19:38.620 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 197 M 2023-10-11T16:19:38.620 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-10-11T16:19:38.635 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-11T16:19:38.636 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-11T16:19:38.651 INFO:teuthology.orchestra.run.smithi130.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-11T16:19:38.652 INFO:teuthology.orchestra.run.smithi130.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-11T16:19:38.653 INFO:teuthology.orchestra.run.smithi130.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-11T16:19:38.655 INFO:teuthology.orchestra.run.smithi130.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-11T16:19:38.657 INFO:teuthology.orchestra.run.smithi130.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-11T16:19:38.660 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-11T16:19:38.662 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-11T16:19:38.770 INFO:teuthology.orchestra.run.smithi150.stdout:(1/131): ceph-18.0.0-3908.g05a43c8f.el8.x86_64. 49 kB/s | 6.4 kB 00:00 2023-10-11T16:19:38.791 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2023-10-11T16:19:38.803 INFO:teuthology.orchestra.run.smithi130.stdout:=========================================================================================================================== 2023-10-11T16:19:38.803 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2023-10-11T16:19:38.803 INFO:teuthology.orchestra.run.smithi130.stdout:=========================================================================================================================== 2023-10-11T16:19:38.804 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2023-10-11T16:19:38.804 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-11T16:19:38.804 INFO:teuthology.orchestra.run.smithi130.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-11T16:19:38.804 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.4 k 2023-10-11T16:19:38.804 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 5.3 M 2023-10-11T16:19:38.805 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 848 k 2023-10-11T16:19:38.805 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 166 k 2023-10-11T16:19:38.805 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 1.5 M 2023-10-11T16:19:38.805 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 127 k 2023-10-11T16:19:38.805 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 1.7 M 2023-10-11T16:19:38.806 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 7.4 M 2023-10-11T16:19:38.806 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 47 k 2023-10-11T16:19:38.806 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 13 M 2023-10-11T16:19:38.806 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 41 M 2023-10-11T16:19:38.807 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 265 k 2023-10-11T16:19:38.807 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 83 k 2023-10-11T16:19:38.807 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 258 k 2023-10-11T16:19:38.807 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 18 k 2023-10-11T16:19:38.807 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-11T16:19:38.808 INFO:teuthology.orchestra.run.smithi130.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-11T16:19:38.808 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-11T16:19:38.808 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 31 k 2023-10-11T16:19:38.808 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 770 k 2023-10-11T16:19:38.808 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 139 k 2023-10-11T16:19:38.809 INFO:teuthology.orchestra.run.smithi130.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-11T16:19:38.809 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-11T16:19:38.809 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-11T16:19:38.809 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-11T16:19:38.809 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-11T16:19:38.810 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-11T16:19:38.810 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 198 k 2023-10-11T16:19:38.810 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 371 k 2023-10-11T16:19:38.810 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 364 k 2023-10-11T16:19:38.810 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 106 k 2023-10-11T16:19:38.811 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 94 k 2023-10-11T16:19:38.811 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.4 M 2023-10-11T16:19:38.811 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 179 k 2023-10-11T16:19:38.811 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-11T16:19:38.811 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2023-10-11T16:19:38.811 INFO:teuthology.orchestra.run.smithi130.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-11T16:19:38.812 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.7 M 2023-10-11T16:19:38.812 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.6 M 2023-10-11T16:19:38.812 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2023-10-11T16:19:38.812 INFO:teuthology.orchestra.run.smithi130.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-11T16:19:38.812 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 22 M 2023-10-11T16:19:38.813 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 23 k 2023-10-11T16:19:38.813 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 2.3 M 2023-10-11T16:19:38.813 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 240 k 2023-10-11T16:19:38.813 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 4.6 M 2023-10-11T16:19:38.813 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 18 M 2023-10-11T16:19:38.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 15 k 2023-10-11T16:19:38.814 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 24 k 2023-10-11T16:19:38.814 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-11T16:19:38.814 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-11T16:19:38.814 INFO:teuthology.orchestra.run.smithi130.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-11T16:19:38.815 INFO:teuthology.orchestra.run.smithi130.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-11T16:19:38.815 INFO:teuthology.orchestra.run.smithi130.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-11T16:19:38.815 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 137 k 2023-10-11T16:19:38.815 INFO:teuthology.orchestra.run.smithi130.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-11T16:19:38.815 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-11T16:19:38.816 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-11T16:19:38.816 INFO:teuthology.orchestra.run.smithi130.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-11T16:19:38.816 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-11T16:19:38.816 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-11T16:19:38.816 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 473 k 2023-10-11T16:19:38.817 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-11T16:19:38.817 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.5 M 2023-10-11T16:19:38.817 INFO:teuthology.orchestra.run.smithi130.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-11T16:19:38.817 INFO:teuthology.orchestra.run.smithi130.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-11T16:19:38.817 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-11T16:19:38.818 INFO:teuthology.orchestra.run.smithi130.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-11T16:19:38.818 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-11T16:19:38.818 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-11T16:19:38.818 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-11T16:19:38.818 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-11T16:19:38.819 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-11T16:19:38.819 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-11T16:19:38.819 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-11T16:19:38.819 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-11T16:19:38.819 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-11T16:19:38.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-11T16:19:38.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-11T16:19:38.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-11T16:19:38.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 45 k 2023-10-11T16:19:38.820 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 119 k 2023-10-11T16:19:38.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-11T16:19:38.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-11T16:19:38.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-11T16:19:38.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-11T16:19:38.821 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-11T16:19:38.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-11T16:19:38.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-11T16:19:38.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-11T16:19:38.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-11T16:19:38.822 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-11T16:19:38.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-11T16:19:38.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T16:19:38.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-11T16:19:38.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-11T16:19:38.823 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-11T16:19:38.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-11T16:19:38.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-11T16:19:38.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-11T16:19:38.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-11T16:19:38.824 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-11T16:19:38.825 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-11T16:19:38.825 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-11T16:19:38.825 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-11T16:19:38.825 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-11T16:19:38.825 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-11T16:19:38.826 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-11T16:19:38.826 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-11T16:19:38.826 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-11T16:19:38.826 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-11T16:19:38.826 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-11T16:19:38.826 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T16:19:38.827 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-11T16:19:38.827 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-11T16:19:38.827 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-11T16:19:38.827 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-11T16:19:38.828 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-11T16:19:38.828 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-11T16:19:38.828 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-11T16:19:38.828 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-11T16:19:38.828 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-11T16:19:38.829 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-11T16:19:38.829 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-11T16:19:38.829 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-11T16:19:38.829 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-11T16:19:38.829 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-11T16:19:38.829 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-11T16:19:38.830 INFO:teuthology.orchestra.run.smithi130.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-11T16:19:38.830 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-11T16:19:38.830 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-11T16:19:38.830 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling module streams: 2023-10-11T16:19:38.831 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql 10 2023-10-11T16:19:38.831 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:19:38.831 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2023-10-11T16:19:38.831 INFO:teuthology.orchestra.run.smithi130.stdout:=========================================================================================================================== 2023-10-11T16:19:38.831 INFO:teuthology.orchestra.run.smithi130.stdout:Install 128 Packages 2023-10-11T16:19:38.831 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 3 Packages 2023-10-11T16:19:38.832 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:19:38.832 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 197 M 2023-10-11T16:19:38.832 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2023-10-11T16:19:38.979 INFO:teuthology.orchestra.run.smithi150.stdout:(2/131): ceph-fuse-18.0.0-3908.g05a43c8f.el8.x8 4.0 MB/s | 848 kB 00:00 2023-10-11T16:19:39.004 INFO:teuthology.orchestra.run.smithi130.stdout:(1/131): ceph-18.0.0-3908.g05a43c8f.el8.x86_64. 51 kB/s | 6.4 kB 00:00 2023-10-11T16:19:39.037 INFO:teuthology.orchestra.run.smithi150.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 2.8 MB/s | 166 kB 00:00 2023-10-11T16:19:39.404 INFO:teuthology.orchestra.run.smithi150.stdout:(4/131): ceph-mds-18.0.0-3908.g05a43c8f.el8.x86 6.2 MB/s | 2.3 MB 00:00 2023-10-11T16:19:39.487 INFO:teuthology.orchestra.run.smithi130.stdout:(2/131): ceph-fuse-18.0.0-3908.g05a43c8f.el8.x8 1.7 MB/s | 848 kB 00:00 2023-10-11T16:19:39.662 INFO:teuthology.orchestra.run.smithi150.stdout:(5/131): ceph-mgr-18.0.0-3908.g05a43c8f.el8.x86 6.0 MB/s | 1.5 MB 00:00 2023-10-11T16:19:39.663 INFO:teuthology.orchestra.run.smithi130.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 946 kB/s | 166 kB 00:00 2023-10-11T16:19:39.788 INFO:teuthology.orchestra.run.smithi150.stdout:(6/131): ceph-base-18.0.0-3908.g05a43c8f.el8.x8 4.6 MB/s | 5.3 MB 00:01 2023-10-11T16:19:40.371 INFO:teuthology.orchestra.run.smithi150.stdout:(7/131): ceph-mon-18.0.0-3908.g05a43c8f.el8.x86 6.5 MB/s | 4.6 MB 00:00 2023-10-11T16:19:40.446 INFO:teuthology.orchestra.run.smithi130.stdout:(4/131): ceph-mds-18.0.0-3908.g05a43c8f.el8.x86 2.9 MB/s | 2.3 MB 00:00 2023-10-11T16:19:40.621 INFO:teuthology.orchestra.run.smithi150.stdout:(8/131): ceph-common-18.0.0-3908.g05a43c8f.el8. 11 MB/s | 22 MB 00:01 2023-10-11T16:19:40.671 INFO:teuthology.orchestra.run.smithi130.stdout:(5/131): ceph-mgr-18.0.0-3908.g05a43c8f.el8.x86 6.9 MB/s | 1.5 MB 00:00 2023-10-11T16:19:40.696 INFO:teuthology.orchestra.run.smithi150.stdout:(9/131): ceph-selinux-18.0.0-3908.g05a43c8f.el8 326 kB/s | 24 kB 00:00 2023-10-11T16:19:41.102 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-11T16:19:41.104 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-11T16:19:41.119 INFO:teuthology.orchestra.run.smithi111.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-11T16:19:41.120 INFO:teuthology.orchestra.run.smithi111.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-11T16:19:41.121 INFO:teuthology.orchestra.run.smithi111.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-11T16:19:41.122 INFO:teuthology.orchestra.run.smithi111.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-11T16:19:41.125 INFO:teuthology.orchestra.run.smithi111.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-11T16:19:41.128 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-11T16:19:41.130 INFO:teuthology.orchestra.run.smithi111.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-11T16:19:41.271 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2023-10-11T16:19:41.283 INFO:teuthology.orchestra.run.smithi111.stdout:=========================================================================================================================== 2023-10-11T16:19:41.283 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2023-10-11T16:19:41.284 INFO:teuthology.orchestra.run.smithi111.stdout:=========================================================================================================================== 2023-10-11T16:19:41.284 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2023-10-11T16:19:41.284 INFO:teuthology.orchestra.run.smithi111.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-11T16:19:41.284 INFO:teuthology.orchestra.run.smithi111.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-11T16:19:41.284 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.4 k 2023-10-11T16:19:41.285 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 5.3 M 2023-10-11T16:19:41.285 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 848 k 2023-10-11T16:19:41.285 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 166 k 2023-10-11T16:19:41.285 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 1.5 M 2023-10-11T16:19:41.285 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 127 k 2023-10-11T16:19:41.286 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 1.7 M 2023-10-11T16:19:41.286 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 7.4 M 2023-10-11T16:19:41.286 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 47 k 2023-10-11T16:19:41.286 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 13 M 2023-10-11T16:19:41.286 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 41 M 2023-10-11T16:19:41.287 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 265 k 2023-10-11T16:19:41.287 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 83 k 2023-10-11T16:19:41.287 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 258 k 2023-10-11T16:19:41.287 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 18 k 2023-10-11T16:19:41.287 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-11T16:19:41.288 INFO:teuthology.orchestra.run.smithi111.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-11T16:19:41.288 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-11T16:19:41.288 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 31 k 2023-10-11T16:19:41.288 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 770 k 2023-10-11T16:19:41.288 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 139 k 2023-10-11T16:19:41.289 INFO:teuthology.orchestra.run.smithi111.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-11T16:19:41.289 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-11T16:19:41.289 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-11T16:19:41.289 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-11T16:19:41.289 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-11T16:19:41.289 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-11T16:19:41.290 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 198 k 2023-10-11T16:19:41.290 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 371 k 2023-10-11T16:19:41.290 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 364 k 2023-10-11T16:19:41.290 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 106 k 2023-10-11T16:19:41.291 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 94 k 2023-10-11T16:19:41.291 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.4 M 2023-10-11T16:19:41.291 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 179 k 2023-10-11T16:19:41.291 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-11T16:19:41.291 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2023-10-11T16:19:41.291 INFO:teuthology.orchestra.run.smithi111.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-11T16:19:41.292 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.7 M 2023-10-11T16:19:41.292 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.6 M 2023-10-11T16:19:41.292 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2023-10-11T16:19:41.292 INFO:teuthology.orchestra.run.smithi111.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-11T16:19:41.292 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 22 M 2023-10-11T16:19:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 23 k 2023-10-11T16:19:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 2.3 M 2023-10-11T16:19:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 240 k 2023-10-11T16:19:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 4.6 M 2023-10-11T16:19:41.293 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 18 M 2023-10-11T16:19:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 15 k 2023-10-11T16:19:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 24 k 2023-10-11T16:19:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-11T16:19:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-11T16:19:41.294 INFO:teuthology.orchestra.run.smithi111.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-11T16:19:41.295 INFO:teuthology.orchestra.run.smithi111.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-11T16:19:41.295 INFO:teuthology.orchestra.run.smithi111.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-11T16:19:41.295 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 137 k 2023-10-11T16:19:41.295 INFO:teuthology.orchestra.run.smithi111.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-11T16:19:41.295 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-11T16:19:41.296 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-11T16:19:41.296 INFO:teuthology.orchestra.run.smithi111.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-11T16:19:41.296 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-11T16:19:41.296 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-11T16:19:41.296 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 473 k 2023-10-11T16:19:41.296 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-11T16:19:41.297 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.5 M 2023-10-11T16:19:41.297 INFO:teuthology.orchestra.run.smithi111.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-11T16:19:41.297 INFO:teuthology.orchestra.run.smithi111.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-11T16:19:41.297 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-11T16:19:41.297 INFO:teuthology.orchestra.run.smithi111.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-11T16:19:41.298 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-11T16:19:41.298 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-11T16:19:41.298 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-11T16:19:41.298 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-11T16:19:41.298 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-11T16:19:41.299 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-11T16:19:41.299 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-11T16:19:41.299 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-11T16:19:41.299 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-11T16:19:41.299 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-11T16:19:41.299 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-11T16:19:41.300 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-11T16:19:41.300 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 45 k 2023-10-11T16:19:41.300 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 119 k 2023-10-11T16:19:41.300 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-11T16:19:41.300 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-11T16:19:41.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-11T16:19:41.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-11T16:19:41.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-11T16:19:41.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-11T16:19:41.301 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-11T16:19:41.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-11T16:19:41.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-11T16:19:41.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-11T16:19:41.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-11T16:19:41.302 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T16:19:41.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-11T16:19:41.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-11T16:19:41.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-11T16:19:41.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-11T16:19:41.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-11T16:19:41.303 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-11T16:19:41.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-11T16:19:41.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-11T16:19:41.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-11T16:19:41.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-11T16:19:41.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-11T16:19:41.304 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-11T16:19:41.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-11T16:19:41.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-11T16:19:41.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-11T16:19:41.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-11T16:19:41.305 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-11T16:19:41.306 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-11T16:19:41.306 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-11T16:19:41.306 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-11T16:19:41.306 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-11T16:19:41.306 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-11T16:19:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-11T16:19:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-11T16:19:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-11T16:19:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-11T16:19:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-11T16:19:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-11T16:19:41.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-11T16:19:41.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-11T16:19:41.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-11T16:19:41.308 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-11T16:19:41.308 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-11T16:19:41.309 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-11T16:19:41.309 INFO:teuthology.orchestra.run.smithi111.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-11T16:19:41.309 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-11T16:19:41.309 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-11T16:19:41.309 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling module streams: 2023-10-11T16:19:41.309 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql 10 2023-10-11T16:19:41.310 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:19:41.310 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2023-10-11T16:19:41.310 INFO:teuthology.orchestra.run.smithi111.stdout:=========================================================================================================================== 2023-10-11T16:19:41.310 INFO:teuthology.orchestra.run.smithi111.stdout:Install 128 Packages 2023-10-11T16:19:41.310 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 3 Packages 2023-10-11T16:19:41.311 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:19:41.311 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 197 M 2023-10-11T16:19:41.311 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2023-10-11T16:19:41.513 INFO:teuthology.orchestra.run.smithi130.stdout:(6/131): ceph-mon-18.0.0-3908.g05a43c8f.el8.x86 5.5 MB/s | 4.6 MB 00:00 2023-10-11T16:19:41.787 INFO:teuthology.orchestra.run.smithi111.stdout:(1/131): ceph-18.0.0-3908.g05a43c8f.el8.x86_64. 20 kB/s | 6.4 kB 00:00 2023-10-11T16:19:42.047 INFO:teuthology.orchestra.run.smithi130.stdout:(7/131): ceph-base-18.0.0-3908.g05a43c8f.el8.x8 1.7 MB/s | 5.3 MB 00:03 2023-10-11T16:19:42.337 INFO:teuthology.orchestra.run.smithi111.stdout:(2/131): ceph-fuse-18.0.0-3908.g05a43c8f.el8.x8 1.5 MB/s | 848 kB 00:00 2023-10-11T16:19:42.404 INFO:teuthology.orchestra.run.smithi111.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 2.4 MB/s | 166 kB 00:00 2023-10-11T16:19:42.438 INFO:teuthology.orchestra.run.smithi150.stdout:(10/131): ceph-radosgw-18.0.0-3908.g05a43c8f.el 6.3 MB/s | 13 MB 00:02 2023-10-11T16:19:42.705 INFO:teuthology.orchestra.run.smithi150.stdout:(11/131): cephfs-mirror-18.0.0-3908.g05a43c8f.e 968 kB/s | 258 kB 00:00 2023-10-11T16:19:42.797 INFO:teuthology.orchestra.run.smithi150.stdout:(12/131): libcephfs-devel-18.0.0-3908.g05a43c8f 338 kB/s | 31 kB 00:00 2023-10-11T16:19:43.171 INFO:teuthology.orchestra.run.smithi111.stdout:(4/131): ceph-mds-18.0.0-3908.g05a43c8f.el8.x86 3.0 MB/s | 2.3 MB 00:00 2023-10-11T16:19:43.322 INFO:teuthology.orchestra.run.smithi150.stdout:(13/131): libcephfs2-18.0.0-3908.g05a43c8f.el8. 1.4 MB/s | 770 kB 00:00 2023-10-11T16:19:43.481 INFO:teuthology.orchestra.run.smithi150.stdout:(14/131): libcephsqlite-18.0.0-3908.g05a43c8f.e 868 kB/s | 137 kB 00:00 2023-10-11T16:19:43.639 INFO:teuthology.orchestra.run.smithi150.stdout:(15/131): librados-devel-18.0.0-3908.g05a43c8f. 877 kB/s | 139 kB 00:00 2023-10-11T16:19:43.688 INFO:teuthology.orchestra.run.smithi111.stdout:(5/131): ceph-mgr-18.0.0-3908.g05a43c8f.el8.x86 3.0 MB/s | 1.5 MB 00:00 2023-10-11T16:19:43.931 INFO:teuthology.orchestra.run.smithi150.stdout:(16/131): libradosstriper1-18.0.0-3908.g05a43c8 1.6 MB/s | 473 kB 00:00 2023-10-11T16:19:43.938 INFO:teuthology.orchestra.run.smithi111.stdout:(6/131): ceph-base-18.0.0-3908.g05a43c8f.el8.x8 2.2 MB/s | 5.3 MB 00:02 2023-10-11T16:19:44.730 INFO:teuthology.orchestra.run.smithi130.stdout:(8/131): ceph-osd-18.0.0-3908.g05a43c8f.el8.x86 5.5 MB/s | 18 MB 00:03 2023-10-11T16:19:44.814 INFO:teuthology.orchestra.run.smithi130.stdout:(9/131): ceph-selinux-18.0.0-3908.g05a43c8f.el8 294 kB/s | 24 kB 00:00 2023-10-11T16:19:45.122 INFO:teuthology.orchestra.run.smithi111.stdout:(7/131): ceph-mon-18.0.0-3908.g05a43c8f.el8.x86 3.2 MB/s | 4.6 MB 00:01 2023-10-11T16:19:45.315 INFO:teuthology.orchestra.run.smithi150.stdout:(17/131): ceph-osd-18.0.0-3908.g05a43c8f.el8.x8 3.2 MB/s | 18 MB 00:05 2023-10-11T16:19:45.432 INFO:teuthology.orchestra.run.smithi150.stdout:(18/131): python3-ceph-argparse-18.0.0-3908.g05 384 kB/s | 45 kB 00:00 2023-10-11T16:19:45.565 INFO:teuthology.orchestra.run.smithi150.stdout:(19/131): python3-ceph-common-18.0.0-3908.g05a4 891 kB/s | 119 kB 00:00 2023-10-11T16:19:45.732 INFO:teuthology.orchestra.run.smithi150.stdout:(20/131): python3-cephfs-18.0.0-3908.g05a43c8f. 1.2 MB/s | 198 kB 00:00 2023-10-11T16:19:45.933 INFO:teuthology.orchestra.run.smithi150.stdout:(21/131): python3-rados-18.0.0-3908.g05a43c8f.e 1.8 MB/s | 371 kB 00:00 2023-10-11T16:19:46.125 INFO:teuthology.orchestra.run.smithi150.stdout:(22/131): python3-rbd-18.0.0-3908.g05a43c8f.el8 1.9 MB/s | 364 kB 00:00 2023-10-11T16:19:46.225 INFO:teuthology.orchestra.run.smithi150.stdout:(23/131): librgw2-18.0.0-3908.g05a43c8f.el8.x86 2.8 MB/s | 6.5 MB 00:02 2023-10-11T16:19:46.242 INFO:teuthology.orchestra.run.smithi150.stdout:(24/131): python3-rgw-18.0.0-3908.g05a43c8f.el8 912 kB/s | 106 kB 00:00 2023-10-11T16:19:46.309 INFO:teuthology.orchestra.run.smithi150.stdout:(25/131): rbd-fuse-18.0.0-3908.g05a43c8f.el8.x8 1.1 MB/s | 94 kB 00:00 2023-10-11T16:19:46.400 INFO:teuthology.orchestra.run.smithi150.stdout:(26/131): rbd-nbd-18.0.0-3908.g05a43c8f.el8.x86 1.9 MB/s | 179 kB 00:00 2023-10-11T16:19:46.493 INFO:teuthology.orchestra.run.smithi150.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3908.g 251 kB/s | 23 kB 00:00 2023-10-11T16:19:46.585 INFO:teuthology.orchestra.run.smithi150.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3908.g05a43c8 1.4 MB/s | 127 kB 00:00 2023-10-11T16:19:47.310 INFO:teuthology.orchestra.run.smithi150.stdout:(29/131): ceph-mgr-dashboard-18.0.0-3908.g05a43 2.4 MB/s | 1.7 MB 00:00 2023-10-11T16:19:47.619 INFO:teuthology.orchestra.run.smithi150.stdout:(30/131): rbd-mirror-18.0.0-3908.g05a43c8f.el8. 2.5 MB/s | 3.4 MB 00:01 2023-10-11T16:19:47.735 INFO:teuthology.orchestra.run.smithi150.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3908.g05 2.0 MB/s | 240 kB 00:00 2023-10-11T16:19:47.844 INFO:teuthology.orchestra.run.smithi150.stdout:(32/131): ceph-mgr-rook-18.0.0-3908.g05a43c8f.e 437 kB/s | 47 kB 00:00 2023-10-11T16:19:47.945 INFO:teuthology.orchestra.run.smithi150.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3908.g0 145 kB/s | 15 kB 00:00 2023-10-11T16:19:48.106 INFO:teuthology.orchestra.run.smithi130.stdout:(10/131): ceph-radosgw-18.0.0-3908.g05a43c8f.el 2.1 MB/s | 13 MB 00:06 2023-10-11T16:19:48.264 INFO:teuthology.orchestra.run.smithi130.stdout:(11/131): cephfs-mirror-18.0.0-3908.g05a43c8f.e 1.6 MB/s | 258 kB 00:00 2023-10-11T16:19:48.348 INFO:teuthology.orchestra.run.smithi130.stdout:(12/131): libcephfs-devel-18.0.0-3908.g05a43c8f 373 kB/s | 31 kB 00:00 2023-10-11T16:19:48.428 INFO:teuthology.orchestra.run.smithi150.stdout:(34/131): ceph-test-18.0.0-3908.g05a43c8f.el8.x 5.3 MB/s | 41 MB 00:07 2023-10-11T16:19:48.445 INFO:teuthology.orchestra.run.smithi150.stdout:(35/131): ceph-volume-18.0.0-3908.g05a43c8f.el8 529 kB/s | 265 kB 00:00 2023-10-11T16:19:48.512 INFO:teuthology.orchestra.run.smithi150.stdout:(36/131): cephadm-18.0.0-3908.g05a43c8f.el8.noa 1.0 MB/s | 83 kB 00:00 2023-10-11T16:19:48.545 INFO:teuthology.orchestra.run.smithi150.stdout:(37/131): cephfs-top-18.0.0-3908.g05a43c8f.el8. 179 kB/s | 18 kB 00:00 2023-10-11T16:19:48.640 INFO:teuthology.orchestra.run.smithi130.stdout:(13/131): ceph-common-18.0.0-3908.g05a43c8f.el8 2.2 MB/s | 22 MB 00:09 2023-10-11T16:19:48.679 INFO:teuthology.orchestra.run.smithi150.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 440 kB/s | 59 kB 00:00 2023-10-11T16:19:48.738 INFO:teuthology.orchestra.run.smithi150.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-10-11T16:19:48.798 INFO:teuthology.orchestra.run.smithi130.stdout:(14/131): libcephfs2-18.0.0-3908.g05a43c8f.el8. 1.7 MB/s | 770 kB 00:00 2023-10-11T16:19:48.805 INFO:teuthology.orchestra.run.smithi111.stdout:(8/131): ceph-radosgw-18.0.0-3908.g05a43c8f.el8 3.5 MB/s | 13 MB 00:03 2023-10-11T16:19:48.815 INFO:teuthology.orchestra.run.smithi130.stdout:(15/131): libcephsqlite-18.0.0-3908.g05a43c8f.e 785 kB/s | 137 kB 00:00 2023-10-11T16:19:48.880 INFO:teuthology.orchestra.run.smithi111.stdout:(9/131): ceph-selinux-18.0.0-3908.g05a43c8f.el8 328 kB/s | 24 kB 00:00 2023-10-11T16:19:48.888 INFO:teuthology.orchestra.run.smithi150.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 3.1 MB/s | 476 kB 00:00 2023-10-11T16:19:48.898 INFO:teuthology.orchestra.run.smithi130.stdout:(16/131): librados-devel-18.0.0-3908.g05a43c8f. 1.4 MB/s | 139 kB 00:00 2023-10-11T16:19:48.988 INFO:teuthology.orchestra.run.smithi150.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 437 kB/s | 44 kB 00:00 2023-10-11T16:19:49.116 INFO:teuthology.orchestra.run.smithi130.stdout:(17/131): libradosstriper1-18.0.0-3908.g05a43c8 1.5 MB/s | 473 kB 00:00 2023-10-11T16:19:49.216 INFO:teuthology.orchestra.run.smithi130.stdout:(18/131): python3-ceph-argparse-18.0.0-3908.g05 447 kB/s | 45 kB 00:00 2023-10-11T16:19:49.282 INFO:teuthology.orchestra.run.smithi150.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 631 kB/s | 185 kB 00:00 2023-10-11T16:19:49.314 INFO:teuthology.orchestra.run.smithi150.stdout:(43/131): gperftools-libs-2.7-9.el8.x86_64.rpm 382 kB/s | 306 kB 00:00 2023-10-11T16:19:49.333 INFO:teuthology.orchestra.run.smithi130.stdout:(19/131): python3-ceph-common-18.0.0-3908.g05a4 1.0 MB/s | 119 kB 00:00 2023-10-11T16:19:49.339 INFO:teuthology.orchestra.run.smithi150.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 579 kB/s | 33 kB 00:00 2023-10-11T16:19:49.398 INFO:teuthology.orchestra.run.smithi150.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 195 kB/s | 16 kB 00:00 2023-10-11T16:19:49.449 INFO:teuthology.orchestra.run.smithi150.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2023-10-11T16:19:49.532 INFO:teuthology.orchestra.run.smithi150.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 977 kB/s | 81 kB 00:00 2023-10-11T16:19:49.541 INFO:teuthology.orchestra.run.smithi130.stdout:(20/131): python3-cephfs-18.0.0-3908.g05a43c8f. 949 kB/s | 198 kB 00:00 2023-10-11T16:19:49.566 INFO:teuthology.orchestra.run.smithi150.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 334 kB/s | 11 kB 00:00 2023-10-11T16:19:49.599 INFO:teuthology.orchestra.run.smithi150.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 529 kB/s | 18 kB 00:00 2023-10-11T16:19:49.834 INFO:teuthology.orchestra.run.smithi130.stdout:(21/131): python3-rados-18.0.0-3908.g05a43c8f.e 1.2 MB/s | 371 kB 00:00 2023-10-11T16:19:49.841 INFO:teuthology.orchestra.run.smithi150.stdout:(50/131): ceph-mgr-diskprediction-local-18.0.0- 2.9 MB/s | 7.4 MB 00:02 2023-10-11T16:19:49.867 INFO:teuthology.orchestra.run.smithi150.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 822 kB/s | 384 kB 00:00 2023-10-11T16:19:49.950 INFO:teuthology.orchestra.run.smithi150.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 745 kB/s | 59 kB 00:00 2023-10-11T16:19:49.984 INFO:teuthology.orchestra.run.smithi150.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 3.0 MB/s | 1.2 MB 00:00 2023-10-11T16:19:50.002 INFO:teuthology.orchestra.run.smithi150.stdout:(54/131): python3-logutils-0.3.5-11.el8.noarch. 307 kB/s | 49 kB 00:00 2023-10-11T16:19:50.017 INFO:teuthology.orchestra.run.smithi150.stdout:(55/131): python3-natsort-7.1.1-2.el8.noarch.rp 900 kB/s | 60 kB 00:00 2023-10-11T16:19:50.035 INFO:teuthology.orchestra.run.smithi150.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 459 kB/s | 16 kB 00:00 2023-10-11T16:19:50.042 INFO:teuthology.orchestra.run.smithi130.stdout:(22/131): python3-rbd-18.0.0-3908.g05a43c8f.el8 1.7 MB/s | 364 kB 00:00 2023-10-11T16:19:50.077 INFO:teuthology.orchestra.run.smithi150.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-10-11T16:19:50.102 INFO:teuthology.orchestra.run.smithi150.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 401 kB/s | 34 kB 00:00 2023-10-11T16:19:50.119 INFO:teuthology.orchestra.run.smithi150.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 1.5 MB/s | 64 kB 00:00 2023-10-11T16:19:50.144 INFO:teuthology.orchestra.run.smithi150.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2023-10-11T16:19:50.159 INFO:teuthology.orchestra.run.smithi130.stdout:(23/131): python3-rgw-18.0.0-3908.g05a43c8f.el8 914 kB/s | 106 kB 00:00 2023-10-11T16:19:50.169 INFO:teuthology.orchestra.run.smithi150.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2023-10-11T16:19:50.194 INFO:teuthology.orchestra.run.smithi150.stdout:(62/131): python3-simplegeneric-0.8.1-17.el8.no 203 kB/s | 19 kB 00:00 2023-10-11T16:19:50.220 INFO:teuthology.orchestra.run.smithi150.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2023-10-11T16:19:50.245 INFO:teuthology.orchestra.run.smithi150.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 366 kB/s | 27 kB 00:00 2023-10-11T16:19:50.284 INFO:teuthology.orchestra.run.smithi130.stdout:(24/131): rbd-fuse-18.0.0-3908.g05a43c8f.el8.x8 755 kB/s | 94 kB 00:00 2023-10-11T16:19:50.287 INFO:teuthology.orchestra.run.smithi150.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.4 MB/s | 61 kB 00:00 2023-10-11T16:19:50.312 INFO:teuthology.orchestra.run.smithi150.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.7 MB/s | 251 kB 00:00 2023-10-11T16:19:50.346 INFO:teuthology.orchestra.run.smithi150.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2023-10-11T16:19:50.371 INFO:teuthology.orchestra.run.smithi150.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 392 kB/s | 23 kB 00:00 2023-10-11T16:19:50.405 INFO:teuthology.orchestra.run.smithi150.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2023-10-11T16:19:50.480 INFO:teuthology.orchestra.run.smithi150.stdout:(70/131): python3-waitress-1.4.3-1.el8.noarch.r 983 kB/s | 255 kB 00:00 2023-10-11T16:19:50.639 INFO:teuthology.orchestra.run.smithi150.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 696 kB/s | 110 kB 00:00 2023-10-11T16:19:50.739 INFO:teuthology.orchestra.run.smithi150.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 543 kB/s | 54 kB 00:00 2023-10-11T16:19:50.848 INFO:teuthology.orchestra.run.smithi150.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 3.5 MB/s | 1.7 MB 00:00 2023-10-11T16:19:50.898 INFO:teuthology.orchestra.run.smithi150.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2023-10-11T16:19:50.965 INFO:teuthology.orchestra.run.smithi150.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-10-11T16:19:50.990 INFO:teuthology.orchestra.run.smithi150.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 423 kB/s | 39 kB 00:00 2023-10-11T16:19:51.005 INFO:teuthology.orchestra.run.smithi111.stdout:(10/131): ceph-common-18.0.0-3908.g05a43c8f.el8 2.3 MB/s | 22 MB 00:09 2023-10-11T16:19:51.125 INFO:teuthology.orchestra.run.smithi150.stdout:(77/131): python3-scipy-1.0.0-21.module+el8.5.0 19 MB/s | 14 MB 00:00 2023-10-11T16:19:51.158 INFO:teuthology.orchestra.run.smithi150.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.1 MB/s | 538 kB 00:00 2023-10-11T16:19:51.183 INFO:teuthology.orchestra.run.smithi150.stdout:(79/131): bison-3.0.4-10.el8.x86_64.rpm 3.1 MB/s | 688 kB 00:00 2023-10-11T16:19:51.258 INFO:teuthology.orchestra.run.smithi150.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-10-11T16:19:51.272 INFO:teuthology.orchestra.run.smithi111.stdout:(11/131): cephfs-mirror-18.0.0-3908.g05a43c8f.e 968 kB/s | 258 kB 00:00 2023-10-11T16:19:51.275 INFO:teuthology.orchestra.run.smithi150.stdout:(81/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2023-10-11T16:19:51.380 INFO:teuthology.orchestra.run.smithi111.stdout:(12/131): libcephfs-devel-18.0.0-3908.g05a43c8f 287 kB/s | 31 kB 00:00 2023-10-11T16:19:51.417 INFO:teuthology.orchestra.run.smithi150.stdout:(82/131): python3-babel-2.5.1-7.el8.noarch.rpm 30 MB/s | 4.8 MB 00:00 2023-10-11T16:19:51.442 INFO:teuthology.orchestra.run.smithi150.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 159 kB/s | 27 kB 00:00 2023-10-11T16:19:51.542 INFO:teuthology.orchestra.run.smithi150.stdout:(84/131): automake-1.16.1-7.el8.noarch.rpm 1.9 MB/s | 713 kB 00:00 2023-10-11T16:19:51.609 INFO:teuthology.orchestra.run.smithi150.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 263 kB/s | 44 kB 00:00 2023-10-11T16:19:51.635 INFO:teuthology.orchestra.run.smithi150.stdout:(86/131): flex-2.6.1-9.el8.x86_64.rpm 1.4 MB/s | 320 kB 00:00 2023-10-11T16:19:51.710 INFO:teuthology.orchestra.run.smithi150.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 4.4 MB/s | 457 kB 00:00 2023-10-11T16:19:51.785 INFO:teuthology.orchestra.run.smithi150.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 15 MB/s | 3.7 MB 00:00 2023-10-11T16:19:51.801 INFO:teuthology.orchestra.run.smithi130.stdout:(25/131): rbd-mirror-18.0.0-3908.g05a43c8f.el8. 2.3 MB/s | 3.4 MB 00:01 2023-10-11T16:19:51.811 INFO:teuthology.orchestra.run.smithi150.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2023-10-11T16:19:51.839 INFO:teuthology.orchestra.run.smithi111.stdout:(13/131): libcephfs2-18.0.0-3908.g05a43c8f.el8. 1.6 MB/s | 770 kB 00:00 2023-10-11T16:19:51.918 INFO:teuthology.orchestra.run.smithi130.stdout:(26/131): rbd-nbd-18.0.0-3908.g05a43c8f.el8.x86 1.5 MB/s | 179 kB 00:00 2023-10-11T16:19:51.948 INFO:teuthology.orchestra.run.smithi111.stdout:(14/131): libcephsqlite-18.0.0-3908.g05a43c8f.e 1.2 MB/s | 137 kB 00:00 2023-10-11T16:19:52.010 INFO:teuthology.orchestra.run.smithi130.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3908.g 251 kB/s | 23 kB 00:00 2023-10-11T16:19:52.019 INFO:teuthology.orchestra.run.smithi150.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 541 kB/s | 126 kB 00:00 2023-10-11T16:19:52.065 INFO:teuthology.orchestra.run.smithi111.stdout:(15/131): librados-devel-18.0.0-3908.g05a43c8f. 1.2 MB/s | 139 kB 00:00 2023-10-11T16:19:52.078 INFO:teuthology.orchestra.run.smithi150.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 75 kB/s | 20 kB 00:00 2023-10-11T16:19:52.111 INFO:teuthology.orchestra.run.smithi150.stdout:(92/131): uuid-1.6.2-43.el8.x86_64.rpm 160 kB/s | 64 kB 00:00 2023-10-11T16:19:52.119 INFO:teuthology.orchestra.run.smithi130.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3908.g05a43c8 1.1 MB/s | 127 kB 00:00 2023-10-11T16:19:52.195 INFO:teuthology.orchestra.run.smithi150.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 590 kB/s | 103 kB 00:00 2023-10-11T16:19:52.321 INFO:teuthology.orchestra.run.smithi150.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 2.9 MB/s | 709 kB 00:00 2023-10-11T16:19:52.356 INFO:teuthology.orchestra.run.smithi111.stdout:(16/131): libradosstriper1-18.0.0-3908.g05a43c8 1.6 MB/s | 473 kB 00:00 2023-10-11T16:19:52.396 INFO:teuthology.orchestra.run.smithi150.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2023-10-11T16:19:52.403 INFO:teuthology.orchestra.run.smithi130.stdout:(29/131): librgw2-18.0.0-3908.g05a43c8f.el8.x86 1.9 MB/s | 6.5 MB 00:03 2023-10-11T16:19:52.471 INFO:teuthology.orchestra.run.smithi150.stdout:(96/131): libpq-13.5-1.el8.x86_64.rpm 718 kB/s | 198 kB 00:00 2023-10-11T16:19:52.530 INFO:teuthology.orchestra.run.smithi150.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2023-10-11T16:19:52.621 INFO:teuthology.orchestra.run.smithi150.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-10-11T16:19:52.646 INFO:teuthology.orchestra.run.smithi150.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.0 MB/s | 354 kB 00:00 2023-10-11T16:19:52.874 INFO:teuthology.orchestra.run.smithi150.stdout:(100/131): postgresql-server-10.21-2.module+el8 15 MB/s | 5.1 MB 00:00 2023-10-11T16:19:52.906 INFO:teuthology.orchestra.run.smithi150.stdout:(101/131): postgresql-contrib-10.21-2.module+el 2.8 MB/s | 808 kB 00:00 2023-10-11T16:19:52.961 INFO:teuthology.orchestra.run.smithi130.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3908.g05a43 2.0 MB/s | 1.7 MB 00:00 2023-10-11T16:19:52.972 INFO:teuthology.orchestra.run.smithi150.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 4.6 MB/s | 1.5 MB 00:00 2023-10-11T16:19:53.115 INFO:teuthology.orchestra.run.smithi150.stdout:(103/131): python3-jwt-1.6.1-2.el8.noarch.rpm 300 kB/s | 43 kB 00:00 2023-10-11T16:19:53.170 INFO:teuthology.orchestra.run.smithi130.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3908.g05 1.1 MB/s | 240 kB 00:00 2023-10-11T16:19:53.281 INFO:teuthology.orchestra.run.smithi150.stdout:(104/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 114 kB/s | 47 kB 00:00 2023-10-11T16:19:53.307 INFO:teuthology.orchestra.run.smithi150.stdout:(105/131): libselinux-devel-2.9-5.el8.x86_64.rp 499 kB/s | 200 kB 00:00 2023-10-11T16:19:53.309 INFO:teuthology.orchestra.run.smithi130.stdout:(32/131): ceph-mgr-rook-18.0.0-3908.g05a43c8f.e 341 kB/s | 47 kB 00:00 2023-10-11T16:19:53.390 INFO:teuthology.orchestra.run.smithi150.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 800 kB/s | 220 kB 00:00 2023-10-11T16:19:53.420 INFO:teuthology.orchestra.run.smithi130.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3908.g0 131 kB/s | 15 kB 00:00 2023-10-11T16:19:53.482 INFO:teuthology.orchestra.run.smithi150.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 240 kB/s | 48 kB 00:00 2023-10-11T16:19:53.508 INFO:teuthology.orchestra.run.smithi150.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 434 kB/s | 87 kB 00:00 2023-10-11T16:19:53.525 INFO:teuthology.orchestra.run.smithi111.stdout:(17/131): ceph-osd-18.0.0-3908.g05a43c8f.el8.x8 1.8 MB/s | 18 MB 00:09 2023-10-11T16:19:53.600 INFO:teuthology.orchestra.run.smithi150.stdout:(109/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.4 MB/s | 229 kB 00:00 2023-10-11T16:19:53.629 INFO:teuthology.orchestra.run.smithi130.stdout:(34/131): ceph-volume-18.0.0-3908.g05a43c8f.el8 1.2 MB/s | 265 kB 00:00 2023-10-11T16:19:53.633 INFO:teuthology.orchestra.run.smithi150.stdout:(110/131): python3-requests-oauthlib-1.0.0-1.el 176 kB/s | 43 kB 00:00 2023-10-11T16:19:53.649 INFO:teuthology.orchestra.run.smithi111.stdout:(18/131): python3-ceph-argparse-18.0.0-3908.g05 362 kB/s | 45 kB 00:00 2023-10-11T16:19:53.658 INFO:teuthology.orchestra.run.smithi150.stdout:(111/131): python3-pycparser-2.14-14.el8.noarch 621 kB/s | 109 kB 00:00 2023-10-11T16:19:53.754 INFO:teuthology.orchestra.run.smithi130.stdout:(35/131): cephadm-18.0.0-3908.g05a43c8f.el8.noa 670 kB/s | 83 kB 00:00 2023-10-11T16:19:53.850 INFO:teuthology.orchestra.run.smithi150.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.2 MB/s | 560 kB 00:00 2023-10-11T16:19:53.875 INFO:teuthology.orchestra.run.smithi150.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 77 kB/s | 19 kB 00:00 2023-10-11T16:19:53.896 INFO:teuthology.orchestra.run.smithi130.stdout:(36/131): cephfs-top-18.0.0-3908.g05a43c8f.el8. 127 kB/s | 18 kB 00:00 2023-10-11T16:19:53.907 INFO:teuthology.orchestra.run.smithi111.stdout:(19/131): python3-ceph-common-18.0.0-3908.g05a4 460 kB/s | 119 kB 00:00 2023-10-11T16:19:53.917 INFO:teuthology.orchestra.run.smithi150.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 8.2 MB/s | 559 kB 00:00 2023-10-11T16:19:53.951 INFO:teuthology.orchestra.run.smithi150.stdout:(115/131): python3-oauthlib-2.1.0-1.el8.noarch. 529 kB/s | 155 kB 00:00 2023-10-11T16:19:54.127 INFO:teuthology.orchestra.run.smithi150.stdout:(116/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-10-11T16:19:54.241 INFO:teuthology.orchestra.run.smithi111.stdout:(20/131): python3-cephfs-18.0.0-3908.g05a43c8f. 593 kB/s | 198 kB 00:00 2023-10-11T16:19:54.252 INFO:teuthology.orchestra.run.smithi150.stdout:(117/131): libverto-devel-0.3.0-5.el8.x86_64.rp 48 kB/s | 18 kB 00:00 2023-10-11T16:19:54.286 INFO:teuthology.orchestra.run.smithi150.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 1.6 MB/s | 605 kB 00:00 2023-10-11T16:19:54.346 INFO:teuthology.orchestra.run.smithi130.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 681 kB/s | 306 kB 00:00 2023-10-11T16:19:54.369 INFO:teuthology.orchestra.run.smithi150.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 9.6 MB/s | 2.3 MB 00:00 2023-10-11T16:19:54.405 INFO:teuthology.orchestra.run.smithi130.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-10-11T16:19:54.511 INFO:teuthology.orchestra.run.smithi150.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 279 kB/s | 72 kB 00:00 2023-10-11T16:19:54.537 INFO:teuthology.orchestra.run.smithi150.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 330 kB/s | 83 kB 00:00 2023-10-11T16:19:54.625 INFO:teuthology.orchestra.run.smithi111.stdout:(21/131): python3-rados-18.0.0-3908.g05a43c8f.e 969 kB/s | 371 kB 00:00 2023-10-11T16:19:54.628 INFO:teuthology.orchestra.run.smithi150.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 231 kB/s | 60 kB 00:00 2023-10-11T16:19:54.654 INFO:teuthology.orchestra.run.smithi150.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 690 kB/s | 98 kB 00:00 2023-10-11T16:19:54.916 INFO:teuthology.orchestra.run.smithi150.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 152 kB/s | 58 kB 00:00 2023-10-11T16:19:54.922 INFO:teuthology.orchestra.run.smithi130.stdout:(39/131): ceph-test-18.0.0-3908.g05a43c8f.el8.x 4.0 MB/s | 41 MB 00:10 2023-10-11T16:19:54.947 INFO:teuthology.orchestra.run.smithi130.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 138 kB/s | 75 kB 00:00 2023-10-11T16:19:54.975 INFO:teuthology.orchestra.run.smithi111.stdout:(22/131): python3-rbd-18.0.0-3908.g05a43c8f.el8 1.0 MB/s | 364 kB 00:00 2023-10-11T16:19:55.064 INFO:teuthology.orchestra.run.smithi130.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 374 kB/s | 44 kB 00:00 2023-10-11T16:19:55.126 INFO:teuthology.orchestra.run.smithi111.stdout:(23/131): python3-rgw-18.0.0-3908.g05a43c8f.el8 711 kB/s | 106 kB 00:00 2023-10-11T16:19:55.171 INFO:teuthology.orchestra.run.smithi150.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 71 kB/s | 39 kB 00:00 2023-10-11T16:19:55.196 INFO:teuthology.orchestra.run.smithi150.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 611 kB/s | 171 kB 00:00 2023-10-11T16:19:55.230 INFO:teuthology.orchestra.run.smithi150.stdout:(127/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 1.1 MB/s | 644 kB 00:00 2023-10-11T16:19:55.231 INFO:teuthology.orchestra.run.smithi130.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.1 MB/s | 185 kB 00:00 2023-10-11T16:19:55.234 INFO:teuthology.orchestra.run.smithi111.stdout:(24/131): rbd-fuse-18.0.0-3908.g05a43c8f.el8.x8 872 kB/s | 94 kB 00:00 2023-10-11T16:19:55.281 INFO:teuthology.orchestra.run.smithi130.stdout:(43/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.3 MB/s | 476 kB 00:00 2023-10-11T16:19:55.307 INFO:teuthology.orchestra.run.smithi130.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 438 kB/s | 33 kB 00:00 2023-10-11T16:19:55.332 INFO:teuthology.orchestra.run.smithi130.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 324 kB/s | 16 kB 00:00 2023-10-11T16:19:55.399 INFO:teuthology.orchestra.run.smithi130.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.8 MB/s | 173 kB 00:00 2023-10-11T16:19:55.447 INFO:teuthology.orchestra.run.smithi150.stdout:(128/131): NetworkManager-initscripts-updown-1. 500 kB/s | 138 kB 00:00 2023-10-11T16:19:55.483 INFO:teuthology.orchestra.run.smithi130.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 976 kB/s | 81 kB 00:00 2023-10-11T16:19:55.525 INFO:teuthology.orchestra.run.smithi130.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 266 kB/s | 11 kB 00:00 2023-10-11T16:19:55.567 INFO:teuthology.orchestra.run.smithi130.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.6 MB/s | 384 kB 00:00 2023-10-11T16:19:55.617 INFO:teuthology.orchestra.run.smithi130.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 191 kB/s | 18 kB 00:00 2023-10-11T16:19:55.659 INFO:teuthology.orchestra.run.smithi130.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 1.1 MB/s | 49 kB 00:00 2023-10-11T16:19:55.689 INFO:teuthology.orchestra.run.smithi150.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.2 MB/s | 798 kB 00:00 2023-10-11T16:19:55.701 INFO:teuthology.orchestra.run.smithi130.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.4 MB/s | 59 kB 00:00 2023-10-11T16:19:55.743 INFO:teuthology.orchestra.run.smithi130.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2023-10-11T16:19:55.960 INFO:teuthology.orchestra.run.smithi130.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-10-11T16:19:56.035 INFO:teuthology.orchestra.run.smithi130.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 209 kB/s | 16 kB 00:00 2023-10-11T16:19:56.059 INFO:teuthology.orchestra.run.smithi111.stdout:(25/131): librgw2-18.0.0-3908.g05a43c8f.el8.x86 1.8 MB/s | 6.5 MB 00:03 2023-10-11T16:19:56.081 INFO:teuthology.orchestra.run.smithi150.stdout:(130/131): librados2-18.0.0-3908.g05a43c8f.el8. 4.1 MB/s | 3.7 MB 00:00 2023-10-11T16:19:56.111 INFO:teuthology.orchestra.run.smithi130.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 448 kB/s | 34 kB 00:00 2023-10-11T16:19:56.226 INFO:teuthology.orchestra.run.smithi111.stdout:(26/131): rbd-nbd-18.0.0-3908.g05a43c8f.el8.x86 1.1 MB/s | 179 kB 00:00 2023-10-11T16:19:56.302 INFO:teuthology.orchestra.run.smithi130.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 1.0 MB/s | 196 kB 00:00 2023-10-11T16:19:56.343 INFO:teuthology.orchestra.run.smithi111.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3908.g 198 kB/s | 23 kB 00:00 2023-10-11T16:19:56.386 INFO:teuthology.orchestra.run.smithi130.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 771 kB/s | 64 kB 00:00 2023-10-11T16:19:56.461 INFO:teuthology.orchestra.run.smithi130.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 250 kB/s | 19 kB 00:00 2023-10-11T16:19:56.535 INFO:teuthology.orchestra.run.smithi111.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3908.g05a43c8 665 kB/s | 127 kB 00:00 2023-10-11T16:19:56.581 INFO:teuthology.orchestra.run.smithi150.stdout:(131/131): librbd1-18.0.0-3908.g05a43c8f.el8.x8 2.6 MB/s | 3.6 MB 00:01 2023-10-11T16:19:56.586 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-10-11T16:19:56.586 INFO:teuthology.orchestra.run.smithi150.stdout:Total 11 MB/s | 197 MB 00:17 2023-10-11T16:19:56.695 INFO:teuthology.orchestra.run.smithi130.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 102 kB/s | 24 kB 00:00 2023-10-11T16:19:56.729 INFO:teuthology.orchestra.run.smithi130.stdout:(61/131): python3-kubernetes-11.0.0-6.el8.noarc 1.0 MB/s | 1.2 MB 00:01 2023-10-11T16:19:56.754 INFO:teuthology.orchestra.run.smithi130.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 497 kB/s | 29 kB 00:00 2023-10-11T16:19:56.846 INFO:teuthology.orchestra.run.smithi130.stdout:(63/131): ceph-mgr-diskprediction-local-18.0.0- 1.7 MB/s | 7.4 MB 00:04 2023-10-11T16:19:56.863 INFO:teuthology.orchestra.run.smithi130.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 205 kB/s | 27 kB 00:00 2023-10-11T16:19:56.922 INFO:teuthology.orchestra.run.smithi130.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2023-10-11T16:19:56.989 INFO:teuthology.orchestra.run.smithi130.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 1.1 MB/s | 255 kB 00:00 2023-10-11T16:19:57.047 INFO:teuthology.orchestra.run.smithi130.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 682 kB/s | 85 kB 00:00 2023-10-11T16:19:57.064 INFO:teuthology.orchestra.run.smithi130.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 304 kB/s | 23 kB 00:00 2023-10-11T16:19:57.089 INFO:teuthology.orchestra.run.smithi130.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-10-11T16:19:57.165 INFO:teuthology.orchestra.run.smithi130.stdout:(70/131): python3-webob-1.8.5-1.el8.1.noarch.rp 790 kB/s | 251 kB 00:00 2023-10-11T16:19:57.248 INFO:teuthology.orchestra.run.smithi130.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 295 kB/s | 47 kB 00:00 2023-10-11T16:19:57.316 INFO:teuthology.orchestra.run.smithi130.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.3 MB/s | 200 kB 00:00 2023-10-11T16:19:57.357 INFO:teuthology.orchestra.run.smithi130.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 392 kB/s | 43 kB 00:00 2023-10-11T16:19:57.391 INFO:teuthology.orchestra.run.smithi130.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2023-10-11T16:19:57.400 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-10-11T16:19:57.416 INFO:teuthology.orchestra.run.smithi130.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 822 kB/s | 48 kB 00:00 2023-10-11T16:19:57.442 INFO:teuthology.orchestra.run.smithi130.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-10-11T16:19:57.492 INFO:teuthology.orchestra.run.smithi130.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-10-11T16:19:57.517 INFO:teuthology.orchestra.run.smithi130.stdout:(78/131): python3-requests-oauthlib-1.0.0-1.el8 425 kB/s | 43 kB 00:00 2023-10-11T16:19:57.521 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-10-11T16:19:57.522 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-10-11T16:19:57.534 INFO:teuthology.orchestra.run.smithi130.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-10-11T16:19:57.535 INFO:teuthology.orchestra.run.smithi111.stdout:(29/131): ceph-mgr-dashboard-18.0.0-3908.g05a43 1.7 MB/s | 1.7 MB 00:00 2023-10-11T16:19:57.568 INFO:teuthology.orchestra.run.smithi130.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 11 MB/s | 560 kB 00:00 2023-10-11T16:19:57.593 INFO:teuthology.orchestra.run.smithi130.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 319 kB/s | 19 kB 00:00 2023-10-11T16:19:57.669 INFO:teuthology.orchestra.run.smithi130.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.5 MB/s | 155 kB 00:00 2023-10-11T16:19:57.752 INFO:teuthology.orchestra.run.smithi130.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 3.4 MB/s | 559 kB 00:00 2023-10-11T16:19:57.786 INFO:teuthology.orchestra.run.smithi130.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 153 kB/s | 18 kB 00:00 2023-10-11T16:19:57.870 INFO:teuthology.orchestra.run.smithi130.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.1 MB/s | 605 kB 00:00 2023-10-11T16:19:57.920 INFO:teuthology.orchestra.run.smithi130.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2023-10-11T16:19:57.962 INFO:teuthology.orchestra.run.smithi130.stdout:(87/131): thrift-0.13.0-2.el8.x86_64.rpm 1.9 MB/s | 1.7 MB 00:00 2023-10-11T16:19:58.054 INFO:teuthology.orchestra.run.smithi130.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 13 MB/s | 2.3 MB 00:00 2023-10-11T16:19:58.079 INFO:teuthology.orchestra.run.smithi130.stdout:(89/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 455 kB/s | 72 kB 00:00 2023-10-11T16:19:58.104 INFO:teuthology.orchestra.run.smithi130.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 582 kB/s | 83 kB 00:00 2023-10-11T16:19:58.205 INFO:teuthology.orchestra.run.smithi130.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 783 kB/s | 98 kB 00:00 2023-10-11T16:19:58.230 INFO:teuthology.orchestra.run.smithi130.stdout:(92/131): elfutils-libelf-devel-0.186-1.el8.x86 341 kB/s | 60 kB 00:00 2023-10-11T16:19:58.272 INFO:teuthology.orchestra.run.smithi130.stdout:(93/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 345 kB/s | 58 kB 00:00 2023-10-11T16:19:58.297 INFO:teuthology.orchestra.run.smithi130.stdout:(94/131): libcom_err-devel-1.45.6-4.el8.x86_64. 419 kB/s | 39 kB 00:00 2023-10-11T16:19:58.381 INFO:teuthology.orchestra.run.smithi130.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.5 MB/s | 171 kB 00:00 2023-10-11T16:19:58.414 INFO:teuthology.orchestra.run.smithi130.stdout:(96/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.4 MB/s | 644 kB 00:00 2023-10-11T16:19:58.436 INFO:teuthology.orchestra.run.smithi111.stdout:(30/131): rbd-mirror-18.0.0-3908.g05a43c8f.el8. 1.1 MB/s | 3.4 MB 00:03 2023-10-11T16:19:58.440 INFO:teuthology.orchestra.run.smithi130.stdout:(97/131): NetworkManager-initscripts-updown-1.3 967 kB/s | 138 kB 00:00 2023-10-11T16:19:58.582 INFO:teuthology.orchestra.run.smithi130.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 383 kB/s | 54 kB 00:00 2023-10-11T16:19:58.598 INFO:teuthology.orchestra.run.smithi130.stdout:(99/131): python3-pyasn1-modules-0.3.7-6.el8.no 599 kB/s | 110 kB 00:00 2023-10-11T16:19:58.620 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-10-11T16:19:58.624 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-10-11T16:19:58.682 INFO:teuthology.orchestra.run.smithi130.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2023-10-11T16:19:58.724 INFO:teuthology.orchestra.run.smithi130.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2023-10-11T16:19:58.799 INFO:teuthology.orchestra.run.smithi130.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 332 kB/s | 39 kB 00:00 2023-10-11T16:19:58.858 INFO:teuthology.orchestra.run.smithi130.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 5.0 MB/s | 688 kB 00:00 2023-10-11T16:19:58.961 INFO:teuthology.orchestra.run.smithi111.stdout:(31/131): ceph-test-18.0.0-3908.g05a43c8f.el8.x 4.0 MB/s | 41 MB 00:10 2023-10-11T16:19:58.992 INFO:teuthology.orchestra.run.smithi130.stdout:(104/131): python3-scipy-1.0.0-21.module+el8.5. 22 MB/s | 14 MB 00:00 2023-10-11T16:19:59.017 INFO:teuthology.orchestra.run.smithi130.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 990 kB/s | 157 kB 00:00 2023-10-11T16:19:59.019 INFO:teuthology.orchestra.run.smithi111.stdout:(32/131): ceph-mgr-modules-core-18.0.0-3908.g05 412 kB/s | 240 kB 00:00 2023-10-11T16:19:59.045 INFO:teuthology.orchestra.run.smithi111.stdout:(33/131): ceph-mgr-rook-18.0.0-3908.g05a43c8f.e 569 kB/s | 47 kB 00:00 2023-10-11T16:19:59.050 INFO:teuthology.orchestra.run.smithi130.stdout:(106/131): python3-jinja2-2.10.1-3.el8.noarch.r 2.1 MB/s | 538 kB 00:00 2023-10-11T16:19:59.087 INFO:teuthology.orchestra.run.smithi111.stdout:(34/131): ceph-prometheus-alerts-18.0.0-3908.g0 218 kB/s | 15 kB 00:00 2023-10-11T16:19:59.117 INFO:teuthology.orchestra.run.smithi130.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-10-11T16:19:59.145 INFO:teuthology.orchestra.run.smithi111.stdout:(35/131): ceph-volume-18.0.0-3908.g05a43c8f.el8 2.6 MB/s | 265 kB 00:00 2023-10-11T16:19:59.151 INFO:teuthology.orchestra.run.smithi130.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-10-11T16:19:59.196 INFO:teuthology.orchestra.run.smithi111.stdout:(36/131): cephfs-top-18.0.0-3908.g05a43c8f.el8. 359 kB/s | 18 kB 00:00 2023-10-11T16:19:59.221 INFO:teuthology.orchestra.run.smithi111.stdout:(37/131): cephadm-18.0.0-3908.g05a43c8f.el8.noa 624 kB/s | 83 kB 00:00 2023-10-11T16:19:59.260 INFO:teuthology.orchestra.run.smithi130.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2023-10-11T16:19:59.285 INFO:teuthology.orchestra.run.smithi130.stdout:(110/131): python3-jsonpatch-1.21-2.el8.noarch. 158 kB/s | 27 kB 00:00 2023-10-11T16:19:59.310 INFO:teuthology.orchestra.run.smithi130.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2023-10-11T16:19:59.369 INFO:teuthology.orchestra.run.smithi130.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 404 kB/s | 44 kB 00:00 2023-10-11T16:19:59.461 INFO:teuthology.orchestra.run.smithi130.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 3.0 MB/s | 457 kB 00:00 2023-10-11T16:19:59.528 INFO:teuthology.orchestra.run.smithi130.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 15 MB/s | 3.7 MB 00:00 2023-10-11T16:19:59.545 INFO:teuthology.orchestra.run.smithi130.stdout:(115/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.3 MB/s | 225 kB 00:00 2023-10-11T16:19:59.587 INFO:teuthology.orchestra.run.smithi130.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 511 kB/s | 64 kB 00:00 2023-10-11T16:19:59.613 INFO:teuthology.orchestra.run.smithi111.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 150 kB/s | 59 kB 00:00 2023-10-11T16:19:59.662 INFO:teuthology.orchestra.run.smithi130.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 171 kB/s | 20 kB 00:00 2023-10-11T16:19:59.687 INFO:teuthology.orchestra.run.smithi130.stdout:(118/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 793 kB/s | 126 kB 00:00 2023-10-11T16:19:59.746 INFO:teuthology.orchestra.run.smithi130.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 650 kB/s | 103 kB 00:00 2023-10-11T16:19:59.814 INFO:teuthology.orchestra.run.smithi111.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 496 kB/s | 306 kB 00:00 2023-10-11T16:19:59.838 INFO:teuthology.orchestra.run.smithi130.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 3.9 MB/s | 709 kB 00:00 2023-10-11T16:19:59.839 INFO:teuthology.orchestra.run.smithi111.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 333 kB/s | 75 kB 00:00 2023-10-11T16:19:59.888 INFO:teuthology.orchestra.run.smithi130.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.4 MB/s | 198 kB 00:00 2023-10-11T16:19:59.990 INFO:teuthology.orchestra.run.smithi111.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 287 kB/s | 44 kB 00:00 2023-10-11T16:20:00.005 INFO:teuthology.orchestra.run.smithi130.stdout:(122/131): openblas-threads-0.3.15-3.el8.x86_64 15 MB/s | 4.9 MB 00:00 2023-10-11T16:20:00.030 INFO:teuthology.orchestra.run.smithi130.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 1.4 MB/s | 202 kB 00:00 2023-10-11T16:20:00.031 INFO:teuthology.orchestra.run.smithi111.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.1 MB/s | 476 kB 00:00 2023-10-11T16:20:00.098 INFO:teuthology.orchestra.run.smithi130.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-10-11T16:20:00.218 INFO:teuthology.orchestra.run.smithi130.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.6 MB/s | 354 kB 00:00 2023-10-11T16:20:00.340 INFO:teuthology.orchestra.run.smithi130.stdout:(126/131): postgresql-server-10.21-2.module+el8 16 MB/s | 5.1 MB 00:00 2023-10-11T16:20:00.373 INFO:teuthology.orchestra.run.smithi130.stdout:(127/131): postgresql-contrib-10.21-2.module+el 2.9 MB/s | 808 kB 00:00 2023-10-11T16:20:00.440 INFO:teuthology.orchestra.run.smithi130.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 6.8 MB/s | 1.5 MB 00:00 2023-10-11T16:20:00.540 INFO:teuthology.orchestra.run.smithi130.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 7.8 MB/s | 798 kB 00:00 2023-10-11T16:20:00.603 INFO:teuthology.orchestra.run.smithi111.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 58 kB/s | 33 kB 00:00 2023-10-11T16:20:00.632 INFO:teuthology.orchestra.run.smithi111.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 289 kB/s | 185 kB 00:00 2023-10-11T16:20:00.732 INFO:teuthology.orchestra.run.smithi111.stdout:(45/131): ceph-mgr-diskprediction-local-18.0.0- 2.3 MB/s | 7.4 MB 00:03 2023-10-11T16:20:00.749 INFO:teuthology.orchestra.run.smithi111.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 111 kB/s | 16 kB 00:00 2023-10-11T16:20:00.833 INFO:teuthology.orchestra.run.smithi111.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 860 kB/s | 173 kB 00:00 2023-10-11T16:20:00.883 INFO:teuthology.orchestra.run.smithi111.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 609 kB/s | 81 kB 00:00 2023-10-11T16:20:00.942 INFO:teuthology.orchestra.run.smithi111.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-10-11T16:20:00.984 INFO:teuthology.orchestra.run.smithi111.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 175 kB/s | 18 kB 00:00 2023-10-11T16:20:00.990 INFO:teuthology.orchestra.run.smithi130.stdout:(130/131): librados2-18.0.0-3908.g05a43c8f.el8. 5.6 MB/s | 3.7 MB 00:00 2023-10-11T16:20:01.101 INFO:teuthology.orchestra.run.smithi111.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 417 kB/s | 49 kB 00:00 2023-10-11T16:20:01.234 INFO:teuthology.orchestra.run.smithi111.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 441 kB/s | 59 kB 00:00 2023-10-11T16:20:01.301 INFO:teuthology.orchestra.run.smithi111.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 3.2 MB/s | 1.2 MB 00:00 2023-10-11T16:20:01.335 INFO:teuthology.orchestra.run.smithi111.stdout:(54/131): python3-cherrypy-18.4.0-1.el8.noarch. 638 kB/s | 384 kB 00:00 2023-10-11T16:20:01.360 INFO:teuthology.orchestra.run.smithi111.stdout:(55/131): python3-natsort-7.1.1-2.el8.noarch.rp 478 kB/s | 60 kB 00:00 2023-10-11T16:20:01.419 INFO:teuthology.orchestra.run.smithi111.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2023-10-11T16:20:01.444 INFO:teuthology.orchestra.run.smithi111.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 143 kB/s | 16 kB 00:00 2023-10-11T16:20:01.470 INFO:teuthology.orchestra.run.smithi111.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 309 kB/s | 34 kB 00:00 2023-10-11T16:20:01.528 INFO:teuthology.orchestra.run.smithi111.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2023-10-11T16:20:01.620 INFO:teuthology.orchestra.run.smithi111.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 124 kB/s | 19 kB 00:00 2023-10-11T16:20:01.646 INFO:teuthology.orchestra.run.smithi111.stdout:(61/131): python3-rsa-4.9-2.el8.noarch.rpm 319 kB/s | 64 kB 00:00 2023-10-11T16:20:01.671 INFO:teuthology.orchestra.run.smithi111.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 168 kB/s | 24 kB 00:00 2023-10-11T16:20:01.746 INFO:teuthology.orchestra.run.smithi111.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 232 kB/s | 29 kB 00:00 2023-10-11T16:20:01.772 INFO:teuthology.orchestra.run.smithi111.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 220 kB/s | 27 kB 00:00 2023-10-11T16:20:01.797 INFO:teuthology.orchestra.run.smithi111.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 2.0 MB/s | 255 kB 00:00 2023-10-11T16:20:01.889 INFO:teuthology.orchestra.run.smithi111.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.7 MB/s | 251 kB 00:00 2023-10-11T16:20:01.914 INFO:teuthology.orchestra.run.smithi111.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 731 kB/s | 85 kB 00:00 2023-10-11T16:20:01.940 INFO:teuthology.orchestra.run.smithi111.stdout:(68/131): python3-websocket-client-0.56.0-5.el8 365 kB/s | 61 kB 00:00 2023-10-11T16:20:01.965 INFO:teuthology.orchestra.run.smithi111.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-10-11T16:20:01.999 INFO:teuthology.orchestra.run.smithi111.stdout:(70/131): python3-zc-lockfile-2.0-2.el8.noarch. 210 kB/s | 23 kB 00:00 2023-10-11T16:20:02.041 INFO:teuthology.orchestra.run.smithi130.stdout:(131/131): librbd1-18.0.0-3908.g05a43c8f.el8.x8 2.1 MB/s | 3.6 MB 00:01 2023-10-11T16:20:02.044 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2023-10-11T16:20:02.044 INFO:teuthology.orchestra.run.smithi130.stdout:Total 8.5 MB/s | 197 MB 00:23 2023-10-11T16:20:02.115 INFO:teuthology.orchestra.run.smithi111.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 993 kB/s | 110 kB 00:00 2023-10-11T16:20:02.241 INFO:teuthology.orchestra.run.smithi111.stdout:(72/131): thrift-0.13.0-2.el8.x86_64.rpm 5.3 MB/s | 1.7 MB 00:00 2023-10-11T16:20:02.291 INFO:teuthology.orchestra.run.smithi111.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 309 kB/s | 54 kB 00:00 2023-10-11T16:20:02.416 INFO:teuthology.orchestra.run.smithi111.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-10-11T16:20:02.466 INFO:teuthology.orchestra.run.smithi111.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2023-10-11T16:20:02.550 INFO:teuthology.orchestra.run.smithi111.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 291 kB/s | 39 kB 00:00 2023-10-11T16:20:02.742 INFO:teuthology.orchestra.run.smithi111.stdout:(77/131): python3-scipy-1.0.0-21.module+el8.5.0 18 MB/s | 14 MB 00:00 2023-10-11T16:20:02.776 INFO:teuthology.orchestra.run.smithi111.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 2.3 MB/s | 538 kB 00:00 2023-10-11T16:20:02.817 INFO:teuthology.orchestra.run.smithi111.stdout:(79/131): bison-3.0.4-10.el8.x86_64.rpm 1.9 MB/s | 688 kB 00:00 2023-10-11T16:20:02.867 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2023-10-11T16:20:02.894 INFO:teuthology.orchestra.run.smithi111.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2023-10-11T16:20:02.988 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2023-10-11T16:20:02.989 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2023-10-11T16:20:03.035 INFO:teuthology.orchestra.run.smithi111.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 724 kB/s | 187 kB 00:00 2023-10-11T16:20:03.110 INFO:teuthology.orchestra.run.smithi111.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 2.4 MB/s | 713 kB 00:00 2023-10-11T16:20:03.153 INFO:teuthology.orchestra.run.smithi111.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 227 kB/s | 27 kB 00:00 2023-10-11T16:20:03.244 INFO:teuthology.orchestra.run.smithi111.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2023-10-11T16:20:03.278 INFO:teuthology.orchestra.run.smithi111.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.9 MB/s | 320 kB 00:00 2023-10-11T16:20:03.303 INFO:teuthology.orchestra.run.smithi111.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 292 kB/s | 44 kB 00:00 2023-10-11T16:20:03.376 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-10-11T16:20:03.395 INFO:teuthology.orchestra.run.smithi111.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 3.8 MB/s | 457 kB 00:00 2023-10-11T16:20:03.437 INFO:teuthology.orchestra.run.smithi111.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.6 MB/s | 225 kB 00:00 2023-10-11T16:20:03.521 INFO:teuthology.orchestra.run.smithi111.stdout:(89/131): python3-numpy-1.14.3-10.el8.x86_64.rp 14 MB/s | 3.7 MB 00:00 2023-10-11T16:20:03.546 INFO:teuthology.orchestra.run.smithi111.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 425 kB/s | 64 kB 00:00 2023-10-11T16:20:03.596 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-11T16:20:03.605 INFO:teuthology.orchestra.run.smithi111.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 756 kB/s | 126 kB 00:00 2023-10-11T16:20:03.680 INFO:teuthology.orchestra.run.smithi111.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 774 kB/s | 103 kB 00:00 2023-10-11T16:20:03.705 INFO:teuthology.orchestra.run.smithi111.stdout:(93/131): python3-jsonpointer-1.10-11.el8.noarc 108 kB/s | 20 kB 00:00 2023-10-11T16:20:03.777 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-11T16:20:03.797 INFO:teuthology.orchestra.run.smithi111.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.6 MB/s | 709 kB 00:00 2023-10-11T16:20:03.856 INFO:teuthology.orchestra.run.smithi111.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.3 MB/s | 198 kB 00:00 2023-10-11T16:20:03.973 INFO:teuthology.orchestra.run.smithi111.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 1.7 MB/s | 202 kB 00:00 2023-10-11T16:20:03.980 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-11T16:20:04.076 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2023-10-11T16:20:04.078 INFO:teuthology.orchestra.run.smithi111.stdout:(97/131): openblas-threads-0.3.15-3.el8.x86_64. 12 MB/s | 4.9 MB 00:00 2023-10-11T16:20:04.080 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2023-10-11T16:20:04.115 INFO:teuthology.orchestra.run.smithi111.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-10-11T16:20:04.132 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-11T16:20:04.199 INFO:teuthology.orchestra.run.smithi111.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-10-11T16:20:04.235 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-11T16:20:04.266 INFO:teuthology.orchestra.run.smithi111.stdout:(100/131): postgresql-contrib-10.21-2.module+el 5.2 MB/s | 808 kB 00:00 2023-10-11T16:20:04.375 INFO:teuthology.orchestra.run.smithi111.stdout:(101/131): postgresql-server-10.21-2.module+el8 17 MB/s | 5.1 MB 00:00 2023-10-11T16:20:04.425 INFO:teuthology.orchestra.run.smithi111.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 6.7 MB/s | 1.5 MB 00:00 2023-10-11T16:20:04.450 INFO:teuthology.orchestra.run.smithi111.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 255 kB/s | 47 kB 00:00 2023-10-11T16:20:04.457 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-11T16:20:04.509 INFO:teuthology.orchestra.run.smithi111.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.5 MB/s | 200 kB 00:00 2023-10-11T16:20:04.642 INFO:teuthology.orchestra.run.smithi111.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 196 kB/s | 43 kB 00:00 2023-10-11T16:20:04.701 INFO:teuthology.orchestra.run.smithi111.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 880 kB/s | 220 kB 00:00 2023-10-11T16:20:04.785 INFO:teuthology.orchestra.run.smithi111.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 174 kB/s | 48 kB 00:00 2023-10-11T16:20:04.810 INFO:teuthology.orchestra.run.smithi111.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 392 kB/s | 43 kB 00:00 2023-10-11T16:20:04.835 INFO:teuthology.orchestra.run.smithi111.stdout:(109/131): libsepol-devel-2.9-3.el8.x86_64.rpm 451 kB/s | 87 kB 00:00 2023-10-11T16:20:04.861 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-11T16:20:04.952 INFO:teuthology.orchestra.run.smithi111.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 652 kB/s | 109 kB 00:00 2023-10-11T16:20:04.986 INFO:teuthology.orchestra.run.smithi111.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.3 MB/s | 229 kB 00:00 2023-10-11T16:20:05.020 INFO:teuthology.orchestra.run.smithi111.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.0 MB/s | 560 kB 00:00 2023-10-11T16:20:05.037 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:20:05.103 INFO:teuthology.orchestra.run.smithi111.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.3 MB/s | 155 kB 00:00 2023-10-11T16:20:05.128 INFO:teuthology.orchestra.run.smithi111.stdout:(114/131): libaio-devel-0.3.112-1.el8.x86_64.rp 106 kB/s | 19 kB 00:00 2023-10-11T16:20:05.170 INFO:teuthology.orchestra.run.smithi111.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.6 MB/s | 559 kB 00:00 2023-10-11T16:20:05.346 INFO:teuthology.orchestra.run.smithi111.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 74 kB/s | 18 kB 00:00 2023-10-11T16:20:05.379 INFO:teuthology.orchestra.run.smithi111.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2023-10-11T16:20:05.413 INFO:teuthology.orchestra.run.smithi111.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.1 MB/s | 605 kB 00:00 2023-10-11T16:20:05.456 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:20:05.530 INFO:teuthology.orchestra.run.smithi111.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 480 kB/s | 72 kB 00:00 2023-10-11T16:20:05.580 INFO:teuthology.orchestra.run.smithi111.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 495 kB/s | 83 kB 00:00 2023-10-11T16:20:05.656 INFO:teuthology.orchestra.run.smithi111.stdout:(121/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 7.5 MB/s | 2.3 MB 00:00 2023-10-11T16:20:05.689 INFO:teuthology.orchestra.run.smithi111.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 377 kB/s | 60 kB 00:00 2023-10-11T16:20:05.781 INFO:teuthology.orchestra.run.smithi111.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 490 kB/s | 98 kB 00:00 2023-10-11T16:20:05.806 INFO:teuthology.orchestra.run.smithi111.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 383 kB/s | 58 kB 00:00 2023-10-11T16:20:05.848 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:20:05.873 INFO:teuthology.orchestra.run.smithi111.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 210 kB/s | 39 kB 00:00 2023-10-11T16:20:05.907 INFO:teuthology.orchestra.run.smithi111.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 5.0 MB/s | 644 kB 00:00 2023-10-11T16:20:05.932 INFO:teuthology.orchestra.run.smithi111.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.3 MB/s | 171 kB 00:00 2023-10-11T16:20:05.975 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:20:06.049 INFO:teuthology.orchestra.run.smithi111.stdout:(128/131): NetworkManager-initscripts-updown-1. 784 kB/s | 138 kB 00:00 2023-10-11T16:20:06.156 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-11T16:20:06.216 INFO:teuthology.orchestra.run.smithi111.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.7 MB/s | 798 kB 00:00 2023-10-11T16:20:06.258 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-11T16:20:06.383 INFO:teuthology.orchestra.run.smithi111.stdout:(130/131): librados2-18.0.0-3908.g05a43c8f.el8. 7.7 MB/s | 3.7 MB 00:00 2023-10-11T16:20:06.434 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-11T16:20:06.684 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-11T16:20:06.877 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-11T16:20:07.252 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-11T16:20:07.461 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-11T16:20:07.516 INFO:teuthology.orchestra.run.smithi111.stdout:(131/131): librbd1-18.0.0-3908.g05a43c8f.el8.x8 2.2 MB/s | 3.6 MB 00:01 2023-10-11T16:20:07.523 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T16:20:07.523 INFO:teuthology.orchestra.run.smithi111.stdout:Total 7.5 MB/s | 197 MB 00:26 2023-10-11T16:20:07.598 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-11T16:20:07.700 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-11T16:20:07.848 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-11T16:20:07.926 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-11T16:20:08.062 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-11T16:20:08.184 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-11T16:20:08.364 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-11T16:20:08.981 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-11T16:20:09.148 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-11T16:20:09.183 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2023-10-11T16:20:09.317 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2023-10-11T16:20:09.317 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2023-10-11T16:20:09.324 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2023-10-11T16:20:09.524 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-11T16:20:09.716 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-11T16:20:09.747 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-11T16:20:09.885 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-10-11T16:20:09.939 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-11T16:20:09.995 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-10-11T16:20:10.259 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-10-11T16:20:10.628 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-11T16:20:10.662 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-10-11T16:20:10.867 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 5/134 2023-10-11T16:20:10.879 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-11T16:20:10.975 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2023-10-11T16:20:10.980 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2023-10-11T16:20:11.108 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-11T16:20:11.269 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 5/134 2023-10-11T16:20:11.334 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-11T16:20:11.426 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:20:11.531 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:20:11.554 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-11T16:20:11.742 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:20:11.754 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-11T16:20:11.830 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:20:11.967 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-11T16:20:11.981 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-11T16:20:12.134 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-10-11T16:20:12.145 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 26/134 2023-10-11T16:20:12.267 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 27/134 2023-10-11T16:20:12.353 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-11T16:20:12.603 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-11T16:20:12.995 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-11T16:20:13.153 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 28/134 2023-10-11T16:20:13.229 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-11T16:20:13.384 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-11T16:20:13.480 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-11T16:20:13.486 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-11T16:20:13.625 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-11T16:20:13.757 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-11T16:20:13.758 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-11T16:20:13.970 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-10-11T16:20:14.010 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-11T16:20:14.189 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-10-11T16:20:14.214 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-11T16:20:14.381 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-10-11T16:20:14.508 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-10-11T16:20:14.645 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-10-11T16:20:14.825 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-10-11T16:20:15.428 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-10-11T16:20:15.600 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-10-11T16:20:15.979 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-11T16:20:15.995 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2023-10-11T16:20:16.197 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-11T16:20:16.202 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-10-11T16:20:16.352 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-11T16:20:16.365 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-11T16:20:16.383 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-10-11T16:20:16.558 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-11T16:20:16.584 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-11T16:20:16.748 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-11T16:20:16.800 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-11T16:20:16.897 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-11T16:20:16.978 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 36/134 2023-10-11T16:20:17.060 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-10-11T16:20:17.140 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-11T16:20:17.142 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 37/134 2023-10-11T16:20:17.312 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-11T16:20:17.320 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-10-11T16:20:17.490 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-11T16:20:17.523 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-10-11T16:20:17.600 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-11T16:20:17.691 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-11T16:20:17.743 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-11T16:20:17.831 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:20:17.849 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-11T16:20:17.931 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 26/134 2023-10-11T16:20:18.045 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 27/134 2023-10-11T16:20:18.110 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-11T16:20:18.254 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 28/134 2023-10-11T16:20:18.278 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:20:18.309 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-11T16:20:18.433 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-10-11T16:20:18.447 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:20:18.599 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:20:18.649 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-11T16:20:18.789 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-11T16:20:18.903 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-11T16:20:18.925 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-11T16:20:19.109 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-11T16:20:19.124 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-11T16:20:19.353 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-11T16:20:19.362 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-11T16:20:19.584 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-11T16:20:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-11T16:20:19.887 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-11T16:20:20.029 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-11T16:20:20.121 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-11T16:20:20.193 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-10-11T16:20:20.277 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-11T16:20:20.329 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-11T16:20:20.422 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-11T16:20:20.534 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-11T16:20:20.550 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-11T16:20:20.692 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-11T16:20:20.739 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-11T16:20:20.821 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-11T16:20:20.858 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-11T16:20:20.955 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-11T16:20:21.127 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-11T16:20:21.153 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-11T16:20:21.174 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-10-11T16:20:21.289 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-11T16:20:21.409 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-11T16:20:21.463 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-10-11T16:20:21.651 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-11T16:20:21.709 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-10-11T16:20:21.870 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-11T16:20:21.943 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-10-11T16:20:21.982 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-11T16:20:22.036 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-11T16:20:22.152 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-11T16:20:22.172 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-11T16:20:22.205 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-11T16:20:22.329 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-11T16:20:22.351 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-11T16:20:22.398 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-11T16:20:22.547 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 36/134 2023-10-11T16:20:22.642 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-11T16:20:22.697 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 37/134 2023-10-11T16:20:22.847 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-10-11T16:20:22.871 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-11T16:20:22.890 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-11T16:20:23.005 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-10-11T16:20:23.029 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-11T16:20:23.080 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-11T16:20:23.110 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-11T16:20:23.207 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-10-11T16:20:23.305 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-10-11T16:20:23.317 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-11T16:20:23.494 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-11T16:20:23.510 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-10-11T16:20:23.629 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-11T16:20:23.710 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-10-11T16:20:23.788 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-11T16:20:23.865 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-10-11T16:20:23.930 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-11T16:20:23.949 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-10-11T16:20:24.021 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-11T16:20:24.178 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-10-11T16:20:24.221 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-11T16:20:24.236 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-11T16:20:24.344 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-10-11T16:20:24.408 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-11T16:20:24.448 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-11T16:20:24.522 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-10-11T16:20:24.556 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-11T16:20:24.683 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-10-11T16:20:24.695 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-11T16:20:24.754 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-11T16:20:24.901 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-10-11T16:20:24.915 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-11T16:20:24.957 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-11T16:20:25.052 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-10-11T16:20:25.156 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-11T16:20:25.176 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-11T16:20:25.255 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-10-11T16:20:25.307 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-11T16:20:25.443 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-11T16:20:25.452 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-10-11T16:20:25.459 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-11T16:20:25.665 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 26/134 2023-10-11T16:20:25.689 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-11T16:20:25.793 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 27/134 2023-10-11T16:20:25.834 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-10-11T16:20:25.991 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-11T16:20:26.150 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-10-11T16:20:26.176 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-11T16:20:26.401 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-10-11T16:20:26.411 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-11T16:20:26.634 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-10-11T16:20:26.818 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 28/134 2023-10-11T16:20:26.861 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-10-11T16:20:27.038 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-11T16:20:27.169 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-11T16:20:27.172 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-10-11T16:20:27.204 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-11T16:20:27.373 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-10-11T16:20:27.422 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-11T16:20:27.438 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-11T16:20:27.553 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-10-11T16:20:27.769 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-10-11T16:20:27.784 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-11T16:20:27.952 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-10-11T16:20:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-11T16:20:28.126 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-10-11T16:20:28.416 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-10-11T16:20:28.561 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-11T16:20:28.670 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-10-11T16:20:28.824 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-11T16:20:28.902 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-10-11T16:20:29.054 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-11T16:20:29.099 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-10-11T16:20:29.247 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-11T16:20:29.389 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-11T16:20:29.467 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-11T16:20:29.647 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-11T16:20:29.724 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-11T16:20:29.846 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-11T16:20:29.963 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-11T16:20:30.077 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-11T16:20:30.093 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-11T16:20:30.266 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-11T16:20:30.446 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-11T16:20:30.455 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-11T16:20:30.595 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-11T16:20:30.659 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-11T16:20:30.667 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-11T16:20:30.767 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-11T16:20:30.839 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-11T16:20:30.848 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-11T16:20:30.860 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-11T16:20:30.995 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 93/134 2023-10-11T16:20:31.034 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-11T16:20:31.162 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-11T16:20:31.354 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 36/134 2023-10-11T16:20:31.378 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-11T16:20:31.543 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 37/134 2023-10-11T16:20:31.618 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-11T16:20:31.739 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-11T16:20:31.820 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-11T16:20:31.953 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-11T16:20:32.046 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-11T16:20:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-11T16:20:32.238 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-11T16:20:32.378 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-11T16:20:32.387 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-11T16:20:32.594 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-11T16:20:32.629 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-11T16:20:32.843 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-11T16:20:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-11T16:20:33.578 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-11T16:20:33.712 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-11T16:20:33.803 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-11T16:20:34.041 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-11T16:20:34.042 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-11T16:20:34.288 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-11T16:20:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-11T16:20:34.590 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-11T16:20:34.638 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:34.640 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-11T16:20:34.870 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:34.880 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-11T16:20:34.890 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-11T16:20:35.058 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-11T16:20:35.088 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-11T16:20:35.314 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-11T16:20:35.466 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:35.502 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-11T16:20:35.675 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-11T16:20:35.720 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-11T16:20:35.789 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-11T16:20:35.829 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-11T16:20:35.919 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-11T16:20:35.980 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-11T16:20:36.037 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-11T16:20:36.041 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-11T16:20:36.197 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 93/134 2023-10-11T16:20:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-11T16:20:36.618 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-11T16:20:36.845 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-11T16:20:37.045 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-11T16:20:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-11T16:20:37.444 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-11T16:20:37.542 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-11T16:20:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-11T16:20:38.059 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-11T16:20:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-11T16:20:38.363 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-11T16:20:38.590 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-11T16:20:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-11T16:20:38.946 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-11T16:20:39.126 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-11T16:20:39.451 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-11T16:20:39.661 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-11T16:20:39.841 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-11T16:20:39.863 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:39.999 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-11T16:20:40.088 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:40.200 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-11T16:20:40.460 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-11T16:20:40.668 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-11T16:20:40.670 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:40.889 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-11T16:20:40.964 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-11T16:20:41.043 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-11T16:20:41.045 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-11T16:20:41.180 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-11T16:20:41.337 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-11T16:20:41.558 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-11T16:20:41.751 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-11T16:20:42.074 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-11T16:20:42.287 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-11T16:20:42.512 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-11T16:20:42.743 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-11T16:20:44.087 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-11T16:20:44.413 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-11T16:20:44.674 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-11T16:20:45.084 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-11T16:20:45.389 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-11T16:20:45.587 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-11T16:20:46.388 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-11T16:20:46.543 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-11T16:20:46.642 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-11T16:20:46.844 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 93/134 2023-10-11T16:20:51.722 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:52.235 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:52.919 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-11T16:20:53.219 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-11T16:20:53.357 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-11T16:20:53.534 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-11T16:20:54.749 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-11T16:20:54.749 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2023-10-11T16:20:54.749 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2023-10-11T16:20:54.749 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /dev 2023-10-11T16:20:54.749 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /run 2023-10-11T16:20:54.750 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2023-10-11T16:20:54.750 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2023-10-11T16:20:54.750 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2023-10-11T16:20:54.750 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2023-10-11T16:20:54.750 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /dev 2023-10-11T16:20:54.751 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:20:55.084 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-11T16:20:56.649 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-11T16:20:56.884 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-11T16:20:57.107 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-11T16:20:57.284 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-11T16:20:57.733 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-11T16:20:57.892 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-11T16:20:57.908 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-11T16:20:59.764 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-11T16:20:59.820 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-11T16:20:59.821 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2023-10-11T16:20:59.821 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2023-10-11T16:20:59.821 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2023-10-11T16:20:59.821 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /run 2023-10-11T16:20:59.821 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2023-10-11T16:20:59.822 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2023-10-11T16:20:59.822 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2023-10-11T16:20:59.822 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2023-10-11T16:20:59.822 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2023-10-11T16:20:59.822 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:20:59.944 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-11T16:21:00.081 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 102/134 2023-10-11T16:21:00.152 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-11T16:21:00.305 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 103/134 2023-10-11T16:21:00.491 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-11T16:21:00.706 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-11T16:21:01.008 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-11T16:21:01.722 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-11T16:21:01.936 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-11T16:21:01.970 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-11T16:21:02.158 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-11T16:21:02.164 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-11T16:21:02.249 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-11T16:21:02.343 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-11T16:21:02.585 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 107/134 2023-10-11T16:21:02.796 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-11T16:21:02.800 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-11T16:21:02.835 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-11T16:21:02.943 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-11T16:21:02.960 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-11T16:21:02.991 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-11T16:21:03.002 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-11T16:21:04.248 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 110/134 2023-10-11T16:21:04.430 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-11T16:21:04.464 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-11T16:21:04.599 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-11T16:21:04.622 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-11T16:21:04.641 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-11T16:21:04.766 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-11T16:21:04.805 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-11T16:21:04.940 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 102/134 2023-10-11T16:21:05.111 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-11T16:21:05.157 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 103/134 2023-10-11T16:21:05.266 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-11T16:21:05.334 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-11T16:21:05.546 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-11T16:21:05.885 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-11T16:21:06.847 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-11T16:21:07.009 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-11T16:21:07.100 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-11T16:21:07.480 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 107/134 2023-10-11T16:21:07.682 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-11T16:21:07.718 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-11T16:21:07.859 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-11T16:21:07.869 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-11T16:21:09.107 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 110/134 2023-10-11T16:21:09.296 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-11T16:21:09.329 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-11T16:21:09.388 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-11T16:21:09.459 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-11T16:21:09.500 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-11T16:21:09.667 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-11T16:21:09.699 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 115/134 2023-10-11T16:21:09.865 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-11T16:21:10.015 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-11T16:21:10.066 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-11T16:21:10.159 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-11T16:21:10.954 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-11T16:21:11.313 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-11T16:21:11.493 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 119/134 2023-10-11T16:21:11.660 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-11T16:21:11.825 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 121/134 2023-10-11T16:21:11.974 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-11T16:21:13.643 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-11T16:21:13.643 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2023-10-11T16:21:13.643 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2023-10-11T16:21:13.643 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2023-10-11T16:21:13.644 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /run 2023-10-11T16:21:13.644 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2023-10-11T16:21:13.644 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2023-10-11T16:21:13.644 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2023-10-11T16:21:13.644 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2023-10-11T16:21:13.644 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2023-10-11T16:21:13.645 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:21:14.048 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-11T16:21:14.091 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-11T16:21:14.164 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:14.429 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:14.446 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:14.590 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 124/134 2023-10-11T16:21:14.872 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 125/134 2023-10-11T16:21:14.979 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 115/134 2023-10-11T16:21:15.158 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 126/134 2023-10-11T16:21:15.314 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-10-11T16:21:15.458 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-11T16:21:15.494 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 117/134 2023-10-11T16:21:15.597 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-11T16:21:15.606 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-11T16:21:15.751 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-11T16:21:15.752 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-10-11T16:21:15.778 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-11T16:21:15.932 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-11T16:21:15.949 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-10-11T16:21:15.997 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-11T16:21:16.080 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-11T16:21:16.146 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-10-11T16:21:16.197 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-11T16:21:16.264 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-11T16:21:16.276 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-11T16:21:16.276 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:16.353 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-10-11T16:21:16.389 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-11T16:21:16.467 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:16.468 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-11T16:21:16.512 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 121/134 2023-10-11T16:21:16.677 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-11T16:21:16.692 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-11T16:21:16.888 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:16.905 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-11T16:21:17.131 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-11T16:21:17.172 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-11T16:21:17.366 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:17.366 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 1/134 2023-10-11T16:21:17.366 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2/134 2023-10-11T16:21:17.367 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 3/134 2023-10-11T16:21:17.367 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 4/134 2023-10-11T16:21:17.367 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 5/134 2023-10-11T16:21:17.367 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:21:17.367 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:21:17.368 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-11T16:21:17.368 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-11T16:21:17.368 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 10/134 2023-10-11T16:21:17.368 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 11/134 2023-10-11T16:21:17.368 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 12/134 2023-10-11T16:21:17.368 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 13/134 2023-10-11T16:21:17.369 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 14/134 2023-10-11T16:21:17.369 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 15/134 2023-10-11T16:21:17.369 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 16/134 2023-10-11T16:21:17.369 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 17/134 2023-10-11T16:21:17.370 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 18/134 2023-10-11T16:21:17.370 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 19/134 2023-10-11T16:21:17.370 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 20/134 2023-10-11T16:21:17.370 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 21/134 2023-10-11T16:21:17.370 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 22/134 2023-10-11T16:21:17.371 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 23/134 2023-10-11T16:21:17.371 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 24/134 2023-10-11T16:21:17.371 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 25/134 2023-10-11T16:21:17.371 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 26/134 2023-10-11T16:21:17.371 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 27/134 2023-10-11T16:21:17.372 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 28/134 2023-10-11T16:21:17.372 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 29/134 2023-10-11T16:21:17.372 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 30/134 2023-10-11T16:21:17.372 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 31/134 2023-10-11T16:21:17.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 32/134 2023-10-11T16:21:17.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 33/134 2023-10-11T16:21:17.381 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 34/134 2023-10-11T16:21:17.382 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 35/134 2023-10-11T16:21:17.382 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 36/134 2023-10-11T16:21:17.382 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 37/134 2023-10-11T16:21:17.382 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 38/134 2023-10-11T16:21:17.382 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-11T16:21:17.383 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-11T16:21:17.383 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-11T16:21:17.383 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-11T16:21:17.383 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-11T16:21:17.384 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-11T16:21:17.384 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-11T16:21:17.384 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-11T16:21:17.384 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-11T16:21:17.384 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-11T16:21:17.385 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-11T16:21:17.385 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-11T16:21:17.385 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-11T16:21:17.385 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-11T16:21:17.385 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-11T16:21:17.386 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-11T16:21:17.386 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-11T16:21:17.386 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-11T16:21:17.386 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-11T16:21:17.386 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-11T16:21:17.387 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-11T16:21:17.387 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-11T16:21:17.387 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-11T16:21:17.387 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-11T16:21:17.387 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-11T16:21:17.388 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-11T16:21:17.388 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-11T16:21:17.388 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-11T16:21:17.388 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-11T16:21:17.388 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-11T16:21:17.389 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-11T16:21:17.389 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-11T16:21:17.389 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-11T16:21:17.389 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-11T16:21:17.389 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-11T16:21:17.390 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-11T16:21:17.390 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-11T16:21:17.390 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-11T16:21:17.390 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-11T16:21:17.390 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-11T16:21:17.391 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-11T16:21:17.391 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-11T16:21:17.391 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-11T16:21:17.391 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-11T16:21:17.391 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-11T16:21:17.392 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-11T16:21:17.392 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-11T16:21:17.392 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-11T16:21:17.392 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-11T16:21:17.392 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-11T16:21:17.393 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-11T16:21:17.393 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-11T16:21:17.393 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-11T16:21:17.393 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-11T16:21:17.393 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-11T16:21:17.394 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-11T16:21:17.394 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-11T16:21:17.394 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-11T16:21:17.394 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-11T16:21:17.394 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-11T16:21:17.395 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-11T16:21:17.395 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-11T16:21:17.395 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-11T16:21:17.395 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-11T16:21:17.395 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-11T16:21:17.396 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-11T16:21:17.396 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-11T16:21:17.396 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-11T16:21:17.396 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-11T16:21:17.396 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-11T16:21:17.397 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-11T16:21:17.397 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-11T16:21:17.397 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-11T16:21:17.397 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-11T16:21:17.397 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-11T16:21:17.398 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-11T16:21:17.398 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-11T16:21:17.398 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-11T16:21:17.398 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-11T16:21:17.398 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-11T16:21:17.399 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-11T16:21:17.399 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-11T16:21:17.399 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-11T16:21:17.399 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-11T16:21:17.399 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-11T16:21:17.399 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-11T16:21:17.400 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-11T16:21:17.400 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-11T16:21:17.400 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-11T16:21:17.400 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-11T16:21:17.400 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 129/134 2023-10-11T16:21:17.400 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-11T16:21:17.401 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 131/134 2023-10-11T16:21:17.401 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:17.401 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-11T16:21:18.852 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:19.115 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:19.134 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:19.283 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 124/134 2023-10-11T16:21:19.557 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 125/134 2023-10-11T16:21:19.829 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:19.830 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:53 ago on Wed 11 Oct 2023 04:19:26 PM UTC. 2023-10-11T16:21:19.872 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 126/134 2023-10-11T16:21:19.953 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-11T16:21:20.176 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-11T16:21:20.198 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-11T16:21:20.288 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 102/134 2023-10-11T16:21:20.365 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-10-11T16:21:20.455 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-10-11T16:21:20.493 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 103/134 2023-10-11T16:21:20.640 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-10-11T16:21:20.653 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-10-11T16:21:20.688 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-11T16:21:20.798 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-10-11T16:21:20.806 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-10-11T16:21:20.917 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-10-11T16:21:20.918 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:20.952 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-11T16:21:21.093 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:21.093 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-11T16:21:21.239 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-11T16:21:21.328 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-11T16:21:21.522 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:22.025 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:22.025 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 1/134 2023-10-11T16:21:22.026 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2/134 2023-10-11T16:21:22.026 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 3/134 2023-10-11T16:21:22.026 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 4/134 2023-10-11T16:21:22.026 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 5/134 2023-10-11T16:21:22.026 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:21:22.027 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:21:22.027 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-11T16:21:22.027 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-11T16:21:22.027 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 10/134 2023-10-11T16:21:22.027 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 11/134 2023-10-11T16:21:22.028 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 12/134 2023-10-11T16:21:22.028 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 13/134 2023-10-11T16:21:22.028 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 14/134 2023-10-11T16:21:22.028 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 15/134 2023-10-11T16:21:22.028 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 16/134 2023-10-11T16:21:22.029 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 17/134 2023-10-11T16:21:22.029 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 18/134 2023-10-11T16:21:22.029 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 19/134 2023-10-11T16:21:22.029 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 20/134 2023-10-11T16:21:22.029 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 21/134 2023-10-11T16:21:22.029 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 22/134 2023-10-11T16:21:22.030 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 23/134 2023-10-11T16:21:22.030 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 24/134 2023-10-11T16:21:22.030 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 25/134 2023-10-11T16:21:22.030 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 26/134 2023-10-11T16:21:22.031 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 27/134 2023-10-11T16:21:22.031 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 28/134 2023-10-11T16:21:22.031 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 29/134 2023-10-11T16:21:22.040 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 30/134 2023-10-11T16:21:22.040 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 31/134 2023-10-11T16:21:22.040 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 32/134 2023-10-11T16:21:22.041 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 33/134 2023-10-11T16:21:22.041 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 34/134 2023-10-11T16:21:22.041 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 35/134 2023-10-11T16:21:22.041 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 36/134 2023-10-11T16:21:22.041 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 37/134 2023-10-11T16:21:22.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 38/134 2023-10-11T16:21:22.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-11T16:21:22.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-11T16:21:22.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-11T16:21:22.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-11T16:21:22.042 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-11T16:21:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-11T16:21:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-11T16:21:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-11T16:21:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-11T16:21:22.043 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-11T16:21:22.044 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-11T16:21:22.044 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-11T16:21:22.044 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-11T16:21:22.044 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-11T16:21:22.044 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-11T16:21:22.045 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-11T16:21:22.045 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-11T16:21:22.045 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-11T16:21:22.045 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-11T16:21:22.045 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-11T16:21:22.045 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-11T16:21:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-11T16:21:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-11T16:21:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-11T16:21:22.046 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-11T16:21:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-11T16:21:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-11T16:21:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-11T16:21:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-11T16:21:22.047 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-11T16:21:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-11T16:21:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-11T16:21:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-11T16:21:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-10-11T16:21:22.048 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-10-11T16:21:22.049 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-10-11T16:21:22.049 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-10-11T16:21:22.049 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-10-11T16:21:22.049 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-10-11T16:21:22.049 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-10-11T16:21:22.049 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-10-11T16:21:22.050 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-10-11T16:21:22.050 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-10-11T16:21:22.050 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-10-11T16:21:22.050 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-10-11T16:21:22.051 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-10-11T16:21:22.051 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-10-11T16:21:22.051 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-10-11T16:21:22.051 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-10-11T16:21:22.051 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-10-11T16:21:22.051 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-10-11T16:21:22.052 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-10-11T16:21:22.052 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-10-11T16:21:22.052 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-10-11T16:21:22.052 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-10-11T16:21:22.052 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-10-11T16:21:22.053 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-10-11T16:21:22.053 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-10-11T16:21:22.053 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-10-11T16:21:22.053 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-10-11T16:21:22.053 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-10-11T16:21:22.054 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-10-11T16:21:22.054 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-10-11T16:21:22.054 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-10-11T16:21:22.054 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-10-11T16:21:22.054 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-10-11T16:21:22.054 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-10-11T16:21:22.055 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-10-11T16:21:22.055 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-10-11T16:21:22.055 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-10-11T16:21:22.055 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-10-11T16:21:22.055 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-10-11T16:21:22.056 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-10-11T16:21:22.056 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-10-11T16:21:22.056 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-10-11T16:21:22.056 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-10-11T16:21:22.056 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-10-11T16:21:22.057 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-10-11T16:21:22.057 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-10-11T16:21:22.057 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-10-11T16:21:22.057 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-10-11T16:21:22.057 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-10-11T16:21:22.057 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-10-11T16:21:22.058 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-10-11T16:21:22.058 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-10-11T16:21:22.058 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-10-11T16:21:22.058 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-10-11T16:21:22.058 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-10-11T16:21:22.059 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-10-11T16:21:22.059 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-10-11T16:21:22.059 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 129/134 2023-10-11T16:21:22.059 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-11T16:21:22.059 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 131/134 2023-10-11T16:21:22.060 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:22.060 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-11T16:21:22.326 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-11T16:21:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-11T16:21:22.646 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-11T16:21:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 107/134 2023-10-11T16:21:23.281 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-11T16:21:23.347 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-11T16:21:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-11T16:21:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-11T16:21:24.510 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:24.510 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Oct 2023 04:19:27 PM UTC. 2023-10-11T16:21:24.934 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 110/134 2023-10-11T16:21:25.156 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-11T16:21:25.230 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-11T16:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-11T16:21:25.441 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-11T16:21:25.582 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-10-11T16:21:25.653 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-11T16:21:25.795 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:21:25.796 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2023-10-11T16:21:25.796 INFO:teuthology.orchestra.run.smithi150.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-11T16:21:25.796 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.796 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.797 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-10-11T16:21:25.797 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-11T16:21:25.797 INFO:teuthology.orchestra.run.smithi150.stdout: automake-1.16.1-7.el8.noarch 2023-10-11T16:21:25.797 INFO:teuthology.orchestra.run.smithi150.stdout: bison-3.0.4-10.el8.x86_64 2023-10-11T16:21:25.797 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.798 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.798 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.798 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.798 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.798 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.799 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.799 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.799 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.799 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.799 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.800 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.800 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.800 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.800 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.800 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.801 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.801 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.802 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.802 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:25.802 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-11T16:21:25.802 INFO:teuthology.orchestra.run.smithi150.stdout: flex-2.6.1-9.el8.x86_64 2023-10-11T16:21:25.802 INFO:teuthology.orchestra.run.smithi150.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-11T16:21:25.803 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-11T16:21:25.803 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-3.el8.x86_64 2023-10-11T16:21:25.803 INFO:teuthology.orchestra.run.smithi150.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-11T16:21:25.803 INFO:teuthology.orchestra.run.smithi150.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-11T16:21:25.803 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-11T16:21:25.804 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.804 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.804 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.804 INFO:teuthology.orchestra.run.smithi150.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-11T16:21:25.804 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-11T16:21:25.804 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-11T16:21:25.805 INFO:teuthology.orchestra.run.smithi150.stdout: libpq-13.5-1.el8.x86_64 2023-10-11T16:21:25.805 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-11T16:21:25.805 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-11T16:21:25.805 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.805 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.806 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-11T16:21:25.806 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.806 INFO:teuthology.orchestra.run.smithi150.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-11T16:21:25.806 INFO:teuthology.orchestra.run.smithi150.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-11T16:21:25.807 INFO:teuthology.orchestra.run.smithi150.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-11T16:21:25.807 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-11T16:21:25.807 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-11T16:21:25.807 INFO:teuthology.orchestra.run.smithi150.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-11T16:21:25.807 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-11T16:21:25.808 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-11T16:21:25.808 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-11T16:21:25.808 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-11T16:21:25.808 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-11T16:21:25.808 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-11T16:21:25.809 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-11T16:21:25.809 INFO:teuthology.orchestra.run.smithi150.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-11T16:21:25.809 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:25.809 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:25.809 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:25.810 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-11T16:21:25.810 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-11T16:21:25.810 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-11T16:21:25.810 INFO:teuthology.orchestra.run.smithi150.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-11T16:21:25.810 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-11T16:21:25.810 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.811 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.811 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.811 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-11T16:21:25.811 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-11T16:21:25.811 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-11T16:21:25.812 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-11T16:21:25.812 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-11T16:21:25.812 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-11T16:21:25.812 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-11T16:21:25.812 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-11T16:21:25.813 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-11T16:21:25.813 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-11T16:21:25.813 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-11T16:21:25.813 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-11T16:21:25.813 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-11T16:21:25.813 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-11T16:21:25.814 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-11T16:21:25.814 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-11T16:21:25.814 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-11T16:21:25.814 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-11T16:21:25.814 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-11T16:21:25.815 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-11T16:21:25.815 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-11T16:21:25.815 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-11T16:21:25.815 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-2.6-1.el8.noarch 2023-10-11T16:21:25.815 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-11T16:21:25.816 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-11T16:21:25.816 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-11T16:21:25.816 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-11T16:21:25.816 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-11T16:21:25.816 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-11T16:21:25.816 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.817 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.817 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-11T16:21:25.817 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-11T16:21:25.817 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.817 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-11T16:21:25.818 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-11T16:21:25.818 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-11T16:21:25.818 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-11T16:21:25.818 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-11T16:21:25.818 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-11T16:21:25.819 INFO:teuthology.orchestra.run.smithi150.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-11T16:21:25.819 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-11T16:21:25.819 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-11T16:21:25.819 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-11T16:21:25.819 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-11T16:21:25.820 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-11T16:21:25.820 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-11T16:21:25.820 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.820 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.820 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:25.820 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-11T16:21:25.821 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-11T16:21:25.821 INFO:teuthology.orchestra.run.smithi150.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-11T16:21:25.821 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-11T16:21:25.821 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-11T16:21:25.822 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-11T16:21:25.822 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:21:25.822 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-10-11T16:21:25.979 DEBUG:teuthology.parallel:result is None 2023-10-11T16:21:26.058 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-11T16:21:26.213 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-11T16:21:30.479 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-11T16:21:30.837 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 115/134 2023-10-11T16:21:31.053 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-11T16:21:31.260 INFO:teuthology.orchestra.run.smithi130.stdout:Installed products updated. 2023-10-11T16:21:31.342 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-11T16:21:31.474 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:21:31.475 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2023-10-11T16:21:31.475 INFO:teuthology.orchestra.run.smithi130.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-11T16:21:31.475 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.475 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.476 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2023-10-11T16:21:31.476 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-11T16:21:31.476 INFO:teuthology.orchestra.run.smithi130.stdout: automake-1.16.1-7.el8.noarch 2023-10-11T16:21:31.476 INFO:teuthology.orchestra.run.smithi130.stdout: bison-3.0.4-10.el8.x86_64 2023-10-11T16:21:31.476 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.477 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.477 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.477 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.477 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.477 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.477 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.478 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.478 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.478 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.478 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.478 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.479 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.479 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.479 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.479 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.479 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.480 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.480 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.480 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.480 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.480 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.480 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:31.481 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-11T16:21:31.481 INFO:teuthology.orchestra.run.smithi130.stdout: flex-2.6.1-9.el8.x86_64 2023-10-11T16:21:31.481 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-11T16:21:31.481 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-11T16:21:31.481 INFO:teuthology.orchestra.run.smithi130.stdout: jq-1.6-3.el8.x86_64 2023-10-11T16:21:31.481 INFO:teuthology.orchestra.run.smithi130.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-11T16:21:31.482 INFO:teuthology.orchestra.run.smithi130.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-11T16:21:31.482 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-11T16:21:31.482 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.482 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.483 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.483 INFO:teuthology.orchestra.run.smithi130.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-11T16:21:31.483 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-11T16:21:31.484 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-11T16:21:31.484 INFO:teuthology.orchestra.run.smithi130.stdout: libpq-13.5-1.el8.x86_64 2023-10-11T16:21:31.484 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-11T16:21:31.485 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-11T16:21:31.485 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.485 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.486 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-11T16:21:31.486 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.486 INFO:teuthology.orchestra.run.smithi130.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-11T16:21:31.487 INFO:teuthology.orchestra.run.smithi130.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-11T16:21:31.487 INFO:teuthology.orchestra.run.smithi130.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-11T16:21:31.488 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-11T16:21:31.488 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-11T16:21:31.488 INFO:teuthology.orchestra.run.smithi130.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-11T16:21:31.489 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-11T16:21:31.489 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-11T16:21:31.489 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-11T16:21:31.490 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-11T16:21:31.490 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-11T16:21:31.490 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-11T16:21:31.491 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-11T16:21:31.491 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-11T16:21:31.491 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:31.492 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:31.492 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:31.492 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-11T16:21:31.493 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-11T16:21:31.493 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-11T16:21:31.493 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-11T16:21:31.494 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-11T16:21:31.494 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.494 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.495 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.495 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-11T16:21:31.495 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-11T16:21:31.496 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-11T16:21:31.496 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-11T16:21:31.496 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-11T16:21:31.497 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-11T16:21:31.497 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-11T16:21:31.498 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-11T16:21:31.498 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-11T16:21:31.498 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-11T16:21:31.499 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-11T16:21:31.499 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-11T16:21:31.499 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-11T16:21:31.500 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-11T16:21:31.500 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-11T16:21:31.500 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-11T16:21:31.501 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-11T16:21:31.501 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-11T16:21:31.501 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-11T16:21:31.501 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-11T16:21:31.502 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-11T16:21:31.502 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-11T16:21:31.502 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-2.6-1.el8.noarch 2023-10-11T16:21:31.503 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-11T16:21:31.503 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-11T16:21:31.503 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-11T16:21:31.504 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-11T16:21:31.504 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-11T16:21:31.505 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-11T16:21:31.505 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.505 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.506 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-11T16:21:31.506 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-11T16:21:31.506 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.507 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-11T16:21:31.507 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-11T16:21:31.507 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-11T16:21:31.507 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-11T16:21:31.508 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-11T16:21:31.508 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-11T16:21:31.508 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-11T16:21:31.508 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-11T16:21:31.508 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-11T16:21:31.509 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-11T16:21:31.509 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-11T16:21:31.509 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-11T16:21:31.509 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-11T16:21:31.509 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.510 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.510 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:31.510 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-11T16:21:31.510 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-11T16:21:31.510 INFO:teuthology.orchestra.run.smithi130.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-11T16:21:31.511 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-11T16:21:31.511 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-11T16:21:31.511 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-11T16:21:31.512 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-10-11T16:21:31.512 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2023-10-11T16:21:31.672 DEBUG:teuthology.parallel:result is None 2023-10-11T16:21:32.624 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-11T16:21:32.957 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-11T16:21:33.175 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 119/134 2023-10-11T16:21:33.391 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-11T16:21:33.624 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 121/134 2023-10-11T16:21:33.779 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-11T16:21:37.230 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:37.523 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:37.561 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-11T16:21:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 124/134 2023-10-11T16:21:38.019 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 125/134 2023-10-11T16:21:38.400 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 126/134 2023-10-11T16:21:38.704 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-11T16:21:38.902 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-11T16:21:38.925 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-11T16:21:39.203 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-11T16:21:39.410 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-11T16:21:39.522 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-11T16:21:39.777 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-11T16:21:39.806 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-11T16:21:39.806 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:40.008 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:40.008 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-11T16:21:40.316 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-11T16:21:40.535 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:40.857 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:40.857 INFO:teuthology.orchestra.run.smithi111.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-11T16:21:40.857 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:21:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 1/134 2023-10-11T16:21:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2/134 2023-10-11T16:21:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 3/134 2023-10-11T16:21:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 4/134 2023-10-11T16:21:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 5/134 2023-10-11T16:21:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-11T16:21:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-11T16:21:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-11T16:21:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-11T16:21:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 10/134 2023-10-11T16:21:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 11/134 2023-10-11T16:21:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 12/134 2023-10-11T16:21:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 13/134 2023-10-11T16:21:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 14/134 2023-10-11T16:21:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 15/134 2023-10-11T16:21:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 16/134 2023-10-11T16:21:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 17/134 2023-10-11T16:21:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 18/134 2023-10-11T16:21:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 19/134 2023-10-11T16:21:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 20/134 2023-10-11T16:21:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 21/134 2023-10-11T16:21:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 22/134 2023-10-11T16:21:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 23/134 2023-10-11T16:21:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 24/134 2023-10-11T16:21:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 25/134 2023-10-11T16:21:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 26/134 2023-10-11T16:21:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 27/134 2023-10-11T16:21:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 28/134 2023-10-11T16:21:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 29/134 2023-10-11T16:21:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 30/134 2023-10-11T16:21:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 31/134 2023-10-11T16:21:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 32/134 2023-10-11T16:21:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 33/134 2023-10-11T16:21:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 34/134 2023-10-11T16:21:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 35/134 2023-10-11T16:21:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 36/134 2023-10-11T16:21:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 37/134 2023-10-11T16:21:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 38/134 2023-10-11T16:21:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-11T16:21:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-11T16:21:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-11T16:21:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-11T16:21:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-11T16:21:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-11T16:21:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-11T16:21:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-11T16:21:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-11T16:21:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-11T16:21:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-11T16:21:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-11T16:21:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-11T16:21:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-11T16:21:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-11T16:21:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-11T16:21:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-11T16:21:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-11T16:21:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-11T16:21:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-11T16:21:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-11T16:21:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-11T16:21:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-11T16:21:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-11T16:21:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-11T16:21:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-11T16:21:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-11T16:21:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-11T16:21:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-11T16:21:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-11T16:21:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-11T16:21:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-11T16:21:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-11T16:21:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-11T16:21:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-11T16:21:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-11T16:21:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-11T16:21:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-11T16:21:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-11T16:21:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-11T16:21:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-11T16:21:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-11T16:21:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-11T16:21:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-11T16:21:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-11T16:21:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-11T16:21:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-11T16:21:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-11T16:21:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-11T16:21:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-11T16:21:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-11T16:21:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-11T16:21:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-11T16:21:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-11T16:21:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-11T16:21:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-11T16:21:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-11T16:21:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-11T16:21:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-11T16:21:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-11T16:21:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-11T16:21:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-11T16:21:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-11T16:21:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-11T16:21:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-11T16:21:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-11T16:21:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-11T16:21:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-11T16:21:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-11T16:21:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-11T16:21:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-11T16:21:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-11T16:21:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-11T16:21:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-11T16:21:41.547 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-11T16:21:41.547 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-11T16:21:41.547 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-11T16:21:41.547 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-11T16:21:41.547 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-11T16:21:41.548 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-11T16:21:41.548 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-11T16:21:41.548 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-11T16:21:41.548 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-11T16:21:41.548 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-11T16:21:41.549 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-11T16:21:41.549 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-11T16:21:41.549 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-11T16:21:41.549 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-11T16:21:41.549 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-11T16:21:41.550 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-11T16:21:41.550 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 129/134 2023-10-11T16:21:41.550 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-11T16:21:41.550 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 131/134 2023-10-11T16:21:41.550 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-11T16:21:41.551 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-11T16:21:44.260 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-11T16:21:44.260 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:16 ago on Wed 11 Oct 2023 04:19:28 PM UTC. 2023-10-11T16:21:49.528 INFO:teuthology.orchestra.run.smithi111.stdout:Installed products updated. 2023-10-11T16:21:49.752 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:21:49.753 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2023-10-11T16:21:49.753 INFO:teuthology.orchestra.run.smithi111.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-11T16:21:49.753 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.753 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.753 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2023-10-11T16:21:49.754 INFO:teuthology.orchestra.run.smithi111.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-11T16:21:49.754 INFO:teuthology.orchestra.run.smithi111.stdout: automake-1.16.1-7.el8.noarch 2023-10-11T16:21:49.754 INFO:teuthology.orchestra.run.smithi111.stdout: bison-3.0.4-10.el8.x86_64 2023-10-11T16:21:49.754 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.755 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.755 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.755 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.755 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.755 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.756 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.756 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.756 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.756 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.756 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.757 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.757 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.757 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.757 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.757 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.758 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.758 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.758 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.758 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.759 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.759 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.759 INFO:teuthology.orchestra.run.smithi111.stdout: cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-11T16:21:49.759 INFO:teuthology.orchestra.run.smithi111.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-11T16:21:49.759 INFO:teuthology.orchestra.run.smithi111.stdout: flex-2.6.1-9.el8.x86_64 2023-10-11T16:21:49.760 INFO:teuthology.orchestra.run.smithi111.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-11T16:21:49.760 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-11T16:21:49.760 INFO:teuthology.orchestra.run.smithi111.stdout: jq-1.6-3.el8.x86_64 2023-10-11T16:21:49.760 INFO:teuthology.orchestra.run.smithi111.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-11T16:21:49.760 INFO:teuthology.orchestra.run.smithi111.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-11T16:21:49.761 INFO:teuthology.orchestra.run.smithi111.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-11T16:21:49.761 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.761 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.761 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.762 INFO:teuthology.orchestra.run.smithi111.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-11T16:21:49.762 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-11T16:21:49.762 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-11T16:21:49.762 INFO:teuthology.orchestra.run.smithi111.stdout: libpq-13.5-1.el8.x86_64 2023-10-11T16:21:49.763 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-11T16:21:49.763 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-11T16:21:49.763 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.763 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.764 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-11T16:21:49.764 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.764 INFO:teuthology.orchestra.run.smithi111.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-11T16:21:49.764 INFO:teuthology.orchestra.run.smithi111.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-11T16:21:49.765 INFO:teuthology.orchestra.run.smithi111.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-11T16:21:49.765 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-11T16:21:49.765 INFO:teuthology.orchestra.run.smithi111.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-11T16:21:49.765 INFO:teuthology.orchestra.run.smithi111.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-11T16:21:49.765 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-11T16:21:49.766 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-11T16:21:49.766 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-11T16:21:49.766 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-11T16:21:49.766 INFO:teuthology.orchestra.run.smithi111.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-11T16:21:49.767 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-11T16:21:49.767 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-11T16:21:49.767 INFO:teuthology.orchestra.run.smithi111.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-11T16:21:49.767 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:49.767 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:49.768 INFO:teuthology.orchestra.run.smithi111.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-11T16:21:49.768 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-11T16:21:49.768 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-11T16:21:49.768 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-11T16:21:49.768 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-11T16:21:49.769 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-11T16:21:49.769 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.769 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.769 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.769 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-11T16:21:49.769 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-11T16:21:49.770 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-11T16:21:49.770 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-11T16:21:49.770 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-11T16:21:49.770 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-11T16:21:49.771 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-11T16:21:49.771 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-11T16:21:49.771 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-11T16:21:49.771 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-11T16:21:49.771 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-11T16:21:49.771 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-11T16:21:49.772 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-11T16:21:49.772 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-11T16:21:49.772 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-11T16:21:49.772 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-11T16:21:49.773 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-11T16:21:49.773 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-11T16:21:49.773 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-11T16:21:49.773 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-11T16:21:49.773 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-11T16:21:49.774 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-11T16:21:49.774 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-2.6-1.el8.noarch 2023-10-11T16:21:49.774 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-11T16:21:49.774 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-11T16:21:49.775 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-11T16:21:49.775 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-11T16:21:49.775 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-11T16:21:49.776 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-11T16:21:49.778 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.779 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.779 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-11T16:21:49.779 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-11T16:21:49.779 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-11T16:21:49.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-11T16:21:49.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-11T16:21:49.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-11T16:21:49.780 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-11T16:21:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-11T16:21:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-11T16:21:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-11T16:21:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-11T16:21:49.781 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-11T16:21:49.782 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-11T16:21:49.782 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-11T16:21:49.782 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-11T16:21:49.782 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.783 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.783 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-11T16:21:49.783 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-11T16:21:49.783 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-11T16:21:49.783 INFO:teuthology.orchestra.run.smithi111.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-11T16:21:49.784 INFO:teuthology.orchestra.run.smithi111.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-11T16:21:49.784 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-11T16:21:49.784 INFO:teuthology.orchestra.run.smithi111.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-11T16:21:49.785 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:21:49.785 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2023-10-11T16:21:50.077 DEBUG:teuthology.parallel:result is None 2023-10-11T16:21:50.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:21:50.231 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-11T16:21:50.284 INFO:teuthology.orchestra.run.smithi111.stdout:18.0.0-3908.g05a43c8f.el8 2023-10-11T16:21:50.285 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3908.g05a43c8f.el8 2023-10-11T16:21:50.286 INFO:teuthology.task.install:The correct ceph version 18.0.0-3908.g05a43c8f is installed. 2023-10-11T16:21:50.286 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:21:50.642 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-11T16:21:50.671 INFO:teuthology.orchestra.run.smithi130.stdout:18.0.0-3908.g05a43c8f.el8 2023-10-11T16:21:50.671 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3908.g05a43c8f.el8 2023-10-11T16:21:50.672 INFO:teuthology.task.install:The correct ceph version 18.0.0-3908.g05a43c8f is installed. 2023-10-11T16:21:50.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:21:50.809 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-11T16:21:50.838 INFO:teuthology.orchestra.run.smithi150.stdout:18.0.0-3908.g05a43c8f.el8 2023-10-11T16:21:50.839 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3908.g05a43c8f.el8 2023-10-11T16:21:50.839 INFO:teuthology.task.install:The correct ceph version 18.0.0-3908.g05a43c8f is installed. 2023-10-11T16:21:50.840 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-11T16:21:50.840 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:21:50.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-11T16:21:50.932 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:21:50.932 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-11T16:21:50.981 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:21:50.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-11T16:21:51.029 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-11T16:21:51.029 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:21:51.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2023-10-11T16:21:51.098 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-11T16:21:51.206 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:21:51.206 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2023-10-11T16:21:51.239 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-11T16:21:51.310 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:21:51.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2023-10-11T16:21:51.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-11T16:21:51.413 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-11T16:21:51.414 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:21:51.414 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-11T16:21:51.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-11T16:21:51.592 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:21:51.592 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-11T16:21:51.625 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-11T16:21:51.696 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:21:51.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-11T16:21:51.728 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-11T16:21:51.800 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-11T16:21:51.801 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:21:51.801 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2023-10-11T16:21:51.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-11T16:21:51.977 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:21:51.977 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2023-10-11T16:21:52.009 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-11T16:21:52.080 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:21:52.080 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2023-10-11T16:21:52.112 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-11T16:21:52.183 INFO:teuthology.run_tasks:Running task cephadm... 2023-10-11T16:21:52.263 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'subvols': {'create': 2}} 2023-10-11T16:21:52.263 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:21:52.264 INFO:tasks.cephadm:Cluster fsid is 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:21:52.264 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-10-11T16:21:52.265 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.111', 'mon.b': '172.21.15.130', 'mon.c': '172.21.15.150'} 2023-10-11T16:21:52.265 INFO:tasks.cephadm:First mon is mon.a on smithi111 2023-10-11T16:21:52.265 INFO:tasks.cephadm:First mgr is x 2023-10-11T16:21:52.265 INFO:tasks.cephadm:Normalizing hostnames... 2023-10-11T16:21:52.266 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2023-10-11T16:21:52.335 DEBUG:teuthology.orchestra.run.smithi130:> sudo hostname $(hostname -s) 2023-10-11T16:21:52.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2023-10-11T16:21:52.399 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-10-11T16:21:52.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:21:52.540 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/72870/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-3908-g05a43c8f', 'node_name': '172.21.2.4+braggi04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-3908.g05a43c8f'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-09-28 20:32:16.258921', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-choffman-62793', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-10-11T16:21:52.798 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-choffman-62793, sha1 05a43c8fb3988914c879a50970f9054a098e0612 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-11T16:21:52.799 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/x86_64/flavors/default/cephadm 2023-10-11T16:21:52.799 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:53.800 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Oct 11 16:21 /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:53.801 DEBUG:teuthology.orchestra.run.smithi130:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:54.800 INFO:teuthology.orchestra.run.smithi130.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Oct 11 16:21 /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:54.801 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:55.430 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Oct 11 16:21 /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:55.431 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:55.483 DEBUG:teuthology.orchestra.run.smithi130:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:55.505 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-11T16:21:55.540 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 on all hosts... 2023-10-11T16:21:55.541 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 pull 2023-10-11T16:21:55.546 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 pull 2023-10-11T16:21:55.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 pull 2023-10-11T16:21:56.938 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-11T16:21:56.969 INFO:teuthology.orchestra.run.smithi130.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-11T16:21:59.924 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-11T16:22:23.421 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-10-11T16:22:23.421 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev)", 2023-10-11T16:22:23.421 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b", 2023-10-11T16:22:23.421 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2023-10-11T16:22:23.422 INFO:teuthology.orchestra.run.smithi150.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e7b667c026dce026a4358e8fd0cc7b66586a86d2cb1765f49867fecd8b222e58" 2023-10-11T16:22:23.422 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2023-10-11T16:22:23.422 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-10-11T16:22:30.791 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:22:30.792 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev)", 2023-10-11T16:22:30.792 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b", 2023-10-11T16:22:30.792 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2023-10-11T16:22:30.792 INFO:teuthology.orchestra.run.smithi111.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e7b667c026dce026a4358e8fd0cc7b66586a86d2cb1765f49867fecd8b222e58" 2023-10-11T16:22:30.792 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:22:30.793 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:22:33.600 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2023-10-11T16:22:33.601 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev)", 2023-10-11T16:22:33.601 INFO:teuthology.orchestra.run.smithi130.stdout: "image_id": "2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b", 2023-10-11T16:22:33.601 INFO:teuthology.orchestra.run.smithi130.stdout: "repo_digests": [ 2023-10-11T16:22:33.601 INFO:teuthology.orchestra.run.smithi130.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e7b667c026dce026a4358e8fd0cc7b66586a86d2cb1765f49867fecd8b222e58" 2023-10-11T16:22:33.602 INFO:teuthology.orchestra.run.smithi130.stdout: ] 2023-10-11T16:22:33.602 INFO:teuthology.orchestra.run.smithi130.stdout:} 2023-10-11T16:22:33.620 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2023-10-11T16:22:33.699 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph 2023-10-11T16:22:33.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2023-10-11T16:22:33.766 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2023-10-11T16:22:33.836 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /etc/ceph 2023-10-11T16:22:33.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2023-10-11T16:22:33.894 INFO:tasks.cephadm:Writing seed config... 2023-10-11T16:22:33.896 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-10-11T16:22:33.896 INFO:tasks.cephadm: override: [client] debug client = 20 2023-10-11T16:22:33.896 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-10-11T16:22:33.896 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-10-11T16:22:33.897 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-10-11T16:22:33.897 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-10-11T16:22:33.897 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-10-11T16:22:33.897 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-10-11T16:22:33.898 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-10-11T16:22:33.898 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-10-11T16:22:33.898 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-10-11T16:22:33.899 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-10-11T16:22:33.899 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-10-11T16:22:33.899 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-10-11T16:22:33.899 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-10-11T16:22:33.900 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-10-11T16:22:33.900 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-10-11T16:22:33.900 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-10-11T16:22:33.900 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-10-11T16:22:33.901 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-10-11T16:22:33.901 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-10-11T16:22:33.901 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-10-11T16:22:33.902 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-10-11T16:22:33.902 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-10-11T16:22:33.902 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-10-11T16:22:33.902 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-10-11T16:22:33.903 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-10-11T16:22:33.903 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-10-11T16:22:33.903 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-10-11T16:22:33.903 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-10-11T16:22:33.904 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-10-11T16:22:33.904 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-10-11T16:22:33.904 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-10-11T16:22:33.904 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-10-11T16:22:33.905 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-10-11T16:22:33.905 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-10-11T16:22:33.905 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-10-11T16:22:33.906 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-10-11T16:22:33.906 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-10-11T16:22:33.906 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-10-11T16:22:33.906 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-10-11T16:22:33.907 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-10-11T16:22:33.907 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-10-11T16:22:33.907 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-10-11T16:22:33.907 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-10-11T16:22:33.908 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:22:33.909 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-10-11T16:22:33.952 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 495f265e-6852-11ee-8db6-212e2dc638e7 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-10-11T16:22:33.953 DEBUG:teuthology.orchestra.run.smithi111:mon.a> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.a.service 2023-10-11T16:22:33.997 DEBUG:teuthology.orchestra.run.smithi111:mgr.x> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.x.service 2023-10-11T16:22:34.044 INFO:tasks.cephadm:Bootstrapping... 2023-10-11T16:22:34.044 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 -v bootstrap --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.111 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:22:34.065 INFO:journalctl@ceph.mon.a.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:22:34.112 INFO:journalctl@ceph.mgr.x.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:22:34.382 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2023-10-11T16:22:34.382 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612', '-v', 'bootstrap', '--fsid', '495f265e-6852-11ee-8db6-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.111', '--skip-admin-label'] 2023-10-11T16:22:34.557 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 4.1.1 2023-10-11T16:22:34.558 INFO:teuthology.orchestra.run.smithi111.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-10-11T16:22:34.558 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying podman|docker is present... 2023-10-11T16:22:34.729 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 4.1.1 2023-10-11T16:22:34.729 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying lvm2 is present... 2023-10-11T16:22:34.730 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying time synchronization is in place... 2023-10-11T16:22:34.744 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-10-11T16:22:34.744 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-10-11T16:22:34.756 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-10-11T16:22:34.757 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2023-10-11T16:22:34.770 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout enabled 2023-10-11T16:22:34.783 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout active 2023-10-11T16:22:34.783 INFO:teuthology.orchestra.run.smithi111.stdout:Unit chronyd.service is enabled and running 2023-10-11T16:22:34.783 INFO:teuthology.orchestra.run.smithi111.stdout:Repeating the final host check... 2023-10-11T16:22:34.953 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 4.1.1 2023-10-11T16:22:34.954 INFO:teuthology.orchestra.run.smithi111.stdout:podman (/bin/podman) version 4.1.1 is present 2023-10-11T16:22:34.954 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl is present 2023-10-11T16:22:34.954 INFO:teuthology.orchestra.run.smithi111.stdout:lvcreate is present 2023-10-11T16:22:34.966 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-10-11T16:22:34.967 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-10-11T16:22:34.979 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-10-11T16:22:34.979 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2023-10-11T16:22:34.993 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout enabled 2023-10-11T16:22:35.006 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout active 2023-10-11T16:22:35.006 INFO:teuthology.orchestra.run.smithi111.stdout:Unit chronyd.service is enabled and running 2023-10-11T16:22:35.006 INFO:teuthology.orchestra.run.smithi111.stdout:Host looks OK 2023-10-11T16:22:35.006 INFO:teuthology.orchestra.run.smithi111.stdout:Cluster fsid: 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:22:35.007 INFO:teuthology.orchestra.run.smithi111.stdout:Acquiring lock 140444152998376 on /run/cephadm/495f265e-6852-11ee-8db6-212e2dc638e7.lock 2023-10-11T16:22:35.007 INFO:teuthology.orchestra.run.smithi111.stdout:Lock 140444152998376 acquired on /run/cephadm/495f265e-6852-11ee-8db6-212e2dc638e7.lock 2023-10-11T16:22:35.007 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying IP 172.21.15.111 port 3300 ... 2023-10-11T16:22:35.008 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying IP 172.21.15.111 port 6789 ... 2023-10-11T16:22:35.008 INFO:teuthology.orchestra.run.smithi111.stdout:Base mon IP(s) is [172.21.15.111:3300, 172.21.15.111:6789], mon addrv is [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2023-10-11T16:22:35.015 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-10-11T16:22:35.015 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.111 metric 100 2023-10-11T16:22:35.022 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-10-11T16:22:35.022 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-10-11T16:22:35.022 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-10-11T16:22:35.029 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-10-11T16:22:35.029 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-10-11T16:22:35.029 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-10-11T16:22:35.029 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-10-11T16:22:35.030 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout inet6 fe80::9056:a3ce:d946:5f52/64 scope link noprefixroute 2023-10-11T16:22:35.030 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-10-11T16:22:35.031 INFO:teuthology.orchestra.run.smithi111.stdout:Mon IP `172.21.15.111` is in CIDR network `172.21.0.0/20` 2023-10-11T16:22:35.031 INFO:teuthology.orchestra.run.smithi111.stdout:Mon IP `172.21.15.111` is in CIDR network `172.21.0.0/20` 2023-10-11T16:22:35.031 INFO:teuthology.orchestra.run.smithi111.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-10-11T16:22:35.031 INFO:teuthology.orchestra.run.smithi111.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-10-11T16:22:35.032 INFO:teuthology.orchestra.run.smithi111.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-11T16:22:35.506 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b 2023-10-11T16:22:35.506 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-11T16:22:35.507 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Getting image source signatures 2023-10-11T16:22:35.507 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying blob sha256:788df1f70699415e5b73051cdcd2e0fd42e95c6bdebf4a5149631a5f4ae456ad 2023-10-11T16:22:35.507 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying blob sha256:c16254b0810587cdd81d0647eccf1c3e4d200bf4eeba6596492e593ad063d4ee 2023-10-11T16:22:35.507 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying config sha256:2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b 2023-10-11T16:22:35.507 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Writing manifest to image destination 2023-10-11T16:22:35.508 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Storing signatures 2023-10-11T16:22:36.382 INFO:teuthology.orchestra.run.smithi111.stdout:ceph: stdout ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev) 2023-10-11T16:22:36.382 INFO:teuthology.orchestra.run.smithi111.stdout:Ceph version: ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev) 2023-10-11T16:22:36.382 INFO:teuthology.orchestra.run.smithi111.stdout:Extracting ceph user uid/gid from container image... 2023-10-11T16:22:37.162 INFO:teuthology.orchestra.run.smithi111.stdout:stat: stdout 167 167 2023-10-11T16:22:37.162 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial keys... 2023-10-11T16:22:39.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQDOyyZl1X+NLBAA+DNdA5Y9fTMiSz108gWwPg== 2023-10-11T16:22:39.926 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQDPyyZleUR7HxAAUjKRpTmQkto2AfI8198ybw== 2023-10-11T16:22:40.703 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQDQyyZlraMWEhAAH2DG7lk46I/HulRRWZKyMw== 2023-10-11T16:22:40.705 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial monmap... 2023-10-11T16:22:41.382 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-11T16:22:41.382 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-10-11T16:22:41.383 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:22:41.383 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-11T16:22:41.383 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool for a [v2:172.21.15.111:3300,v1:172.21.15.111:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-11T16:22:41.383 INFO:teuthology.orchestra.run.smithi111.stdout:setting min_mon_release = pacific 2023-10-11T16:22:41.383 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: set fsid to 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:22:41.384 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-11T16:22:41.384 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:22:41.384 INFO:teuthology.orchestra.run.smithi111.stdout:Creating mon... 2023-10-11T16:22:42.160 INFO:teuthology.orchestra.run.smithi111.stdout:create mon.a on 2023-10-11T16:22:42.371 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-10-11T16:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-10-11T16:22:42.771 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-495f265e-6852-11ee-8db6-212e2dc638e7.target → /etc/systemd/system/ceph-495f265e-6852-11ee-8db6-212e2dc638e7.target. 2023-10-11T16:22:42.771 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-495f265e-6852-11ee-8db6-212e2dc638e7.target → /etc/systemd/system/ceph-495f265e-6852-11ee-8db6-212e2dc638e7.target. 2023-10-11T16:22:43.052 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.a 2023-10-11T16:22:43.053 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to reset failed state of unit ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.a.service: Unit ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.a.service not loaded. 2023-10-11T16:22:43.245 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-495f265e-6852-11ee-8db6-212e2dc638e7.target.wants/ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.a.service → /etc/systemd/system/ceph-495f265e-6852-11ee-8db6-212e2dc638e7@.service. 2023-10-11T16:22:43.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:43 smithi111 systemd[1]: Starting Ceph mon.a for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:22:44.518 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:44 smithi111 podman[97982]: 2023-10-11T16:22:44.518 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:44 smithi111 bash[97982]: 726acd90fd56c99c263b5df96501230d4f59897ebbd333dd029e25403893fccb 2023-10-11T16:22:44.575 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-11T16:22:44.575 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2023-10-11T16:22:44.591 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-11T16:22:44.591 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2023-10-11T16:22:44.591 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2023-10-11T16:22:44.591 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to enable service . firewalld.service is not available 2023-10-11T16:22:44.592 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mon to start... 2023-10-11T16:22:44.592 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mon... 2023-10-11T16:22:44.875 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:44 smithi111 ceph-mon[98023]: mkfs 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:22:44.875 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:44 smithi111 ceph-mon[98023]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-11T16:22:44.876 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:44 smithi111 systemd[1]: Started Ceph mon.a for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:22:45.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout cluster: 2023-10-11T16:22:45.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout id: 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:22:45.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-10-11T16:22:45.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:22:45.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout services: 2023-10-11T16:22:45.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.694834s) 2023-10-11T16:22:45.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-10-11T16:22:45.683 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-10-11T16:22:45.683 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:22:45.683 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout data: 2023-10-11T16:22:45.683 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-10-11T16:22:45.683 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-10-11T16:22:45.684 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-10-11T16:22:45.684 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout pgs: 2023-10-11T16:22:45.684 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:22:45.684 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.122+0000 7fb0b7a92700 1 Processor -- start 2023-10-11T16:22:45.684 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.123+0000 7fb0b7a92700 1 -- start start 2023-10-11T16:22:45.684 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.123+0000 7fb0b7a92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b01046f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:45.685 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.123+0000 7fb0b7a92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0079e00 con 0x7fb0b01042e0 2023-10-11T16:22:45.685 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.124+0000 7fb0b582e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b01046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:45.685 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.124+0000 7fb0b582e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b01046f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50330/0 (socket says 172.21.15.111:50330) 2023-10-11T16:22:45.685 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.124+0000 7fb0b582e700 1 -- 172.21.15.111:0/879422453 learned_addr learned my addr 172.21.15.111:0/879422453 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:45.685 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.124+0000 7fb0b582e700 1 -- 172.21.15.111:0/879422453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0104c30 con 0x7fb0b01042e0 2023-10-11T16:22:45.686 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.124+0000 7fb0b582e700 1 --2- 172.21.15.111:0/879422453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b01046f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac0098a0 tx=0x7fb0ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530f75ba6bb7d335 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:45.686 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b482c700 1 -- 172.21.15.111:0/879422453 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0ac004160 con 0x7fb0b01042e0 2023-10-11T16:22:45.686 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b482c700 1 -- 172.21.15.111:0/879422453 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac0042c0 con 0x7fb0b01042e0 2023-10-11T16:22:45.686 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b482c700 1 -- 172.21.15.111:0/879422453 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0ac0045a0 con 0x7fb0b01042e0 2023-10-11T16:22:45.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b7a92700 1 -- 172.21.15.111:0/879422453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 msgr2=0x7fb0b01046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:45.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b7a92700 1 --2- 172.21.15.111:0/879422453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b01046f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac0098a0 tx=0x7fb0ac009bb0 comp rx=0 tx=0).stop 2023-10-11T16:22:45.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b7a92700 1 -- 172.21.15.111:0/879422453 shutdown_connections 2023-10-11T16:22:45.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b7a92700 1 --2- 172.21.15.111:0/879422453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b01046f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:45.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b7a92700 1 -- 172.21.15.111:0/879422453 >> 172.21.15.111:0/879422453 conn(0x7fb0b00ffed0 msgr2=0x7fb0b0102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:45.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b7a92700 1 -- 172.21.15.111:0/879422453 shutdown_connections 2023-10-11T16:22:45.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.125+0000 7fb0b7a92700 1 -- 172.21.15.111:0/879422453 wait complete. 2023-10-11T16:22:45.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.126+0000 7fb0b7a92700 1 Processor -- start 2023-10-11T16:22:45.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.126+0000 7fb0b7a92700 1 -- start start 2023-10-11T16:22:45.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.127+0000 7fb0b7a92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b019b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:45.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.127+0000 7fb0b7a92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b019b6a0 con 0x7fb0b01042e0 2023-10-11T16:22:45.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.127+0000 7fb0b582e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b019b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:45.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.127+0000 7fb0b582e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b019b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50334/0 (socket says 172.21.15.111:50334) 2023-10-11T16:22:45.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.127+0000 7fb0b582e700 1 -- 172.21.15.111:0/3078204832 learned_addr learned my addr 172.21.15.111:0/3078204832 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:45.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.127+0000 7fb0b582e700 1 -- 172.21.15.111:0/3078204832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac009550 con 0x7fb0b01042e0 2023-10-11T16:22:45.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.128+0000 7fb0b582e700 1 --2- 172.21.15.111:0/3078204832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b019b160 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009fb0 tx=0x7fb0ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:45.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.128+0000 7fb0a6ffd700 1 -- 172.21.15.111:0/3078204832 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0ac003dd0 con 0x7fb0b01042e0 2023-10-11T16:22:45.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.128+0000 7fb0a6ffd700 1 -- 172.21.15.111:0/3078204832 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac011630 con 0x7fb0b01042e0 2023-10-11T16:22:45.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.128+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b0105130 con 0x7fb0b01042e0 2023-10-11T16:22:45.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.128+0000 7fb0a6ffd700 1 -- 172.21.15.111:0/3078204832 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0ac011910 con 0x7fb0b01042e0 2023-10-11T16:22:45.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.128+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b0104fa0 con 0x7fb0b01042e0 2023-10-11T16:22:45.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.128+0000 7fb0a6ffd700 1 -- 172.21.15.111:0/3078204832 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb0ac01c460 con 0x7fb0b01042e0 2023-10-11T16:22:45.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.128+0000 7fb0a6ffd700 1 -- 172.21.15.111:0/3078204832 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb0ac01a880 con 0x7fb0b01042e0 2023-10-11T16:22:45.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.129+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b0194d70 con 0x7fb0b01042e0 2023-10-11T16:22:45.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.132+0000 7fb0a6ffd700 1 -- 172.21.15.111:0/3078204832 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fb0ac01aa40 con 0x7fb0b01042e0 2023-10-11T16:22:45.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.205+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb0b002cfe0 con 0x7fb0b01042e0 2023-10-11T16:22:45.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.205+0000 7fb0a6ffd700 1 -- 172.21.15.111:0/3078204832 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fb0ac01abe0 con 0x7fb0b01042e0 2023-10-11T16:22:45.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.206+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 msgr2=0x7fb0b019b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:45.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.206+0000 7fb0b7a92700 1 --2- 172.21.15.111:0/3078204832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b019b160 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac009fb0 tx=0x7fb0ac004750 comp rx=0 tx=0).stop 2023-10-11T16:22:45.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.206+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 shutdown_connections 2023-10-11T16:22:45.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.206+0000 7fb0b7a92700 1 --2- 172.21.15.111:0/3078204832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0b01042e0 0x7fb0b019b160 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:45.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.206+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 >> 172.21.15.111:0/3078204832 conn(0x7fb0b00ffed0 msgr2=0x7fb0b0191eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:45.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.206+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 shutdown_connections 2023-10-11T16:22:45.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:45.207+0000 7fb0b7a92700 1 -- 172.21.15.111:0/3078204832 wait complete. 2023-10-11T16:22:45.693 INFO:teuthology.orchestra.run.smithi111.stdout:mon is available 2023-10-11T16:22:45.693 INFO:teuthology.orchestra.run.smithi111.stdout:Assimilating anything we can from ceph.conf... 2023-10-11T16:22:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:45 smithi111 ceph-mon[98023]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-11T16:22:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:45 smithi111 ceph-mon[98023]: monmap e1: 1 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} 2023-10-11T16:22:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:45 smithi111 ceph-mon[98023]: fsmap 2023-10-11T16:22:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:45 smithi111 ceph-mon[98023]: osdmap e1: 0 total, 0 up, 0 in 2023-10-11T16:22:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:45 smithi111 ceph-mon[98023]: mgrmap e1: no daemons active 2023-10-11T16:22:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:45 smithi111 ceph-mon[98023]: from='client.? 172.21.15.111:0/3078204832' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-11T16:22:46.663 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:22:46.663 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [global] 2023-10-11T16:22:46.663 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout fsid = 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:22:46.664 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2023-10-11T16:22:46.664 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-10-11T16:22:46.664 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-10-11T16:22:46.664 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-10-11T16:22:46.664 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-10-11T16:22:46.665 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:22:46.665 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [mgr] 2023-10-11T16:22:46.665 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-10-11T16:22:46.665 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:22:46.665 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [osd] 2023-10-11T16:22:46.666 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-10-11T16:22:46.666 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-10-11T16:22:46.666 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-10-11T16:22:46.666 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-10-11T16:22:46.666 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-10-11T16:22:46.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.164+0000 7f0fd2057700 1 Processor -- start 2023-10-11T16:22:46.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.164+0000 7f0fd2057700 1 -- start start 2023-10-11T16:22:46.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.165+0000 7f0fd2057700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:46.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.165+0000 7f0fd2057700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc07a8a0 con 0x7f0fcc07be70 2023-10-11T16:22:46.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.165+0000 7f0fcb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:46.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.165+0000 7f0fcb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50346/0 (socket says 172.21.15.111:50346) 2023-10-11T16:22:46.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.165+0000 7f0fcb7fe700 1 -- 172.21.15.111:0/3427717193 learned_addr learned my addr 172.21.15.111:0/3427717193 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:46.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.166+0000 7f0fcb7fe700 1 -- 172.21.15.111:0/3427717193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc07a9e0 con 0x7f0fcc07be70 2023-10-11T16:22:46.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.166+0000 7f0fcb7fe700 1 --2- 172.21.15.111:0/3427717193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc07a2d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0098a0 tx=0x7f0fbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b14a84d26190ece server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:46.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.166+0000 7f0fca7fc700 1 -- 172.21.15.111:0/3427717193 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fbc004160 con 0x7f0fcc07be70 2023-10-11T16:22:46.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.166+0000 7f0fca7fc700 1 -- 172.21.15.111:0/3427717193 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc0042c0 con 0x7f0fcc07be70 2023-10-11T16:22:46.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.166+0000 7f0fca7fc700 1 -- 172.21.15.111:0/3427717193 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fbc0045a0 con 0x7f0fcc07be70 2023-10-11T16:22:46.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.167+0000 7f0fd2057700 1 -- 172.21.15.111:0/3427717193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 msgr2=0x7f0fcc07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:46.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.167+0000 7f0fd2057700 1 --2- 172.21.15.111:0/3427717193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc07a2d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0098a0 tx=0x7f0fbc009bb0 comp rx=0 tx=0).stop 2023-10-11T16:22:46.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.167+0000 7f0fd2057700 1 -- 172.21.15.111:0/3427717193 shutdown_connections 2023-10-11T16:22:46.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.167+0000 7f0fd2057700 1 --2- 172.21.15.111:0/3427717193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc07a2d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:46.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.167+0000 7f0fd2057700 1 -- 172.21.15.111:0/3427717193 >> 172.21.15.111:0/3427717193 conn(0x7f0fcc100fd0 msgr2=0x7f0fcc103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:46.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.167+0000 7f0fd2057700 1 -- 172.21.15.111:0/3427717193 shutdown_connections 2023-10-11T16:22:46.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.167+0000 7f0fd2057700 1 -- 172.21.15.111:0/3427717193 wait complete. 2023-10-11T16:22:46.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.168+0000 7f0fd2057700 1 Processor -- start 2023-10-11T16:22:46.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.169+0000 7f0fd2057700 1 -- start start 2023-10-11T16:22:46.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.169+0000 7f0fd2057700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc19b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:46.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.169+0000 7f0fd2057700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc19b6c0 con 0x7f0fcc07be70 2023-10-11T16:22:46.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.169+0000 7f0fcb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc19b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:46.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.169+0000 7f0fcb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc19b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50356/0 (socket says 172.21.15.111:50356) 2023-10-11T16:22:46.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.169+0000 7f0fcb7fe700 1 -- 172.21.15.111:0/1548298432 learned_addr learned my addr 172.21.15.111:0/1548298432 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:46.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.169+0000 7f0fcb7fe700 1 -- 172.21.15.111:0/1548298432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc009550 con 0x7f0fcc07be70 2023-10-11T16:22:46.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.170+0000 7f0fcb7fe700 1 --2- 172.21.15.111:0/1548298432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc19b180 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc07b340 tx=0x7f0fbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:46.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.170+0000 7f0fc8ff9700 1 -- 172.21.15.111:0/1548298432 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fbc011630 con 0x7f0fcc07be70 2023-10-11T16:22:46.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.170+0000 7f0fc8ff9700 1 -- 172.21.15.111:0/1548298432 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc011790 con 0x7f0fcc07be70 2023-10-11T16:22:46.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.170+0000 7f0fc8ff9700 1 -- 172.21.15.111:0/1548298432 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fbc011a30 con 0x7f0fcc07be70 2023-10-11T16:22:46.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.170+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc07aee0 con 0x7f0fcc07be70 2023-10-11T16:22:46.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.170+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc07ad50 con 0x7f0fcc07be70 2023-10-11T16:22:46.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.170+0000 7f0fc8ff9700 1 -- 172.21.15.111:0/1548298432 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0fbc022070 con 0x7f0fcc07be70 2023-10-11T16:22:46.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.170+0000 7f0fc8ff9700 1 -- 172.21.15.111:0/1548298432 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0fbc01c450 con 0x7f0fcc07be70 2023-10-11T16:22:46.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.171+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc194fb0 con 0x7f0fcc07be70 2023-10-11T16:22:46.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.174+0000 7f0fc8ff9700 1 -- 172.21.15.111:0/1548298432 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f0fbc01b450 con 0x7f0fcc07be70 2023-10-11T16:22:46.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.243+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0fcc062120 con 0x7f0fcc07be70 2023-10-11T16:22:46.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.247+0000 7f0fc8ff9700 1 -- 172.21.15.111:0/1548298432 <== mon.0 v2:172.21.15.111:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fbc01b5f0 con 0x7f0fcc07be70 2023-10-11T16:22:46.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.247+0000 7f0fc8ff9700 1 -- 172.21.15.111:0/1548298432 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f0fcc062120 con 0x7f0fcc07be70 2023-10-11T16:22:46.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.248+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 msgr2=0x7f0fcc19b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:46.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.248+0000 7f0fd2057700 1 --2- 172.21.15.111:0/1548298432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc19b180 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc07b340 tx=0x7f0fbc004750 comp rx=0 tx=0).stop 2023-10-11T16:22:46.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.249+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 shutdown_connections 2023-10-11T16:22:46.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.249+0000 7f0fd2057700 1 --2- 172.21.15.111:0/1548298432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fcc07be70 0x7f0fcc19b180 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:46.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.249+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 >> 172.21.15.111:0/1548298432 conn(0x7f0fcc100fd0 msgr2=0x7f0fcc1029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:46.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.249+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 shutdown_connections 2023-10-11T16:22:46.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:46.249+0000 7f0fd2057700 1 -- 172.21.15.111:0/1548298432 wait complete. 2023-10-11T16:22:46.676 INFO:teuthology.orchestra.run.smithi111.stdout:Generating new minimal ceph.conf... 2023-10-11T16:22:46.875 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:46 smithi111 ceph-mon[98023]: from='client.? 172.21.15.111:0/1548298432' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-11T16:22:46.875 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:46 smithi111 ceph-mon[98023]: from='client.? 172.21.15.111:0/1548298432' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-10-11T16:22:47.643 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.131+0000 7f0a36474700 1 Processor -- start 2023-10-11T16:22:47.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.131+0000 7f0a36474700 1 -- start start 2023-10-11T16:22:47.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.131+0000 7f0a36474700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3007a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:47.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.131+0000 7f0a36474700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3007a880 con 0x7f0a3007be50 2023-10-11T16:22:47.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.131+0000 7f0a2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3007a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.131+0000 7f0a2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3007a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50372/0 (socket says 172.21.15.111:50372) 2023-10-11T16:22:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.131+0000 7f0a2ffff700 1 -- 172.21.15.111:0/3609169187 learned_addr learned my addr 172.21.15.111:0/3609169187 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.132+0000 7f0a2ffff700 1 -- 172.21.15.111:0/3609169187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3007a9c0 con 0x7f0a3007be50 2023-10-11T16:22:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.132+0000 7f0a2ffff700 1 --2- 172.21.15.111:0/3609169187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3007a2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0a200098a0 tx=0x7f0a20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d81cbd652df479 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.132+0000 7f0a2effd700 1 -- 172.21.15.111:0/3609169187 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a20004160 con 0x7f0a3007be50 2023-10-11T16:22:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.132+0000 7f0a2effd700 1 -- 172.21.15.111:0/3609169187 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a200042c0 con 0x7f0a3007be50 2023-10-11T16:22:47.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.132+0000 7f0a2effd700 1 -- 172.21.15.111:0/3609169187 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a20011a00 con 0x7f0a3007be50 2023-10-11T16:22:47.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.133+0000 7f0a36474700 1 -- 172.21.15.111:0/3609169187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 msgr2=0x7f0a3007a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:47.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.133+0000 7f0a36474700 1 --2- 172.21.15.111:0/3609169187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3007a2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0a200098a0 tx=0x7f0a20009bb0 comp rx=0 tx=0).stop 2023-10-11T16:22:47.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.133+0000 7f0a36474700 1 -- 172.21.15.111:0/3609169187 shutdown_connections 2023-10-11T16:22:47.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.133+0000 7f0a36474700 1 --2- 172.21.15.111:0/3609169187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3007a2b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:47.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.133+0000 7f0a36474700 1 -- 172.21.15.111:0/3609169187 >> 172.21.15.111:0/3609169187 conn(0x7f0a30100fd0 msgr2=0x7f0a30103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:47.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.133+0000 7f0a36474700 1 -- 172.21.15.111:0/3609169187 shutdown_connections 2023-10-11T16:22:47.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.133+0000 7f0a36474700 1 -- 172.21.15.111:0/3609169187 wait complete. 2023-10-11T16:22:47.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.134+0000 7f0a36474700 1 Processor -- start 2023-10-11T16:22:47.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.134+0000 7f0a36474700 1 -- start start 2023-10-11T16:22:47.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.134+0000 7f0a36474700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3019b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:47.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.134+0000 7f0a36474700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3019bec0 con 0x7f0a3007be50 2023-10-11T16:22:47.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3019b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:47.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3019b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50382/0 (socket says 172.21.15.111:50382) 2023-10-11T16:22:47.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2ffff700 1 -- 172.21.15.111:0/2861000690 learned_addr learned my addr 172.21.15.111:0/2861000690 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:47.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2ffff700 1 -- 172.21.15.111:0/2861000690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20009550 con 0x7f0a3007be50 2023-10-11T16:22:47.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2ffff700 1 --2- 172.21.15.111:0/2861000690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3019b980 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0a20006af0 tx=0x7f0a20011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:47.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2d7fa700 1 -- 172.21.15.111:0/2861000690 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a20003a40 con 0x7f0a3007be50 2023-10-11T16:22:47.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2d7fa700 1 -- 172.21.15.111:0/2861000690 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a20003ba0 con 0x7f0a3007be50 2023-10-11T16:22:47.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a3010bf80 con 0x7f0a3007be50 2023-10-11T16:22:47.650 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2d7fa700 1 -- 172.21.15.111:0/2861000690 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a2001a6e0 con 0x7f0a3007be50 2023-10-11T16:22:47.650 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a3010bdf0 con 0x7f0a3007be50 2023-10-11T16:22:47.650 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2d7fa700 1 -- 172.21.15.111:0/2861000690 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0a2001ab40 con 0x7f0a3007be50 2023-10-11T16:22:47.650 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.136+0000 7f0a2d7fa700 1 -- 172.21.15.111:0/2861000690 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0a20019700 con 0x7f0a3007be50 2023-10-11T16:22:47.651 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.137+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a30062120 con 0x7f0a3007be50 2023-10-11T16:22:47.651 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.140+0000 7f0a2d7fa700 1 -- 172.21.15.111:0/2861000690 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f0a2002de20 con 0x7f0a3007be50 2023-10-11T16:22:47.651 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.209+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0a3019c400 con 0x7f0a3007be50 2023-10-11T16:22:47.651 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.209+0000 7f0a2d7fa700 1 -- 172.21.15.111:0/2861000690 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f0a20024a60 con 0x7f0a3007be50 2023-10-11T16:22:47.651 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.210+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 msgr2=0x7f0a3019b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:47.651 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.210+0000 7f0a36474700 1 --2- 172.21.15.111:0/2861000690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3019b980 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0a20006af0 tx=0x7f0a20011ad0 comp rx=0 tx=0).stop 2023-10-11T16:22:47.652 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.210+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 shutdown_connections 2023-10-11T16:22:47.652 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.210+0000 7f0a36474700 1 --2- 172.21.15.111:0/2861000690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a3007be50 0x7f0a3019b980 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:47.652 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.210+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 >> 172.21.15.111:0/2861000690 conn(0x7f0a30100fd0 msgr2=0x7f0a301026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:47.652 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.210+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 shutdown_connections 2023-10-11T16:22:47.652 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:47.210+0000 7f0a36474700 1 -- 172.21.15.111:0/2861000690 wait complete. 2023-10-11T16:22:47.652 INFO:teuthology.orchestra.run.smithi111.stdout:Restarting the monitor... 2023-10-11T16:22:47.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:47 smithi111 ceph-mon[98023]: from='client.? 172.21.15.111:0/2861000690' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:22:47.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:47 smithi111 systemd[1]: Stopping Ceph mon.a for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:22:48.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:48 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-a[97998]: 2023-10-11T16:22:48.093+0000 7ffb5baf5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:22:48.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:48 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-a[97998]: 2023-10-11T16:22:48.093+0000 7ffb5baf5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-11T16:22:48.830 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:48 smithi111 bash[98362]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-a 2023-10-11T16:22:49.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:48 smithi111 systemd[1]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.a.service: Succeeded. 2023-10-11T16:22:49.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:48 smithi111 systemd[1]: Stopped Ceph mon.a for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:22:49.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:48 smithi111 systemd[1]: Starting Ceph mon.a for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:22:49.534 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 podman[98487]: 2023-10-11T16:22:49.534 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 bash[98487]: e020767c462c53af5fcb07ec681e67e9e29506f5a6c17d23fbed6f610a4f48f8 2023-10-11T16:22:49.534 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: set uid:gid to 167:167 (ceph:ceph) 2023-10-11T16:22:49.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev), process ceph-mon, pid 7 2023-10-11T16:22:49.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: pidfile_write: ignore empty --pid-file 2023-10-11T16:22:49.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: load: jerasure load: lrc 2023-10-11T16:22:49.608 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: RocksDB version: 7.9.2 2023-10-11T16:22:49.609 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Git sha 0 2023-10-11T16:22:49.609 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Compile date 2023-09-28 19:19:12 2023-10-11T16:22:49.609 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: DB SUMMARY 2023-10-11T16:22:49.609 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: DB Session ID: F3C98UGHI7KKT2MQZ31R 2023-10-11T16:22:49.610 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: CURRENT file: CURRENT 2023-10-11T16:22:49.610 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: IDENTITY file: IDENTITY 2023-10-11T16:22:49.610 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-10-11T16:22:49.610 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-10-11T16:22:49.610 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98773 ; 2023-10-11T16:22:49.613 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.error_if_exists: 0 2023-10-11T16:22:49.613 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.create_if_missing: 0 2023-10-11T16:22:49.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.paranoid_checks: 1 2023-10-11T16:22:49.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-11T16:22:49.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-11T16:22:49.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-11T16:22:49.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.env: 0x55bb68148720 2023-10-11T16:22:49.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.fs: PosixFileSystem 2023-10-11T16:22:49.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.info_log: 0x55bb69c8d3c0 2023-10-11T16:22:49.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_file_opening_threads: 16 2023-10-11T16:22:49.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.statistics: (nil) 2023-10-11T16:22:49.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.use_fsync: 0 2023-10-11T16:22:49.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_log_file_size: 0 2023-10-11T16:22:49.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-11T16:22:49.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-11T16:22:49.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.keep_log_file_num: 1000 2023-10-11T16:22:49.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.recycle_log_file_num: 0 2023-10-11T16:22:49.617 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.allow_fallocate: 1 2023-10-11T16:22:49.617 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.allow_mmap_reads: 0 2023-10-11T16:22:49.617 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.allow_mmap_writes: 0 2023-10-11T16:22:49.617 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.use_direct_reads: 0 2023-10-11T16:22:49.617 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-11T16:22:49.618 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.create_missing_column_families: 0 2023-10-11T16:22:49.618 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.db_log_dir: 2023-10-11T16:22:49.618 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.wal_dir: 2023-10-11T16:22:49.618 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-11T16:22:49.618 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-11T16:22:49.619 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-11T16:22:49.619 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-11T16:22:49.619 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-11T16:22:49.619 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-11T16:22:49.619 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.advise_random_on_open: 1 2023-10-11T16:22:49.620 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.db_write_buffer_size: 0 2023-10-11T16:22:49.620 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.write_buffer_manager: 0x55bb68efe3c0 2023-10-11T16:22:49.620 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-11T16:22:49.620 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-11T16:22:49.620 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-11T16:22:49.621 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.rate_limiter: (nil) 2023-10-11T16:22:49.621 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-11T16:22:49.621 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.wal_recovery_mode: 2 2023-10-11T16:22:49.621 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.enable_thread_tracking: 0 2023-10-11T16:22:49.621 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.enable_pipelined_write: 0 2023-10-11T16:22:49.622 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.unordered_write: 0 2023-10-11T16:22:49.622 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-11T16:22:49.622 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-11T16:22:49.622 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-11T16:22:49.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-11T16:22:49.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.row_cache: None 2023-10-11T16:22:49.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.wal_filter: None 2023-10-11T16:22:49.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-11T16:22:49.623 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.allow_ingest_behind: 0 2023-10-11T16:22:49.624 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.two_write_queues: 0 2023-10-11T16:22:49.624 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.manual_wal_flush: 0 2023-10-11T16:22:49.624 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.wal_compression: 0 2023-10-11T16:22:49.624 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.atomic_flush: 0 2023-10-11T16:22:49.624 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-11T16:22:49.625 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-11T16:22:49.625 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-11T16:22:49.625 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.log_readahead_size: 0 2023-10-11T16:22:49.625 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-11T16:22:49.625 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.best_efforts_recovery: 0 2023-10-11T16:22:49.626 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-11T16:22:49.626 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-11T16:22:49.626 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.allow_data_in_errors: 0 2023-10-11T16:22:49.626 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.db_host_id: __hostname__ 2023-10-11T16:22:49.626 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-11T16:22:49.627 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_background_jobs: 2 2023-10-11T16:22:49.627 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_background_compactions: -1 2023-10-11T16:22:49.627 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_subcompactions: 1 2023-10-11T16:22:49.627 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-11T16:22:49.627 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-11T16:22:49.628 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-11T16:22:49.628 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_total_wal_size: 0 2023-10-11T16:22:49.628 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-11T16:22:49.628 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-11T16:22:49.628 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-11T16:22:49.629 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-11T16:22:49.629 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_open_files: -1 2023-10-11T16:22:49.629 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bytes_per_sync: 0 2023-10-11T16:22:49.629 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-11T16:22:49.629 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-11T16:22:49.630 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_readahead_size: 0 2023-10-11T16:22:49.630 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_background_flushes: -1 2023-10-11T16:22:49.630 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Compression algorithms supported: 2023-10-11T16:22:49.630 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-11T16:22:49.630 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: kZSTD supported: 0 2023-10-11T16:22:49.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: kXpressCompression supported: 0 2023-10-11T16:22:49.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: kLZ4HCCompression supported: 1 2023-10-11T16:22:49.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: kZlibCompression supported: 1 2023-10-11T16:22:49.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: kSnappyCompression supported: 1 2023-10-11T16:22:49.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: kLZ4Compression supported: 1 2023-10-11T16:22:49.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: kBZip2Compression supported: 0 2023-10-11T16:22:49.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-11T16:22:49.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-11T16:22:49.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-10-11T16:22:49.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-11T16:22:49.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-11T16:22:49.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.merge_operator: 2023-10-11T16:22:49.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_filter: None 2023-10-11T16:22:49.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_filter_factory: None 2023-10-11T16:22:49.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.sst_partitioner_factory: None 2023-10-11T16:22:49.634 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-11T16:22:49.634 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-11T16:22:49.634 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bb69c8d4e0) 2023-10-11T16:22:49.634 INFO:journalctl@ceph.mon.a.smithi111.stdout: cache_index_and_filter_blocks: 1 2023-10-11T16:22:49.634 INFO:journalctl@ceph.mon.a.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-11T16:22:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-11T16:22:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout: pin_top_level_index_and_filter: 1 2023-10-11T16:22:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout: index_type: 0 2023-10-11T16:22:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout: data_block_index_type: 0 2023-10-11T16:22:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout: index_shortening: 1 2023-10-11T16:22:49.636 INFO:journalctl@ceph.mon.a.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-11T16:22:49.636 INFO:journalctl@ceph.mon.a.smithi111.stdout: checksum: 4 2023-10-11T16:22:49.636 INFO:journalctl@ceph.mon.a.smithi111.stdout: no_block_cache: 0 2023-10-11T16:22:49.636 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_cache: 0x55bb68f9f1f0 2023-10-11T16:22:49.636 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_cache_name: BinnedLRUCache 2023-10-11T16:22:49.637 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_cache_options: 2023-10-11T16:22:49.637 INFO:journalctl@ceph.mon.a.smithi111.stdout: capacity : 536870912 2023-10-11T16:22:49.637 INFO:journalctl@ceph.mon.a.smithi111.stdout: num_shard_bits : 4 2023-10-11T16:22:49.637 INFO:journalctl@ceph.mon.a.smithi111.stdout: strict_capacity_limit : 0 2023-10-11T16:22:49.637 INFO:journalctl@ceph.mon.a.smithi111.stdout: high_pri_pool_ratio: 0.000 2023-10-11T16:22:49.637 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_cache_compressed: (nil) 2023-10-11T16:22:49.638 INFO:journalctl@ceph.mon.a.smithi111.stdout: persistent_cache: (nil) 2023-10-11T16:22:49.638 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_size: 4096 2023-10-11T16:22:49.638 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_size_deviation: 10 2023-10-11T16:22:49.638 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_restart_interval: 16 2023-10-11T16:22:49.638 INFO:journalctl@ceph.mon.a.smithi111.stdout: index_block_restart_interval: 1 2023-10-11T16:22:49.639 INFO:journalctl@ceph.mon.a.smithi111.stdout: metadata_block_size: 4096 2023-10-11T16:22:49.639 INFO:journalctl@ceph.mon.a.smithi111.stdout: partition_filters: 0 2023-10-11T16:22:49.639 INFO:journalctl@ceph.mon.a.smithi111.stdout: use_delta_encoding: 1 2023-10-11T16:22:49.639 INFO:journalctl@ceph.mon.a.smithi111.stdout: filter_policy: bloomfilter 2023-10-11T16:22:49.639 INFO:journalctl@ceph.mon.a.smithi111.stdout: whole_key_filtering: 1 2023-10-11T16:22:49.640 INFO:journalctl@ceph.mon.a.smithi111.stdout: verify_compression: 0 2023-10-11T16:22:49.640 INFO:journalctl@ceph.mon.a.smithi111.stdout: read_amp_bytes_per_bit: 0 2023-10-11T16:22:49.640 INFO:journalctl@ceph.mon.a.smithi111.stdout: format_version: 5 2023-10-11T16:22:49.640 INFO:journalctl@ceph.mon.a.smithi111.stdout: enable_index_compression: 1 2023-10-11T16:22:49.640 INFO:journalctl@ceph.mon.a.smithi111.stdout: block_align: 0 2023-10-11T16:22:49.641 INFO:journalctl@ceph.mon.a.smithi111.stdout: max_auto_readahead_size: 262144 2023-10-11T16:22:49.641 INFO:journalctl@ceph.mon.a.smithi111.stdout: prepopulate_block_cache: 0 2023-10-11T16:22:49.641 INFO:journalctl@ceph.mon.a.smithi111.stdout: initial_auto_readahead_size: 8192 2023-10-11T16:22:49.641 INFO:journalctl@ceph.mon.a.smithi111.stdout: num_file_reads_for_auto_readahead: 2 2023-10-11T16:22:49.641 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.write_buffer_size: 33554432 2023-10-11T16:22:49.642 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_write_buffer_number: 2 2023-10-11T16:22:49.642 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression: NoCompression 2023-10-11T16:22:49.642 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression: Disabled 2023-10-11T16:22:49.642 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.prefix_extractor: nullptr 2023-10-11T16:22:49.642 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-11T16:22:49.643 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.num_levels: 7 2023-10-11T16:22:49.643 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-11T16:22:49.643 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-11T16:22:49.643 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-11T16:22:49.643 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-11T16:22:49.644 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-11T16:22:49.644 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-11T16:22:49.644 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-11T16:22:49.644 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-11T16:22:49.644 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-11T16:22:49.645 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-11T16:22:49.646 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-11T16:22:49.646 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-11T16:22:49.646 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-11T16:22:49.646 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.level: 32767 2023-10-11T16:22:49.647 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.strategy: 0 2023-10-11T16:22:49.647 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-11T16:22:49.647 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-11T16:22:49.647 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-11T16:22:49.647 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-11T16:22:49.648 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.enabled: false 2023-10-11T16:22:49.648 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-11T16:22:49.648 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-11T16:22:49.648 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-11T16:22:49.648 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-11T16:22:49.649 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.target_file_size_base: 67108864 2023-10-11T16:22:49.649 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-11T16:22:49.649 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-11T16:22:49.649 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-11T16:22:49.649 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-11T16:22:49.650 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-11T16:22:49.650 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-11T16:22:49.650 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-11T16:22:49.650 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-11T16:22:49.650 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-11T16:22:49.651 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-11T16:22:49.651 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-11T16:22:49.651 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-11T16:22:49.651 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-11T16:22:49.651 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-11T16:22:49.652 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.arena_block_size: 1048576 2023-10-11T16:22:49.652 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-11T16:22:49.652 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-11T16:22:49.652 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.disable_auto_compactions: 0 2023-10-11T16:22:49.652 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-11T16:22:49.653 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-11T16:22:49.653 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-11T16:22:49.653 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-11T16:22:49.653 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-11T16:22:49.653 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-11T16:22:49.654 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-11T16:22:49.654 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-11T16:22:49.654 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-11T16:22:49.654 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-11T16:22:49.654 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.table_properties_collectors: 2023-10-11T16:22:49.655 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.inplace_update_support: 0 2023-10-11T16:22:49.655 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-11T16:22:49.655 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-11T16:22:49.655 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-11T16:22:49.655 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-11T16:22:49.656 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.bloom_locality: 0 2023-10-11T16:22:49.656 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.max_successive_merges: 0 2023-10-11T16:22:49.656 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-11T16:22:49.656 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.paranoid_file_checks: 0 2023-10-11T16:22:49.656 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.force_consistency_checks: 1 2023-10-11T16:22:49.657 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.report_bg_io_stats: 0 2023-10-11T16:22:49.657 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.ttl: 2592000 2023-10-11T16:22:49.657 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-11T16:22:49.657 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-11T16:22:49.657 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-11T16:22:49.658 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.enable_blob_files: false 2023-10-11T16:22:49.658 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.min_blob_size: 0 2023-10-11T16:22:49.658 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.blob_file_size: 268435456 2023-10-11T16:22:49.658 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-11T16:22:49.658 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-11T16:22:49.659 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-11T16:22:49.659 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-11T16:22:49.659 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-11T16:22:49.659 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.blob_file_starting_level: 0 2023-10-11T16:22:49.659 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-11T16:22:49.660 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-10-11T16:22:49.660 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-10-11T16:22:49.660 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 818f084f-4cf8-4b4c-85cd-3501c052848f 2023-10-11T16:22:49.660 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041369541902, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-10-11T16:22:49.660 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-10-11T16:22:49.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041369543600, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94026, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92025, "index_size": 283, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13658, "raw_average_key_size": 50, "raw_value_size": 84566, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1697041369, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "818f084f-4cf8-4b4c-85cd-3501c052848f", "db_session_id": "F3C98UGHI7KKT2MQZ31R", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-10-11T16:22:49.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041369543728, "job": 1, "event": "recovery_finished"} 2023-10-11T16:22:49.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-10-11T16:22:49.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-11T16:22:49.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bb6903c000 2023-10-11T16:22:49.662 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: DB pointer 0x55bb69028000 2023-10-11T16:22:49.662 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-11T16:22:49.662 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-11T16:22:49.662 INFO:journalctl@ceph.mon.a.smithi111.stdout: ** DB Stats ** 2023-10-11T16:22:49.663 INFO:journalctl@ceph.mon.a.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-11T16:22:49.663 INFO:journalctl@ceph.mon.a.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-11T16:22:49.663 INFO:journalctl@ceph.mon.a.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-11T16:22:49.663 INFO:journalctl@ceph.mon.a.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-11T16:22:49.663 INFO:journalctl@ceph.mon.a.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-11T16:22:49.664 INFO:journalctl@ceph.mon.a.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-11T16:22:49.664 INFO:journalctl@ceph.mon.a.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-11T16:22:49.664 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2023-10-11T16:22:49.664 INFO:journalctl@ceph.mon.a.smithi111.stdout: ** Compaction Stats [default] ** 2023-10-11T16:22:49.664 INFO:journalctl@ceph.mon.a.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-11T16:22:49.665 INFO:journalctl@ceph.mon.a.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-11T16:22:49.665 INFO:journalctl@ceph.mon.a.smithi111.stdout: L0 2/0 93.66 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-11T16:22:49.665 INFO:journalctl@ceph.mon.a.smithi111.stdout: Sum 2/0 93.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-11T16:22:49.665 INFO:journalctl@ceph.mon.a.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-11T16:22:49.665 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2023-10-11T16:22:49.666 INFO:journalctl@ceph.mon.a.smithi111.stdout: ** Compaction Stats [default] ** 2023-10-11T16:22:49.666 INFO:journalctl@ceph.mon.a.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-11T16:22:49.666 INFO:journalctl@ceph.mon.a.smithi111.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-11T16:22:49.666 INFO:journalctl@ceph.mon.a.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 85.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-11T16:22:49.666 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2023-10-11T16:22:49.667 INFO:journalctl@ceph.mon.a.smithi111.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-11T16:22:49.667 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2023-10-11T16:22:49.667 INFO:journalctl@ceph.mon.a.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-11T16:22:49.667 INFO:journalctl@ceph.mon.a.smithi111.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-11T16:22:49.667 INFO:journalctl@ceph.mon.a.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-11T16:22:49.668 INFO:journalctl@ceph.mon.a.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-11T16:22:49.668 INFO:journalctl@ceph.mon.a.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-11T16:22:49.668 INFO:journalctl@ceph.mon.a.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-11T16:22:49.668 INFO:journalctl@ceph.mon.a.smithi111.stdout: Cumulative compaction: 0.00 GB write, 15.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-11T16:22:49.668 INFO:journalctl@ceph.mon.a.smithi111.stdout: Interval compaction: 0.00 GB write, 15.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-11T16:22:49.669 INFO:journalctl@ceph.mon.a.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-11T16:22:49.669 INFO:journalctl@ceph.mon.a.smithi111.stdout: Block cache BinnedLRUCache@0x55bb68f9f1f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-10-11T16:22:49.669 INFO:journalctl@ceph.mon.a.smithi111.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-10-11T16:22:49.669 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2023-10-11T16:22:49.669 INFO:journalctl@ceph.mon.a.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-11T16:22:49.670 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: starting mon.a rank 0 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:22:49.670 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a@-1(???) e1 preinit fsid 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:22:49.670 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a@-1(???).mds e1 new map 2023-10-11T16:22:49.670 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a@-1(???).mds e1 print_map 2023-10-11T16:22:49.670 INFO:journalctl@ceph.mon.a.smithi111.stdout: e1 2023-10-11T16:22:49.671 INFO:journalctl@ceph.mon.a.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-11T16:22:49.671 INFO:journalctl@ceph.mon.a.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-11T16:22:49.671 INFO:journalctl@ceph.mon.a.smithi111.stdout: legacy client fscid: -1 2023-10-11T16:22:49.671 INFO:journalctl@ceph.mon.a.smithi111.stdout: 2023-10-11T16:22:49.671 INFO:journalctl@ceph.mon.a.smithi111.stdout: No filesystems configured 2023-10-11T16:22:49.672 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-11T16:22:49.672 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:22:49.672 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:22:49.672 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:22:49.672 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-10-11T16:22:49.673 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta expand map: {default=false} 2023-10-11T16:22:49.673 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta from 'false' to 'false' 2023-10-11T16:22:49.673 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta expanded map: {default=false} 2023-10-11T16:22:49.673 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta expand map: {default=info} 2023-10-11T16:22:49.673 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta from 'info' to 'info' 2023-10-11T16:22:49.674 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta expanded map: {default=info} 2023-10-11T16:22:49.674 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta expand map: {default=daemon} 2023-10-11T16:22:49.674 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-11T16:22:49.674 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta expanded map: {default=daemon} 2023-10-11T16:22:49.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta expand map: {default=debug} 2023-10-11T16:22:49.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta from 'debug' to 'debug' 2023-10-11T16:22:49.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: expand_channel_meta expanded map: {default=debug} 2023-10-11T16:22:49.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-11T16:22:49.675 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: monmap e1: 1 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} 2023-10-11T16:22:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: fsmap 2023-10-11T16:22:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: osdmap e1: 0 total, 0 up, 0 in 2023-10-11T16:22:49.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 ceph-mon[98527]: mgrmap e1: no daemons active 2023-10-11T16:22:49.676 INFO:teuthology.orchestra.run.smithi111.stdout:Setting mon public_network to 172.21.0.0/20 2023-10-11T16:22:49.856 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:49 smithi111 systemd[1]: Started Ceph mon.a for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:22:50.602 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.103+0000 7fea843dd700 1 Processor -- start 2023-10-11T16:22:50.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.104+0000 7fea843dd700 1 -- start start 2023-10-11T16:22:50.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.104+0000 7fea843dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:50.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.104+0000 7fea843dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7c07a860 con 0x7fea7c07be30 2023-10-11T16:22:50.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.104+0000 7fea82179700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:50.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.105+0000 7fea82179700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50384/0 (socket says 172.21.15.111:50384) 2023-10-11T16:22:50.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.105+0000 7fea82179700 1 -- 172.21.15.111:0/2039219558 learned_addr learned my addr 172.21.15.111:0/2039219558 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:50.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.105+0000 7fea82179700 1 -- 172.21.15.111:0/2039219558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c07a9a0 con 0x7fea7c07be30 2023-10-11T16:22:50.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.105+0000 7fea82179700 1 --2- 172.21.15.111:0/2039219558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c07a290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fea780098a0 tx=0x7fea78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288f2f56fe9de10d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:50.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.105+0000 7fea81177700 1 -- 172.21.15.111:0/2039219558 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea78004160 con 0x7fea7c07be30 2023-10-11T16:22:50.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.105+0000 7fea81177700 1 -- 172.21.15.111:0/2039219558 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea780042c0 con 0x7fea7c07be30 2023-10-11T16:22:50.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.106+0000 7fea81177700 1 -- 172.21.15.111:0/2039219558 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea78011a00 con 0x7fea7c07be30 2023-10-11T16:22:50.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.106+0000 7fea843dd700 1 -- 172.21.15.111:0/2039219558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 msgr2=0x7fea7c07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:50.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.106+0000 7fea843dd700 1 --2- 172.21.15.111:0/2039219558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c07a290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fea780098a0 tx=0x7fea78009bb0 comp rx=0 tx=0).stop 2023-10-11T16:22:50.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.106+0000 7fea843dd700 1 -- 172.21.15.111:0/2039219558 shutdown_connections 2023-10-11T16:22:50.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.106+0000 7fea843dd700 1 --2- 172.21.15.111:0/2039219558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c07a290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:50.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.106+0000 7fea843dd700 1 -- 172.21.15.111:0/2039219558 >> 172.21.15.111:0/2039219558 conn(0x7fea7c100fd0 msgr2=0x7fea7c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:50.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.106+0000 7fea843dd700 1 -- 172.21.15.111:0/2039219558 shutdown_connections 2023-10-11T16:22:50.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.106+0000 7fea843dd700 1 -- 172.21.15.111:0/2039219558 wait complete. 2023-10-11T16:22:50.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.107+0000 7fea843dd700 1 Processor -- start 2023-10-11T16:22:50.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.107+0000 7fea843dd700 1 -- start start 2023-10-11T16:22:50.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.108+0000 7fea843dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:50.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.108+0000 7fea843dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7c19be70 con 0x7fea7c07be30 2023-10-11T16:22:50.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.108+0000 7fea82179700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:50.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.108+0000 7fea82179700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c19b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50388/0 (socket says 172.21.15.111:50388) 2023-10-11T16:22:50.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.108+0000 7fea82179700 1 -- 172.21.15.111:0/3945667719 learned_addr learned my addr 172.21.15.111:0/3945667719 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:50.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.108+0000 7fea82179700 1 -- 172.21.15.111:0/3945667719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea78009550 con 0x7fea7c07be30 2023-10-11T16:22:50.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.108+0000 7fea82179700 1 --2- 172.21.15.111:0/3945667719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c19b930 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fea78006af0 tx=0x7fea78011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:50.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.109+0000 7fea737fe700 1 -- 172.21.15.111:0/3945667719 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea78003a40 con 0x7fea7c07be30 2023-10-11T16:22:50.609 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.109+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7c10bf40 con 0x7fea7c07be30 2023-10-11T16:22:50.609 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.109+0000 7fea737fe700 1 -- 172.21.15.111:0/3945667719 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea78003ba0 con 0x7fea7c07be30 2023-10-11T16:22:50.609 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.109+0000 7fea737fe700 1 -- 172.21.15.111:0/3945667719 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea7801a6e0 con 0x7fea7c07be30 2023-10-11T16:22:50.609 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.109+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7c10bdb0 con 0x7fea7c07be30 2023-10-11T16:22:50.609 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.109+0000 7fea737fe700 1 -- 172.21.15.111:0/3945667719 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fea7801ac50 con 0x7fea7c07be30 2023-10-11T16:22:50.610 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.109+0000 7fea737fe700 1 -- 172.21.15.111:0/3945667719 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fea78019700 con 0x7fea7c07be30 2023-10-11T16:22:50.610 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.110+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7c062120 con 0x7fea7c07be30 2023-10-11T16:22:50.610 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.113+0000 7fea737fe700 1 -- 172.21.15.111:0/3945667719 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fea7802de30 con 0x7fea7c07be30 2023-10-11T16:22:50.610 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.182+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fea7c19c3b0 con 0x7fea7c07be30 2023-10-11T16:22:50.610 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.185+0000 7fea737fe700 1 -- 172.21.15.111:0/3945667719 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fea78024a60 con 0x7fea7c07be30 2023-10-11T16:22:50.611 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.186+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 msgr2=0x7fea7c19b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:50.611 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.186+0000 7fea843dd700 1 --2- 172.21.15.111:0/3945667719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c19b930 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fea78006af0 tx=0x7fea78011ad0 comp rx=0 tx=0).stop 2023-10-11T16:22:50.611 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.186+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 shutdown_connections 2023-10-11T16:22:50.611 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.186+0000 7fea843dd700 1 --2- 172.21.15.111:0/3945667719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea7c07be30 0x7fea7c19b930 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:50.611 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.186+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 >> 172.21.15.111:0/3945667719 conn(0x7fea7c100fd0 msgr2=0x7fea7c1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:50.612 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.186+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 shutdown_connections 2023-10-11T16:22:50.612 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:50.186+0000 7fea843dd700 1 -- 172.21.15.111:0/3945667719 wait complete. 2023-10-11T16:22:50.612 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote config to /etc/ceph/ceph.conf 2023-10-11T16:22:50.612 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:22:50.612 INFO:teuthology.orchestra.run.smithi111.stdout:Creating mgr... 2023-10-11T16:22:50.613 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying port 9283 ... 2023-10-11T16:22:50.613 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying port 8765 ... 2023-10-11T16:22:50.916 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.x 2023-10-11T16:22:50.916 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to reset failed state of unit ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.x.service: Unit ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.x.service not loaded. 2023-10-11T16:22:51.118 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-495f265e-6852-11ee-8db6-212e2dc638e7.target.wants/ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.x.service → /etc/systemd/system/ceph-495f265e-6852-11ee-8db6-212e2dc638e7@.service. 2023-10-11T16:22:51.185 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:51 smithi111 systemd[1]: Starting Ceph mgr.x for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:22:51.605 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:51 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3945667719' entity='client.admin' 2023-10-11T16:22:51.856 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:51 smithi111 podman[98761]: 2023-10-11T16:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-11T16:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2023-10-11T16:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-11T16:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2023-10-11T16:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2023-10-11T16:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to enable service . firewalld.service is not available 2023-10-11T16:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-11T16:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2023-10-11T16:22:51.972 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-11T16:22:51.972 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2023-10-11T16:22:51.973 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2023-10-11T16:22:51.973 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-10-11T16:22:51.973 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr to start... 2023-10-11T16:22:51.973 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr... 2023-10-11T16:22:52.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:51 smithi111 bash[98761]: 9e6dab0c08cc91cc571a97b9856d57328d0671120eb51e43b7b57a2e4603be4b 2023-10-11T16:22:52.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:51 smithi111 systemd[1]: Started Ceph mgr.x for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:22:52.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:52.042+0000 7fea891f1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T16:22:52.629 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:52.148+0000 7fea891f1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T16:22:52.629 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:52.267+0000 7fea891f1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T16:22:52.629 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:52.397+0000 7fea891f1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T16:22:52.630 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:52.628+0000 7fea891f1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T16:22:52.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:22:52 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2744541071' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-11T16:22:53.057 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:22:53.060 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2023-10-11T16:22:53.060 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:22:53.060 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "health": { 2023-10-11T16:22:53.061 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-11T16:22:53.061 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-11T16:22:53.061 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-11T16:22:53.061 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:22:53.061 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-11T16:22:53.061 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-11T16:22:53.062 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 0 2023-10-11T16:22:53.062 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2023-10-11T16:22:53.062 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-11T16:22:53.062 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "a" 2023-10-11T16:22:53.062 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2023-10-11T16:22:53.063 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2023-10-11T16:22:53.063 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-11T16:22:53.063 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-11T16:22:53.063 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-11T16:22:53.063 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-11T16:22:53.064 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:22:53.064 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-11T16:22:53.064 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-11T16:22:53.064 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-11T16:22:53.064 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-11T16:22:53.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-11T16:22:53.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-11T16:22:53.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-11T16:22:53.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-11T16:22:53.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:22:53.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-11T16:22:53.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-11T16:22:53.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-11T16:22:53.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-11T16:22:53.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-11T16:22:53.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-11T16:22:53.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-11T16:22:53.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-11T16:22:53.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-11T16:22:53.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:22:53.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-11T16:22:53.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-11T16:22:53.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-11T16:22:53.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-11T16:22:53.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:22:53.076 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-11T16:22:53.076 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-11T16:22:53.076 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-11T16:22:53.076 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-11T16:22:53.076 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-11T16:22:53.077 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-11T16:22:53.077 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "restful" 2023-10-11T16:22:53.077 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2023-10-11T16:22:53.077 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-11T16:22:53.077 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:22:53.078 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-11T16:22:53.078 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-11T16:22:53.078 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modified": "2023-10-11T16:22:44.513795+0000", 2023-10-11T16:22:53.078 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-11T16:22:53.078 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:22:53.079 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-11T16:22:53.079 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2023-10-11T16:22:53.079 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.535+0000 7fed85594700 1 Processor -- start 2023-10-11T16:22:53.079 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.535+0000 7fed85594700 1 -- start start 2023-10-11T16:22:53.079 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.535+0000 7fed85594700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8007a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:53.079 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.535+0000 7fed85594700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8007a840 con 0x7fed8007be10 2023-10-11T16:22:53.080 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.536+0000 7fed7effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8007a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:53.080 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.536+0000 7fed7effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8007a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50402/0 (socket says 172.21.15.111:50402) 2023-10-11T16:22:53.080 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.536+0000 7fed7effd700 1 -- 172.21.15.111:0/1732784846 learned_addr learned my addr 172.21.15.111:0/1732784846 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:53.080 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.536+0000 7fed7effd700 1 -- 172.21.15.111:0/1732784846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8007a980 con 0x7fed8007be10 2023-10-11T16:22:53.080 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.536+0000 7fed7effd700 1 --2- 172.21.15.111:0/1732784846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8007a270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fed700098a0 tx=0x7fed70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4778cce0f9c3abf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:53.081 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.536+0000 7fed7dffb700 1 -- 172.21.15.111:0/1732784846 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed70004160 con 0x7fed8007be10 2023-10-11T16:22:53.081 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.537+0000 7fed7dffb700 1 -- 172.21.15.111:0/1732784846 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed700042c0 con 0x7fed8007be10 2023-10-11T16:22:53.081 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.537+0000 7fed7dffb700 1 -- 172.21.15.111:0/1732784846 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed70011a00 con 0x7fed8007be10 2023-10-11T16:22:53.081 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.537+0000 7fed85594700 1 -- 172.21.15.111:0/1732784846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 msgr2=0x7fed8007a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:53.081 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.537+0000 7fed85594700 1 --2- 172.21.15.111:0/1732784846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8007a270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fed700098a0 tx=0x7fed70009bb0 comp rx=0 tx=0).stop 2023-10-11T16:22:53.082 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.537+0000 7fed85594700 1 -- 172.21.15.111:0/1732784846 shutdown_connections 2023-10-11T16:22:53.082 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.537+0000 7fed85594700 1 --2- 172.21.15.111:0/1732784846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8007a270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:53.082 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.537+0000 7fed85594700 1 -- 172.21.15.111:0/1732784846 >> 172.21.15.111:0/1732784846 conn(0x7fed80100fd0 msgr2=0x7fed80103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:53.082 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.537+0000 7fed85594700 1 -- 172.21.15.111:0/1732784846 shutdown_connections 2023-10-11T16:22:53.083 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.538+0000 7fed85594700 1 -- 172.21.15.111:0/1732784846 wait complete. 2023-10-11T16:22:53.083 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.539+0000 7fed85594700 1 Processor -- start 2023-10-11T16:22:53.083 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.539+0000 7fed85594700 1 -- start start 2023-10-11T16:22:53.083 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.540+0000 7fed85594700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:22:53.083 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.540+0000 7fed85594700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8019be90 con 0x7fed8007be10 2023-10-11T16:22:53.083 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.540+0000 7fed7effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:22:53.084 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.540+0000 7fed7effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50412/0 (socket says 172.21.15.111:50412) 2023-10-11T16:22:53.084 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.540+0000 7fed7effd700 1 -- 172.21.15.111:0/2744541071 learned_addr learned my addr 172.21.15.111:0/2744541071 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:22:53.084 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.540+0000 7fed7effd700 1 -- 172.21.15.111:0/2744541071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70009550 con 0x7fed8007be10 2023-10-11T16:22:53.084 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.540+0000 7fed7effd700 1 --2- 172.21.15.111:0/2744541071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8019b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fed70006af0 tx=0x7fed70011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:22:53.084 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.541+0000 7fed67fff700 1 -- 172.21.15.111:0/2744541071 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed70003a40 con 0x7fed8007be10 2023-10-11T16:22:53.085 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.541+0000 7fed67fff700 1 -- 172.21.15.111:0/2744541071 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed70003ba0 con 0x7fed8007be10 2023-10-11T16:22:53.085 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.541+0000 7fed67fff700 1 -- 172.21.15.111:0/2744541071 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed7001a6e0 con 0x7fed8007be10 2023-10-11T16:22:53.085 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.541+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8010bf60 con 0x7fed8007be10 2023-10-11T16:22:53.085 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.541+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8010bdd0 con 0x7fed8007be10 2023-10-11T16:22:53.085 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.541+0000 7fed67fff700 1 -- 172.21.15.111:0/2744541071 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fed7001ab60 con 0x7fed8007be10 2023-10-11T16:22:53.086 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.542+0000 7fed67fff700 1 -- 172.21.15.111:0/2744541071 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fed70019700 con 0x7fed8007be10 2023-10-11T16:22:53.086 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.542+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed80062120 con 0x7fed8007be10 2023-10-11T16:22:53.086 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.545+0000 7fed67fff700 1 -- 172.21.15.111:0/2744541071 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fed7002dde0 con 0x7fed8007be10 2023-10-11T16:22:53.086 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.620+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fed8019c3d0 con 0x7fed8007be10 2023-10-11T16:22:53.086 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.620+0000 7fed67fff700 1 -- 172.21.15.111:0/2744541071 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fed70024a60 con 0x7fed8007be10 2023-10-11T16:22:53.087 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.621+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 msgr2=0x7fed8019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:22:53.087 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.621+0000 7fed85594700 1 --2- 172.21.15.111:0/2744541071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8019b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fed70006af0 tx=0x7fed70011ad0 comp rx=0 tx=0).stop 2023-10-11T16:22:53.087 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.622+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 shutdown_connections 2023-10-11T16:22:53.087 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.622+0000 7fed85594700 1 --2- 172.21.15.111:0/2744541071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed8007be10 0x7fed8019b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:22:53.087 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.622+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 >> 172.21.15.111:0/2744541071 conn(0x7fed80100fd0 msgr2=0x7fed801026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:22:53.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.622+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 shutdown_connections 2023-10-11T16:22:53.088 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:22:52.622+0000 7fed85594700 1 -- 172.21.15.111:0/2744541071 wait complete. 2023-10-11T16:22:53.088 INFO:teuthology.orchestra.run.smithi111.stdout:mgr not available, waiting (1/15)... 2023-10-11T16:22:53.325 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:53.045+0000 7fea891f1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T16:22:53.325 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:53.163+0000 7fea891f1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T16:22:53.325 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:53.324+0000 7fea891f1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T16:22:53.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:53.447+0000 7fea891f1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T16:22:54.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:53.986+0000 7fea891f1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T16:22:55.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:54 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:54.769+0000 7fea891f1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T16:22:55.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:54 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:54.982+0000 7fea891f1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T16:22:56.307 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:55.967+0000 7fea891f1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:22:56.308 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:56.085+0000 7fea891f1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T16:22:56.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:56.306+0000 7fea891f1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:22:56.950 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:56.668+0000 7fea891f1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T16:22:57.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:56.950+0000 7fea891f1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T16:22:57.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:57 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:57.143+0000 7fea891f1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T16:22:57.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:57 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:57.494+0000 7fea891f1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T16:22:58.765 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:58.425+0000 7fea891f1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T16:22:59.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:58.597+0000 7fea891f1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T16:22:59.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:58.839+0000 7fea891f1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T16:22:59.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:58.986+0000 7fea891f1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T16:22:59.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:22:59 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:22:59.260+0000 7fea891f1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T16:23:00.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:00.329+0000 7fea891f1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T16:23:00.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: Activating manager daemon x 2023-10-11T16:23:00.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: mgrmap e2: x(active, starting, since 0.00447978s) 2023-10-11T16:23:00.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-11T16:23:00.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-11T16:23:00.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-11T16:23:00.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:23:00.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-11T16:23:00.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: Manager daemon x is now available 2023-10-11T16:23:00.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-11T16:23:00.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-11T16:23:00.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' 2023-10-11T16:23:00.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' 2023-10-11T16:23:00.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:00 smithi111 ceph-mon[98527]: from='mgr.14100 172.21.15.111:0/143159979' entity='mgr.x' 2023-10-11T16:23:02.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:02 smithi111 ceph-mon[98527]: mgrmap e3: x(active, since 1.00819s) 2023-10-11T16:23:03.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:23:03.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2023-10-11T16:23:03.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:23:03.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "health": { 2023-10-11T16:23:03.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-11T16:23:03.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-11T16:23:03.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-11T16:23:03.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:23:03.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-11T16:23:03.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-11T16:23:03.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 0 2023-10-11T16:23:03.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2023-10-11T16:23:03.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-11T16:23:03.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "a" 2023-10-11T16:23:03.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2023-10-11T16:23:03.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2023-10-11T16:23:03.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-11T16:23:03.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-11T16:23:03.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-11T16:23:03.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-11T16:23:03.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:23:03.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-11T16:23:03.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-11T16:23:03.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-11T16:23:03.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-11T16:23:03.494 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-11T16:23:03.503 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-11T16:23:03.503 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-11T16:23:03.503 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-11T16:23:03.503 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:23:03.504 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-11T16:23:03.504 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-11T16:23:03.504 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-11T16:23:03.504 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-11T16:23:03.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-11T16:23:03.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-11T16:23:03.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-11T16:23:03.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-11T16:23:03.505 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-11T16:23:03.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:23:03.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-11T16:23:03.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-11T16:23:03.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-11T16:23:03.506 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-11T16:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-11T16:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-11T16:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-11T16:23:03.507 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-11T16:23:03.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-11T16:23:03.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-11T16:23:03.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "restful" 2023-10-11T16:23:03.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2023-10-11T16:23:03.508 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-11T16:23:03.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:23:03.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-11T16:23:03.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-11T16:23:03.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modified": "2023-10-11T16:22:44.513795+0000", 2023-10-11T16:23:03.509 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-11T16:23:03.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2023-10-11T16:23:03.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-11T16:23:03.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2023-10-11T16:23:03.510 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.777+0000 7f3b0ab93700 1 Processor -- start 2023-10-11T16:23:03.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.777+0000 7f3b0ab93700 1 -- start start 2023-10-11T16:23:03.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.777+0000 7f3b0ab93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b04105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:03.511 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.777+0000 7f3b0ab93700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04106400 con 0x7f3b04105a20 2023-10-11T16:23:03.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.778+0000 7f3b0892f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b04105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:03.512 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.778+0000 7f3b0892f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b04105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44826/0 (socket says 172.21.15.111:44826) 2023-10-11T16:23:03.513 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.778+0000 7f3b0892f700 1 -- 172.21.15.111:0/2048925824 learned_addr learned my addr 172.21.15.111:0/2048925824 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:03.513 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.778+0000 7f3b0892f700 1 -- 172.21.15.111:0/2048925824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b04106c20 con 0x7f3b04105a20 2023-10-11T16:23:03.513 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.778+0000 7f3b0892f700 1 --2- 172.21.15.111:0/2048925824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b04105e30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3af80098a0 tx=0x7f3af8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8550346c97e0cf server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:03.514 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b037fe700 1 -- 172.21.15.111:0/2048925824 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3af8004160 con 0x7f3b04105a20 2023-10-11T16:23:03.514 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b037fe700 1 -- 172.21.15.111:0/2048925824 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3af80042c0 con 0x7f3b04105a20 2023-10-11T16:23:03.515 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b037fe700 1 -- 172.21.15.111:0/2048925824 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3af8011a00 con 0x7f3b04105a20 2023-10-11T16:23:03.515 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b0ab93700 1 -- 172.21.15.111:0/2048925824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 msgr2=0x7f3b04105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:03.515 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b0ab93700 1 --2- 172.21.15.111:0/2048925824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b04105e30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3af80098a0 tx=0x7f3af8009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:03.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b0ab93700 1 -- 172.21.15.111:0/2048925824 shutdown_connections 2023-10-11T16:23:03.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b0ab93700 1 --2- 172.21.15.111:0/2048925824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b04105e30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:03.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b0ab93700 1 -- 172.21.15.111:0/2048925824 >> 172.21.15.111:0/2048925824 conn(0x7f3b04100fd0 msgr2=0x7f3b04103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:03.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b0ab93700 1 -- 172.21.15.111:0/2048925824 shutdown_connections 2023-10-11T16:23:03.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.779+0000 7f3b0ab93700 1 -- 172.21.15.111:0/2048925824 wait complete. 2023-10-11T16:23:03.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.780+0000 7f3b0ab93700 1 Processor -- start 2023-10-11T16:23:03.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.781+0000 7f3b0ab93700 1 -- start start 2023-10-11T16:23:03.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.781+0000 7f3b0ab93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b0407bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:03.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.781+0000 7f3b0ab93700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0407c140 con 0x7f3b04105a20 2023-10-11T16:23:03.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.781+0000 7f3b0892f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b0407bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:03.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.781+0000 7f3b0892f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b0407bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44842/0 (socket says 172.21.15.111:44842) 2023-10-11T16:23:03.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.781+0000 7f3b0892f700 1 -- 172.21.15.111:0/1854592672 learned_addr learned my addr 172.21.15.111:0/1854592672 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:03.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.781+0000 7f3b0892f700 1 -- 172.21.15.111:0/1854592672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af8009550 con 0x7f3b04105a20 2023-10-11T16:23:03.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.782+0000 7f3b0892f700 1 --2- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b0407bc00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3af8006af0 tx=0x7f3af8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:03.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.782+0000 7f3b01ffb700 1 -- 172.21.15.111:0/1854592672 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3af8003a40 con 0x7f3b04105a20 2023-10-11T16:23:03.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.782+0000 7f3b01ffb700 1 -- 172.21.15.111:0/1854592672 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3af8003ba0 con 0x7f3b04105a20 2023-10-11T16:23:03.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.782+0000 7f3b01ffb700 1 -- 172.21.15.111:0/1854592672 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3af801a700 con 0x7f3b04105a20 2023-10-11T16:23:03.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.782+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b04109e50 con 0x7f3b04105a20 2023-10-11T16:23:03.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.782+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b04109fe0 con 0x7f3b04105a20 2023-10-11T16:23:03.520 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.783+0000 7f3b01ffb700 1 -- 172.21.15.111:0/1854592672 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f3af8024ad0 con 0x7f3b04105a20 2023-10-11T16:23:03.520 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.783+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b04107a90 con 0x7f3b04105a20 2023-10-11T16:23:03.520 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.784+0000 7f3b01ffb700 1 --2- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f3af4038710 0x7f3af403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:03.520 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.784+0000 7f3b01ffb700 1 -- 172.21.15.111:0/1854592672 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3af804c630 con 0x7f3b04105a20 2023-10-11T16:23:03.521 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.784+0000 7f3b03fff700 1 --2- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f3af4038710 0x7f3af403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:03.521 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.784+0000 7f3b03fff700 1 --2- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f3af4038710 0x7f3af403abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3af0006fd0 tx=0x7f3af0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:03.521 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:02.789+0000 7f3b01ffb700 1 -- 172.21.15.111:0/1854592672 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3b04107a90 con 0x7f3b04105a20 2023-10-11T16:23:03.521 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.062+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3b04107a90 con 0x7f3b04105a20 2023-10-11T16:23:03.521 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.062+0000 7f3b01ffb700 1 -- 172.21.15.111:0/1854592672 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f3b04107a90 con 0x7f3b04105a20 2023-10-11T16:23:03.522 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f3af4038710 msgr2=0x7f3af403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:03.522 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 --2- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f3af4038710 0x7f3af403abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3af0006fd0 tx=0x7f3af0006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:03.522 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 msgr2=0x7f3b0407bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:03.522 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 --2- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b0407bc00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3af8006af0 tx=0x7f3af8011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:03.522 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 shutdown_connections 2023-10-11T16:23:03.523 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 --2- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f3af4038710 0x7f3af403abc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:03.523 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 --2- 172.21.15.111:0/1854592672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b04105a20 0x7f3b0407bc00 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:03.523 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 >> 172.21.15.111:0/1854592672 conn(0x7f3b04100fd0 msgr2=0x7f3b04102640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:03.523 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 shutdown_connections 2023-10-11T16:23:03.523 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.065+0000 7f3b0ab93700 1 -- 172.21.15.111:0/1854592672 wait complete. 2023-10-11T16:23:03.524 INFO:teuthology.orchestra.run.smithi111.stdout:mgr is available 2023-10-11T16:23:03.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:03 smithi111 ceph-mon[98527]: mgrmap e4: x(active, since 2s) 2023-10-11T16:23:03.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:03 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1854592672' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-11T16:23:04.576 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:23:04.576 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [global] 2023-10-11T16:23:04.577 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout fsid = 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:23:04.577 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-10-11T16:23:04.577 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-10-11T16:23:04.577 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-10-11T16:23:04.577 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-10-11T16:23:04.578 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:23:04.578 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [mgr] 2023-10-11T16:23:04.578 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-10-11T16:23:04.578 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2023-10-11T16:23:04.578 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [osd] 2023-10-11T16:23:04.579 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-10-11T16:23:04.579 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-10-11T16:23:04.579 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-10-11T16:23:04.579 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-10-11T16:23:04.579 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-10-11T16:23:04.580 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.994+0000 7fb29d7e5700 1 Processor -- start 2023-10-11T16:23:04.580 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.994+0000 7fb29d7e5700 1 -- start start 2023-10-11T16:23:04.580 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.994+0000 7fb29d7e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29807a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:04.580 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.994+0000 7fb29d7e5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29807a810 con 0x7fb29807bdd0 2023-10-11T16:23:04.580 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.995+0000 7fb297fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29807a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:04.581 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.995+0000 7fb297fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29807a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44844/0 (socket says 172.21.15.111:44844) 2023-10-11T16:23:04.581 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.995+0000 7fb297fff700 1 -- 172.21.15.111:0/3903741032 learned_addr learned my addr 172.21.15.111:0/3903741032 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:04.581 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.995+0000 7fb297fff700 1 -- 172.21.15.111:0/3903741032 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29807a950 con 0x7fb29807bdd0 2023-10-11T16:23:04.581 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.995+0000 7fb297fff700 1 --2- 172.21.15.111:0/3903741032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29807a2d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb2880098a0 tx=0x7fb288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bbdd9378b2e724 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:04.581 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb296ffd700 1 -- 172.21.15.111:0/3903741032 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb288004160 con 0x7fb29807bdd0 2023-10-11T16:23:04.582 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb296ffd700 1 -- 172.21.15.111:0/3903741032 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2880042c0 con 0x7fb29807bdd0 2023-10-11T16:23:04.582 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb296ffd700 1 -- 172.21.15.111:0/3903741032 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb288011a00 con 0x7fb29807bdd0 2023-10-11T16:23:04.582 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb29d7e5700 1 -- 172.21.15.111:0/3903741032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 msgr2=0x7fb29807a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:04.582 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb29d7e5700 1 --2- 172.21.15.111:0/3903741032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29807a2d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb2880098a0 tx=0x7fb288009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:04.582 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb29d7e5700 1 -- 172.21.15.111:0/3903741032 shutdown_connections 2023-10-11T16:23:04.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb29d7e5700 1 --2- 172.21.15.111:0/3903741032 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29807a2d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:04.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb29d7e5700 1 -- 172.21.15.111:0/3903741032 >> 172.21.15.111:0/3903741032 conn(0x7fb298100f40 msgr2=0x7fb298103350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:04.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb29d7e5700 1 -- 172.21.15.111:0/3903741032 shutdown_connections 2023-10-11T16:23:04.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.996+0000 7fb29d7e5700 1 -- 172.21.15.111:0/3903741032 wait complete. 2023-10-11T16:23:04.583 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.997+0000 7fb29d7e5700 1 Processor -- start 2023-10-11T16:23:04.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.998+0000 7fb29d7e5700 1 -- start start 2023-10-11T16:23:04.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.998+0000 7fb29d7e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29819bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:04.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.998+0000 7fb29d7e5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29819c0e0 con 0x7fb29807bdd0 2023-10-11T16:23:04.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.998+0000 7fb297fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29819bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:04.584 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.998+0000 7fb297fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29819bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44860/0 (socket says 172.21.15.111:44860) 2023-10-11T16:23:04.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.998+0000 7fb297fff700 1 -- 172.21.15.111:0/4115119167 learned_addr learned my addr 172.21.15.111:0/4115119167 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:04.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.998+0000 7fb297fff700 1 -- 172.21.15.111:0/4115119167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288009550 con 0x7fb29807bdd0 2023-10-11T16:23:04.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.999+0000 7fb297fff700 1 --2- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29819bba0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb288009130 tx=0x7fb288011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:04.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.999+0000 7fb2957fa700 1 -- 172.21.15.111:0/4115119167 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb288003ab0 con 0x7fb29807bdd0 2023-10-11T16:23:04.585 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.999+0000 7fb2957fa700 1 -- 172.21.15.111:0/4115119167 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb288003c10 con 0x7fb29807bdd0 2023-10-11T16:23:04.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.999+0000 7fb2957fa700 1 -- 172.21.15.111:0/4115119167 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2880247d0 con 0x7fb29807bdd0 2023-10-11T16:23:04.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.999+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29810bd80 con 0x7fb29807bdd0 2023-10-11T16:23:04.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:03.999+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29810bbf0 con 0x7fb29807bdd0 2023-10-11T16:23:04.586 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.000+0000 7fb2957fa700 1 -- 172.21.15.111:0/4115119167 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fb28801abb0 con 0x7fb29807bdd0 2023-10-11T16:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.000+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb298195860 con 0x7fb29807bdd0 2023-10-11T16:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.001+0000 7fb2957fa700 1 --2- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fb280038710 0x7fb28003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.001+0000 7fb2957fa700 1 -- 172.21.15.111:0/4115119167 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb28804c3e0 con 0x7fb29807bdd0 2023-10-11T16:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.001+0000 7fb2977fe700 1 --2- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fb280038710 0x7fb28003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.001+0000 7fb2977fe700 1 --2- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fb280038710 0x7fb28003abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb28c006fd0 tx=0x7fb28c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.005+0000 7fb2957fa700 1 -- 172.21.15.111:0/4115119167 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb288018710 con 0x7fb29807bdd0 2023-10-11T16:23:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.201+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb298062170 con 0x7fb29807bdd0 2023-10-11T16:23:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.202+0000 7fb2957fa700 1 -- 172.21.15.111:0/4115119167 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fb288018350 con 0x7fb29807bdd0 2023-10-11T16:23:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fb280038710 msgr2=0x7fb28003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 --2- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fb280038710 0x7fb28003abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb28c006fd0 tx=0x7fb28c006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 msgr2=0x7fb29819bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:04.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 --2- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29819bba0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb288009130 tx=0x7fb288011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:04.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 shutdown_connections 2023-10-11T16:23:04.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 --2- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fb280038710 0x7fb28003abc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:04.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 --2- 172.21.15.111:0/4115119167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb29807bdd0 0x7fb29819bba0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:04.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 >> 172.21.15.111:0/4115119167 conn(0x7fb298100f40 msgr2=0x7fb2981032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:04.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 shutdown_connections 2023-10-11T16:23:04.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:04.205+0000 7fb29d7e5700 1 -- 172.21.15.111:0/4115119167 wait complete. 2023-10-11T16:23:04.592 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling cephadm module... 2023-10-11T16:23:04.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:04 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4115119167' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-11T16:23:05.832 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:05 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4216581154' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-10-11T16:23:05.833 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:05 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: ignoring --setuser ceph since I am not root 2023-10-11T16:23:05.833 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:05 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: ignoring --setgroup ceph since I am not root 2023-10-11T16:23:05.951 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.062+0000 7f9bea26b700 1 Processor -- start 2023-10-11T16:23:05.952 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.062+0000 7f9bea26b700 1 -- start start 2023-10-11T16:23:05.952 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.062+0000 7f9bea26b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be407a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:05.952 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.062+0000 7f9bea26b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be407a860 con 0x7f9be407be30 2023-10-11T16:23:05.952 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.062+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be407a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:05.953 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.062+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be407a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48728/0 (socket says 172.21.15.111:48728) 2023-10-11T16:23:05.957 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.062+0000 7f9be3fff700 1 -- 172.21.15.111:0/4030892985 learned_addr learned my addr 172.21.15.111:0/4030892985 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:05.957 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.063+0000 7f9be3fff700 1 -- 172.21.15.111:0/4030892985 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be407a9a0 con 0x7f9be407be30 2023-10-11T16:23:05.958 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.063+0000 7f9be3fff700 1 --2- 172.21.15.111:0/4030892985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be407a290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40098a0 tx=0x7f9bd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68c333d62c2c540 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:05.958 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.063+0000 7f9be2ffd700 1 -- 172.21.15.111:0/4030892985 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bd4004160 con 0x7f9be407be30 2023-10-11T16:23:05.958 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.063+0000 7f9be2ffd700 1 -- 172.21.15.111:0/4030892985 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bd40042c0 con 0x7f9be407be30 2023-10-11T16:23:05.958 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.063+0000 7f9be2ffd700 1 -- 172.21.15.111:0/4030892985 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bd4011a00 con 0x7f9be407be30 2023-10-11T16:23:05.959 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.064+0000 7f9bea26b700 1 -- 172.21.15.111:0/4030892985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 msgr2=0x7f9be407a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:05.959 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.064+0000 7f9bea26b700 1 --2- 172.21.15.111:0/4030892985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be407a290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40098a0 tx=0x7f9bd4009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:05.959 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.064+0000 7f9bea26b700 1 -- 172.21.15.111:0/4030892985 shutdown_connections 2023-10-11T16:23:05.959 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.064+0000 7f9bea26b700 1 --2- 172.21.15.111:0/4030892985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be407a290 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:05.959 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.064+0000 7f9bea26b700 1 -- 172.21.15.111:0/4030892985 >> 172.21.15.111:0/4030892985 conn(0x7f9be4100fd0 msgr2=0x7f9be4103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:05.960 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.064+0000 7f9bea26b700 1 -- 172.21.15.111:0/4030892985 shutdown_connections 2023-10-11T16:23:05.960 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.064+0000 7f9bea26b700 1 -- 172.21.15.111:0/4030892985 wait complete. 2023-10-11T16:23:05.960 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.065+0000 7f9bea26b700 1 Processor -- start 2023-10-11T16:23:05.960 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.065+0000 7f9bea26b700 1 -- start start 2023-10-11T16:23:05.960 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.065+0000 7f9bea26b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:05.961 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.065+0000 7f9bea26b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be419be70 con 0x7f9be407be30 2023-10-11T16:23:05.961 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:05.961 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be419b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48740/0 (socket says 172.21.15.111:48740) 2023-10-11T16:23:05.961 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9be3fff700 1 -- 172.21.15.111:0/4216581154 learned_addr learned my addr 172.21.15.111:0/4216581154 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:05.961 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9be3fff700 1 -- 172.21.15.111:0/4216581154 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd4009550 con 0x7f9be407be30 2023-10-11T16:23:05.962 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9be3fff700 1 --2- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be419b930 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4006af0 tx=0x7f9bd4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:05.962 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9be17fa700 1 -- 172.21.15.111:0/4216581154 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bd4003a40 con 0x7f9be407be30 2023-10-11T16:23:05.962 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9be17fa700 1 -- 172.21.15.111:0/4216581154 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bd4003ba0 con 0x7f9be407be30 2023-10-11T16:23:05.962 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9be17fa700 1 -- 172.21.15.111:0/4216581154 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bd401a6e0 con 0x7f9be407be30 2023-10-11T16:23:05.963 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.066+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be410bf40 con 0x7f9be407be30 2023-10-11T16:23:05.963 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.067+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be410bdb0 con 0x7f9be407be30 2023-10-11T16:23:05.963 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.068+0000 7f9be17fa700 1 -- 172.21.15.111:0/4216581154 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f9bd4024a60 con 0x7f9be407be30 2023-10-11T16:23:05.963 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.068+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be4062120 con 0x7f9be407be30 2023-10-11T16:23:05.963 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.068+0000 7f9be17fa700 1 --2- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f9bcc038710 0x7f9bcc03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:05.964 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.068+0000 7f9be17fa700 1 -- 172.21.15.111:0/4216581154 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9bd404c690 con 0x7f9be407be30 2023-10-11T16:23:05.964 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.068+0000 7f9be37fe700 1 --2- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f9bcc038710 0x7f9bcc03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:05.964 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.069+0000 7f9be37fe700 1 --2- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f9bcc038710 0x7f9bcc03abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8006fd0 tx=0x7f9bd8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:05.964 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.073+0000 7f9be17fa700 1 -- 172.21.15.111:0/4216581154 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9bd4024430 con 0x7f9be407be30 2023-10-11T16:23:05.964 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.322+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f9be419c3b0 con 0x7f9be407be30 2023-10-11T16:23:05.965 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.561+0000 7f9be17fa700 1 -- 172.21.15.111:0/4216581154 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f9bd402d910 con 0x7f9be407be30 2023-10-11T16:23:05.965 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.562+0000 7f9be17fa700 1 -- 172.21.15.111:0/4216581154 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f9bd402c350 con 0x7f9be407be30 2023-10-11T16:23:05.965 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f9bcc038710 msgr2=0x7f9bcc03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:05.965 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 --2- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f9bcc038710 0x7f9bcc03abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8006fd0 tx=0x7f9bd8006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:05.965 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 msgr2=0x7f9be419b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:05.966 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 --2- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be419b930 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4006af0 tx=0x7f9bd4011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:05.966 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 shutdown_connections 2023-10-11T16:23:05.966 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 --2- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7f9bcc038710 0x7f9bcc03abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:05.966 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 --2- 172.21.15.111:0/4216581154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9be407be30 0x7f9be419b930 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:05.966 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 >> 172.21.15.111:0/4216581154 conn(0x7f9be4100fd0 msgr2=0x7f9be41026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:05.967 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 shutdown_connections 2023-10-11T16:23:05.967 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:05.565+0000 7f9bea26b700 1 -- 172.21.15.111:0/4216581154 wait complete. 2023-10-11T16:23:06.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:05 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:05.831+0000 7fee59d54200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T16:23:06.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:05 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:05.935+0000 7fee59d54200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T16:23:06.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:06.068+0000 7fee59d54200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T16:23:06.561 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:06.196+0000 7fee59d54200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T16:23:06.561 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:06.425+0000 7fee59d54200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T16:23:06.818 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:06 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4216581154' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-10-11T16:23:06.818 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:06 smithi111 ceph-mon[98527]: mgrmap e5: x(active, since 5s) 2023-10-11T16:23:07.122 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:06.824+0000 7fee59d54200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T16:23:07.122 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:06.952+0000 7fee59d54200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T16:23:07.122 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:07 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:07.121+0000 7fee59d54200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T16:23:07.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2023-10-11T16:23:07.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-10-11T16:23:07.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-11T16:23:07.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-10-11T16:23:07.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-10-11T16:23:07.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2023-10-11T16:23:07.143 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.521+0000 7fec2f1d2700 1 Processor -- start 2023-10-11T16:23:07.143 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.522+0000 7fec2f1d2700 1 -- start start 2023-10-11T16:23:07.143 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.522+0000 7fec2f1d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec28108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:07.143 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.522+0000 7fec2f1d2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec28108600 con 0x7fec28107c20 2023-10-11T16:23:07.144 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.522+0000 7fec2cf6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec28108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:07.144 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.522+0000 7fec2cf6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec28108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48764/0 (socket says 172.21.15.111:48764) 2023-10-11T16:23:07.144 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.522+0000 7fec2cf6e700 1 -- 172.21.15.111:0/2758469568 learned_addr learned my addr 172.21.15.111:0/2758469568 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:07.144 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.523+0000 7fec2cf6e700 1 -- 172.21.15.111:0/2758469568 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec28108e20 con 0x7fec28107c20 2023-10-11T16:23:07.144 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.523+0000 7fec2cf6e700 1 --2- 172.21.15.111:0/2758469568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec28108030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0098a0 tx=0x7fec1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff06b6bda777e4a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:07.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.523+0000 7fec277fe700 1 -- 172.21.15.111:0/2758469568 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec1c004160 con 0x7fec28107c20 2023-10-11T16:23:07.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.523+0000 7fec277fe700 1 -- 172.21.15.111:0/2758469568 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec1c0042c0 con 0x7fec28107c20 2023-10-11T16:23:07.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.523+0000 7fec277fe700 1 -- 172.21.15.111:0/2758469568 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec1c011a00 con 0x7fec28107c20 2023-10-11T16:23:07.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.524+0000 7fec2f1d2700 1 -- 172.21.15.111:0/2758469568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 msgr2=0x7fec28108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:07.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.524+0000 7fec2f1d2700 1 --2- 172.21.15.111:0/2758469568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec28108030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0098a0 tx=0x7fec1c009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:07.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.524+0000 7fec2f1d2700 1 -- 172.21.15.111:0/2758469568 shutdown_connections 2023-10-11T16:23:07.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.524+0000 7fec2f1d2700 1 --2- 172.21.15.111:0/2758469568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec28108030 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:07.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.524+0000 7fec2f1d2700 1 -- 172.21.15.111:0/2758469568 >> 172.21.15.111:0/2758469568 conn(0x7fec2807b000 msgr2=0x7fec2807b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:07.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.524+0000 7fec2f1d2700 1 -- 172.21.15.111:0/2758469568 shutdown_connections 2023-10-11T16:23:07.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.524+0000 7fec2f1d2700 1 -- 172.21.15.111:0/2758469568 wait complete. 2023-10-11T16:23:07.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.525+0000 7fec2f1d2700 1 Processor -- start 2023-10-11T16:23:07.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.525+0000 7fec2f1d2700 1 -- start start 2023-10-11T16:23:07.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.525+0000 7fec2f1d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec2819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:07.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.526+0000 7fec2f1d2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2819c070 con 0x7fec28107c20 2023-10-11T16:23:07.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.526+0000 7fec2cf6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec2819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:07.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.526+0000 7fec2cf6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec2819bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48772/0 (socket says 172.21.15.111:48772) 2023-10-11T16:23:07.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.526+0000 7fec2cf6e700 1 -- 172.21.15.111:0/392180937 learned_addr learned my addr 172.21.15.111:0/392180937 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:07.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.526+0000 7fec2cf6e700 1 -- 172.21.15.111:0/392180937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c009550 con 0x7fec28107c20 2023-10-11T16:23:07.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.526+0000 7fec2cf6e700 1 --2- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec2819bb30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fec1c006af0 tx=0x7fec1c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:07.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.527+0000 7fec25ffb700 1 -- 172.21.15.111:0/392180937 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec1c003a40 con 0x7fec28107c20 2023-10-11T16:23:07.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.527+0000 7fec25ffb700 1 -- 172.21.15.111:0/392180937 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec1c003ba0 con 0x7fec28107c20 2023-10-11T16:23:07.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.527+0000 7fec25ffb700 1 -- 172.21.15.111:0/392180937 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec1c01a700 con 0x7fec28107c20 2023-10-11T16:23:07.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.527+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2810c050 con 0x7fec28107c20 2023-10-11T16:23:07.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.527+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2810c1e0 con 0x7fec28107c20 2023-10-11T16:23:07.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.528+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec28109c90 con 0x7fec28107c20 2023-10-11T16:23:07.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.528+0000 7fec25ffb700 1 -- 172.21.15.111:0/392180937 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fec1c024ad0 con 0x7fec28107c20 2023-10-11T16:23:07.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.529+0000 7fec25ffb700 1 --2- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fec18038760 0x7fec1803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:07.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.529+0000 7fec25ffb700 1 -- 172.21.15.111:0/392180937 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fec1c04c6c0 con 0x7fec28107c20 2023-10-11T16:23:07.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.529+0000 7fec27fff700 1 -- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fec18038760 msgr2=0x7fec1803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/699479694 2023-10-11T16:23:07.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.529+0000 7fec27fff700 1 --2- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fec18038760 0x7fec1803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:23:07.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.534+0000 7fec25ffb700 1 -- 172.21.15.111:0/392180937 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fec28109c90 con 0x7fec28107c20 2023-10-11T16:23:07.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.730+0000 7fec27fff700 1 -- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fec18038760 msgr2=0x7fec1803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/699479694 2023-10-11T16:23:07.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.730+0000 7fec27fff700 1 --2- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fec18038760 0x7fec1803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-11T16:23:07.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.793+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fec28109c90 con 0x7fec28107c20 2023-10-11T16:23:07.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.794+0000 7fec25ffb700 1 -- 172.21.15.111:0/392180937 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fec28109c90 con 0x7fec28107c20 2023-10-11T16:23:07.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fec18038760 msgr2=0x7fec1803ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:23:07.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 --2- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fec18038760 0x7fec1803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:07.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 msgr2=0x7fec2819bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:07.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 --2- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec2819bb30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fec1c006af0 tx=0x7fec1c011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:07.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 shutdown_connections 2023-10-11T16:23:07.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 --2- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fec18038760 0x7fec1803ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:07.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 --2- 172.21.15.111:0/392180937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec28107c20 0x7fec2819bb30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:07.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 >> 172.21.15.111:0/392180937 conn(0x7fec2807b000 msgr2=0x7fec28105840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:07.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 shutdown_connections 2023-10-11T16:23:07.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:06.796+0000 7fec2f1d2700 1 -- 172.21.15.111:0/392180937 wait complete. 2023-10-11T16:23:07.153 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for the mgr to restart... 2023-10-11T16:23:07.154 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr epoch 5... 2023-10-11T16:23:07.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:07 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:07.275+0000 7fee59d54200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T16:23:07.830 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:07 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/392180937' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-11T16:23:08.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:07 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:07.830+0000 7fee59d54200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T16:23:08.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:08 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:08.567+0000 7fee59d54200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T16:23:08.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:08 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:08.773+0000 7fee59d54200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T16:23:10.045 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:09 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:09.705+0000 7fee59d54200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:23:10.046 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:09 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:09.823+0000 7fee59d54200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T16:23:10.046 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:10.043+0000 7fee59d54200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:23:10.679 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:10.403+0000 7fee59d54200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T16:23:11.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:10.677+0000 7fee59d54200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T16:23:11.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:10.865+0000 7fee59d54200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T16:23:11.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:11 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:11.204+0000 7fee59d54200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T16:23:12.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:12.119+0000 7fee59d54200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T16:23:12.386 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:12.286+0000 7fee59d54200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T16:23:12.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:12.529+0000 7fee59d54200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T16:23:12.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:12.674+0000 7fee59d54200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T16:23:13.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:12.948+0000 7fee59d54200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T16:23:14.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:13 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:13.993+0000 7fee59d54200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T16:23:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: Active manager daemon x restarted 2023-10-11T16:23:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: Activating manager daemon x 2023-10-11T16:23:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: osdmap e2: 0 total, 0 up, 0 in 2023-10-11T16:23:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: mgrmap e6: x(active, starting, since 0.00567537s) 2023-10-11T16:23:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:23:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-11T16:23:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-11T16:23:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-11T16:23:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-11T16:23:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: Manager daemon x is now available 2023-10-11T16:23:14.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:14.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:14.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:14.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:14 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:15.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:15 smithi111 ceph-mon[98527]: Found migration_current of "None". Setting to last migration. 2023-10-11T16:23:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:15 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:15 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-11T16:23:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:15 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-11T16:23:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:15 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:15 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:15.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:15 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:15.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:15 smithi111 ceph-mon[98527]: mgrmap e7: x(active, since 1.00678s) 2023-10-11T16:23:15.458 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2023-10-11T16:23:15.459 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-10-11T16:23:15.459 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "initialized": true 2023-10-11T16:23:15.459 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2023-10-11T16:23:15.459 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.981+0000 7fa4a1544700 1 Processor -- start 2023-10-11T16:23:15.459 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.981+0000 7fa4a1544700 1 -- start start 2023-10-11T16:23:15.460 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.982+0000 7fa4a1544700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:15.460 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.982+0000 7fa4a1544700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c07a860 con 0x7fa49c07be30 2023-10-11T16:23:15.460 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.982+0000 7fa49affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:15.460 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.982+0000 7fa49affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48782/0 (socket says 172.21.15.111:48782) 2023-10-11T16:23:15.460 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.982+0000 7fa49affd700 1 -- 172.21.15.111:0/1146801302 learned_addr learned my addr 172.21.15.111:0/1146801302 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:15.461 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.983+0000 7fa49affd700 1 -- 172.21.15.111:0/1146801302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c07a9a0 con 0x7fa49c07be30 2023-10-11T16:23:15.461 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.983+0000 7fa49affd700 1 --2- 172.21.15.111:0/1146801302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c07a290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0098a0 tx=0x7fa48c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0d25a32c130d5b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:15.461 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.983+0000 7fa499ffb700 1 -- 172.21.15.111:0/1146801302 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa48c004160 con 0x7fa49c07be30 2023-10-11T16:23:15.461 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.983+0000 7fa499ffb700 1 -- 172.21.15.111:0/1146801302 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa48c0042c0 con 0x7fa49c07be30 2023-10-11T16:23:15.461 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.983+0000 7fa499ffb700 1 -- 172.21.15.111:0/1146801302 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa48c011a00 con 0x7fa49c07be30 2023-10-11T16:23:15.462 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.984+0000 7fa4a1544700 1 -- 172.21.15.111:0/1146801302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 msgr2=0x7fa49c07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:15.462 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.984+0000 7fa4a1544700 1 --2- 172.21.15.111:0/1146801302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c07a290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0098a0 tx=0x7fa48c009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:15.462 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.984+0000 7fa4a1544700 1 -- 172.21.15.111:0/1146801302 shutdown_connections 2023-10-11T16:23:15.462 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.984+0000 7fa4a1544700 1 --2- 172.21.15.111:0/1146801302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c07a290 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:15.462 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.984+0000 7fa4a1544700 1 -- 172.21.15.111:0/1146801302 >> 172.21.15.111:0/1146801302 conn(0x7fa49c100fd0 msgr2=0x7fa49c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:15.463 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.984+0000 7fa4a1544700 1 -- 172.21.15.111:0/1146801302 shutdown_connections 2023-10-11T16:23:15.463 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.984+0000 7fa4a1544700 1 -- 172.21.15.111:0/1146801302 wait complete. 2023-10-11T16:23:15.463 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.985+0000 7fa4a1544700 1 Processor -- start 2023-10-11T16:23:15.463 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.986+0000 7fa4a1544700 1 -- start start 2023-10-11T16:23:15.463 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.986+0000 7fa4a1544700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.986+0000 7fa4a1544700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c19be70 con 0x7fa49c07be30 2023-10-11T16:23:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.986+0000 7fa49affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.986+0000 7fa49affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c19b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48798/0 (socket says 172.21.15.111:48798) 2023-10-11T16:23:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.986+0000 7fa49affd700 1 -- 172.21.15.111:0/1054297871 learned_addr learned my addr 172.21.15.111:0/1054297871 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.987+0000 7fa49affd700 1 -- 172.21.15.111:0/1054297871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c009550 con 0x7fa49c07be30 2023-10-11T16:23:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.987+0000 7fa49affd700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c19b930 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa48c006af0 tx=0x7fa48c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.987+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa48c003a40 con 0x7fa49c07be30 2023-10-11T16:23:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.987+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa48c003ba0 con 0x7fa49c07be30 2023-10-11T16:23:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.987+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49c10bf40 con 0x7fa49c07be30 2023-10-11T16:23:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.987+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa48c01a6e0 con 0x7fa49c07be30 2023-10-11T16:23:15.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.987+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49c10bdb0 con 0x7fa49c07be30 2023-10-11T16:23:15.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.989+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fa48c024430 con 0x7fa49c07be30 2023-10-11T16:23:15.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.989+0000 7fa483fff700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 0x7fa48403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:15.466 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.989+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 --> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa48403b320 con 0x7fa484038760 2023-10-11T16:23:15.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.989+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa48c04c6e0 con 0x7fa49c07be30 2023-10-11T16:23:15.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.989+0000 7fa49a7fc700 1 -- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 msgr2=0x7fa48403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/699479694 2023-10-11T16:23:15.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:07.989+0000 7fa49a7fc700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 0x7fa48403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:23:15.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:08.190+0000 7fa49a7fc700 1 -- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 msgr2=0x7fa48403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/699479694 2023-10-11T16:23:15.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:08.190+0000 7fa49a7fc700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 0x7fa48403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-11T16:23:15.467 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:08.591+0000 7fa49a7fc700 1 -- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 msgr2=0x7fa48403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/699479694 2023-10-11T16:23:15.468 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:08.591+0000 7fa49a7fc700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 0x7fa48403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-11T16:23:15.468 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:09.392+0000 7fa49a7fc700 1 -- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 msgr2=0x7fa48403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/699479694 2023-10-11T16:23:15.468 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:09.392+0000 7fa49a7fc700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 0x7fa48403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-11T16:23:15.468 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:10.992+0000 7fa49a7fc700 1 -- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 msgr2=0x7fa48403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/699479694 2023-10-11T16:23:15.468 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:10.993+0000 7fa49a7fc700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 0x7fa48403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-11T16:23:15.469 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:13.999+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fa48c024a60 con 0x7fa49c07be30 2023-10-11T16:23:15.469 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:13.999+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 msgr2=0x7fa48403ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:23:15.469 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:13.999+0000 7fa483fff700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 0x7fa48403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:15.469 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.000+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fa48c04dc70 con 0x7fa49c07be30 2023-10-11T16:23:15.469 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.000+0000 7fa483fff700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fa48403bfe0 0x7fa48403e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:15.470 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.000+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa48403b320 con 0x7fa48403bfe0 2023-10-11T16:23:15.470 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.000+0000 7fa49a7fc700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fa48403bfe0 0x7fa48403e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:15.470 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.001+0000 7fa49a7fc700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fa48403bfe0 0x7fa48403e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa490003a10 tx=0x7fa4900092b0 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:15.470 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.002+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mgr.14116 v2:172.21.15.111:6800/4086392563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fa48403b320 con 0x7fa48403bfe0 2023-10-11T16:23:15.470 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.012+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa49c19cc70 con 0x7fa48403bfe0 2023-10-11T16:23:15.471 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.012+0000 7fa483fff700 1 -- 172.21.15.111:0/1054297871 <== mgr.14116 v2:172.21.15.111:6800/4086392563 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa49c19cc70 con 0x7fa48403bfe0 2023-10-11T16:23:15.471 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fa48403bfe0 msgr2=0x7fa48403e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:15.471 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fa48403bfe0 0x7fa48403e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa490003a10 tx=0x7fa4900092b0 comp rx=0 tx=0).stop 2023-10-11T16:23:15.471 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 msgr2=0x7fa49c19b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:15.471 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c19b930 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa48c006af0 tx=0x7fa48c011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:15.472 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 shutdown_connections 2023-10-11T16:23:15.472 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fa48403bfe0 0x7fa48403e3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:15.472 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa49c07be30 0x7fa49c19b930 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:15.472 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 --2- 172.21.15.111:0/1054297871 >> [v2:172.21.15.111:6800/699479694,v1:172.21.15.111:6801/699479694] conn(0x7fa484038760 0x7fa48403ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:15.472 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 >> 172.21.15.111:0/1054297871 conn(0x7fa49c100fd0 msgr2=0x7fa49c1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:15.473 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 shutdown_connections 2023-10-11T16:23:15.473 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.013+0000 7fa4a1544700 1 -- 172.21.15.111:0/1054297871 wait complete. 2023-10-11T16:23:15.473 INFO:teuthology.orchestra.run.smithi111.stdout:mgr epoch 5 is available 2023-10-11T16:23:15.473 INFO:teuthology.orchestra.run.smithi111.stdout:Setting orchestrator backend to cephadm... 2023-10-11T16:23:16.371 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:16 smithi111 ceph-mon[98527]: [11/Oct/2023:16:23:14] ENGINE Bus STARTING 2023-10-11T16:23:16.371 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:16 smithi111 ceph-mon[98527]: [11/Oct/2023:16:23:14] ENGINE Serving on http://172.21.15.111:8765 2023-10-11T16:23:16.372 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:16 smithi111 ceph-mon[98527]: [11/Oct/2023:16:23:14] ENGINE Serving on https://172.21.15.111:7150 2023-10-11T16:23:16.372 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:16 smithi111 ceph-mon[98527]: [11/Oct/2023:16:23:14] ENGINE Bus STARTED 2023-10-11T16:23:16.372 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:16 smithi111 ceph-mon[98527]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-11T16:23:16.372 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:16 smithi111 ceph-mon[98527]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-11T16:23:16.639 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.985+0000 7fe36e55a700 1 Processor -- start 2023-10-11T16:23:16.639 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.985+0000 7fe36e55a700 1 -- start start 2023-10-11T16:23:16.639 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.985+0000 7fe36e55a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe368105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:16.640 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.985+0000 7fe36e55a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe368106220 con 0x7fe3681058d0 2023-10-11T16:23:16.640 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.986+0000 7fe36d558700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe368105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:16.640 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.986+0000 7fe36d558700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe368105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39850/0 (socket says 172.21.15.111:39850) 2023-10-11T16:23:16.640 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.986+0000 7fe36d558700 1 -- 172.21.15.111:0/3971613159 learned_addr learned my addr 172.21.15.111:0/3971613159 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:16.640 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.986+0000 7fe36d558700 1 -- 172.21.15.111:0/3971613159 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe368106360 con 0x7fe3681058d0 2023-10-11T16:23:16.641 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.986+0000 7fe36d558700 1 --2- 172.21.15.111:0/3971613159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe368105ce0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe3640098a0 tx=0x7fe364009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692ff167f065b434 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:16.641 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.986+0000 7fe35ffff700 1 -- 172.21.15.111:0/3971613159 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe364004160 con 0x7fe3681058d0 2023-10-11T16:23:16.641 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.986+0000 7fe35ffff700 1 -- 172.21.15.111:0/3971613159 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3640042c0 con 0x7fe3681058d0 2023-10-11T16:23:16.641 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.986+0000 7fe35ffff700 1 -- 172.21.15.111:0/3971613159 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe364011a00 con 0x7fe3681058d0 2023-10-11T16:23:16.641 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.987+0000 7fe36e55a700 1 -- 172.21.15.111:0/3971613159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 msgr2=0x7fe368105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:16.642 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.987+0000 7fe36e55a700 1 --2- 172.21.15.111:0/3971613159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe368105ce0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe3640098a0 tx=0x7fe364009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:16.642 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.987+0000 7fe36e55a700 1 -- 172.21.15.111:0/3971613159 shutdown_connections 2023-10-11T16:23:16.642 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.987+0000 7fe36e55a700 1 --2- 172.21.15.111:0/3971613159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe368105ce0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:16.642 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.987+0000 7fe36e55a700 1 -- 172.21.15.111:0/3971613159 >> 172.21.15.111:0/3971613159 conn(0x7fe368100f40 msgr2=0x7fe368103350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:16.642 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.987+0000 7fe36e55a700 1 -- 172.21.15.111:0/3971613159 shutdown_connections 2023-10-11T16:23:16.643 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.987+0000 7fe36e55a700 1 -- 172.21.15.111:0/3971613159 wait complete. 2023-10-11T16:23:16.643 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.989+0000 7fe36e55a700 1 Processor -- start 2023-10-11T16:23:16.643 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.989+0000 7fe36e55a700 1 -- start start 2023-10-11T16:23:16.643 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.989+0000 7fe36e55a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe36819bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:16.643 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.989+0000 7fe36e55a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36819c060 con 0x7fe3681058d0 2023-10-11T16:23:16.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.989+0000 7fe36d558700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe36819bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:16.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.989+0000 7fe36d558700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe36819bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39852/0 (socket says 172.21.15.111:39852) 2023-10-11T16:23:16.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.990+0000 7fe36d558700 1 -- 172.21.15.111:0/228914879 learned_addr learned my addr 172.21.15.111:0/228914879 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:16.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.990+0000 7fe36d558700 1 -- 172.21.15.111:0/228914879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364009550 con 0x7fe3681058d0 2023-10-11T16:23:16.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.990+0000 7fe36d558700 1 --2- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe36819bb20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe364009130 tx=0x7fe364011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:16.644 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.990+0000 7fe35e7fc700 1 -- 172.21.15.111:0/228914879 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe364003b70 con 0x7fe3681058d0 2023-10-11T16:23:16.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.990+0000 7fe35e7fc700 1 -- 172.21.15.111:0/228914879 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe364003cd0 con 0x7fe3681058d0 2023-10-11T16:23:16.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.990+0000 7fe35e7fc700 1 -- 172.21.15.111:0/228914879 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3640197d0 con 0x7fe3681058d0 2023-10-11T16:23:16.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.990+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe368107920 con 0x7fe3681058d0 2023-10-11T16:23:16.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.990+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe368107790 con 0x7fe3681058d0 2023-10-11T16:23:16.645 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.991+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe368109da0 con 0x7fe3681058d0 2023-10-11T16:23:16.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.991+0000 7fe35e7fc700 1 -- 172.21.15.111:0/228914879 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fe36401abb0 con 0x7fe3681058d0 2023-10-11T16:23:16.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.992+0000 7fe35e7fc700 1 --2- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fe354038630 0x7fe35403aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:16.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.992+0000 7fe35e7fc700 1 -- 172.21.15.111:0/228914879 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe36404c790 con 0x7fe3681058d0 2023-10-11T16:23:16.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.992+0000 7fe36cd57700 1 --2- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fe354038630 0x7fe35403aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:16.646 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.993+0000 7fe36cd57700 1 --2- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fe354038630 0x7fe35403aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe358006fd0 tx=0x7fe358006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:16.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:15.997+0000 7fe35e7fc700 1 -- 172.21.15.111:0/228914879 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe36401a430 con 0x7fe3681058d0 2023-10-11T16:23:16.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.051+0000 7fe35e7fc700 1 -- 172.21.15.111:0/228914879 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe36402d350 con 0x7fe3681058d0 2023-10-11T16:23:16.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.207+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fe36819c6d0 con 0x7fe354038630 2023-10-11T16:23:16.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.214+0000 7fe35e7fc700 1 -- 172.21.15.111:0/228914879 <== mgr.14116 v2:172.21.15.111:6800/4086392563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe36819c6d0 con 0x7fe354038630 2023-10-11T16:23:16.647 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fe354038630 msgr2=0x7fe35403aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:16.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 --2- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fe354038630 0x7fe35403aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe358006fd0 tx=0x7fe358006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:16.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 msgr2=0x7fe36819bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:16.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 --2- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe36819bb20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe364009130 tx=0x7fe364011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:16.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 shutdown_connections 2023-10-11T16:23:16.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 --2- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fe354038630 0x7fe35403aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:16.648 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 --2- 172.21.15.111:0/228914879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3681058d0 0x7fe36819bb20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:16.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 >> 172.21.15.111:0/228914879 conn(0x7fe368100f40 msgr2=0x7fe3681031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:16.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 shutdown_connections 2023-10-11T16:23:16.649 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:16.217+0000 7fe36e55a700 1 -- 172.21.15.111:0/228914879 wait complete. 2023-10-11T16:23:17.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:17 smithi111 ceph-mon[98527]: mgrmap e8: x(active, since 2s) 2023-10-11T16:23:17.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:17 smithi111 ceph-mon[98527]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:17.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:17 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:17 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:17.817 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout value unchanged 2023-10-11T16:23:17.817 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.151+0000 7f465130f700 1 Processor -- start 2023-10-11T16:23:17.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.151+0000 7f465130f700 1 -- start start 2023-10-11T16:23:17.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.151+0000 7f465130f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:17.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.151+0000 7f465130f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c106400 con 0x7f464c105a20 2023-10-11T16:23:17.818 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.152+0000 7f464affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:17.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.152+0000 7f464affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39854/0 (socket says 172.21.15.111:39854) 2023-10-11T16:23:17.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.152+0000 7f464affd700 1 -- 172.21.15.111:0/520228772 learned_addr learned my addr 172.21.15.111:0/520228772 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:17.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.152+0000 7f464affd700 1 -- 172.21.15.111:0/520228772 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c106c20 con 0x7f464c105a20 2023-10-11T16:23:17.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.152+0000 7f464affd700 1 --2- 172.21.15.111:0/520228772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c105e30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f463c0098a0 tx=0x7f463c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa771157544e9cd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:17.819 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.152+0000 7f4649ffb700 1 -- 172.21.15.111:0/520228772 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f463c004160 con 0x7f464c105a20 2023-10-11T16:23:17.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f4649ffb700 1 -- 172.21.15.111:0/520228772 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f463c0042c0 con 0x7f464c105a20 2023-10-11T16:23:17.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f4649ffb700 1 -- 172.21.15.111:0/520228772 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f463c011a00 con 0x7f464c105a20 2023-10-11T16:23:17.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f465130f700 1 -- 172.21.15.111:0/520228772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 msgr2=0x7f464c105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:17.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f465130f700 1 --2- 172.21.15.111:0/520228772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c105e30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f463c0098a0 tx=0x7f463c009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:17.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f465130f700 1 -- 172.21.15.111:0/520228772 shutdown_connections 2023-10-11T16:23:17.820 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f465130f700 1 --2- 172.21.15.111:0/520228772 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c105e30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:17.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f465130f700 1 -- 172.21.15.111:0/520228772 >> 172.21.15.111:0/520228772 conn(0x7f464c100fd0 msgr2=0x7f464c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:17.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f465130f700 1 -- 172.21.15.111:0/520228772 shutdown_connections 2023-10-11T16:23:17.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.153+0000 7f465130f700 1 -- 172.21.15.111:0/520228772 wait complete. 2023-10-11T16:23:17.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.154+0000 7f465130f700 1 Processor -- start 2023-10-11T16:23:17.821 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.154+0000 7f465130f700 1 -- start start 2023-10-11T16:23:17.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.155+0000 7f465130f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:17.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.155+0000 7f465130f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c199e70 con 0x7f464c105a20 2023-10-11T16:23:17.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.155+0000 7f464affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:17.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.155+0000 7f464affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39858/0 (socket says 172.21.15.111:39858) 2023-10-11T16:23:17.822 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.155+0000 7f464affd700 1 -- 172.21.15.111:0/2895998847 learned_addr learned my addr 172.21.15.111:0/2895998847 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:17.823 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.155+0000 7f464affd700 1 -- 172.21.15.111:0/2895998847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009550 con 0x7f464c105a20 2023-10-11T16:23:17.823 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.156+0000 7f464affd700 1 --2- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c199930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f463c009fb0 tx=0x7f463c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:17.823 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.156+0000 7f4633fff700 1 -- 172.21.15.111:0/2895998847 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f463c003930 con 0x7f464c105a20 2023-10-11T16:23:17.823 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.156+0000 7f4633fff700 1 -- 172.21.15.111:0/2895998847 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f463c003a90 con 0x7f464c105a20 2023-10-11T16:23:17.823 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.156+0000 7f4633fff700 1 -- 172.21.15.111:0/2895998847 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f463c0195e0 con 0x7f464c105a20 2023-10-11T16:23:17.823 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.156+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c109e50 con 0x7f464c105a20 2023-10-11T16:23:17.824 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.156+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c109fe0 con 0x7f464c105a20 2023-10-11T16:23:17.824 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.157+0000 7f4633fff700 1 -- 172.21.15.111:0/2895998847 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f463c01a9d0 con 0x7f464c105a20 2023-10-11T16:23:17.824 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.157+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c107a90 con 0x7f464c105a20 2023-10-11T16:23:17.824 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.157+0000 7f4633fff700 1 --2- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f4634038710 0x7f463403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:17.824 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.157+0000 7f4633fff700 1 -- 172.21.15.111:0/2895998847 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f463c04c890 con 0x7f464c105a20 2023-10-11T16:23:17.825 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.158+0000 7f464a7fc700 1 --2- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f4634038710 0x7f463403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:17.825 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.158+0000 7f464a7fc700 1 --2- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f4634038710 0x7f463403abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4640006fd0 tx=0x7f4640006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:17.825 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.163+0000 7f4633fff700 1 -- 172.21.15.111:0/2895998847 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f464c107a90 con 0x7f464c105a20 2023-10-11T16:23:17.825 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.359+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f464c107a90 con 0x7f4634038710 2023-10-11T16:23:17.825 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.360+0000 7f4633fff700 1 -- 172.21.15.111:0/2895998847 <== mgr.14116 v2:172.21.15.111:6800/4086392563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f464c107a90 con 0x7f4634038710 2023-10-11T16:23:17.826 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f4634038710 msgr2=0x7f463403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:17.826 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 --2- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f4634038710 0x7f463403abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4640006fd0 tx=0x7f4640006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:17.826 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 msgr2=0x7f464c199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:17.826 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 --2- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c199930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f463c009fb0 tx=0x7f463c011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:17.826 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 shutdown_connections 2023-10-11T16:23:17.827 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 --2- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f4634038710 0x7f463403abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:17.827 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 --2- 172.21.15.111:0/2895998847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f464c105a20 0x7f464c199930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:17.827 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 >> 172.21.15.111:0/2895998847 conn(0x7f464c100fd0 msgr2=0x7f464c102640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:17.827 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 shutdown_connections 2023-10-11T16:23:17.827 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:17.362+0000 7f465130f700 1 -- 172.21.15.111:0/2895998847 wait complete. 2023-10-11T16:23:17.828 INFO:teuthology.orchestra.run.smithi111.stdout:Generating ssh key... 2023-10-11T16:23:18.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:18 smithi111 ceph-mon[98527]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:19.318 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-mon[98527]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:19.319 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-mon[98527]: Generating ssh key... 2023-10-11T16:23:19.319 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: Generating public/private rsa key pair. 2023-10-11T16:23:19.319 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: Your identification has been saved in /tmp/tmp58l05px9/key. 2023-10-11T16:23:19.319 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: Your public key has been saved in /tmp/tmp58l05px9/key.pub. 2023-10-11T16:23:19.320 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: The key fingerprint is: 2023-10-11T16:23:19.320 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: SHA256:YTWm6lPUA4/pIKzEZLSUDXZmAlaTCpgTB3Iz39Ro+Gk ceph-495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:23:19.320 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: The key's randomart image is: 2023-10-11T16:23:19.320 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: +---[RSA 3072]----+ 2023-10-11T16:23:19.320 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: |*B#*=..o. + | 2023-10-11T16:23:19.320 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: |BO.%+oo .X . | 2023-10-11T16:23:19.321 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: |..= +oo.B + | 2023-10-11T16:23:19.321 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: | o . .E* . . | 2023-10-11T16:23:19.321 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: | . .. S | 2023-10-11T16:23:19.321 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: | . . | 2023-10-11T16:23:19.321 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: | o | 2023-10-11T16:23:19.322 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: | . | 2023-10-11T16:23:19.322 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: | | 2023-10-11T16:23:19.322 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: +----[SHA256]-----+ 2023-10-11T16:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.315+0000 7fc57bfd0700 1 Processor -- start 2023-10-11T16:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.316+0000 7fc57bfd0700 1 -- start start 2023-10-11T16:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.316+0000 7fc57bfd0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc574105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.316+0000 7fc57bfd0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc574106400 con 0x7fc574105a20 2023-10-11T16:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.316+0000 7fc579d6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc574105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.316+0000 7fc579d6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc574105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39868/0 (socket says 172.21.15.111:39868) 2023-10-11T16:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.316+0000 7fc579d6c700 1 -- 172.21.15.111:0/3135533502 learned_addr learned my addr 172.21.15.111:0/3135533502 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.317+0000 7fc579d6c700 1 -- 172.21.15.111:0/3135533502 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574106c20 con 0x7fc574105a20 2023-10-11T16:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.317+0000 7fc579d6c700 1 --2- 172.21.15.111:0/3135533502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc574105e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc5700098a0 tx=0x7fc570009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e76a82506582c3e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.317+0000 7fc578d6a700 1 -- 172.21.15.111:0/3135533502 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc570004160 con 0x7fc574105a20 2023-10-11T16:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.317+0000 7fc578d6a700 1 -- 172.21.15.111:0/3135533502 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5700042c0 con 0x7fc574105a20 2023-10-11T16:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.317+0000 7fc578d6a700 1 -- 172.21.15.111:0/3135533502 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc570011a00 con 0x7fc574105a20 2023-10-11T16:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.318+0000 7fc57bfd0700 1 -- 172.21.15.111:0/3135533502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 msgr2=0x7fc574105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.318+0000 7fc57bfd0700 1 --2- 172.21.15.111:0/3135533502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc574105e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc5700098a0 tx=0x7fc570009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.318+0000 7fc57bfd0700 1 -- 172.21.15.111:0/3135533502 shutdown_connections 2023-10-11T16:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.318+0000 7fc57bfd0700 1 --2- 172.21.15.111:0/3135533502 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc574105e30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.318+0000 7fc57bfd0700 1 -- 172.21.15.111:0/3135533502 >> 172.21.15.111:0/3135533502 conn(0x7fc574100fd0 msgr2=0x7fc574103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.318+0000 7fc57bfd0700 1 -- 172.21.15.111:0/3135533502 shutdown_connections 2023-10-11T16:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.318+0000 7fc57bfd0700 1 -- 172.21.15.111:0/3135533502 wait complete. 2023-10-11T16:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.319+0000 7fc57bfd0700 1 Processor -- start 2023-10-11T16:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.319+0000 7fc57bfd0700 1 -- start start 2023-10-11T16:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.319+0000 7fc57bfd0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc57419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.319+0000 7fc57bfd0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57419c090 con 0x7fc574105a20 2023-10-11T16:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc579d6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc57419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc579d6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc57419bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39874/0 (socket says 172.21.15.111:39874) 2023-10-11T16:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc579d6c700 1 -- 172.21.15.111:0/4219305004 learned_addr learned my addr 172.21.15.111:0/4219305004 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc579d6c700 1 -- 172.21.15.111:0/4219305004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc570009550 con 0x7fc574105a20 2023-10-11T16:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc579d6c700 1 --2- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc57419bb50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc570006af0 tx=0x7fc570011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc56affd700 1 -- 172.21.15.111:0/4219305004 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc570003a40 con 0x7fc574105a20 2023-10-11T16:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc56affd700 1 -- 172.21.15.111:0/4219305004 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc570003ba0 con 0x7fc574105a20 2023-10-11T16:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc574109e50 con 0x7fc574105a20 2023-10-11T16:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.320+0000 7fc56affd700 1 -- 172.21.15.111:0/4219305004 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc57001a700 con 0x7fc574105a20 2023-10-11T16:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.321+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc574109fe0 con 0x7fc574105a20 2023-10-11T16:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.321+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc574107a90 con 0x7fc574105a20 2023-10-11T16:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.322+0000 7fc56affd700 1 -- 172.21.15.111:0/4219305004 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc570024ad0 con 0x7fc574105a20 2023-10-11T16:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.322+0000 7fc56affd700 1 --2- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc560038760 0x7fc56003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.322+0000 7fc56affd700 1 -- 172.21.15.111:0/4219305004 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc57004c780 con 0x7fc574105a20 2023-10-11T16:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.322+0000 7fc57956b700 1 --2- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc560038760 0x7fc56003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.323+0000 7fc57956b700 1 --2- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc560038760 0x7fc56003ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc564006fd0 tx=0x7fc564006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.327+0000 7fc56affd700 1 -- 172.21.15.111:0/4219305004 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc574107a90 con 0x7fc574105a20 2023-10-11T16:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:18.524+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fc574107a90 con 0x7fc560038760 2023-10-11T16:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.140+0000 7fc56affd700 1 -- 172.21.15.111:0/4219305004 <== mgr.14116 v2:172.21.15.111:6800/4086392563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc574107a90 con 0x7fc560038760 2023-10-11T16:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc560038760 msgr2=0x7fc56003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 --2- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc560038760 0x7fc56003ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc564006fd0 tx=0x7fc564006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 msgr2=0x7fc57419bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 --2- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc57419bb50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc570006af0 tx=0x7fc570011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 shutdown_connections 2023-10-11T16:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 --2- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc560038760 0x7fc56003ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 --2- 172.21.15.111:0/4219305004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc574105a20 0x7fc57419bb50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 >> 172.21.15.111:0/4219305004 conn(0x7fc574100fd0 msgr2=0x7fc574102640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 shutdown_connections 2023-10-11T16:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:19.143+0000 7fc57bfd0700 1 -- 172.21.15.111:0/4219305004 wait complete. 2023-10-11T16:23:20.503 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:20 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:20.503 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:20 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:20.779 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv4rY7gmsg9O2tzRVG9zZJ1YZK32mb9wRcpFJ89gxtvzvPYHxNiwDM9SweS686KkE7t+zGqrfTIBi/mWBRNaI0RDWOII7Vy324q8VfP/1SdBUBNRBOk6AAKq6Gg/9NGkQq7q2O1zerwtbaxW76cGYAIlKKdDVHDC1ihuPCE1+If/xOTT3TK/kwwNdg4ihbRb4LUKHznRSvh4/yMm6uxpfEDBsy9h3KSc5zc5Sm2aohdqiFnYfLd34bVGTWRKPTUmmLkPyiUcs1cLihNCjo1DTxNHYcxXFVw+DoZYbPWri6/PpOCtL1MGSV1OlL8lLbr7Sl/nVmvcb7Z9JV8X3gSu8GgsSzAcdfoFoA57QShRlZqnW23kAlyUaki81A8g9hh+mSi6m57tgew2K/uQwmtF6h7Kj1eVPvqjK9xNZeygPEoC/p/AXk48YZtqADF5m2e+UMvyhTPy814ELOLmdBCZM50lYa6yiIKimblvsPocCgHW9LxfMxMHmfT7giOS+QQZM= ceph-495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:23:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.108+0000 7ff798828700 1 Processor -- start 2023-10-11T16:23:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.109+0000 7ff798828700 1 -- start start 2023-10-11T16:23:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.109+0000 7ff798828700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff7941066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:20.780 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.109+0000 7ff798828700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff794079e60 con 0x7ff794104280 2023-10-11T16:23:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.109+0000 7ff79259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff7941066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.109+0000 7ff79259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff7941066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39884/0 (socket says 172.21.15.111:39884) 2023-10-11T16:23:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.109+0000 7ff79259c700 1 -- 172.21.15.111:0/3587202390 learned_addr learned my addr 172.21.15.111:0/3587202390 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.110+0000 7ff79259c700 1 -- 172.21.15.111:0/3587202390 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff794106be0 con 0x7ff794104280 2023-10-11T16:23:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.110+0000 7ff79259c700 1 --2- 172.21.15.111:0/3587202390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff7941066a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7840098a0 tx=0x7ff784009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2e86db5277bcc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:20.781 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.110+0000 7ff79159a700 1 -- 172.21.15.111:0/3587202390 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff784004160 con 0x7ff794104280 2023-10-11T16:23:20.782 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.110+0000 7ff79159a700 1 -- 172.21.15.111:0/3587202390 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7840042c0 con 0x7ff794104280 2023-10-11T16:23:20.782 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.110+0000 7ff79159a700 1 -- 172.21.15.111:0/3587202390 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff784011a00 con 0x7ff794104280 2023-10-11T16:23:20.782 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.111+0000 7ff798828700 1 -- 172.21.15.111:0/3587202390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 msgr2=0x7ff7941066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:20.782 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.111+0000 7ff798828700 1 --2- 172.21.15.111:0/3587202390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff7941066a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7840098a0 tx=0x7ff784009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:20.783 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.111+0000 7ff798828700 1 -- 172.21.15.111:0/3587202390 shutdown_connections 2023-10-11T16:23:20.783 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.111+0000 7ff798828700 1 --2- 172.21.15.111:0/3587202390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff7941066a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:20.783 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.111+0000 7ff798828700 1 -- 172.21.15.111:0/3587202390 >> 172.21.15.111:0/3587202390 conn(0x7ff7940ffed0 msgr2=0x7ff794102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:20.783 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.111+0000 7ff798828700 1 -- 172.21.15.111:0/3587202390 shutdown_connections 2023-10-11T16:23:20.783 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.111+0000 7ff798828700 1 -- 172.21.15.111:0/3587202390 wait complete. 2023-10-11T16:23:20.784 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.112+0000 7ff798828700 1 Processor -- start 2023-10-11T16:23:20.784 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.112+0000 7ff798828700 1 -- start start 2023-10-11T16:23:20.784 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.112+0000 7ff798828700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff79419fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:20.784 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.112+0000 7ff798828700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7941a0290 con 0x7ff794104280 2023-10-11T16:23:20.784 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff79259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff79419fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:20.785 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff79259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff79419fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39894/0 (socket says 172.21.15.111:39894) 2023-10-11T16:23:20.785 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff79259c700 1 -- 172.21.15.111:0/827867801 learned_addr learned my addr 172.21.15.111:0/827867801 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:20.785 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff79259c700 1 -- 172.21.15.111:0/827867801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff784009550 con 0x7ff794104280 2023-10-11T16:23:20.785 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff79259c700 1 --2- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff79419fd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff784006af0 tx=0x7ff784011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:20.785 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff7837fe700 1 -- 172.21.15.111:0/827867801 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff784003a40 con 0x7ff794104280 2023-10-11T16:23:20.786 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff7837fe700 1 -- 172.21.15.111:0/827867801 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff784003ba0 con 0x7ff794104280 2023-10-11T16:23:20.786 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff7837fe700 1 -- 172.21.15.111:0/827867801 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff78401a6e0 con 0x7ff794104280 2023-10-11T16:23:20.786 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff794107be0 con 0x7ff794104280 2023-10-11T16:23:20.786 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.113+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff794107a50 con 0x7ff794104280 2023-10-11T16:23:20.786 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.114+0000 7ff7837fe700 1 -- 172.21.15.111:0/827867801 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff784024430 con 0x7ff794104280 2023-10-11T16:23:20.787 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.114+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff794062120 con 0x7ff794104280 2023-10-11T16:23:20.787 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.115+0000 7ff7837fe700 1 --2- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7ff77c038760 0x7ff77c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:20.787 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.115+0000 7ff7837fe700 1 -- 172.21.15.111:0/827867801 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff78404c7b0 con 0x7ff794104280 2023-10-11T16:23:20.787 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.115+0000 7ff791d9b700 1 --2- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7ff77c038760 0x7ff77c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:20.787 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.116+0000 7ff791d9b700 1 --2- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7ff77c038760 0x7ff77c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff788006fd0 tx=0x7ff788006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:20.788 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.120+0000 7ff7837fe700 1 -- 172.21.15.111:0/827867801 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff7840246e0 con 0x7ff794104280 2023-10-11T16:23:20.788 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.318+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff79410a0b0 con 0x7ff77c038760 2023-10-11T16:23:20.788 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.318+0000 7ff7837fe700 1 -- 172.21.15.111:0/827867801 <== mgr.14116 v2:172.21.15.111:6800/4086392563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff79410a0b0 con 0x7ff77c038760 2023-10-11T16:23:20.788 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.320+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7ff77c038760 msgr2=0x7ff77c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:20.788 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.320+0000 7ff798828700 1 --2- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7ff77c038760 0x7ff77c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff788006fd0 tx=0x7ff788006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:20.789 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.320+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 msgr2=0x7ff79419fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:20.789 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.320+0000 7ff798828700 1 --2- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff79419fd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff784006af0 tx=0x7ff784011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:20.789 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.321+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 shutdown_connections 2023-10-11T16:23:20.789 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.321+0000 7ff798828700 1 --2- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7ff77c038760 0x7ff77c03ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:20.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.321+0000 7ff798828700 1 --2- 172.21.15.111:0/827867801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff794104280 0x7ff79419fd50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:20.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.321+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 >> 172.21.15.111:0/827867801 conn(0x7ff7940ffed0 msgr2=0x7ff794102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:20.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.321+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 shutdown_connections 2023-10-11T16:23:20.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:20.321+0000 7ff798828700 1 -- 172.21.15.111:0/827867801 wait complete. 2023-10-11T16:23:20.791 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-10-11T16:23:20.791 INFO:teuthology.orchestra.run.smithi111.stdout:Adding key to root@localhost authorized_keys... 2023-10-11T16:23:20.791 INFO:teuthology.orchestra.run.smithi111.stdout:Adding host smithi111... 2023-10-11T16:23:21.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:21 smithi111 ceph-mon[98527]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:22.170 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:22 smithi111 ceph-mon[98527]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:23.528 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:23 smithi111 ceph-mon[98527]: Deploying cephadm binary to smithi111 2023-10-11T16:23:24.767 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Added host 'smithi111' with addr '172.21.15.111' 2023-10-11T16:23:24.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.276+0000 7f0d967e3700 1 Processor -- start 2023-10-11T16:23:24.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.277+0000 7f0d967e3700 1 -- start start 2023-10-11T16:23:24.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.277+0000 7f0d967e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9007a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:24.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.277+0000 7f0d967e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9007a880 con 0x7f0d9007be50 2023-10-11T16:23:24.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.278+0000 7f0d8ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9007a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:24.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.278+0000 7f0d8ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9007a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39898/0 (socket says 172.21.15.111:39898) 2023-10-11T16:23:24.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.278+0000 7f0d8ffff700 1 -- 172.21.15.111:0/1523128414 learned_addr learned my addr 172.21.15.111:0/1523128414 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:24.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.278+0000 7f0d8ffff700 1 -- 172.21.15.111:0/1523128414 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9007a9c0 con 0x7f0d9007be50 2023-10-11T16:23:24.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.278+0000 7f0d8ffff700 1 --2- 172.21.15.111:0/1523128414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9007a2b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d800098a0 tx=0x7f0d80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e59df75237c8d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:24.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.278+0000 7f0d8effd700 1 -- 172.21.15.111:0/1523128414 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d80004160 con 0x7f0d9007be50 2023-10-11T16:23:24.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.278+0000 7f0d8effd700 1 -- 172.21.15.111:0/1523128414 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d800042c0 con 0x7f0d9007be50 2023-10-11T16:23:24.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.278+0000 7f0d8effd700 1 -- 172.21.15.111:0/1523128414 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d80011a00 con 0x7f0d9007be50 2023-10-11T16:23:24.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.279+0000 7f0d967e3700 1 -- 172.21.15.111:0/1523128414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 msgr2=0x7f0d9007a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:24.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.279+0000 7f0d967e3700 1 --2- 172.21.15.111:0/1523128414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9007a2b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d800098a0 tx=0x7f0d80009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:24.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.279+0000 7f0d967e3700 1 -- 172.21.15.111:0/1523128414 shutdown_connections 2023-10-11T16:23:24.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.279+0000 7f0d967e3700 1 --2- 172.21.15.111:0/1523128414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9007a2b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:24.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.279+0000 7f0d967e3700 1 -- 172.21.15.111:0/1523128414 >> 172.21.15.111:0/1523128414 conn(0x7f0d90100fd0 msgr2=0x7f0d90103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:24.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.279+0000 7f0d967e3700 1 -- 172.21.15.111:0/1523128414 shutdown_connections 2023-10-11T16:23:24.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.279+0000 7f0d967e3700 1 -- 172.21.15.111:0/1523128414 wait complete. 2023-10-11T16:23:24.772 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.280+0000 7f0d967e3700 1 Processor -- start 2023-10-11T16:23:24.772 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.280+0000 7f0d967e3700 1 -- start start 2023-10-11T16:23:24.772 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.281+0000 7f0d967e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9019b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:24.772 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.281+0000 7f0d967e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9019bec0 con 0x7f0d9007be50 2023-10-11T16:23:24.772 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.281+0000 7f0d8ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9019b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:24.773 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.281+0000 7f0d8ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9019b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39902/0 (socket says 172.21.15.111:39902) 2023-10-11T16:23:24.773 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.281+0000 7f0d8ffff700 1 -- 172.21.15.111:0/487163854 learned_addr learned my addr 172.21.15.111:0/487163854 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:24.773 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.281+0000 7f0d8ffff700 1 -- 172.21.15.111:0/487163854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80009550 con 0x7f0d9007be50 2023-10-11T16:23:24.773 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.281+0000 7f0d8ffff700 1 --2- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9019b980 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0d80006af0 tx=0x7f0d80011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:24.773 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.282+0000 7f0d8d7fa700 1 -- 172.21.15.111:0/487163854 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d80003a40 con 0x7f0d9007be50 2023-10-11T16:23:24.774 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.282+0000 7f0d8d7fa700 1 -- 172.21.15.111:0/487163854 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d80003ba0 con 0x7f0d9007be50 2023-10-11T16:23:24.774 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.282+0000 7f0d8d7fa700 1 -- 172.21.15.111:0/487163854 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d8001a6e0 con 0x7f0d9007be50 2023-10-11T16:23:24.774 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.282+0000 7f0d967e3700 1 -- 172.21.15.111:0/487163854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9010bf80 con 0x7f0d9007be50 2023-10-11T16:23:24.774 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.282+0000 7f0d967e3700 1 -- 172.21.15.111:0/487163854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9010bdf0 con 0x7f0d9007be50 2023-10-11T16:23:24.774 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.283+0000 7f0d8d7fa700 1 -- 172.21.15.111:0/487163854 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f0d80024430 con 0x7f0d9007be50 2023-10-11T16:23:24.775 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.283+0000 7f0d76ffd700 1 -- 172.21.15.111:0/487163854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d90062120 con 0x7f0d9007be50 2023-10-11T16:23:24.775 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.283+0000 7f0d8d7fa700 1 --2- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f0d78038760 0x7f0d7803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:24.775 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.283+0000 7f0d8d7fa700 1 -- 172.21.15.111:0/487163854 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0d8004c880 con 0x7f0d9007be50 2023-10-11T16:23:24.775 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.284+0000 7f0d8f7fe700 1 --2- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f0d78038760 0x7f0d7803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:24.775 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.284+0000 7f0d8f7fe700 1 --2- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f0d78038760 0x7f0d7803ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0d84006fd0 tx=0x7f0d84006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:24.776 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.288+0000 7f0d8d7fa700 1 -- 172.21.15.111:0/487163854 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0d800246e0 con 0x7f0d9007be50 2023-10-11T16:23:24.776 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:21.489+0000 7f0d76ffd700 1 -- 172.21.15.111:0/487163854 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d9019c400 con 0x7f0d78038760 2023-10-11T16:23:24.776 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.442+0000 7f0d8d7fa700 1 -- 172.21.15.111:0/487163854 <== mgr.14116 v2:172.21.15.111:6800/4086392563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0d9019c400 con 0x7f0d78038760 2023-10-11T16:23:24.776 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.444+0000 7f0d76ffd700 1 -- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f0d78038760 msgr2=0x7f0d7803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:24.776 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.444+0000 7f0d76ffd700 1 --2- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f0d78038760 0x7f0d7803ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0d84006fd0 tx=0x7f0d84006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:24.777 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.445+0000 7f0d76ffd700 1 -- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 msgr2=0x7f0d9019b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:24.777 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.445+0000 7f0d76ffd700 1 --2- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9019b980 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0d80006af0 tx=0x7f0d80011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:24.777 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.445+0000 7f0d76ffd700 1 -- 172.21.15.111:0/487163854 shutdown_connections 2023-10-11T16:23:24.777 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.445+0000 7f0d76ffd700 1 --2- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f0d78038760 0x7f0d7803ac10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:24.777 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.445+0000 7f0d76ffd700 1 --2- 172.21.15.111:0/487163854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9007be50 0x7f0d9019b980 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:24.778 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.445+0000 7f0d76ffd700 1 -- 172.21.15.111:0/487163854 >> 172.21.15.111:0/487163854 conn(0x7f0d90100fd0 msgr2=0x7f0d901026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:24.778 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.445+0000 7f0d76ffd700 1 -- 172.21.15.111:0/487163854 shutdown_connections 2023-10-11T16:23:24.778 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:24.445+0000 7f0d76ffd700 1 -- 172.21.15.111:0/487163854 wait complete. 2023-10-11T16:23:24.778 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying unmanaged mon service... 2023-10-11T16:23:25.711 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:25 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:25.712 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:25 smithi111 ceph-mon[98527]: Added host smithi111 2023-10-11T16:23:25.712 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:25 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:26.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-10-11T16:23:26.098 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.296+0000 7fc82aebf700 1 Processor -- start 2023-10-11T16:23:26.098 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.296+0000 7fc82aebf700 1 -- start start 2023-10-11T16:23:26.098 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.296+0000 7fc82aebf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc824108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:26.099 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.296+0000 7fc82aebf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc824108600 con 0x7fc824107c20 2023-10-11T16:23:26.099 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.296+0000 7fc828c5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc824108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:26.099 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.296+0000 7fc828c5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc824108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48652/0 (socket says 172.21.15.111:48652) 2023-10-11T16:23:26.099 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.296+0000 7fc828c5b700 1 -- 172.21.15.111:0/1274143509 learned_addr learned my addr 172.21.15.111:0/1274143509 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:26.100 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.297+0000 7fc828c5b700 1 -- 172.21.15.111:0/1274143509 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc824108e20 con 0x7fc824107c20 2023-10-11T16:23:26.100 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.297+0000 7fc828c5b700 1 --2- 172.21.15.111:0/1274143509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc824108030 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8180098a0 tx=0x7fc818009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4593183c64dbde server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:26.100 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.297+0000 7fc8237fe700 1 -- 172.21.15.111:0/1274143509 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc818004160 con 0x7fc824107c20 2023-10-11T16:23:26.100 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.297+0000 7fc8237fe700 1 -- 172.21.15.111:0/1274143509 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8180042c0 con 0x7fc824107c20 2023-10-11T16:23:26.100 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.297+0000 7fc8237fe700 1 -- 172.21.15.111:0/1274143509 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc818011a00 con 0x7fc824107c20 2023-10-11T16:23:26.101 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.298+0000 7fc82aebf700 1 -- 172.21.15.111:0/1274143509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 msgr2=0x7fc824108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:26.101 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.298+0000 7fc82aebf700 1 --2- 172.21.15.111:0/1274143509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc824108030 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc8180098a0 tx=0x7fc818009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:26.101 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.298+0000 7fc82aebf700 1 -- 172.21.15.111:0/1274143509 shutdown_connections 2023-10-11T16:23:26.101 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.298+0000 7fc82aebf700 1 --2- 172.21.15.111:0/1274143509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc824108030 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:26.101 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.298+0000 7fc82aebf700 1 -- 172.21.15.111:0/1274143509 >> 172.21.15.111:0/1274143509 conn(0x7fc82407b000 msgr2=0x7fc82407b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:26.102 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.298+0000 7fc82aebf700 1 -- 172.21.15.111:0/1274143509 shutdown_connections 2023-10-11T16:23:26.102 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.298+0000 7fc82aebf700 1 -- 172.21.15.111:0/1274143509 wait complete. 2023-10-11T16:23:26.102 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.299+0000 7fc82aebf700 1 Processor -- start 2023-10-11T16:23:26.102 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.299+0000 7fc82aebf700 1 -- start start 2023-10-11T16:23:26.102 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.300+0000 7fc82aebf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc82419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:26.103 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.300+0000 7fc82aebf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82419c070 con 0x7fc824107c20 2023-10-11T16:23:26.103 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.300+0000 7fc828c5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc82419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:26.103 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.300+0000 7fc828c5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc82419bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48668/0 (socket says 172.21.15.111:48668) 2023-10-11T16:23:26.103 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.300+0000 7fc828c5b700 1 -- 172.21.15.111:0/2262956726 learned_addr learned my addr 172.21.15.111:0/2262956726 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:26.104 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.300+0000 7fc828c5b700 1 -- 172.21.15.111:0/2262956726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc818009550 con 0x7fc824107c20 2023-10-11T16:23:26.104 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.300+0000 7fc828c5b700 1 --2- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc82419bb30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc818006af0 tx=0x7fc818011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:26.104 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.300+0000 7fc821ffb700 1 -- 172.21.15.111:0/2262956726 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc818003a40 con 0x7fc824107c20 2023-10-11T16:23:26.104 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.301+0000 7fc821ffb700 1 -- 172.21.15.111:0/2262956726 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc818003ba0 con 0x7fc824107c20 2023-10-11T16:23:26.104 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.301+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82410c050 con 0x7fc824107c20 2023-10-11T16:23:26.105 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.301+0000 7fc821ffb700 1 -- 172.21.15.111:0/2262956726 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc81801a700 con 0x7fc824107c20 2023-10-11T16:23:26.105 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.301+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82410c1e0 con 0x7fc824107c20 2023-10-11T16:23:26.105 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.302+0000 7fc821ffb700 1 -- 172.21.15.111:0/2262956726 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc81801a930 con 0x7fc824107c20 2023-10-11T16:23:26.105 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.302+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82410c370 con 0x7fc824107c20 2023-10-11T16:23:26.105 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.302+0000 7fc821ffb700 1 --2- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc814038760 0x7fc81403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:26.106 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.302+0000 7fc821ffb700 1 -- 172.21.15.111:0/2262956726 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc81804c8e0 con 0x7fc824107c20 2023-10-11T16:23:26.106 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.302+0000 7fc823fff700 1 --2- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc814038760 0x7fc81403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:26.106 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.303+0000 7fc823fff700 1 --2- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc814038760 0x7fc81403ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc810006fd0 tx=0x7fc810006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:26.106 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.307+0000 7fc821ffb700 1 -- 172.21.15.111:0/2262956726 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc818018b80 con 0x7fc824107c20 2023-10-11T16:23:26.107 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.525+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc824105f50 con 0x7fc814038760 2023-10-11T16:23:26.107 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.530+0000 7fc821ffb700 1 -- 172.21.15.111:0/2262956726 <== mgr.14116 v2:172.21.15.111:6800/4086392563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc824105f50 con 0x7fc814038760 2023-10-11T16:23:26.107 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc814038760 msgr2=0x7fc81403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:26.107 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 --2- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc814038760 0x7fc81403ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc810006fd0 tx=0x7fc810006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:26.107 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 msgr2=0x7fc82419bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:26.108 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 --2- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc82419bb30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc818006af0 tx=0x7fc818011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:26.108 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 shutdown_connections 2023-10-11T16:23:26.108 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 --2- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7fc814038760 0x7fc81403ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:26.108 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 --2- 172.21.15.111:0/2262956726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc824107c20 0x7fc82419bb30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:26.109 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 >> 172.21.15.111:0/2262956726 conn(0x7fc82407b000 msgr2=0x7fc824105840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:26.109 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 shutdown_connections 2023-10-11T16:23:26.109 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:25.532+0000 7fc82aebf700 1 -- 172.21.15.111:0/2262956726 wait complete. 2023-10-11T16:23:26.109 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying unmanaged mgr service... 2023-10-11T16:23:26.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:26 smithi111 ceph-mon[98527]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:26.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:26 smithi111 ceph-mon[98527]: Saving service mon spec with placement count:5 2023-10-11T16:23:26.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:26 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:27.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-10-11T16:23:27.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.794+0000 7f7e8a0e0700 1 Processor -- start 2023-10-11T16:23:27.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.795+0000 7f7e8a0e0700 1 -- start start 2023-10-11T16:23:27.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.795+0000 7f7e8a0e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8407a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:27.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.795+0000 7f7e8a0e0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8407a860 con 0x7f7e8407be30 2023-10-11T16:23:27.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.796+0000 7f7e837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8407a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:27.487 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.796+0000 7f7e837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8407a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48672/0 (socket says 172.21.15.111:48672) 2023-10-11T16:23:27.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.796+0000 7f7e837fe700 1 -- 172.21.15.111:0/2581019221 learned_addr learned my addr 172.21.15.111:0/2581019221 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:27.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.796+0000 7f7e837fe700 1 -- 172.21.15.111:0/2581019221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8407a9a0 con 0x7f7e8407be30 2023-10-11T16:23:27.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.796+0000 7f7e837fe700 1 --2- 172.21.15.111:0/2581019221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8407a290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e740098a0 tx=0x7f7e74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f46b439f8e4c13 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:27.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.796+0000 7f7e827fc700 1 -- 172.21.15.111:0/2581019221 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e74004160 con 0x7f7e8407be30 2023-10-11T16:23:27.488 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e827fc700 1 -- 172.21.15.111:0/2581019221 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e740042c0 con 0x7f7e8407be30 2023-10-11T16:23:27.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e827fc700 1 -- 172.21.15.111:0/2581019221 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e74011a00 con 0x7f7e8407be30 2023-10-11T16:23:27.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/2581019221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 msgr2=0x7f7e8407a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:27.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e8a0e0700 1 --2- 172.21.15.111:0/2581019221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8407a290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e740098a0 tx=0x7f7e74009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:27.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/2581019221 shutdown_connections 2023-10-11T16:23:27.489 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e8a0e0700 1 --2- 172.21.15.111:0/2581019221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8407a290 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:27.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/2581019221 >> 172.21.15.111:0/2581019221 conn(0x7f7e84100fd0 msgr2=0x7f7e84103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:27.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/2581019221 shutdown_connections 2023-10-11T16:23:27.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.797+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/2581019221 wait complete. 2023-10-11T16:23:27.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.798+0000 7f7e8a0e0700 1 Processor -- start 2023-10-11T16:23:27.490 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.798+0000 7f7e8a0e0700 1 -- start start 2023-10-11T16:23:27.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.799+0000 7f7e8a0e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:27.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.799+0000 7f7e8a0e0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8419be70 con 0x7f7e8407be30 2023-10-11T16:23:27.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.799+0000 7f7e837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:27.491 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.799+0000 7f7e837fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8419b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48682/0 (socket says 172.21.15.111:48682) 2023-10-11T16:23:27.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.799+0000 7f7e837fe700 1 -- 172.21.15.111:0/1211326876 learned_addr learned my addr 172.21.15.111:0/1211326876 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:27.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.799+0000 7f7e837fe700 1 -- 172.21.15.111:0/1211326876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74009550 con 0x7f7e8407be30 2023-10-11T16:23:27.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.800+0000 7f7e837fe700 1 --2- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8419b930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e74006af0 tx=0x7f7e74011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:27.492 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.800+0000 7f7e80ff9700 1 -- 172.21.15.111:0/1211326876 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e74003a40 con 0x7f7e8407be30 2023-10-11T16:23:27.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.800+0000 7f7e80ff9700 1 -- 172.21.15.111:0/1211326876 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e74003ba0 con 0x7f7e8407be30 2023-10-11T16:23:27.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.800+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8410bf40 con 0x7f7e8407be30 2023-10-11T16:23:27.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.800+0000 7f7e80ff9700 1 -- 172.21.15.111:0/1211326876 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e7401a6e0 con 0x7f7e8407be30 2023-10-11T16:23:27.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.800+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8410bdb0 con 0x7f7e8407be30 2023-10-11T16:23:27.493 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.801+0000 7f7e80ff9700 1 -- 172.21.15.111:0/1211326876 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f7e74024430 con 0x7f7e8407be30 2023-10-11T16:23:27.494 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.801+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e84062120 con 0x7f7e8407be30 2023-10-11T16:23:27.494 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.802+0000 7f7e80ff9700 1 --2- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f7e6c038760 0x7f7e6c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:27.494 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.802+0000 7f7e80ff9700 1 -- 172.21.15.111:0/1211326876 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7e7404c7e0 con 0x7f7e8407be30 2023-10-11T16:23:27.494 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.802+0000 7f7e82ffd700 1 --2- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f7e6c038760 0x7f7e6c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:27.494 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.803+0000 7f7e82ffd700 1 --2- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f7e6c038760 0x7f7e6c03ac10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e78006fd0 tx=0x7f7e78006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:27.495 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:26.807+0000 7f7e80ff9700 1 -- 172.21.15.111:0/1211326876 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7e740246e0 con 0x7f7e8407be30 2023-10-11T16:23:27.495 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.012+0000 7f7e80ff9700 1 -- 172.21.15.111:0/1211326876 <== mon.0 v2:172.21.15.111:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e740246e0 con 0x7f7e8407be30 2023-10-11T16:23:27.495 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.031+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7e84103300 con 0x7f7e6c038760 2023-10-11T16:23:27.495 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.039+0000 7f7e80ff9700 1 -- 172.21.15.111:0/1211326876 <== mgr.14116 v2:172.21.15.111:6800/4086392563 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7e84103300 con 0x7f7e6c038760 2023-10-11T16:23:27.495 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.041+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f7e6c038760 msgr2=0x7f7e6c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:27.497 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.041+0000 7f7e8a0e0700 1 --2- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f7e6c038760 0x7f7e6c03ac10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e78006fd0 tx=0x7f7e78006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:27.498 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.041+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 msgr2=0x7f7e8419b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:27.498 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.041+0000 7f7e8a0e0700 1 --2- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8419b930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e74006af0 tx=0x7f7e74011ad0 comp rx=0 tx=0).stop 2023-10-11T16:23:27.498 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.041+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 shutdown_connections 2023-10-11T16:23:27.498 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.041+0000 7f7e8a0e0700 1 --2- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f7e6c038760 0x7f7e6c03ac10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:27.499 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.041+0000 7f7e8a0e0700 1 --2- 172.21.15.111:0/1211326876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e8407be30 0x7f7e8419b930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:27.499 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.041+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 >> 172.21.15.111:0/1211326876 conn(0x7f7e84100fd0 msgr2=0x7f7e841026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:27.499 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.042+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 shutdown_connections 2023-10-11T16:23:27.499 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.042+0000 7f7e8a0e0700 1 -- 172.21.15.111:0/1211326876 wait complete. 2023-10-11T16:23:28.361 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:28 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:28.361 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:28 smithi111 ceph-mon[98527]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:28.362 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:28 smithi111 ceph-mon[98527]: Saving service mgr spec with placement count:2 2023-10-11T16:23:28.362 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:28 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:28.362 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:28 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:28.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.984+0000 7f5a0b999700 1 Processor -- start 2023-10-11T16:23:28.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.984+0000 7f5a0b999700 1 -- start start 2023-10-11T16:23:28.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.985+0000 7f5a0b999700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0407a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:28.667 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.985+0000 7f5a0b999700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0407a860 con 0x7f5a0407be30 2023-10-11T16:23:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.985+0000 7f5a09735700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0407a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.985+0000 7f5a09735700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0407a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48696/0 (socket says 172.21.15.111:48696) 2023-10-11T16:23:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.985+0000 7f5a09735700 1 -- 172.21.15.111:0/1607314276 learned_addr learned my addr 172.21.15.111:0/1607314276 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.985+0000 7f5a09735700 1 -- 172.21.15.111:0/1607314276 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0407a9a0 con 0x7f5a0407be30 2023-10-11T16:23:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.986+0000 7f5a09735700 1 --2- 172.21.15.111:0/1607314276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0407a290 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a000098a0 tx=0x7f5a00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca82524439c7f3de server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.986+0000 7f59fbfff700 1 -- 172.21.15.111:0/1607314276 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a00004160 con 0x7f5a0407be30 2023-10-11T16:23:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.986+0000 7f59fbfff700 1 -- 172.21.15.111:0/1607314276 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a000042c0 con 0x7f5a0407be30 2023-10-11T16:23:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.986+0000 7f59fbfff700 1 -- 172.21.15.111:0/1607314276 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a000119b0 con 0x7f5a0407be30 2023-10-11T16:23:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.987+0000 7f5a0b999700 1 -- 172.21.15.111:0/1607314276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 msgr2=0x7f5a0407a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.987+0000 7f5a0b999700 1 --2- 172.21.15.111:0/1607314276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0407a290 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5a000098a0 tx=0x7f5a00009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:28.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.987+0000 7f5a0b999700 1 -- 172.21.15.111:0/1607314276 shutdown_connections 2023-10-11T16:23:28.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.987+0000 7f5a0b999700 1 --2- 172.21.15.111:0/1607314276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0407a290 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:28.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.987+0000 7f5a0b999700 1 -- 172.21.15.111:0/1607314276 >> 172.21.15.111:0/1607314276 conn(0x7f5a04100fd0 msgr2=0x7f5a04103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:28.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.987+0000 7f5a0b999700 1 -- 172.21.15.111:0/1607314276 shutdown_connections 2023-10-11T16:23:28.670 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.987+0000 7f5a0b999700 1 -- 172.21.15.111:0/1607314276 wait complete. 2023-10-11T16:23:28.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.988+0000 7f5a0b999700 1 Processor -- start 2023-10-11T16:23:28.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.988+0000 7f5a0b999700 1 -- start start 2023-10-11T16:23:28.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.988+0000 7f5a0b999700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0419b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:28.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.988+0000 7f5a0b999700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0419bed0 con 0x7f5a0407be30 2023-10-11T16:23:28.671 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f5a09735700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0419b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:28.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f5a09735700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0419b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48700/0 (socket says 172.21.15.111:48700) 2023-10-11T16:23:28.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f5a09735700 1 -- 172.21.15.111:0/603566305 learned_addr learned my addr 172.21.15.111:0/603566305 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:28.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f5a09735700 1 -- 172.21.15.111:0/603566305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a00009550 con 0x7f5a0407be30 2023-10-11T16:23:28.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f5a09735700 1 --2- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0419b990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5a00011980 tx=0x7f5a00011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:28.672 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f59fa7fc700 1 -- 172.21.15.111:0/603566305 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a00003a60 con 0x7f5a0407be30 2023-10-11T16:23:28.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f59fa7fc700 1 -- 172.21.15.111:0/603566305 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a00003bc0 con 0x7f5a0407be30 2023-10-11T16:23:28.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f59fa7fc700 1 -- 172.21.15.111:0/603566305 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a0001a670 con 0x7f5a0407be30 2023-10-11T16:23:28.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.989+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0410bf90 con 0x7f5a0407be30 2023-10-11T16:23:28.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.990+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0410be00 con 0x7f5a0407be30 2023-10-11T16:23:28.673 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.991+0000 7f59fa7fc700 1 -- 172.21.15.111:0/603566305 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5a00024a00 con 0x7f5a0407be30 2023-10-11T16:23:28.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.991+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0410c120 con 0x7f5a0407be30 2023-10-11T16:23:28.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.991+0000 7f59fa7fc700 1 --2- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f59f0038710 0x7f59f003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:28.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.991+0000 7f59fa7fc700 1 -- 172.21.15.111:0/603566305 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5a0004c8f0 con 0x7f5a0407be30 2023-10-11T16:23:28.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.991+0000 7f5a08f34700 1 --2- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f59f0038710 0x7f59f003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:28.674 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:27.992+0000 7f5a08f34700 1 --2- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f59f0038710 0x7f59f003abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f59f4006fd0 tx=0x7f59f4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:28.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.000+0000 7f59fa7fc700 1 -- 172.21.15.111:0/603566305 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5a00018b80 con 0x7f5a0407be30 2023-10-11T16:23:28.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.208+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5a0419f5a0 con 0x7f5a0407be30 2023-10-11T16:23:28.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.214+0000 7f59fa7fc700 1 -- 172.21.15.111:0/603566305 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5a00018350 con 0x7f5a0407be30 2023-10-11T16:23:28.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f59f0038710 msgr2=0x7f59f003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:28.675 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 --2- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f59f0038710 0x7f59f003abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f59f4006fd0 tx=0x7f59f4006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:28.676 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 msgr2=0x7f5a0419b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:28.676 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 --2- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0419b990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5a00011980 tx=0x7f5a00011a60 comp rx=0 tx=0).stop 2023-10-11T16:23:28.676 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 shutdown_connections 2023-10-11T16:23:28.676 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 --2- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f59f0038710 0x7f59f003abc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:28.676 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 --2- 172.21.15.111:0/603566305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a0407be30 0x7f5a0419b990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:28.677 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 >> 172.21.15.111:0/603566305 conn(0x7f5a04100fd0 msgr2=0x7f5a04102710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:28.677 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 shutdown_connections 2023-10-11T16:23:28.677 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:28.217+0000 7f5a0b999700 1 -- 172.21.15.111:0/603566305 wait complete. 2023-10-11T16:23:29.472 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:29 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/603566305' entity='client.admin' 2023-10-11T16:23:29.954 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.213+0000 7f25da988700 1 Processor -- start 2023-10-11T16:23:29.955 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.214+0000 7f25da988700 1 -- start start 2023-10-11T16:23:29.955 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.214+0000 7f25da988700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d407a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:29.956 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.214+0000 7f25da988700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d407a860 con 0x7f25d407be30 2023-10-11T16:23:29.956 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.214+0000 7f25d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d407a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:29.956 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.214+0000 7f25d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d407a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48710/0 (socket says 172.21.15.111:48710) 2023-10-11T16:23:29.957 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.214+0000 7f25d3fff700 1 -- 172.21.15.111:0/1428291412 learned_addr learned my addr 172.21.15.111:0/1428291412 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:29.957 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.215+0000 7f25d3fff700 1 -- 172.21.15.111:0/1428291412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d407a9a0 con 0x7f25d407be30 2023-10-11T16:23:29.957 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.215+0000 7f25d3fff700 1 --2- 172.21.15.111:0/1428291412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d407a290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f25c40098a0 tx=0x7f25c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c3df515be5a96c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:29.958 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.217+0000 7f25d2ffd700 1 -- 172.21.15.111:0/1428291412 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25c4004160 con 0x7f25d407be30 2023-10-11T16:23:29.958 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.217+0000 7f25d2ffd700 1 -- 172.21.15.111:0/1428291412 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25c40042c0 con 0x7f25d407be30 2023-10-11T16:23:29.959 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.217+0000 7f25d2ffd700 1 -- 172.21.15.111:0/1428291412 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25c40119b0 con 0x7f25d407be30 2023-10-11T16:23:29.959 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.217+0000 7f25da988700 1 -- 172.21.15.111:0/1428291412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 msgr2=0x7f25d407a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:29.959 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.217+0000 7f25da988700 1 --2- 172.21.15.111:0/1428291412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d407a290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f25c40098a0 tx=0x7f25c4009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:29.960 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.218+0000 7f25da988700 1 -- 172.21.15.111:0/1428291412 shutdown_connections 2023-10-11T16:23:29.960 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.218+0000 7f25da988700 1 --2- 172.21.15.111:0/1428291412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d407a290 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:29.960 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.218+0000 7f25da988700 1 -- 172.21.15.111:0/1428291412 >> 172.21.15.111:0/1428291412 conn(0x7f25d4100fd0 msgr2=0x7f25d4103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:29.961 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.218+0000 7f25da988700 1 -- 172.21.15.111:0/1428291412 shutdown_connections 2023-10-11T16:23:29.961 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.218+0000 7f25da988700 1 -- 172.21.15.111:0/1428291412 wait complete. 2023-10-11T16:23:29.961 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.219+0000 7f25da988700 1 Processor -- start 2023-10-11T16:23:29.962 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.219+0000 7f25da988700 1 -- start start 2023-10-11T16:23:29.962 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.219+0000 7f25da988700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:29.962 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.219+0000 7f25da988700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d419be90 con 0x7f25d407be30 2023-10-11T16:23:29.963 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.219+0000 7f25d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:29.963 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.219+0000 7f25d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d419b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48714/0 (socket says 172.21.15.111:48714) 2023-10-11T16:23:29.963 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.219+0000 7f25d3fff700 1 -- 172.21.15.111:0/1275955291 learned_addr learned my addr 172.21.15.111:0/1275955291 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:29.964 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.220+0000 7f25d3fff700 1 -- 172.21.15.111:0/1275955291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c4009550 con 0x7f25d407be30 2023-10-11T16:23:29.964 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.220+0000 7f25d3fff700 1 --2- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d419b950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f25c4011980 tx=0x7f25c4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:29.965 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.220+0000 7f25d17fa700 1 -- 172.21.15.111:0/1275955291 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25c4003a60 con 0x7f25d407be30 2023-10-11T16:23:29.965 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.220+0000 7f25d17fa700 1 -- 172.21.15.111:0/1275955291 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25c4003bc0 con 0x7f25d407be30 2023-10-11T16:23:29.965 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.220+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d410bf90 con 0x7f25d407be30 2023-10-11T16:23:29.966 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.220+0000 7f25d17fa700 1 -- 172.21.15.111:0/1275955291 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25c401a670 con 0x7f25d407be30 2023-10-11T16:23:29.966 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.220+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d410be00 con 0x7f25d407be30 2023-10-11T16:23:29.966 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.221+0000 7f25d17fa700 1 -- 172.21.15.111:0/1275955291 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f25c4024440 con 0x7f25d407be30 2023-10-11T16:23:29.967 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.221+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d410c120 con 0x7f25d407be30 2023-10-11T16:23:29.967 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.222+0000 7f25d17fa700 1 --2- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f25bc038760 0x7f25bc03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:29.968 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.222+0000 7f25d17fa700 1 -- 172.21.15.111:0/1275955291 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f25c404c8d0 con 0x7f25d407be30 2023-10-11T16:23:29.968 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.222+0000 7f25d37fe700 1 --2- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f25bc038760 0x7f25bc03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:29.968 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.223+0000 7f25d37fe700 1 --2- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f25bc038760 0x7f25bc03ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f25c8006fd0 tx=0x7f25c8006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.226+0000 7f25d17fa700 1 -- 172.21.15.111:0/1275955291 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f25c40246f0 con 0x7f25d407be30 2023-10-11T16:23:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.439+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f25d40621a0 con 0x7f25d407be30 2023-10-11T16:23:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.445+0000 7f25d17fa700 1 -- 172.21.15.111:0/1275955291 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f25c4018350 con 0x7f25d407be30 2023-10-11T16:23:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f25bc038760 msgr2=0x7f25bc03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 --2- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f25bc038760 0x7f25bc03ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f25c8006fd0 tx=0x7f25c8006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 msgr2=0x7f25d419b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 --2- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d419b950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f25c4011980 tx=0x7f25c4011a60 comp rx=0 tx=0).stop 2023-10-11T16:23:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 shutdown_connections 2023-10-11T16:23:29.972 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 --2- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f25bc038760 0x7f25bc03ac10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:29.972 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 --2- 172.21.15.111:0/1275955291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25d407be30 0x7f25d419b950 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f25c4011980 tx=0x7f25c4011a60 comp rx=0 tx=0).stop 2023-10-11T16:23:29.972 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 >> 172.21.15.111:0/1275955291 conn(0x7f25d4100fd0 msgr2=0x7f25d41026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:29.973 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 shutdown_connections 2023-10-11T16:23:29.973 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:29.448+0000 7f25da988700 1 -- 172.21.15.111:0/1275955291 wait complete. 2023-10-11T16:23:29.973 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling the dashboard module... 2023-10-11T16:23:30.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:30 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1275955291' entity='client.admin' 2023-10-11T16:23:30.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:30 smithi111 ceph-mon[98527]: from='mgr.14116 172.21.15.111:0/2973830112' entity='mgr.x' 2023-10-11T16:23:31.776 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:31 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1149878119' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-10-11T16:23:31.777 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:31 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: ignoring --setuser ceph since I am not root 2023-10-11T16:23:31.777 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:31 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: ignoring --setgroup ceph since I am not root 2023-10-11T16:23:31.938 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.480+0000 7f8fbaf6f700 1 Processor -- start 2023-10-11T16:23:31.938 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.481+0000 7f8fbaf6f700 1 -- start start 2023-10-11T16:23:31.938 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.481+0000 7f8fbaf6f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb407a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:31.938 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.481+0000 7f8fbaf6f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb407a860 con 0x7f8fb407be30 2023-10-11T16:23:31.939 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.481+0000 7f8fb8d0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb407a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:31.939 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.481+0000 7f8fb8d0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb407a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48720/0 (socket says 172.21.15.111:48720) 2023-10-11T16:23:31.939 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.481+0000 7f8fb8d0b700 1 -- 172.21.15.111:0/229047737 learned_addr learned my addr 172.21.15.111:0/229047737 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:31.939 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.482+0000 7f8fb8d0b700 1 -- 172.21.15.111:0/229047737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb407a9a0 con 0x7f8fb407be30 2023-10-11T16:23:31.939 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.482+0000 7f8fb8d0b700 1 --2- 172.21.15.111:0/229047737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb407a290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80098a0 tx=0x7f8fa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280a1d30bfdcede3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:31.940 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.482+0000 7f8fb37fe700 1 -- 172.21.15.111:0/229047737 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8fa8004160 con 0x7f8fb407be30 2023-10-11T16:23:31.940 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.482+0000 7f8fb37fe700 1 -- 172.21.15.111:0/229047737 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fa80042c0 con 0x7f8fb407be30 2023-10-11T16:23:31.940 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.482+0000 7f8fb37fe700 1 -- 172.21.15.111:0/229047737 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8fa80119b0 con 0x7f8fb407be30 2023-10-11T16:23:31.940 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.483+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/229047737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 msgr2=0x7f8fb407a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:31.940 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.483+0000 7f8fbaf6f700 1 --2- 172.21.15.111:0/229047737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb407a290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80098a0 tx=0x7f8fa8009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:31.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.483+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/229047737 shutdown_connections 2023-10-11T16:23:31.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.483+0000 7f8fbaf6f700 1 --2- 172.21.15.111:0/229047737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb407a290 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:31.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.483+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/229047737 >> 172.21.15.111:0/229047737 conn(0x7f8fb4100fd0 msgr2=0x7f8fb4103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:31.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.483+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/229047737 shutdown_connections 2023-10-11T16:23:31.941 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.483+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/229047737 wait complete. 2023-10-11T16:23:31.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.484+0000 7f8fbaf6f700 1 Processor -- start 2023-10-11T16:23:31.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.484+0000 7f8fbaf6f700 1 -- start start 2023-10-11T16:23:31.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.485+0000 7f8fbaf6f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:31.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.485+0000 7f8fbaf6f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb419be90 con 0x7f8fb407be30 2023-10-11T16:23:31.942 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.485+0000 7f8fb8d0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:31.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.485+0000 7f8fb8d0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb419b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48724/0 (socket says 172.21.15.111:48724) 2023-10-11T16:23:31.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.485+0000 7f8fb8d0b700 1 -- 172.21.15.111:0/1149878119 learned_addr learned my addr 172.21.15.111:0/1149878119 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:31.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.485+0000 7f8fb8d0b700 1 -- 172.21.15.111:0/1149878119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa8009550 con 0x7f8fb407be30 2023-10-11T16:23:31.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.485+0000 7f8fb8d0b700 1 --2- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb419b950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8011980 tx=0x7f8fa8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:31.943 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.486+0000 7f8fb1ffb700 1 -- 172.21.15.111:0/1149878119 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8fa8003a60 con 0x7f8fb407be30 2023-10-11T16:23:31.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.486+0000 7f8fb1ffb700 1 -- 172.21.15.111:0/1149878119 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fa8003bc0 con 0x7f8fb407be30 2023-10-11T16:23:31.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.486+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb410bf90 con 0x7f8fb407be30 2023-10-11T16:23:31.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.486+0000 7f8fb1ffb700 1 -- 172.21.15.111:0/1149878119 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8fa801a670 con 0x7f8fb407be30 2023-10-11T16:23:31.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.486+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb410be00 con 0x7f8fb407be30 2023-10-11T16:23:31.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.487+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb410c120 con 0x7f8fb407be30 2023-10-11T16:23:31.944 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.487+0000 7f8fb1ffb700 1 -- 172.21.15.111:0/1149878119 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f8fa8024a00 con 0x7f8fb407be30 2023-10-11T16:23:31.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.488+0000 7f8fb1ffb700 1 --2- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f8fa4038760 0x7f8fa403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:31.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.488+0000 7f8fb1ffb700 1 -- 172.21.15.111:0/1149878119 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8fa804c8b0 con 0x7f8fb407be30 2023-10-11T16:23:31.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.488+0000 7f8fb3fff700 1 --2- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f8fa4038760 0x7f8fa403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:31.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.489+0000 7f8fb3fff700 1 --2- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f8fa4038760 0x7f8fa403ac10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0006fd0 tx=0x7f8fa0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:31.945 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.493+0000 7f8fb1ffb700 1 -- 172.21.15.111:0/1149878119 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8fa8024440 con 0x7f8fb407be30 2023-10-11T16:23:31.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:30.755+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8fb419c4d0 con 0x7f8fb407be30 2023-10-11T16:23:31.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.453+0000 7f8fb1ffb700 1 -- 172.21.15.111:0/1149878119 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f8fa80246d0 con 0x7f8fb407be30 2023-10-11T16:23:31.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.455+0000 7f8fb1ffb700 1 -- 172.21.15.111:0/1149878119 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f8fa804c200 con 0x7f8fb407be30 2023-10-11T16:23:31.946 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f8fa4038760 msgr2=0x7f8fa403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:31.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 --2- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f8fa4038760 0x7f8fa403ac10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0006fd0 tx=0x7f8fa0006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:31.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 msgr2=0x7f8fb419b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:31.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 --2- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb419b950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8011980 tx=0x7f8fa8011a60 comp rx=0 tx=0).stop 2023-10-11T16:23:31.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 shutdown_connections 2023-10-11T16:23:31.947 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 --2- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f8fa4038760 0x7f8fa403ac10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:31.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 --2- 172.21.15.111:0/1149878119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fb407be30 0x7f8fb419b950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:31.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 >> 172.21.15.111:0/1149878119 conn(0x7f8fb4100fd0 msgr2=0x7f8fb41026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:31.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 shutdown_connections 2023-10-11T16:23:31.948 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:31.458+0000 7f8fbaf6f700 1 -- 172.21.15.111:0/1149878119 wait complete. 2023-10-11T16:23:32.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:31 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:31.770+0000 7effc3f9b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T16:23:32.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:31 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:31.876+0000 7effc3f9b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T16:23:32.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:32 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:32.014+0000 7effc3f9b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T16:23:32.455 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:32 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:32.144+0000 7effc3f9b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T16:23:32.455 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:32 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:32.380+0000 7effc3f9b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T16:23:32.800 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:32 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1149878119' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-10-11T16:23:32.800 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:32 smithi111 ceph-mon[98527]: mgrmap e9: x(active, since 17s) 2023-10-11T16:23:33.096 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:32 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:32.800+0000 7effc3f9b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T16:23:33.097 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:32 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:32.928+0000 7effc3f9b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T16:23:33.144 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2023-10-11T16:23:33.144 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-10-11T16:23:33.144 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-11T16:23:33.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-10-11T16:23:33.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-10-11T16:23:33.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2023-10-11T16:23:33.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.472+0000 7f1ec0920700 1 Processor -- start 2023-10-11T16:23:33.145 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.473+0000 7f1ec0920700 1 -- start start 2023-10-11T16:23:33.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.473+0000 7f1ec0920700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:33.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.473+0000 7f1ec0920700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc07a860 con 0x7f1ebc07be30 2023-10-11T16:23:33.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.473+0000 7f1eba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:33.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.474+0000 7f1eba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48746/0 (socket says 172.21.15.111:48746) 2023-10-11T16:23:33.146 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.474+0000 7f1eba59c700 1 -- 172.21.15.111:0/447908448 learned_addr learned my addr 172.21.15.111:0/447908448 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:33.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.474+0000 7f1eba59c700 1 -- 172.21.15.111:0/447908448 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc07a9a0 con 0x7f1ebc07be30 2023-10-11T16:23:33.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.474+0000 7f1eba59c700 1 --2- 172.21.15.111:0/447908448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc07a290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0098a0 tx=0x7f1eac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229dde1a4a4a2852 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:33.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.474+0000 7f1eb959a700 1 -- 172.21.15.111:0/447908448 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1eac004160 con 0x7f1ebc07be30 2023-10-11T16:23:33.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.474+0000 7f1eb959a700 1 -- 172.21.15.111:0/447908448 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eac0042c0 con 0x7f1ebc07be30 2023-10-11T16:23:33.147 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.474+0000 7f1eb959a700 1 -- 172.21.15.111:0/447908448 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1eac0119b0 con 0x7f1ebc07be30 2023-10-11T16:23:33.148 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.475+0000 7f1ec0920700 1 -- 172.21.15.111:0/447908448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 msgr2=0x7f1ebc07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:33.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.475+0000 7f1ec0920700 1 --2- 172.21.15.111:0/447908448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc07a290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1eac0098a0 tx=0x7f1eac009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:33.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.475+0000 7f1ec0920700 1 -- 172.21.15.111:0/447908448 shutdown_connections 2023-10-11T16:23:33.149 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.475+0000 7f1ec0920700 1 --2- 172.21.15.111:0/447908448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc07a290 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:33.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.475+0000 7f1ec0920700 1 -- 172.21.15.111:0/447908448 >> 172.21.15.111:0/447908448 conn(0x7f1ebc100fd0 msgr2=0x7f1ebc103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:33.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.475+0000 7f1ec0920700 1 -- 172.21.15.111:0/447908448 shutdown_connections 2023-10-11T16:23:33.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.475+0000 7f1ec0920700 1 -- 172.21.15.111:0/447908448 wait complete. 2023-10-11T16:23:33.150 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.476+0000 7f1ec0920700 1 Processor -- start 2023-10-11T16:23:33.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.476+0000 7f1ec0920700 1 -- start start 2023-10-11T16:23:33.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.477+0000 7f1ec0920700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc19b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:33.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.477+0000 7f1ec0920700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc19be90 con 0x7f1ebc07be30 2023-10-11T16:23:33.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.477+0000 7f1eba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc19b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:33.151 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.477+0000 7f1eba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc19b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48750/0 (socket says 172.21.15.111:48750) 2023-10-11T16:23:33.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.477+0000 7f1eba59c700 1 -- 172.21.15.111:0/782692049 learned_addr learned my addr 172.21.15.111:0/782692049 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:33.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.477+0000 7f1eba59c700 1 -- 172.21.15.111:0/782692049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac009550 con 0x7f1ebc07be30 2023-10-11T16:23:33.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.478+0000 7f1eba59c700 1 --2- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc19b950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1eac011980 tx=0x7f1eac011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:33.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.478+0000 7f1eab7fe700 1 -- 172.21.15.111:0/782692049 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1eac003a60 con 0x7f1ebc07be30 2023-10-11T16:23:33.152 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.478+0000 7f1eab7fe700 1 -- 172.21.15.111:0/782692049 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eac003bc0 con 0x7f1ebc07be30 2023-10-11T16:23:33.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.478+0000 7f1eab7fe700 1 -- 172.21.15.111:0/782692049 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1eac01a670 con 0x7f1ebc07be30 2023-10-11T16:23:33.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.478+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc10bf90 con 0x7f1ebc07be30 2023-10-11T16:23:33.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.478+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc10be00 con 0x7f1ebc07be30 2023-10-11T16:23:33.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.479+0000 7f1eab7fe700 1 -- 172.21.15.111:0/782692049 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f1eac024440 con 0x7f1ebc07be30 2023-10-11T16:23:33.153 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.480+0000 7f1eab7fe700 1 --2- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ea40387b0 0x7f1ea403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:33.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.479+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc10c120 con 0x7f1ebc07be30 2023-10-11T16:23:33.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.480+0000 7f1eab7fe700 1 -- 172.21.15.111:0/782692049 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1eac04c910 con 0x7f1ebc07be30 2023-10-11T16:23:33.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.480+0000 7f1eb9d9b700 1 -- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ea40387b0 msgr2=0x7f1ea403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4086392563 2023-10-11T16:23:33.154 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.480+0000 7f1eb9d9b700 1 --2- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ea40387b0 0x7f1ea403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:23:33.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.489+0000 7f1eab7fe700 1 -- 172.21.15.111:0/782692049 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1eac0189a0 con 0x7f1ebc07be30 2023-10-11T16:23:33.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.681+0000 7f1eb9d9b700 1 -- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ea40387b0 msgr2=0x7f1ea403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4086392563 2023-10-11T16:23:33.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.681+0000 7f1eb9d9b700 1 --2- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ea40387b0 0x7f1ea403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-11T16:23:33.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.773+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1ebc19c4d0 con 0x7f1ebc07be30 2023-10-11T16:23:33.155 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.774+0000 7f1eab7fe700 1 -- 172.21.15.111:0/782692049 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f1eac018350 con 0x7f1ebc07be30 2023-10-11T16:23:33.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ea40387b0 msgr2=0x7f1ea403ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:23:33.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 --2- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ea40387b0 0x7f1ea403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:33.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 msgr2=0x7f1ebc19b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:33.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 --2- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc19b950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1eac011980 tx=0x7f1eac011a60 comp rx=0 tx=0).stop 2023-10-11T16:23:33.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 shutdown_connections 2023-10-11T16:23:33.156 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 --2- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ea40387b0 0x7f1ea403ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:33.157 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 --2- 172.21.15.111:0/782692049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ebc07be30 0x7f1ebc19b950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:33.157 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 >> 172.21.15.111:0/782692049 conn(0x7f1ebc100fd0 msgr2=0x7f1ebc1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:33.157 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.776+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 shutdown_connections 2023-10-11T16:23:33.157 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:32.777+0000 7f1ec0920700 1 -- 172.21.15.111:0/782692049 wait complete. 2023-10-11T16:23:33.157 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for the mgr to restart... 2023-10-11T16:23:33.158 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr epoch 9... 2023-10-11T16:23:33.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:33 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:33.094+0000 7effc3f9b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T16:23:33.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:33 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:33.226+0000 7effc3f9b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T16:23:33.785 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:33 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/782692049' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-11T16:23:34.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:33 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:33.784+0000 7effc3f9b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T16:23:34.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:34.517+0000 7effc3f9b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T16:23:34.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:34.726+0000 7effc3f9b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T16:23:36.003 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:35 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:35.662+0000 7effc3f9b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:23:36.004 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:35 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:35.781+0000 7effc3f9b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T16:23:36.363 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:36.002+0000 7effc3f9b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:23:36.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:36.360+0000 7effc3f9b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T16:23:37.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:36.635+0000 7effc3f9b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T16:23:37.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:36.825+0000 7effc3f9b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T16:23:37.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:37.161+0000 7effc3f9b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T16:23:38.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:38.075+0000 7effc3f9b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T16:23:38.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:38.245+0000 7effc3f9b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T16:23:38.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:38.493+0000 7effc3f9b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T16:23:38.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:38.641+0000 7effc3f9b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T16:23:39.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:38.915+0000 7effc3f9b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T16:23:40.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:23:39 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:23:39.967+0000 7effc3f9b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T16:23:40.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: Active manager daemon x restarted 2023-10-11T16:23:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: Activating manager daemon x 2023-10-11T16:23:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: osdmap e3: 0 total, 0 up, 0 in 2023-10-11T16:23:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: mgrmap e10: x(active, starting, since 0.0038082s) 2023-10-11T16:23:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:23:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-11T16:23:40.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-11T16:23:40.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-11T16:23:40.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-11T16:23:40.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: Manager daemon x is now available 2023-10-11T16:23:40.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:40.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:40.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-11T16:23:40.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-11T16:23:41.346 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2023-10-11T16:23:41.346 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-10-11T16:23:41.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "initialized": true 2023-10-11T16:23:41.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2023-10-11T16:23:41.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.695+0000 7f1bc4573700 1 Processor -- start 2023-10-11T16:23:41.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.695+0000 7f1bc4573700 1 -- start start 2023-10-11T16:23:41.347 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.695+0000 7f1bc4573700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc108870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:41.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.695+0000 7f1bc4573700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc079e60 con 0x7f1bbc106490 2023-10-11T16:23:41.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.696+0000 7f1bc230f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc108870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:41.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.696+0000 7f1bc230f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc108870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48754/0 (socket says 172.21.15.111:48754) 2023-10-11T16:23:41.348 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.696+0000 7f1bc230f700 1 -- 172.21.15.111:0/2972086960 learned_addr learned my addr 172.21.15.111:0/2972086960 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:41.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.696+0000 7f1bc230f700 1 -- 172.21.15.111:0/2972086960 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc108db0 con 0x7f1bbc106490 2023-10-11T16:23:41.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.696+0000 7f1bc230f700 1 --2- 172.21.15.111:0/2972086960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc108870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80098a0 tx=0x7f1bb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f88a7db4b94838 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:41.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.696+0000 7f1bc130d700 1 -- 172.21.15.111:0/2972086960 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bb8004160 con 0x7f1bbc106490 2023-10-11T16:23:41.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc130d700 1 -- 172.21.15.111:0/2972086960 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bb80042c0 con 0x7f1bbc106490 2023-10-11T16:23:41.349 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc130d700 1 -- 172.21.15.111:0/2972086960 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bb80119b0 con 0x7f1bbc106490 2023-10-11T16:23:41.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc4573700 1 -- 172.21.15.111:0/2972086960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 msgr2=0x7f1bbc108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:41.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc4573700 1 --2- 172.21.15.111:0/2972086960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc108870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1bb80098a0 tx=0x7f1bb8009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:41.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc4573700 1 -- 172.21.15.111:0/2972086960 shutdown_connections 2023-10-11T16:23:41.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc4573700 1 --2- 172.21.15.111:0/2972086960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc108870 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:41.350 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc4573700 1 -- 172.21.15.111:0/2972086960 >> 172.21.15.111:0/2972086960 conn(0x7f1bbc0ffed0 msgr2=0x7f1bbc102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:41.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc4573700 1 -- 172.21.15.111:0/2972086960 shutdown_connections 2023-10-11T16:23:41.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.697+0000 7f1bc4573700 1 -- 172.21.15.111:0/2972086960 wait complete. 2023-10-11T16:23:41.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.698+0000 7f1bc4573700 1 Processor -- start 2023-10-11T16:23:41.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.699+0000 7f1bc4573700 1 -- start start 2023-10-11T16:23:41.351 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.699+0000 7f1bc4573700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc105df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.699+0000 7f1bc4573700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc106330 con 0x7f1bbc106490 2023-10-11T16:23:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.699+0000 7f1bc230f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc105df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.699+0000 7f1bc230f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc105df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48758/0 (socket says 172.21.15.111:48758) 2023-10-11T16:23:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.699+0000 7f1bc230f700 1 -- 172.21.15.111:0/1435328680 learned_addr learned my addr 172.21.15.111:0/1435328680 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.699+0000 7f1bc230f700 1 -- 172.21.15.111:0/1435328680 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8009550 con 0x7f1bbc106490 2023-10-11T16:23:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.700+0000 7f1bc230f700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc105df0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8003e60 tx=0x7f1bb8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:41.353 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.700+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bb8003970 con 0x7f1bbc106490 2023-10-11T16:23:41.353 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.700+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bb8003ad0 con 0x7f1bbc106490 2023-10-11T16:23:41.353 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.700+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bbc109d80 con 0x7f1bbc106490 2023-10-11T16:23:41.353 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.700+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bb80195e0 con 0x7f1bbc106490 2023-10-11T16:23:41.353 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.700+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bbc109bf0 con 0x7f1bbc106490 2023-10-11T16:23:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.701+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f1bb80198a0 con 0x7f1bbc106490 2023-10-11T16:23:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.702+0000 7f1bb37fe700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.702+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 --> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ba803b2d0 con 0x7f1ba8038710 2023-10-11T16:23:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.702+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1bb804cb60 con 0x7f1bbc106490 2023-10-11T16:23:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.702+0000 7f1bc1b0e700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 msgr2=0x7f1ba803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4086392563 2023-10-11T16:23:41.355 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.702+0000 7f1bc1b0e700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:23:41.355 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.903+0000 7f1bc1b0e700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 msgr2=0x7f1ba803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4086392563 2023-10-11T16:23:41.355 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:33.903+0000 7f1bc1b0e700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-11T16:23:41.355 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:34.304+0000 7f1bc1b0e700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 msgr2=0x7f1ba803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4086392563 2023-10-11T16:23:41.355 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:34.304+0000 7f1bc1b0e700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-11T16:23:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:35.105+0000 7f1bc1b0e700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 msgr2=0x7f1ba803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4086392563 2023-10-11T16:23:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:35.105+0000 7f1bc1b0e700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-11T16:23:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:36.706+0000 7f1bc1b0e700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 msgr2=0x7f1ba803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4086392563 2023-10-11T16:23:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:36.707+0000 7f1bc1b0e700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-11T16:23:41.356 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:39.910+0000 7f1bc1b0e700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 msgr2=0x7f1ba803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4086392563 2023-10-11T16:23:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:39.910+0000 7f1bc1b0e700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-10-11T16:23:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:39.972+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f1bb802da00 con 0x7f1bbc106490 2023-10-11T16:23:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:39.972+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 msgr2=0x7f1ba803abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:23:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:39.972+0000 7f1bb37fe700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:41.357 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.974+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f1bb804e110 con 0x7f1bbc106490 2023-10-11T16:23:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.974+0000 7f1bb37fe700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ba803bff0 0x7f1ba803e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.974+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ba803b2d0 con 0x7f1ba803bff0 2023-10-11T16:23:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.975+0000 7f1bc1b0e700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ba803bff0 0x7f1ba803e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:41.358 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.975+0000 7f1bc1b0e700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ba803bff0 0x7f1ba803e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1bac003a10 tx=0x7f1bac0092b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.976+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f1ba803b2d0 con 0x7f1ba803bff0 2023-10-11T16:23:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.984+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f1bbc105990 con 0x7f1ba803bff0 2023-10-11T16:23:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.985+0000 7f1bb37fe700 1 -- 172.21.15.111:0/1435328680 <== mgr.14148 v2:172.21.15.111:6800/2838247405 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f1bbc105990 con 0x7f1ba803bff0 2023-10-11T16:23:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ba803bff0 msgr2=0x7f1ba803e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:41.359 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ba803bff0 0x7f1ba803e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1bac003a10 tx=0x7f1bac0092b0 comp rx=0 tx=0).stop 2023-10-11T16:23:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 msgr2=0x7f1bbc105df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc105df0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8003e60 tx=0x7f1bb8011aa0 comp rx=0 tx=0).stop 2023-10-11T16:23:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 shutdown_connections 2023-10-11T16:23:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ba803bff0 0x7f1ba803e3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:41.360 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bbc106490 0x7f1bbc105df0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 --2- 172.21.15.111:0/1435328680 >> [v2:172.21.15.111:6800/4086392563,v1:172.21.15.111:6801/4086392563] conn(0x7f1ba8038710 0x7f1ba803abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 >> 172.21.15.111:0/1435328680 conn(0x7f1bbc0ffed0 msgr2=0x7f1bbc100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 shutdown_connections 2023-10-11T16:23:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:40.986+0000 7f1bc4573700 1 -- 172.21.15.111:0/1435328680 wait complete. 2023-10-11T16:23:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:mgr epoch 9 is available 2023-10-11T16:23:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:Generating a dashboard self-signed certificate... 2023-10-11T16:23:41.625 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:41 smithi111 ceph-mon[98527]: [11/Oct/2023:16:23:40] ENGINE Bus STARTING 2023-10-11T16:23:41.625 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:41 smithi111 ceph-mon[98527]: [11/Oct/2023:16:23:40] ENGINE Serving on https://172.21.15.111:7150 2023-10-11T16:23:41.625 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:41 smithi111 ceph-mon[98527]: [11/Oct/2023:16:23:40] ENGINE Serving on http://172.21.15.111:8765 2023-10-11T16:23:41.626 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:41 smithi111 ceph-mon[98527]: [11/Oct/2023:16:23:40] ENGINE Bus STARTED 2023-10-11T16:23:41.626 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:41 smithi111 ceph-mon[98527]: mgrmap e11: x(active, since 1.00699s) 2023-10-11T16:23:42.625 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-10-11T16:23:42.625 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.960+0000 7f808daa9700 1 Processor -- start 2023-10-11T16:23:42.626 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.960+0000 7f808daa9700 1 -- start start 2023-10-11T16:23:42.626 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.960+0000 7f808daa9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808807a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:42.626 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.960+0000 7f808daa9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808807a860 con 0x7f808807be30 2023-10-11T16:23:42.626 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.961+0000 7f80877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808807a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:42.627 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.961+0000 7f80877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808807a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48952/0 (socket says 172.21.15.111:48952) 2023-10-11T16:23:42.627 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.961+0000 7f80877fe700 1 -- 172.21.15.111:0/2291261779 learned_addr learned my addr 172.21.15.111:0/2291261779 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:42.627 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.961+0000 7f80877fe700 1 -- 172.21.15.111:0/2291261779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808807a9a0 con 0x7f808807be30 2023-10-11T16:23:42.627 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.961+0000 7f80877fe700 1 --2- 172.21.15.111:0/2291261779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808807a290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f80780098a0 tx=0x7f8078009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9332b13d7d5b939 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:42.627 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.962+0000 7f80867fc700 1 -- 172.21.15.111:0/2291261779 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8078004160 con 0x7f808807be30 2023-10-11T16:23:42.628 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.962+0000 7f80867fc700 1 -- 172.21.15.111:0/2291261779 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80780042c0 con 0x7f808807be30 2023-10-11T16:23:42.628 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.962+0000 7f80867fc700 1 -- 172.21.15.111:0/2291261779 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80780119b0 con 0x7f808807be30 2023-10-11T16:23:42.628 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.962+0000 7f808daa9700 1 -- 172.21.15.111:0/2291261779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 msgr2=0x7f808807a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:42.628 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.962+0000 7f808daa9700 1 --2- 172.21.15.111:0/2291261779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808807a290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f80780098a0 tx=0x7f8078009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:42.629 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.963+0000 7f808daa9700 1 -- 172.21.15.111:0/2291261779 shutdown_connections 2023-10-11T16:23:42.629 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.963+0000 7f808daa9700 1 --2- 172.21.15.111:0/2291261779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808807a290 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:42.629 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.963+0000 7f808daa9700 1 -- 172.21.15.111:0/2291261779 >> 172.21.15.111:0/2291261779 conn(0x7f8088100fd0 msgr2=0x7f8088103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:42.629 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.963+0000 7f808daa9700 1 -- 172.21.15.111:0/2291261779 shutdown_connections 2023-10-11T16:23:42.629 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.963+0000 7f808daa9700 1 -- 172.21.15.111:0/2291261779 wait complete. 2023-10-11T16:23:42.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.964+0000 7f808daa9700 1 Processor -- start 2023-10-11T16:23:42.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.964+0000 7f808daa9700 1 -- start start 2023-10-11T16:23:42.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.965+0000 7f808daa9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808819b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:42.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.965+0000 7f808daa9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808819be90 con 0x7f808807be30 2023-10-11T16:23:42.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.965+0000 7f80877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808819b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:42.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.965+0000 7f80877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808819b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48954/0 (socket says 172.21.15.111:48954) 2023-10-11T16:23:42.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.965+0000 7f80877fe700 1 -- 172.21.15.111:0/3098392572 learned_addr learned my addr 172.21.15.111:0/3098392572 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:42.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.965+0000 7f80877fe700 1 -- 172.21.15.111:0/3098392572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8078009550 con 0x7f808807be30 2023-10-11T16:23:42.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.966+0000 7f80877fe700 1 --2- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808819b950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8078003e60 tx=0x7f8078011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:42.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.966+0000 7f8084ff9700 1 -- 172.21.15.111:0/3098392572 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8078003970 con 0x7f808807be30 2023-10-11T16:23:42.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.966+0000 7f8084ff9700 1 -- 172.21.15.111:0/3098392572 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8078003ad0 con 0x7f808807be30 2023-10-11T16:23:42.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.966+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f808810bf90 con 0x7f808807be30 2023-10-11T16:23:42.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.966+0000 7f8084ff9700 1 -- 172.21.15.111:0/3098392572 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80780195e0 con 0x7f808807be30 2023-10-11T16:23:42.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.967+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f808810be00 con 0x7f808807be30 2023-10-11T16:23:42.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.967+0000 7f8084ff9700 1 -- 172.21.15.111:0/3098392572 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f8078019850 con 0x7f808807be30 2023-10-11T16:23:42.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.968+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808810c120 con 0x7f808807be30 2023-10-11T16:23:42.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.968+0000 7f8084ff9700 1 --2- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8070038640 0x7f807003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:42.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.968+0000 7f8084ff9700 1 -- 172.21.15.111:0/3098392572 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f807804c970 con 0x7f808807be30 2023-10-11T16:23:42.634 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.968+0000 7f8086ffd700 1 --2- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8070038640 0x7f807003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:42.634 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.969+0000 7f8086ffd700 1 --2- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8070038640 0x7f807003aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f807c006fd0 tx=0x7f807c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:42.634 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:41.973+0000 7f8084ff9700 1 -- 172.21.15.111:0/3098392572 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8078018d20 con 0x7f808807be30 2023-10-11T16:23:42.634 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.189+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8088103300 con 0x7f8070038640 2023-10-11T16:23:42.635 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.275+0000 7f8084ff9700 1 -- 172.21.15.111:0/3098392572 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8088103300 con 0x7f8070038640 2023-10-11T16:23:42.635 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8070038640 msgr2=0x7f807003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:42.635 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 --2- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8070038640 0x7f807003aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f807c006fd0 tx=0x7f807c006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:42.635 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 msgr2=0x7f808819b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:42.635 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 --2- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808819b950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8078003e60 tx=0x7f8078011aa0 comp rx=0 tx=0).stop 2023-10-11T16:23:42.636 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 shutdown_connections 2023-10-11T16:23:42.636 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 --2- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8070038640 0x7f807003aaf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:42.636 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 --2- 172.21.15.111:0/3098392572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f808807be30 0x7f808819b950 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:42.636 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 >> 172.21.15.111:0/3098392572 conn(0x7f8088100fd0 msgr2=0x7f80881026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:42.636 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 shutdown_connections 2023-10-11T16:23:42.637 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:42.277+0000 7f808daa9700 1 -- 172.21.15.111:0/3098392572 wait complete. 2023-10-11T16:23:42.637 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial admin user... 2023-10-11T16:23:42.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:42 smithi111 ceph-mon[98527]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-11T16:23:42.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:42 smithi111 ceph-mon[98527]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-11T16:23:42.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:42.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:42.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:43.828 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:43 smithi111 ceph-mon[98527]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:43.829 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:43 smithi111 ceph-mon[98527]: mgrmap e12: x(active, since 2s) 2023-10-11T16:23:44.110 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$v2gmQtsW8ZfE9FENAFVJO.cUxKP/8nkRqe6Hbk4VW1.4Zuj4mk6aO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1697041423, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-10-11T16:23:44.111 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.182+0000 7f7082eec700 1 Processor -- start 2023-10-11T16:23:44.111 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.182+0000 7f7082eec700 1 -- start start 2023-10-11T16:23:44.111 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.183+0000 7f7082eec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:44.111 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.183+0000 7f7082eec700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c106400 con 0x7f707c105a20 2023-10-11T16:23:44.111 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.183+0000 7f7080c88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:44.111 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.183+0000 7f7080c88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48960/0 (socket says 172.21.15.111:48960) 2023-10-11T16:23:44.112 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.183+0000 7f7080c88700 1 -- 172.21.15.111:0/2575669170 learned_addr learned my addr 172.21.15.111:0/2575669170 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:44.112 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.183+0000 7f7080c88700 1 -- 172.21.15.111:0/2575669170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c106c20 con 0x7f707c105a20 2023-10-11T16:23:44.112 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.184+0000 7f7080c88700 1 --2- 172.21.15.111:0/2575669170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c105e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70700098a0 tx=0x7f7070009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd721206223f593 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:44.112 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.184+0000 7f707b7fe700 1 -- 172.21.15.111:0/2575669170 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7070004160 con 0x7f707c105a20 2023-10-11T16:23:44.112 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.184+0000 7f707b7fe700 1 -- 172.21.15.111:0/2575669170 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70700042c0 con 0x7f707c105a20 2023-10-11T16:23:44.113 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.184+0000 7f707b7fe700 1 -- 172.21.15.111:0/2575669170 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70700119b0 con 0x7f707c105a20 2023-10-11T16:23:44.113 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.185+0000 7f7082eec700 1 -- 172.21.15.111:0/2575669170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 msgr2=0x7f707c105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:44.113 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.185+0000 7f7082eec700 1 --2- 172.21.15.111:0/2575669170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c105e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70700098a0 tx=0x7f7070009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:44.113 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.185+0000 7f7082eec700 1 -- 172.21.15.111:0/2575669170 shutdown_connections 2023-10-11T16:23:44.113 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.185+0000 7f7082eec700 1 --2- 172.21.15.111:0/2575669170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c105e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:44.113 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.185+0000 7f7082eec700 1 -- 172.21.15.111:0/2575669170 >> 172.21.15.111:0/2575669170 conn(0x7f707c100fd0 msgr2=0x7f707c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:44.114 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.185+0000 7f7082eec700 1 -- 172.21.15.111:0/2575669170 shutdown_connections 2023-10-11T16:23:44.114 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.185+0000 7f7082eec700 1 -- 172.21.15.111:0/2575669170 wait complete. 2023-10-11T16:23:44.114 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.186+0000 7f7082eec700 1 Processor -- start 2023-10-11T16:23:44.114 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.186+0000 7f7082eec700 1 -- start start 2023-10-11T16:23:44.114 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.186+0000 7f7082eec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c19b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:44.115 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.186+0000 7f7082eec700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c19bf00 con 0x7f707c105a20 2023-10-11T16:23:44.115 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.187+0000 7f7080c88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c19b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:44.115 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.187+0000 7f7080c88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c19b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48976/0 (socket says 172.21.15.111:48976) 2023-10-11T16:23:44.115 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.187+0000 7f7080c88700 1 -- 172.21.15.111:0/2240346266 learned_addr learned my addr 172.21.15.111:0/2240346266 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:44.115 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.187+0000 7f7080c88700 1 -- 172.21.15.111:0/2240346266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070009550 con 0x7f707c105a20 2023-10-11T16:23:44.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.187+0000 7f7080c88700 1 --2- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c19b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7070000c00 tx=0x7f7070011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:44.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.188+0000 7f7079ffb700 1 -- 172.21.15.111:0/2240346266 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70700039f0 con 0x7f707c105a20 2023-10-11T16:23:44.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.188+0000 7f7079ffb700 1 -- 172.21.15.111:0/2240346266 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7070003b50 con 0x7f707c105a20 2023-10-11T16:23:44.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.188+0000 7f7079ffb700 1 -- 172.21.15.111:0/2240346266 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f707001a780 con 0x7f707c105a20 2023-10-11T16:23:44.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.188+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c109fe0 con 0x7f707c105a20 2023-10-11T16:23:44.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.188+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707c109e50 con 0x7f707c105a20 2023-10-11T16:23:44.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.189+0000 7f7079ffb700 1 -- 172.21.15.111:0/2240346266 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f7070024440 con 0x7f707c105a20 2023-10-11T16:23:44.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.189+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707c062190 con 0x7f707c105a20 2023-10-11T16:23:44.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.189+0000 7f7079ffb700 1 --2- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f706c038760 0x7f706c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:44.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.189+0000 7f7079ffb700 1 -- 172.21.15.111:0/2240346266 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f707004c9b0 con 0x7f707c105a20 2023-10-11T16:23:44.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.190+0000 7f707bfff700 1 --2- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f706c038760 0x7f706c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:44.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.190+0000 7f707bfff700 1 --2- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f706c038760 0x7f706c03ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7068006fd0 tx=0x7f7068006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:44.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.194+0000 7f7079ffb700 1 -- 172.21.15.111:0/2240346266 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f707c062190 con 0x7f707c105a20 2023-10-11T16:23:44.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.415+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f707c19f380 con 0x7f706c038760 2023-10-11T16:23:44.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.644+0000 7f7079ffb700 1 -- 172.21.15.111:0/2240346266 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f707c19f380 con 0x7f706c038760 2023-10-11T16:23:44.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f706c038760 msgr2=0x7f706c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:44.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 --2- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f706c038760 0x7f706c03ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7068006fd0 tx=0x7f7068006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:44.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 msgr2=0x7f707c19b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:44.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 --2- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c19b9c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7070000c00 tx=0x7f7070011a30 comp rx=0 tx=0).stop 2023-10-11T16:23:44.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 shutdown_connections 2023-10-11T16:23:44.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 --2- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f706c038760 0x7f706c03ac10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:44.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 --2- 172.21.15.111:0/2240346266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f707c105a20 0x7f707c19b9c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:44.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 >> 172.21.15.111:0/2240346266 conn(0x7f707c100fd0 msgr2=0x7f707c102630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:44.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 shutdown_connections 2023-10-11T16:23:44.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:43.647+0000 7f7082eec700 1 -- 172.21.15.111:0/2240346266 wait complete. 2023-10-11T16:23:44.120 INFO:teuthology.orchestra.run.smithi111.stdout:Fetching dashboard port number... 2023-10-11T16:23:44.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:44 smithi111 ceph-mon[98527]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:44.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:44.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:45.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 8443 2023-10-11T16:23:45.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.665+0000 7f48f2016700 1 Processor -- start 2023-10-11T16:23:45.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.665+0000 7f48f2016700 1 -- start start 2023-10-11T16:23:45.392 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.665+0000 7f48f2016700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:45.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.665+0000 7f48f2016700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec07a8a0 con 0x7f48ec07be70 2023-10-11T16:23:45.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.665+0000 7f48eb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:45.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.666+0000 7f48eb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48988/0 (socket says 172.21.15.111:48988) 2023-10-11T16:23:45.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.666+0000 7f48eb7fe700 1 -- 172.21.15.111:0/4222810289 learned_addr learned my addr 172.21.15.111:0/4222810289 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:45.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.666+0000 7f48eb7fe700 1 -- 172.21.15.111:0/4222810289 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec07a9e0 con 0x7f48ec07be70 2023-10-11T16:23:45.393 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.666+0000 7f48eb7fe700 1 --2- 172.21.15.111:0/4222810289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec07a2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0098a0 tx=0x7f48dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea298362ee3f65f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:45.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.666+0000 7f48ea7fc700 1 -- 172.21.15.111:0/4222810289 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48dc004160 con 0x7f48ec07be70 2023-10-11T16:23:45.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.666+0000 7f48ea7fc700 1 -- 172.21.15.111:0/4222810289 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48dc0042c0 con 0x7f48ec07be70 2023-10-11T16:23:45.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.666+0000 7f48ea7fc700 1 -- 172.21.15.111:0/4222810289 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48dc0119b0 con 0x7f48ec07be70 2023-10-11T16:23:45.394 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.667+0000 7f48f2016700 1 -- 172.21.15.111:0/4222810289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 msgr2=0x7f48ec07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:45.395 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.667+0000 7f48f2016700 1 --2- 172.21.15.111:0/4222810289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec07a2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0098a0 tx=0x7f48dc009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:45.395 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.667+0000 7f48f2016700 1 -- 172.21.15.111:0/4222810289 shutdown_connections 2023-10-11T16:23:45.395 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.667+0000 7f48f2016700 1 --2- 172.21.15.111:0/4222810289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec07a2d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:45.395 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.667+0000 7f48f2016700 1 -- 172.21.15.111:0/4222810289 >> 172.21.15.111:0/4222810289 conn(0x7f48ec100fd0 msgr2=0x7f48ec103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:45.395 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.667+0000 7f48f2016700 1 -- 172.21.15.111:0/4222810289 shutdown_connections 2023-10-11T16:23:45.396 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.667+0000 7f48f2016700 1 -- 172.21.15.111:0/4222810289 wait complete. 2023-10-11T16:23:45.396 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.668+0000 7f48f2016700 1 Processor -- start 2023-10-11T16:23:45.396 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.668+0000 7f48f2016700 1 -- start start 2023-10-11T16:23:45.396 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.669+0000 7f48f2016700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec19b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:45.396 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.669+0000 7f48f2016700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec19bf10 con 0x7f48ec07be70 2023-10-11T16:23:45.397 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.669+0000 7f48eb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec19b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:45.397 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.669+0000 7f48eb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec19b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49004/0 (socket says 172.21.15.111:49004) 2023-10-11T16:23:45.397 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.669+0000 7f48eb7fe700 1 -- 172.21.15.111:0/1929860279 learned_addr learned my addr 172.21.15.111:0/1929860279 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:45.397 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.669+0000 7f48eb7fe700 1 -- 172.21.15.111:0/1929860279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc009550 con 0x7f48ec07be70 2023-10-11T16:23:45.397 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.669+0000 7f48eb7fe700 1 --2- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec19b9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f48dc011980 tx=0x7f48dc011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:45.398 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.669+0000 7f48e8ff9700 1 -- 172.21.15.111:0/1929860279 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48dc003a60 con 0x7f48ec07be70 2023-10-11T16:23:45.398 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.670+0000 7f48e8ff9700 1 -- 172.21.15.111:0/1929860279 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48dc003bc0 con 0x7f48ec07be70 2023-10-11T16:23:45.398 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.670+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ec10bfb0 con 0x7f48ec07be70 2023-10-11T16:23:45.398 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.670+0000 7f48e8ff9700 1 -- 172.21.15.111:0/1929860279 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48dc01a670 con 0x7f48ec07be70 2023-10-11T16:23:45.398 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.670+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ec10be20 con 0x7f48ec07be70 2023-10-11T16:23:45.399 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.671+0000 7f48e8ff9700 1 -- 172.21.15.111:0/1929860279 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f48dc01a8e0 con 0x7f48ec07be70 2023-10-11T16:23:45.399 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.671+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ec10c140 con 0x7f48ec07be70 2023-10-11T16:23:45.399 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.671+0000 7f48e8ff9700 1 --2- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d40387b0 0x7f48d403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:45.399 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.671+0000 7f48e8ff9700 1 -- 172.21.15.111:0/1929860279 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f48dc04cb90 con 0x7f48ec07be70 2023-10-11T16:23:45.399 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.671+0000 7f48eaffd700 1 --2- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d40387b0 0x7f48d403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:45.400 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.672+0000 7f48eaffd700 1 --2- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d40387b0 0x7f48d403ac60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f48e0006fd0 tx=0x7f48e0006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:45.400 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.677+0000 7f48e8ff9700 1 -- 172.21.15.111:0/1929860279 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f48dc0233a0 con 0x7f48ec07be70 2023-10-11T16:23:45.400 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.887+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f48ec19f5e0 con 0x7f48ec07be70 2023-10-11T16:23:45.400 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.888+0000 7f48e8ff9700 1 -- 172.21.15.111:0/1929860279 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f48dc018b40 con 0x7f48ec07be70 2023-10-11T16:23:45.400 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d40387b0 msgr2=0x7f48d403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:45.400 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 --2- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d40387b0 0x7f48d403ac60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f48e0006fd0 tx=0x7f48e0006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:45.401 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 msgr2=0x7f48ec19b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:45.401 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 --2- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec19b9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f48dc011980 tx=0x7f48dc011a60 comp rx=0 tx=0).stop 2023-10-11T16:23:45.401 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 shutdown_connections 2023-10-11T16:23:45.401 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 --2- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d40387b0 0x7f48d403ac60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:45.401 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 --2- 172.21.15.111:0/1929860279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48ec07be70 0x7f48ec19b9d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:45.402 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 >> 172.21.15.111:0/1929860279 conn(0x7f48ec100fd0 msgr2=0x7f48ec1026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:45.402 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 shutdown_connections 2023-10-11T16:23:45.402 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:44.890+0000 7f48f2016700 1 -- 172.21.15.111:0/1929860279 wait complete. 2023-10-11T16:23:45.409 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-11T16:23:45.409 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2023-10-11T16:23:45.427 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-11T16:23:45.428 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2023-10-11T16:23:45.428 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2023-10-11T16:23:45.428 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-10-11T16:23:45.430 INFO:teuthology.orchestra.run.smithi111.stdout:Ceph Dashboard is now available at: 2023-10-11T16:23:45.430 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:45.431 INFO:teuthology.orchestra.run.smithi111.stdout: URL: https://smithi111.front.sepia.ceph.com:8443/ 2023-10-11T16:23:45.431 INFO:teuthology.orchestra.run.smithi111.stdout: User: admin 2023-10-11T16:23:45.431 INFO:teuthology.orchestra.run.smithi111.stdout: Password: vykduur9aq 2023-10-11T16:23:45.431 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:45.432 INFO:teuthology.orchestra.run.smithi111.stdout:Saving cluster configuration to /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config directory 2023-10-11T16:23:45.432 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling autotune for osd_memory_target 2023-10-11T16:23:45.877 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:45 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1929860279' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-10-11T16:23:46.751 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.135+0000 7fa6ed781700 1 Processor -- start 2023-10-11T16:23:46.751 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.135+0000 7fa6ed781700 1 -- start start 2023-10-11T16:23:46.751 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.135+0000 7fa6ed781700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e807a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:46.751 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.135+0000 7fa6ed781700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e807a880 con 0x7fa6e807be50 2023-10-11T16:23:46.752 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.136+0000 7fa6e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e807a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:46.752 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.136+0000 7fa6e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e807a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46538/0 (socket says 172.21.15.111:46538) 2023-10-11T16:23:46.752 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.136+0000 7fa6e6ffd700 1 -- 172.21.15.111:0/1614815700 learned_addr learned my addr 172.21.15.111:0/1614815700 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:46.752 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.136+0000 7fa6e6ffd700 1 -- 172.21.15.111:0/1614815700 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e807a9c0 con 0x7fa6e807be50 2023-10-11T16:23:46.752 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.137+0000 7fa6e6ffd700 1 --2- 172.21.15.111:0/1614815700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e807a2b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80098a0 tx=0x7fa6d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9288d17c622677 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:46.753 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.137+0000 7fa6e5ffb700 1 -- 172.21.15.111:0/1614815700 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6d8004160 con 0x7fa6e807be50 2023-10-11T16:23:46.753 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.137+0000 7fa6e5ffb700 1 -- 172.21.15.111:0/1614815700 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6d80042c0 con 0x7fa6e807be50 2023-10-11T16:23:46.753 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.137+0000 7fa6e5ffb700 1 -- 172.21.15.111:0/1614815700 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6d80119b0 con 0x7fa6e807be50 2023-10-11T16:23:46.753 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.138+0000 7fa6ed781700 1 -- 172.21.15.111:0/1614815700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 msgr2=0x7fa6e807a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:46.753 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.138+0000 7fa6ed781700 1 --2- 172.21.15.111:0/1614815700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e807a2b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80098a0 tx=0x7fa6d8009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.138+0000 7fa6ed781700 1 -- 172.21.15.111:0/1614815700 shutdown_connections 2023-10-11T16:23:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.138+0000 7fa6ed781700 1 --2- 172.21.15.111:0/1614815700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e807a2b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.138+0000 7fa6ed781700 1 -- 172.21.15.111:0/1614815700 >> 172.21.15.111:0/1614815700 conn(0x7fa6e8100fd0 msgr2=0x7fa6e8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.138+0000 7fa6ed781700 1 -- 172.21.15.111:0/1614815700 shutdown_connections 2023-10-11T16:23:46.754 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.138+0000 7fa6ed781700 1 -- 172.21.15.111:0/1614815700 wait complete. 2023-10-11T16:23:46.755 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.139+0000 7fa6ed781700 1 Processor -- start 2023-10-11T16:23:46.755 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.139+0000 7fa6ed781700 1 -- start start 2023-10-11T16:23:46.755 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.139+0000 7fa6ed781700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e819b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:46.756 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.139+0000 7fa6ed781700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e819bee0 con 0x7fa6e807be50 2023-10-11T16:23:46.756 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e819b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:46.756 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e819b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46542/0 (socket says 172.21.15.111:46542) 2023-10-11T16:23:46.756 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6e6ffd700 1 -- 172.21.15.111:0/3137965233 learned_addr learned my addr 172.21.15.111:0/3137965233 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:46.756 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6e6ffd700 1 -- 172.21.15.111:0/3137965233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d8009550 con 0x7fa6e807be50 2023-10-11T16:23:46.757 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6e6ffd700 1 --2- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e819b9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8011980 tx=0x7fa6d8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:46.757 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6cffff700 1 -- 172.21.15.111:0/3137965233 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6d8003a60 con 0x7fa6e807be50 2023-10-11T16:23:46.757 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6cffff700 1 -- 172.21.15.111:0/3137965233 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6d8003bc0 con 0x7fa6e807be50 2023-10-11T16:23:46.757 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6cffff700 1 -- 172.21.15.111:0/3137965233 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6d801a670 con 0x7fa6e807be50 2023-10-11T16:23:46.757 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.140+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e810bfd0 con 0x7fa6e807be50 2023-10-11T16:23:46.758 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.141+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e810be40 con 0x7fa6e807be50 2023-10-11T16:23:46.758 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.141+0000 7fa6cffff700 1 -- 172.21.15.111:0/3137965233 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fa6d8024a00 con 0x7fa6e807be50 2023-10-11T16:23:46.758 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.141+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e810c160 con 0x7fa6e807be50 2023-10-11T16:23:46.758 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.142+0000 7fa6cffff700 1 --2- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa6d00387b0 0x7fa6d003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:46.758 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.142+0000 7fa6cffff700 1 -- 172.21.15.111:0/3137965233 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa6d804c990 con 0x7fa6e807be50 2023-10-11T16:23:46.759 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.142+0000 7fa6e67fc700 1 --2- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa6d00387b0 0x7fa6d003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:46.759 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.143+0000 7fa6e67fc700 1 --2- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa6d00387b0 0x7fa6d003ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc006fd0 tx=0x7fa6dc006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:46.759 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.147+0000 7fa6cffff700 1 -- 172.21.15.111:0/3137965233 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa6d8024440 con 0x7fa6e807be50 2023-10-11T16:23:46.759 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.348+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fa6e819f620 con 0x7fa6e807be50 2023-10-11T16:23:46.759 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.349+0000 7fa6cffff700 1 -- 172.21.15.111:0/3137965233 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fa6d804c280 con 0x7fa6e807be50 2023-10-11T16:23:46.760 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.351+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa6d00387b0 msgr2=0x7fa6d003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:46.760 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.351+0000 7fa6ed781700 1 --2- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa6d00387b0 0x7fa6d003ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc006fd0 tx=0x7fa6dc006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:46.760 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.351+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 msgr2=0x7fa6e819b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:46.760 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.351+0000 7fa6ed781700 1 --2- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e819b9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8011980 tx=0x7fa6d8011a60 comp rx=0 tx=0).stop 2023-10-11T16:23:46.761 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.351+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 shutdown_connections 2023-10-11T16:23:46.761 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.351+0000 7fa6ed781700 1 --2- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa6d00387b0 0x7fa6d003ac60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:46.761 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.351+0000 7fa6ed781700 1 --2- 172.21.15.111:0/3137965233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6e807be50 0x7fa6e819b9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:46.761 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.351+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 >> 172.21.15.111:0/3137965233 conn(0x7fa6e8100fd0 msgr2=0x7fa6e81026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:46.761 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.352+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 shutdown_connections 2023-10-11T16:23:46.761 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:46.352+0000 7fa6ed781700 1 -- 172.21.15.111:0/3137965233 wait complete. 2023-10-11T16:23:47.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:47 smithi111 ceph-mon[98527]: mgrmap e13: x(active, since 6s) 2023-10-11T16:23:47.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:47 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/512224939' entity='client.admin' 2023-10-11T16:23:48.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.238+0000 7fbcdc8b4700 1 Processor -- start 2023-10-11T16:23:48.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.239+0000 7fbcdc8b4700 1 -- start start 2023-10-11T16:23:48.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.239+0000 7fbcdc8b4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd807a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:48.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.239+0000 7fbcdc8b4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd807a8a0 con 0x7fbcd807be70 2023-10-11T16:23:48.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.239+0000 7fbcd659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd807a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:48.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.239+0000 7fbcd659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd807a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46556/0 (socket says 172.21.15.111:46556) 2023-10-11T16:23:48.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.239+0000 7fbcd659c700 1 -- 172.21.15.111:0/430396133 learned_addr learned my addr 172.21.15.111:0/430396133 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:48.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.240+0000 7fbcd659c700 1 -- 172.21.15.111:0/430396133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd807a9e0 con 0x7fbcd807be70 2023-10-11T16:23:48.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.240+0000 7fbcd659c700 1 --2- 172.21.15.111:0/430396133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd807a2d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80098a0 tx=0x7fbcc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df42bef7b11984a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:48.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.240+0000 7fbcd5d9b700 1 -- 172.21.15.111:0/430396133 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbcc8004160 con 0x7fbcd807be70 2023-10-11T16:23:48.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.240+0000 7fbcd5d9b700 1 -- 172.21.15.111:0/430396133 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcc80042c0 con 0x7fbcd807be70 2023-10-11T16:23:48.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.240+0000 7fbcd5d9b700 1 -- 172.21.15.111:0/430396133 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbcc80119b0 con 0x7fbcd807be70 2023-10-11T16:23:48.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.241+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/430396133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 msgr2=0x7fbcd807a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:48.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.241+0000 7fbcdc8b4700 1 --2- 172.21.15.111:0/430396133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd807a2d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80098a0 tx=0x7fbcc8009bb0 comp rx=0 tx=0).stop 2023-10-11T16:23:48.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.241+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/430396133 shutdown_connections 2023-10-11T16:23:48.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.241+0000 7fbcdc8b4700 1 --2- 172.21.15.111:0/430396133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd807a2d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:48.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.241+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/430396133 >> 172.21.15.111:0/430396133 conn(0x7fbcd8100fd0 msgr2=0x7fbcd8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:48.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.241+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/430396133 shutdown_connections 2023-10-11T16:23:48.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.241+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/430396133 wait complete. 2023-10-11T16:23:48.036 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.242+0000 7fbcdc8b4700 1 Processor -- start 2023-10-11T16:23:48.036 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.242+0000 7fbcdc8b4700 1 -- start start 2023-10-11T16:23:48.036 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.242+0000 7fbcdc8b4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd819b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:48.036 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.242+0000 7fbcdc8b4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd819bf10 con 0x7fbcd807be70 2023-10-11T16:23:48.036 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.243+0000 7fbcd659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd819b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:48.037 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.243+0000 7fbcd659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd819b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46564/0 (socket says 172.21.15.111:46564) 2023-10-11T16:23:48.037 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.243+0000 7fbcd659c700 1 -- 172.21.15.111:0/512224939 learned_addr learned my addr 172.21.15.111:0/512224939 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:48.037 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.243+0000 7fbcd659c700 1 -- 172.21.15.111:0/512224939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc8009550 con 0x7fbcd807be70 2023-10-11T16:23:48.037 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.243+0000 7fbcd659c700 1 --2- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd819b9d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8011980 tx=0x7fbcc8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:48.037 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.243+0000 7fbccf7fe700 1 -- 172.21.15.111:0/512224939 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbcc8003a60 con 0x7fbcd807be70 2023-10-11T16:23:48.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.243+0000 7fbccf7fe700 1 -- 172.21.15.111:0/512224939 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcc8003bc0 con 0x7fbcd807be70 2023-10-11T16:23:48.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.243+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd810bfb0 con 0x7fbcd807be70 2023-10-11T16:23:48.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.244+0000 7fbccf7fe700 1 -- 172.21.15.111:0/512224939 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbcc801a670 con 0x7fbcd807be70 2023-10-11T16:23:48.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.244+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd810be20 con 0x7fbcd807be70 2023-10-11T16:23:48.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.245+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd810c140 con 0x7fbcd807be70 2023-10-11T16:23:48.039 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.245+0000 7fbccf7fe700 1 -- 172.21.15.111:0/512224939 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fbcc8024a00 con 0x7fbcd807be70 2023-10-11T16:23:48.039 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.246+0000 7fbccf7fe700 1 --2- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc4038850 0x7fbcc403ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:48.039 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.246+0000 7fbccf7fe700 1 -- 172.21.15.111:0/512224939 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbcc804ca70 con 0x7fbcd807be70 2023-10-11T16:23:48.039 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.246+0000 7fbccffff700 1 --2- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc4038850 0x7fbcc403ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:48.039 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.246+0000 7fbccffff700 1 --2- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc4038850 0x7fbcc403ad00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0006fd0 tx=0x7fbcc0006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:48.040 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.250+0000 7fbccf7fe700 1 -- 172.21.15.111:0/512224939 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbcc8024440 con 0x7fbcd807be70 2023-10-11T16:23:48.040 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.553+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fbcd819f5e0 con 0x7fbcd807be70 2023-10-11T16:23:48.040 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.555+0000 7fbccf7fe700 1 -- 172.21.15.111:0/512224939 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7fbcc8018350 con 0x7fbcd807be70 2023-10-11T16:23:48.040 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc4038850 msgr2=0x7fbcc403ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:48.040 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 --2- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc4038850 0x7fbcc403ad00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0006fd0 tx=0x7fbcc0006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:48.040 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 msgr2=0x7fbcd819b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:48.041 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 --2- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd819b9d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8011980 tx=0x7fbcc8011a60 comp rx=0 tx=0).stop 2023-10-11T16:23:48.041 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 shutdown_connections 2023-10-11T16:23:48.041 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 --2- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc4038850 0x7fbcc403ad00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:48.041 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 --2- 172.21.15.111:0/512224939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbcd807be70 0x7fbcd819b9d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:48.041 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 >> 172.21.15.111:0/512224939 conn(0x7fbcd8100fd0 msgr2=0x7fbcd81026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:48.042 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 shutdown_connections 2023-10-11T16:23:48.042 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2023-10-11T16:23:47.558+0000 7fbcdc8b4700 1 -- 172.21.15.111:0/512224939 wait complete. 2023-10-11T16:23:48.042 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-10-11T16:23:48.042 INFO:teuthology.orchestra.run.smithi111.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-10-11T16:23:48.042 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:48.043 INFO:teuthology.orchestra.run.smithi111.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:23:48.043 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:48.043 INFO:teuthology.orchestra.run.smithi111.stdout:Or, if you are only running a single cluster on this host: 2023-10-11T16:23:48.043 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:48.043 INFO:teuthology.orchestra.run.smithi111.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-10-11T16:23:48.043 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:48.044 INFO:teuthology.orchestra.run.smithi111.stdout:Please consider enabling telemetry to help improve Ceph: 2023-10-11T16:23:48.044 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:48.044 INFO:teuthology.orchestra.run.smithi111.stdout: ceph telemetry on 2023-10-11T16:23:48.044 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:48.044 INFO:teuthology.orchestra.run.smithi111.stdout:For more information see: 2023-10-11T16:23:48.045 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:48.045 INFO:teuthology.orchestra.run.smithi111.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-10-11T16:23:48.045 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:23:48.045 INFO:teuthology.orchestra.run.smithi111.stdout:Bootstrap complete. 2023-10-11T16:23:48.087 INFO:tasks.cephadm:Fetching config... 2023-10-11T16:23:48.088 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:23:48.088 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-10-11T16:23:48.131 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-10-11T16:23:48.132 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:23:48.132 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-10-11T16:23:48.220 INFO:tasks.cephadm:Fetching mon keyring... 2023-10-11T16:23:48.221 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:23:48.221 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/keyring of=/dev/stdout 2023-10-11T16:23:48.291 INFO:tasks.cephadm:Fetching pub ssh key... 2023-10-11T16:23:48.292 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:23:48.292 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-10-11T16:23:48.374 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-10-11T16:23:48.375 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv4rY7gmsg9O2tzRVG9zZJ1YZK32mb9wRcpFJ89gxtvzvPYHxNiwDM9SweS686KkE7t+zGqrfTIBi/mWBRNaI0RDWOII7Vy324q8VfP/1SdBUBNRBOk6AAKq6Gg/9NGkQq7q2O1zerwtbaxW76cGYAIlKKdDVHDC1ihuPCE1+If/xOTT3TK/kwwNdg4ihbRb4LUKHznRSvh4/yMm6uxpfEDBsy9h3KSc5zc5Sm2aohdqiFnYfLd34bVGTWRKPTUmmLkPyiUcs1cLihNCjo1DTxNHYcxXFVw+DoZYbPWri6/PpOCtL1MGSV1OlL8lLbr7Sl/nVmvcb7Z9JV8X3gSu8GgsSzAcdfoFoA57QShRlZqnW23kAlyUaki81A8g9hh+mSi6m57tgew2K/uQwmtF6h7Kj1eVPvqjK9xNZeygPEoC/p/AXk48YZtqADF5m2e+UMvyhTPy814ELOLmdBCZM50lYa6yiIKimblvsPocCgHW9LxfMxMHmfT7giOS+QQZM= ceph-495f265e-6852-11ee-8db6-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-11T16:23:48.527 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv4rY7gmsg9O2tzRVG9zZJ1YZK32mb9wRcpFJ89gxtvzvPYHxNiwDM9SweS686KkE7t+zGqrfTIBi/mWBRNaI0RDWOII7Vy324q8VfP/1SdBUBNRBOk6AAKq6Gg/9NGkQq7q2O1zerwtbaxW76cGYAIlKKdDVHDC1ihuPCE1+If/xOTT3TK/kwwNdg4ihbRb4LUKHznRSvh4/yMm6uxpfEDBsy9h3KSc5zc5Sm2aohdqiFnYfLd34bVGTWRKPTUmmLkPyiUcs1cLihNCjo1DTxNHYcxXFVw+DoZYbPWri6/PpOCtL1MGSV1OlL8lLbr7Sl/nVmvcb7Z9JV8X3gSu8GgsSzAcdfoFoA57QShRlZqnW23kAlyUaki81A8g9hh+mSi6m57tgew2K/uQwmtF6h7Kj1eVPvqjK9xNZeygPEoC/p/AXk48YZtqADF5m2e+UMvyhTPy814ELOLmdBCZM50lYa6yiIKimblvsPocCgHW9LxfMxMHmfT7giOS+QQZM= ceph-495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:23:48.559 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv4rY7gmsg9O2tzRVG9zZJ1YZK32mb9wRcpFJ89gxtvzvPYHxNiwDM9SweS686KkE7t+zGqrfTIBi/mWBRNaI0RDWOII7Vy324q8VfP/1SdBUBNRBOk6AAKq6Gg/9NGkQq7q2O1zerwtbaxW76cGYAIlKKdDVHDC1ihuPCE1+If/xOTT3TK/kwwNdg4ihbRb4LUKHznRSvh4/yMm6uxpfEDBsy9h3KSc5zc5Sm2aohdqiFnYfLd34bVGTWRKPTUmmLkPyiUcs1cLihNCjo1DTxNHYcxXFVw+DoZYbPWri6/PpOCtL1MGSV1OlL8lLbr7Sl/nVmvcb7Z9JV8X3gSu8GgsSzAcdfoFoA57QShRlZqnW23kAlyUaki81A8g9hh+mSi6m57tgew2K/uQwmtF6h7Kj1eVPvqjK9xNZeygPEoC/p/AXk48YZtqADF5m2e+UMvyhTPy814ELOLmdBCZM50lYa6yiIKimblvsPocCgHW9LxfMxMHmfT7giOS+QQZM= ceph-495f265e-6852-11ee-8db6-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-11T16:23:48.605 INFO:teuthology.orchestra.run.smithi130.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv4rY7gmsg9O2tzRVG9zZJ1YZK32mb9wRcpFJ89gxtvzvPYHxNiwDM9SweS686KkE7t+zGqrfTIBi/mWBRNaI0RDWOII7Vy324q8VfP/1SdBUBNRBOk6AAKq6Gg/9NGkQq7q2O1zerwtbaxW76cGYAIlKKdDVHDC1ihuPCE1+If/xOTT3TK/kwwNdg4ihbRb4LUKHznRSvh4/yMm6uxpfEDBsy9h3KSc5zc5Sm2aohdqiFnYfLd34bVGTWRKPTUmmLkPyiUcs1cLihNCjo1DTxNHYcxXFVw+DoZYbPWri6/PpOCtL1MGSV1OlL8lLbr7Sl/nVmvcb7Z9JV8X3gSu8GgsSzAcdfoFoA57QShRlZqnW23kAlyUaki81A8g9hh+mSi6m57tgew2K/uQwmtF6h7Kj1eVPvqjK9xNZeygPEoC/p/AXk48YZtqADF5m2e+UMvyhTPy814ELOLmdBCZM50lYa6yiIKimblvsPocCgHW9LxfMxMHmfT7giOS+QQZM= ceph-495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:23:48.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv4rY7gmsg9O2tzRVG9zZJ1YZK32mb9wRcpFJ89gxtvzvPYHxNiwDM9SweS686KkE7t+zGqrfTIBi/mWBRNaI0RDWOII7Vy324q8VfP/1SdBUBNRBOk6AAKq6Gg/9NGkQq7q2O1zerwtbaxW76cGYAIlKKdDVHDC1ihuPCE1+If/xOTT3TK/kwwNdg4ihbRb4LUKHznRSvh4/yMm6uxpfEDBsy9h3KSc5zc5Sm2aohdqiFnYfLd34bVGTWRKPTUmmLkPyiUcs1cLihNCjo1DTxNHYcxXFVw+DoZYbPWri6/PpOCtL1MGSV1OlL8lLbr7Sl/nVmvcb7Z9JV8X3gSu8GgsSzAcdfoFoA57QShRlZqnW23kAlyUaki81A8g9hh+mSi6m57tgew2K/uQwmtF6h7Kj1eVPvqjK9xNZeygPEoC/p/AXk48YZtqADF5m2e+UMvyhTPy814ELOLmdBCZM50lYa6yiIKimblvsPocCgHW9LxfMxMHmfT7giOS+QQZM= ceph-495f265e-6852-11ee-8db6-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-11T16:23:48.663 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv4rY7gmsg9O2tzRVG9zZJ1YZK32mb9wRcpFJ89gxtvzvPYHxNiwDM9SweS686KkE7t+zGqrfTIBi/mWBRNaI0RDWOII7Vy324q8VfP/1SdBUBNRBOk6AAKq6Gg/9NGkQq7q2O1zerwtbaxW76cGYAIlKKdDVHDC1ihuPCE1+If/xOTT3TK/kwwNdg4ihbRb4LUKHznRSvh4/yMm6uxpfEDBsy9h3KSc5zc5Sm2aohdqiFnYfLd34bVGTWRKPTUmmLkPyiUcs1cLihNCjo1DTxNHYcxXFVw+DoZYbPWri6/PpOCtL1MGSV1OlL8lLbr7Sl/nVmvcb7Z9JV8X3gSu8GgsSzAcdfoFoA57QShRlZqnW23kAlyUaki81A8g9hh+mSi6m57tgew2K/uQwmtF6h7Kj1eVPvqjK9xNZeygPEoC/p/AXk48YZtqADF5m2e+UMvyhTPy814ELOLmdBCZM50lYa6yiIKimblvsPocCgHW9LxfMxMHmfT7giOS+QQZM= ceph-495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:23:48.676 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-10-11T16:23:49.706 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:23:50.934 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:50.935 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:50.935 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:23:50.935 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:50.935 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:50.936 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:51.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.110+0000 7fd495a67700 1 -- 172.21.15.111:0/3056661157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 msgr2=0x7fd490100130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:51.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.110+0000 7fd495a67700 1 --2- 172.21.15.111:0/3056661157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 0x7fd490100130 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd480009910 tx=0x7fd480009c20 comp rx=0 tx=0).stop 2023-10-11T16:23:51.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.110+0000 7fd495a67700 1 -- 172.21.15.111:0/3056661157 shutdown_connections 2023-10-11T16:23:51.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.110+0000 7fd495a67700 1 --2- 172.21.15.111:0/3056661157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 0x7fd490100130 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:51.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.110+0000 7fd495a67700 1 -- 172.21.15.111:0/3056661157 >> 172.21.15.111:0/3056661157 conn(0x7fd4900fb2b0 msgr2=0x7fd4900fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:51.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.110+0000 7fd495a67700 1 -- 172.21.15.111:0/3056661157 shutdown_connections 2023-10-11T16:23:51.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.110+0000 7fd495a67700 1 -- 172.21.15.111:0/3056661157 wait complete. 2023-10-11T16:23:51.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.111+0000 7fd495a67700 1 Processor -- start 2023-10-11T16:23:51.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.111+0000 7fd495a67700 1 -- start start 2023-10-11T16:23:51.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.112+0000 7fd495a67700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 0x7fd490074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:51.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.112+0000 7fd495a67700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd490074dd0 con 0x7fd4900ffd20 2023-10-11T16:23:51.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.112+0000 7fd48f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 0x7fd490074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:51.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.112+0000 7fd48f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 0x7fd490074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46570/0 (socket says 172.21.15.111:46570) 2023-10-11T16:23:51.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.112+0000 7fd48f7fe700 1 -- 172.21.15.111:0/663066023 learned_addr learned my addr 172.21.15.111:0/663066023 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:51.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.112+0000 7fd48f7fe700 1 -- 172.21.15.111:0/663066023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4800095f0 con 0x7fd4900ffd20 2023-10-11T16:23:51.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.113+0000 7fd48f7fe700 1 --2- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 0x7fd490074890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd480004f40 tx=0x7fd4800057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:51.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.113+0000 7fd48cff9700 1 -- 172.21.15.111:0/663066023 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd48001d070 con 0x7fd4900ffd20 2023-10-11T16:23:51.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.113+0000 7fd48cff9700 1 -- 172.21.15.111:0/663066023 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd48000bc10 con 0x7fd4900ffd20 2023-10-11T16:23:51.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.113+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd490105bf0 con 0x7fd4900ffd20 2023-10-11T16:23:51.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.113+0000 7fd48cff9700 1 -- 172.21.15.111:0/663066023 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd480003a70 con 0x7fd4900ffd20 2023-10-11T16:23:51.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.113+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd490105a60 con 0x7fd4900ffd20 2023-10-11T16:23:51.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.114+0000 7fd48cff9700 1 -- 172.21.15.111:0/663066023 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fd480003ce0 con 0x7fd4900ffd20 2023-10-11T16:23:51.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.114+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd490062120 con 0x7fd4900ffd20 2023-10-11T16:23:51.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.114+0000 7fd48cff9700 1 --2- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd4780388c0 0x7fd47803ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:51.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.115+0000 7fd48cff9700 1 -- 172.21.15.111:0/663066023 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd48004c550 con 0x7fd4900ffd20 2023-10-11T16:23:51.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.115+0000 7fd48effd700 1 --2- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd4780388c0 0x7fd47803ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:51.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.115+0000 7fd48effd700 1 --2- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd4780388c0 0x7fd47803ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd484006fd0 tx=0x7fd484006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:51.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.120+0000 7fd48cff9700 1 -- 172.21.15.111:0/663066023 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd48002abf0 con 0x7fd4900ffd20 2023-10-11T16:23:51.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.314+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd49018e750 con 0x7fd4900ffd20 2023-10-11T16:23:51.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.318+0000 7fd48cff9700 1 -- 172.21.15.111:0/663066023 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd480026070 con 0x7fd4900ffd20 2023-10-11T16:23:51.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.320+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd4780388c0 msgr2=0x7fd47803ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:51.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.320+0000 7fd495a67700 1 --2- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd4780388c0 0x7fd47803ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd484006fd0 tx=0x7fd484006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:51.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.321+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 msgr2=0x7fd490074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:51.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.321+0000 7fd495a67700 1 --2- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 0x7fd490074890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd480004f40 tx=0x7fd4800057c0 comp rx=0 tx=0).stop 2023-10-11T16:23:51.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.321+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 shutdown_connections 2023-10-11T16:23:51.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.321+0000 7fd495a67700 1 --2- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd4780388c0 0x7fd47803ad70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:51.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.321+0000 7fd495a67700 1 --2- 172.21.15.111:0/663066023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4900ffd20 0x7fd490074890 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:51.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.321+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 >> 172.21.15.111:0/663066023 conn(0x7fd4900fb2b0 msgr2=0x7fd4900fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:51.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.321+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 shutdown_connections 2023-10-11T16:23:51.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:51.321+0000 7fd495a67700 1 -- 172.21.15.111:0/663066023 wait complete. 2023-10-11T16:23:51.792 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-10-11T16:23:51.793 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-10-11T16:23:52.632 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:23:52.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:52 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/663066023' entity='client.admin' 2023-10-11T16:23:52.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:52 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:52.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:52 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:54.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.080+0000 7fc4573f2700 1 -- 172.21.15.111:0/3461561875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 msgr2=0x7fc450100130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:54.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.080+0000 7fc4573f2700 1 --2- 172.21.15.111:0/3461561875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 0x7fc450100130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009910 tx=0x7fc44c009c20 comp rx=0 tx=0).stop 2023-10-11T16:23:54.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.080+0000 7fc4573f2700 1 -- 172.21.15.111:0/3461561875 shutdown_connections 2023-10-11T16:23:54.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.080+0000 7fc4573f2700 1 --2- 172.21.15.111:0/3461561875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 0x7fc450100130 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:54.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.080+0000 7fc4573f2700 1 -- 172.21.15.111:0/3461561875 >> 172.21.15.111:0/3461561875 conn(0x7fc4500fb2b0 msgr2=0x7fc4500fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:54.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.080+0000 7fc4573f2700 1 -- 172.21.15.111:0/3461561875 shutdown_connections 2023-10-11T16:23:54.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.080+0000 7fc4573f2700 1 -- 172.21.15.111:0/3461561875 wait complete. 2023-10-11T16:23:54.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.081+0000 7fc4573f2700 1 Processor -- start 2023-10-11T16:23:54.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.081+0000 7fc4573f2700 1 -- start start 2023-10-11T16:23:54.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.082+0000 7fc4573f2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 0x7fc450194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:54.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.082+0000 7fc4573f2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc450195340 con 0x7fc4500ffd20 2023-10-11T16:23:54.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.082+0000 7fc45518e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 0x7fc450194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:54.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.082+0000 7fc45518e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 0x7fc450194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46590/0 (socket says 172.21.15.111:46590) 2023-10-11T16:23:54.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.082+0000 7fc45518e700 1 -- 172.21.15.111:0/1893719347 learned_addr learned my addr 172.21.15.111:0/1893719347 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:54.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.082+0000 7fc45518e700 1 -- 172.21.15.111:0/1893719347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c0095f0 con 0x7fc4500ffd20 2023-10-11T16:23:54.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.082+0000 7fc45518e700 1 --2- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 0x7fc450194e00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc44c004f40 tx=0x7fc44c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:54.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.083+0000 7fc4467fc700 1 -- 172.21.15.111:0/1893719347 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc44c01d070 con 0x7fc4500ffd20 2023-10-11T16:23:54.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.083+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc450105bf0 con 0x7fc4500ffd20 2023-10-11T16:23:54.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.083+0000 7fc4467fc700 1 -- 172.21.15.111:0/1893719347 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc44c00bc10 con 0x7fc4500ffd20 2023-10-11T16:23:54.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.083+0000 7fc4467fc700 1 -- 172.21.15.111:0/1893719347 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc44c003a70 con 0x7fc4500ffd20 2023-10-11T16:23:54.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.083+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc450105a60 con 0x7fc4500ffd20 2023-10-11T16:23:54.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.084+0000 7fc4467fc700 1 -- 172.21.15.111:0/1893719347 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fc44c003d30 con 0x7fc4500ffd20 2023-10-11T16:23:54.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.084+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc450105d80 con 0x7fc4500ffd20 2023-10-11T16:23:54.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.084+0000 7fc4467fc700 1 --2- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc43c038870 0x7fc43c03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:54.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.084+0000 7fc4467fc700 1 -- 172.21.15.111:0/1893719347 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc44c04c5a0 con 0x7fc4500ffd20 2023-10-11T16:23:54.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.085+0000 7fc45498d700 1 --2- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc43c038870 0x7fc43c03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:54.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.085+0000 7fc45498d700 1 --2- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc43c038870 0x7fc43c03ad20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc440006fd0 tx=0x7fc440006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:54.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.089+0000 7fc4467fc700 1 -- 172.21.15.111:0/1893719347 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc44c02ae20 con 0x7fc4500ffd20 2023-10-11T16:23:54.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.291+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fc450060f30 con 0x7fc43c038870 2023-10-11T16:23:54.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.294+0000 7fc4467fc700 1 -- 172.21.15.111:0/1893719347 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc450060f30 con 0x7fc43c038870 2023-10-11T16:23:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc43c038870 msgr2=0x7fc43c03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 --2- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc43c038870 0x7fc43c03ad20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc440006fd0 tx=0x7fc440006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 msgr2=0x7fc450194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 --2- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 0x7fc450194e00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc44c004f40 tx=0x7fc44c0057c0 comp rx=0 tx=0).stop 2023-10-11T16:23:54.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 shutdown_connections 2023-10-11T16:23:54.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 --2- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc43c038870 0x7fc43c03ad20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:54.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 --2- 172.21.15.111:0/1893719347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4500ffd20 0x7fc450194e00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:54.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 >> 172.21.15.111:0/1893719347 conn(0x7fc4500fb2b0 msgr2=0x7fc4500fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:54.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 shutdown_connections 2023-10-11T16:23:54.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:54.297+0000 7fc4573f2700 1 -- 172.21.15.111:0/1893719347 wait complete. 2023-10-11T16:23:54.712 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi130 2023-10-11T16:23:54.713 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:23:54.713 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.conf 2023-10-11T16:23:54.735 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:23:54.735 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:23:54.794 INFO:tasks.cephadm:Adding host smithi130 to orchestrator... 2023-10-11T16:23:54.795 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch host add smithi130 2023-10-11T16:23:55.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:55 smithi111 ceph-mon[98527]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:55.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:55.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:23:55.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:23:55.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:23:55.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:55 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:23:55.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:55 smithi111 ceph-mon[98527]: Updating smithi111:/etc/ceph/ceph.conf 2023-10-11T16:23:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:23:56.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:56 smithi111 ceph-mon[98527]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:23:56.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:56 smithi111 ceph-mon[98527]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:23:57.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.150+0000 7f9c5fd46700 1 -- 172.21.15.111:0/3830365563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 msgr2=0x7f9c581011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:57.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.150+0000 7f9c5fd46700 1 --2- 172.21.15.111:0/3830365563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 0x7f9c581011d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9c54009910 tx=0x7f9c54009c20 comp rx=0 tx=0).stop 2023-10-11T16:23:57.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.150+0000 7f9c5fd46700 1 -- 172.21.15.111:0/3830365563 shutdown_connections 2023-10-11T16:23:57.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.150+0000 7f9c5fd46700 1 --2- 172.21.15.111:0/3830365563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 0x7f9c581011d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:57.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.150+0000 7f9c5fd46700 1 -- 172.21.15.111:0/3830365563 >> 172.21.15.111:0/3830365563 conn(0x7f9c58074940 msgr2=0x7f9c58074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:57.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.150+0000 7f9c5fd46700 1 -- 172.21.15.111:0/3830365563 shutdown_connections 2023-10-11T16:23:57.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.150+0000 7f9c5fd46700 1 -- 172.21.15.111:0/3830365563 wait complete. 2023-10-11T16:23:57.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.151+0000 7f9c5fd46700 1 Processor -- start 2023-10-11T16:23:57.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.151+0000 7f9c5fd46700 1 -- start start 2023-10-11T16:23:57.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.151+0000 7f9c5fd46700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 0x7f9c581973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:57.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.151+0000 7f9c5fd46700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c581978e0 con 0x7f9c58100e00 2023-10-11T16:23:57.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c5dae2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 0x7f9c581973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:57.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c5dae2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 0x7f9c581973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32858/0 (socket says 172.21.15.111:32858) 2023-10-11T16:23:57.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c5dae2700 1 -- 172.21.15.111:0/1469610292 learned_addr learned my addr 172.21.15.111:0/1469610292 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:23:57.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c5dae2700 1 -- 172.21.15.111:0/1469610292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c540095f0 con 0x7f9c58100e00 2023-10-11T16:23:57.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c5dae2700 1 --2- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 0x7f9c581973a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9c54006010 tx=0x7f9c54004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:57.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c4effd700 1 -- 172.21.15.111:0/1469610292 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c5401d070 con 0x7f9c58100e00 2023-10-11T16:23:57.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c4effd700 1 -- 172.21.15.111:0/1469610292 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c540044a0 con 0x7f9c58100e00 2023-10-11T16:23:57.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c4effd700 1 -- 172.21.15.111:0/1469610292 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c54003a70 con 0x7f9c58100e00 2023-10-11T16:23:57.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.152+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5810a1a0 con 0x7f9c58100e00 2023-10-11T16:23:57.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.153+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5810a010 con 0x7f9c58100e00 2023-10-11T16:23:57.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.154+0000 7f9c4effd700 1 -- 172.21.15.111:0/1469610292 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f9c54003c60 con 0x7f9c58100e00 2023-10-11T16:23:57.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.154+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5810a330 con 0x7f9c58100e00 2023-10-11T16:23:57.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.154+0000 7f9c4effd700 1 --2- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9c440388c0 0x7f9c4403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:23:57.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.154+0000 7f9c4effd700 1 -- 172.21.15.111:0/1469610292 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9c5404c620 con 0x7f9c58100e00 2023-10-11T16:23:57.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.154+0000 7f9c5d2e1700 1 --2- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9c440388c0 0x7f9c4403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:23:57.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.155+0000 7f9c5d2e1700 1 --2- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9c440388c0 0x7f9c4403ad70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9c48006fd0 tx=0x7f9c48006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:23:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.159+0000 7f9c4effd700 1 -- 172.21.15.111:0/1469610292 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9c54005c00 con 0x7f9c58100e00 2023-10-11T16:23:57.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:57.360+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c58197f30 con 0x7f9c440388c0 2023-10-11T16:23:57.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:57 smithi111 ceph-mon[98527]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:23:57.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:57 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:57.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:57 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:57.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:57 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:23:58.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:58 smithi111 ceph-mon[98527]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:23:59.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.463+0000 7f9c4effd700 1 -- 172.21.15.111:0/1469610292 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9c58197f30 con 0x7f9c440388c0 2023-10-11T16:23:59.464 INFO:teuthology.orchestra.run.smithi111.stdout:Added host 'smithi130' with addr '172.21.15.130' 2023-10-11T16:23:59.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9c440388c0 msgr2=0x7f9c4403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:59.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 --2- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9c440388c0 0x7f9c4403ad70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9c48006fd0 tx=0x7f9c48006e40 comp rx=0 tx=0).stop 2023-10-11T16:23:59.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 msgr2=0x7f9c581973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:23:59.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 --2- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 0x7f9c581973a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9c54006010 tx=0x7f9c54004ff0 comp rx=0 tx=0).stop 2023-10-11T16:23:59.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 shutdown_connections 2023-10-11T16:23:59.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 --2- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9c440388c0 0x7f9c4403ad70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:59.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 --2- 172.21.15.111:0/1469610292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c58100e00 0x7f9c581973a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:23:59.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 >> 172.21.15.111:0/1469610292 conn(0x7f9c58074940 msgr2=0x7f9c58106a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:23:59.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.465+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 shutdown_connections 2023-10-11T16:23:59.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:23:59.466+0000 7f9c5fd46700 1 -- 172.21.15.111:0/1469610292 wait complete. 2023-10-11T16:23:59.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:23:59 smithi111 ceph-mon[98527]: Deploying cephadm binary to smithi130 2023-10-11T16:23:59.954 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch host ls --format=json 2023-10-11T16:24:00.801 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:24:00.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:00.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:00 smithi111 ceph-mon[98527]: Added host smithi130 2023-10-11T16:24:00.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:00.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:01.602 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:01 smithi111 ceph-mon[98527]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:01.602 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:02.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.210+0000 7f7918362700 1 -- 172.21.15.111:0/110364352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 msgr2=0x7f7910100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:02.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.210+0000 7f7918362700 1 --2- 172.21.15.111:0/110364352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 0x7f7910100360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f790c009910 tx=0x7f790c009c20 comp rx=0 tx=0).stop 2023-10-11T16:24:02.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.210+0000 7f7918362700 1 -- 172.21.15.111:0/110364352 shutdown_connections 2023-10-11T16:24:02.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.210+0000 7f7918362700 1 --2- 172.21.15.111:0/110364352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 0x7f7910100360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:02.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.210+0000 7f7918362700 1 -- 172.21.15.111:0/110364352 >> 172.21.15.111:0/110364352 conn(0x7f79100fb870 msgr2=0x7f79100fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:02.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.210+0000 7f7918362700 1 -- 172.21.15.111:0/110364352 shutdown_connections 2023-10-11T16:24:02.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.210+0000 7f7918362700 1 -- 172.21.15.111:0/110364352 wait complete. 2023-10-11T16:24:02.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.211+0000 7f7918362700 1 Processor -- start 2023-10-11T16:24:02.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.211+0000 7f7918362700 1 -- start start 2023-10-11T16:24:02.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.212+0000 7f7918362700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 0x7f7910072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:02.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.212+0000 7f7918362700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7910074d70 con 0x7f79100fff90 2023-10-11T16:24:02.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.212+0000 7f79160fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 0x7f7910072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:02.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.212+0000 7f79160fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 0x7f7910072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32872/0 (socket says 172.21.15.111:32872) 2023-10-11T16:24:02.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.212+0000 7f79160fe700 1 -- 172.21.15.111:0/2091175452 learned_addr learned my addr 172.21.15.111:0/2091175452 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:24:02.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.212+0000 7f79160fe700 1 -- 172.21.15.111:0/2091175452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c0095f0 con 0x7f79100fff90 2023-10-11T16:24:02.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.212+0000 7f79160fe700 1 --2- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 0x7f7910072df0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f790c004f70 tx=0x7f790c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:02.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.213+0000 7f79077fe700 1 -- 172.21.15.111:0/2091175452 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f790c01d070 con 0x7f79100fff90 2023-10-11T16:24:02.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.213+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791010a170 con 0x7f79100fff90 2023-10-11T16:24:02.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.213+0000 7f79077fe700 1 -- 172.21.15.111:0/2091175452 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f790c00bc10 con 0x7f79100fff90 2023-10-11T16:24:02.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.213+0000 7f79077fe700 1 -- 172.21.15.111:0/2091175452 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f790c003970 con 0x7f79100fff90 2023-10-11T16:24:02.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.213+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7910109fe0 con 0x7f79100fff90 2023-10-11T16:24:02.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.214+0000 7f79077fe700 1 -- 172.21.15.111:0/2091175452 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f790c004390 con 0x7f79100fff90 2023-10-11T16:24:02.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.214+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791010a300 con 0x7f79100fff90 2023-10-11T16:24:02.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.214+0000 7f79077fe700 1 --2- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f78fc038940 0x7f78fc03adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:02.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.215+0000 7f79077fe700 1 -- 172.21.15.111:0/2091175452 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f790c04c510 con 0x7f79100fff90 2023-10-11T16:24:02.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.215+0000 7f79158fd700 1 --2- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f78fc038940 0x7f78fc03adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:02.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.215+0000 7f79158fd700 1 --2- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f78fc038940 0x7f78fc03adf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7900006fd0 tx=0x7f7900006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:02.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.219+0000 7f79077fe700 1 -- 172.21.15.111:0/2091175452 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f790c029360 con 0x7f79100fff90 2023-10-11T16:24:02.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.423+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7910074320 con 0x7f78fc038940 2023-10-11T16:24:02.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.424+0000 7f79077fe700 1 -- 172.21.15.111:0/2091175452 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f7910074320 con 0x7f78fc038940 2023-10-11T16:24:02.426 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:24:02.426 INFO:teuthology.orchestra.run.smithi111.stdout:[{"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}] 2023-10-11T16:24:02.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.426+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f78fc038940 msgr2=0x7f78fc03adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:02.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.426+0000 7f7918362700 1 --2- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f78fc038940 0x7f78fc03adf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7900006fd0 tx=0x7f7900006e40 comp rx=0 tx=0).stop 2023-10-11T16:24:02.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.426+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 msgr2=0x7f7910072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:02.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.426+0000 7f7918362700 1 --2- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 0x7f7910072df0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f790c004f70 tx=0x7f790c005050 comp rx=0 tx=0).stop 2023-10-11T16:24:02.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.427+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 shutdown_connections 2023-10-11T16:24:02.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.427+0000 7f7918362700 1 --2- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f78fc038940 0x7f78fc03adf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:02.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.427+0000 7f7918362700 1 --2- 172.21.15.111:0/2091175452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79100fff90 0x7f7910072df0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:02.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.427+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 >> 172.21.15.111:0/2091175452 conn(0x7f79100fb870 msgr2=0x7f7910104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:02.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.427+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 shutdown_connections 2023-10-11T16:24:02.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:02.427+0000 7f7918362700 1 -- 172.21.15.111:0/2091175452 wait complete. 2023-10-11T16:24:02.885 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi150 2023-10-11T16:24:02.886 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:24:02.886 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.conf 2023-10-11T16:24:02.910 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:24:02.911 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:02.972 INFO:tasks.cephadm:Adding host smithi150 to orchestrator... 2023-10-11T16:24:02.973 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch host add smithi150 2023-10-11T16:24:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:03 smithi111 ceph-mon[98527]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:03 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:03 smithi111 ceph-mon[98527]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:24:03.816 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:24:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.209+0000 7ff11db0d700 1 -- 172.21.15.111:0/3343893205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 msgr2=0x7ff118073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.209+0000 7ff11db0d700 1 --2- 172.21.15.111:0/3343893205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 0x7ff118073260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff108009910 tx=0x7ff108009c20 comp rx=0 tx=0).stop 2023-10-11T16:24:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.210+0000 7ff11db0d700 1 -- 172.21.15.111:0/3343893205 shutdown_connections 2023-10-11T16:24:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.210+0000 7ff11db0d700 1 --2- 172.21.15.111:0/3343893205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 0x7ff118073260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.210+0000 7ff11db0d700 1 -- 172.21.15.111:0/3343893205 >> 172.21.15.111:0/3343893205 conn(0x7ff1180fb870 msgr2=0x7ff1180fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.210+0000 7ff11db0d700 1 -- 172.21.15.111:0/3343893205 shutdown_connections 2023-10-11T16:24:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.210+0000 7ff11db0d700 1 -- 172.21.15.111:0/3343893205 wait complete. 2023-10-11T16:24:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.211+0000 7ff11db0d700 1 Processor -- start 2023-10-11T16:24:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.211+0000 7ff11db0d700 1 -- start start 2023-10-11T16:24:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.212+0000 7ff11db0d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 0x7ff1181a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.212+0000 7ff11db0d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1181a1aa0 con 0x7ff118072e90 2023-10-11T16:24:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.212+0000 7ff1177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 0x7ff1181a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.212+0000 7ff1177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 0x7ff1181a1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53172/0 (socket says 172.21.15.111:53172) 2023-10-11T16:24:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.212+0000 7ff1177fe700 1 -- 172.21.15.111:0/2610754852 learned_addr learned my addr 172.21.15.111:0/2610754852 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:24:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.212+0000 7ff1177fe700 1 -- 172.21.15.111:0/2610754852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1080095f0 con 0x7ff118072e90 2023-10-11T16:24:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.213+0000 7ff1177fe700 1 --2- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 0x7ff1181a1560 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff108006010 tx=0x7ff108004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.213+0000 7ff114ff9700 1 -- 172.21.15.111:0/2610754852 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff10801d070 con 0x7ff118072e90 2023-10-11T16:24:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.213+0000 7ff114ff9700 1 -- 172.21.15.111:0/2610754852 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1080044a0 con 0x7ff118072e90 2023-10-11T16:24:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.213+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff118110780 con 0x7ff118072e90 2023-10-11T16:24:05.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.213+0000 7ff114ff9700 1 -- 172.21.15.111:0/2610754852 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff108003a70 con 0x7ff118072e90 2023-10-11T16:24:05.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.213+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1181105f0 con 0x7ff118072e90 2023-10-11T16:24:05.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.214+0000 7ff114ff9700 1 -- 172.21.15.111:0/2610754852 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7ff108003c60 con 0x7ff118072e90 2023-10-11T16:24:05.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.214+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff118110910 con 0x7ff118072e90 2023-10-11T16:24:05.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.214+0000 7ff114ff9700 1 --2- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff1000388c0 0x7ff10003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:05.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.215+0000 7ff114ff9700 1 -- 172.21.15.111:0/2610754852 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff10804c5c0 con 0x7ff118072e90 2023-10-11T16:24:05.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.215+0000 7ff116ffd700 1 --2- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff1000388c0 0x7ff10003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:05.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.215+0000 7ff116ffd700 1 --2- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff1000388c0 0x7ff10003ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff10c006fd0 tx=0x7ff10c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:05.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.220+0000 7ff114ff9700 1 -- 172.21.15.111:0/2610754852 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff108029540 con 0x7ff118072e90 2023-10-11T16:24:05.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:05.427+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1180008d0 con 0x7ff1000388c0 2023-10-11T16:24:05.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:05 smithi111 ceph-mon[98527]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:24:05.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:05.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:06.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:06.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:24:06.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:06.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:06.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:24:06.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:06.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:06.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:06.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:07.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.533+0000 7ff114ff9700 1 -- 172.21.15.111:0/2610754852 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff1180008d0 con 0x7ff1000388c0 2023-10-11T16:24:07.535 INFO:teuthology.orchestra.run.smithi111.stdout:Added host 'smithi150' with addr '172.21.15.150' 2023-10-11T16:24:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff1000388c0 msgr2=0x7ff10003ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 --2- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff1000388c0 0x7ff10003ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff10c006fd0 tx=0x7ff10c006e40 comp rx=0 tx=0).stop 2023-10-11T16:24:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 msgr2=0x7ff1181a1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 --2- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 0x7ff1181a1560 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff108006010 tx=0x7ff108004ff0 comp rx=0 tx=0).stop 2023-10-11T16:24:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 shutdown_connections 2023-10-11T16:24:07.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 --2- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff1000388c0 0x7ff10003ad70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:07.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 --2- 172.21.15.111:0/2610754852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff118072e90 0x7ff1181a1560 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:07.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 >> 172.21.15.111:0/2610754852 conn(0x7ff1180fb870 msgr2=0x7ff1180fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:07.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.536+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 shutdown_connections 2023-10-11T16:24:07.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:07.537+0000 7ff11db0d700 1 -- 172.21.15.111:0/2610754852 wait complete. 2023-10-11T16:24:07.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:07 smithi111 ceph-mon[98527]: Deploying cephadm binary to smithi150 2023-10-11T16:24:07.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:07 smithi111 ceph-mon[98527]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:07.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:07 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:07.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:07 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:08.004 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch host ls --format=json 2023-10-11T16:24:08.841 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:24:08.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:08 smithi111 ceph-mon[98527]: Added host smithi150 2023-10-11T16:24:08.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:08 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:09.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:09 smithi111 ceph-mon[98527]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:09.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:10.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.223+0000 7f2632e7b700 1 -- 172.21.15.111:0/546824208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 msgr2=0x7f262c100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:10.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.223+0000 7f2632e7b700 1 --2- 172.21.15.111:0/546824208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 0x7f262c100360 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2620009910 tx=0x7f2620009c20 comp rx=0 tx=0).stop 2023-10-11T16:24:10.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.224+0000 7f2632e7b700 1 -- 172.21.15.111:0/546824208 shutdown_connections 2023-10-11T16:24:10.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.224+0000 7f2632e7b700 1 --2- 172.21.15.111:0/546824208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 0x7f262c100360 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:10.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.224+0000 7f2632e7b700 1 -- 172.21.15.111:0/546824208 >> 172.21.15.111:0/546824208 conn(0x7f262c0fb870 msgr2=0x7f262c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:10.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.224+0000 7f2632e7b700 1 -- 172.21.15.111:0/546824208 shutdown_connections 2023-10-11T16:24:10.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.224+0000 7f2632e7b700 1 -- 172.21.15.111:0/546824208 wait complete. 2023-10-11T16:24:10.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.225+0000 7f2632e7b700 1 Processor -- start 2023-10-11T16:24:10.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.225+0000 7f2632e7b700 1 -- start start 2023-10-11T16:24:10.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.225+0000 7f2632e7b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 0x7f262c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:10.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.225+0000 7f2632e7b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c074d70 con 0x7f262c0fff90 2023-10-11T16:24:10.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.225+0000 7f2630c17700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 0x7f262c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:10.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.225+0000 7f2630c17700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 0x7f262c072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53186/0 (socket says 172.21.15.111:53186) 2023-10-11T16:24:10.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.225+0000 7f2630c17700 1 -- 172.21.15.111:0/1406062434 learned_addr learned my addr 172.21.15.111:0/1406062434 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:24:10.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.226+0000 7f2630c17700 1 -- 172.21.15.111:0/1406062434 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200095f0 con 0x7f262c0fff90 2023-10-11T16:24:10.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.226+0000 7f2630c17700 1 --2- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 0x7f262c072df0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2620006010 tx=0x7f2620004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:10.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.226+0000 7f2629ffb700 1 -- 172.21.15.111:0/1406062434 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f262001d070 con 0x7f262c0fff90 2023-10-11T16:24:10.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.226+0000 7f2629ffb700 1 -- 172.21.15.111:0/1406062434 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f262000bc10 con 0x7f262c0fff90 2023-10-11T16:24:10.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.226+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c10a170 con 0x7f262c0fff90 2023-10-11T16:24:10.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.226+0000 7f2629ffb700 1 -- 172.21.15.111:0/1406062434 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2620003910 con 0x7f262c0fff90 2023-10-11T16:24:10.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.226+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c109fe0 con 0x7f262c0fff90 2023-10-11T16:24:10.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.227+0000 7f2629ffb700 1 -- 172.21.15.111:0/1406062434 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f262000bd80 con 0x7f262c0fff90 2023-10-11T16:24:10.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.227+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c10a300 con 0x7f262c0fff90 2023-10-11T16:24:10.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.228+0000 7f2629ffb700 1 --2- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f261c0388a0 0x7f261c03ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:10.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.228+0000 7f2629ffb700 1 -- 172.21.15.111:0/1406062434 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f262004c560 con 0x7f262c0fff90 2023-10-11T16:24:10.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.228+0000 7f262bfff700 1 --2- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f261c0388a0 0x7f261c03ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:10.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.228+0000 7f262bfff700 1 --2- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f261c0388a0 0x7f261c03ad50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2618006fd0 tx=0x7f2618006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:10.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.233+0000 7f2629ffb700 1 -- 172.21.15.111:0/1406062434 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2620029360 con 0x7f262c0fff90 2023-10-11T16:24:10.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.433+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f262c074320 con 0x7f261c0388a0 2023-10-11T16:24:10.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.435+0000 7f2629ffb700 1 -- 172.21.15.111:0/1406062434 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f262c074320 con 0x7f261c0388a0 2023-10-11T16:24:10.437 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:24:10.437 INFO:teuthology.orchestra.run.smithi111.stdout:[{"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}] 2023-10-11T16:24:10.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f261c0388a0 msgr2=0x7f261c03ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:10.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 --2- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f261c0388a0 0x7f261c03ad50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2618006fd0 tx=0x7f2618006e40 comp rx=0 tx=0).stop 2023-10-11T16:24:10.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 msgr2=0x7f262c072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:10.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 --2- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 0x7f262c072df0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2620006010 tx=0x7f2620004ff0 comp rx=0 tx=0).stop 2023-10-11T16:24:10.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 shutdown_connections 2023-10-11T16:24:10.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 --2- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f261c0388a0 0x7f261c03ad50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:10.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 --2- 172.21.15.111:0/1406062434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f262c0fff90 0x7f262c072df0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:10.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 >> 172.21.15.111:0/1406062434 conn(0x7f262c0fb870 msgr2=0x7f262c104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:10.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 shutdown_connections 2023-10-11T16:24:10.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:10.437+0000 7f2632e7b700 1 -- 172.21.15.111:0/1406062434 wait complete. 2023-10-11T16:24:10.915 INFO:tasks.cephadm:Setting crush tunables to default 2023-10-11T16:24:10.916 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd crush tunables default 2023-10-11T16:24:11.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:11 smithi111 ceph-mon[98527]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:11.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:11.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:11 smithi111 ceph-mon[98527]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:24:11.757 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:24:13.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.297+0000 7f574034f700 1 -- 172.21.15.111:0/1911242555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 msgr2=0x7f5738100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:13.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.297+0000 7f574034f700 1 --2- 172.21.15.111:0/1911242555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 0x7f5738100360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5734009910 tx=0x7f5734009c20 comp rx=0 tx=0).stop 2023-10-11T16:24:13.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.297+0000 7f574034f700 1 -- 172.21.15.111:0/1911242555 shutdown_connections 2023-10-11T16:24:13.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.297+0000 7f574034f700 1 --2- 172.21.15.111:0/1911242555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 0x7f5738100360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:13.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.297+0000 7f574034f700 1 -- 172.21.15.111:0/1911242555 >> 172.21.15.111:0/1911242555 conn(0x7f57380fb870 msgr2=0x7f57380fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:13.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.297+0000 7f574034f700 1 -- 172.21.15.111:0/1911242555 shutdown_connections 2023-10-11T16:24:13.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.297+0000 7f574034f700 1 -- 172.21.15.111:0/1911242555 wait complete. 2023-10-11T16:24:13.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.298+0000 7f574034f700 1 Processor -- start 2023-10-11T16:24:13.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.298+0000 7f574034f700 1 -- start start 2023-10-11T16:24:13.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.299+0000 7f574034f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 0x7f5738197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:13.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.299+0000 7f574034f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57381978c0 con 0x7f57380fff90 2023-10-11T16:24:13.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.299+0000 7f573e0eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 0x7f5738197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:13.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.299+0000 7f573e0eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 0x7f5738197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53200/0 (socket says 172.21.15.111:53200) 2023-10-11T16:24:13.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.299+0000 7f573e0eb700 1 -- 172.21.15.111:0/2640446437 learned_addr learned my addr 172.21.15.111:0/2640446437 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:24:13.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.299+0000 7f573e0eb700 1 -- 172.21.15.111:0/2640446437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57340095f0 con 0x7f57380fff90 2023-10-11T16:24:13.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.299+0000 7f573e0eb700 1 --2- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 0x7f5738197380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5734006010 tx=0x7f5734004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:13.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.300+0000 7f572f7fe700 1 -- 172.21.15.111:0/2640446437 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f573401d070 con 0x7f57380fff90 2023-10-11T16:24:13.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.300+0000 7f572f7fe700 1 -- 172.21.15.111:0/2640446437 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f573400bc10 con 0x7f57380fff90 2023-10-11T16:24:13.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.300+0000 7f572f7fe700 1 -- 172.21.15.111:0/2640446437 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5734003910 con 0x7f57380fff90 2023-10-11T16:24:13.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.300+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573810a170 con 0x7f57380fff90 2023-10-11T16:24:13.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.300+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5738109fe0 con 0x7f57380fff90 2023-10-11T16:24:13.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.301+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573810a300 con 0x7f57380fff90 2023-10-11T16:24:13.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.301+0000 7f572f7fe700 1 -- 172.21.15.111:0/2640446437 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f5734003b80 con 0x7f57380fff90 2023-10-11T16:24:13.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.301+0000 7f572f7fe700 1 --2- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57240388f0 0x7f572403ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:13.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.301+0000 7f572f7fe700 1 -- 172.21.15.111:0/2640446437 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f573404c540 con 0x7f57380fff90 2023-10-11T16:24:13.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.301+0000 7f573d8ea700 1 --2- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57240388f0 0x7f572403ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:13.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.302+0000 7f573d8ea700 1 --2- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57240388f0 0x7f572403ada0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5728006fd0 tx=0x7f5728006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:13.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.307+0000 7f572f7fe700 1 -- 172.21.15.111:0/2640446437 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f573400ba70 con 0x7f57380fff90 2023-10-11T16:24:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:13 smithi111 ceph-mon[98527]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:24:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:13.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:13.518+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5738197ec0 con 0x7f57380fff90 2023-10-11T16:24:14.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.117+0000 7f572f7fe700 1 -- 172.21.15.111:0/2640446437 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5734026020 con 0x7f57380fff90 2023-10-11T16:24:14.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57240388f0 msgr2=0x7f572403ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:14.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 --2- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57240388f0 0x7f572403ada0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5728006fd0 tx=0x7f5728006e40 comp rx=0 tx=0).stop 2023-10-11T16:24:14.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 msgr2=0x7f5738197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:14.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 --2- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 0x7f5738197380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5734006010 tx=0x7f5734004ff0 comp rx=0 tx=0).stop 2023-10-11T16:24:14.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 shutdown_connections 2023-10-11T16:24:14.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 --2- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57240388f0 0x7f572403ada0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:14.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 --2- 172.21.15.111:0/2640446437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57380fff90 0x7f5738197380 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:14.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 >> 172.21.15.111:0/2640446437 conn(0x7f57380fb870 msgr2=0x7f57381038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:14.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 shutdown_connections 2023-10-11T16:24:14.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:14.120+0000 7f574034f700 1 -- 172.21.15.111:0/2640446437 wait complete. 2023-10-11T16:24:14.124 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2023-10-11T16:24:14.368 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:14 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:14.369 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:14 smithi111 ceph-mon[98527]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:24:14.369 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:14 smithi111 ceph-mon[98527]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:14.369 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:14 smithi111 ceph-mon[98527]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:14.369 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:14 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2640446437' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-11T16:24:14.370 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:14.370 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:14.370 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:14.642 INFO:tasks.cephadm:Adding mon.a on smithi111 2023-10-11T16:24:14.643 INFO:tasks.cephadm:Adding mon.b on smithi130 2023-10-11T16:24:14.643 INFO:tasks.cephadm:Adding mon.c on smithi150 2023-10-11T16:24:14.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch apply mon '3;smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c' 2023-10-11T16:24:15.309 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:15.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:15 smithi111 ceph-mon[98527]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:24:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:15 smithi111 ceph-mon[98527]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:15 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2640446437' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-11T16:24:15.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:15 smithi111 ceph-mon[98527]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:16.495 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.493+0000 7f2455623700 1 -- 172.21.15.150:0/540375661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 msgr2=0x7f24500fea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:16.496 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.493+0000 7f2455623700 1 --2- 172.21.15.150:0/540375661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 0x7f24500fea10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2444009910 tx=0x7f2444009c20 comp rx=0 tx=0).stop 2023-10-11T16:24:16.496 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.493+0000 7f2455623700 1 -- 172.21.15.150:0/540375661 shutdown_connections 2023-10-11T16:24:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.493+0000 7f2455623700 1 --2- 172.21.15.150:0/540375661 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 0x7f24500fea10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.493+0000 7f2455623700 1 -- 172.21.15.150:0/540375661 >> 172.21.15.150:0/540375661 conn(0x7f24500763c0 msgr2=0x7f24500767c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.493+0000 7f2455623700 1 -- 172.21.15.150:0/540375661 shutdown_connections 2023-10-11T16:24:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.493+0000 7f2455623700 1 -- 172.21.15.150:0/540375661 wait complete. 2023-10-11T16:24:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.494+0000 7f2455623700 1 Processor -- start 2023-10-11T16:24:16.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.494+0000 7f2455623700 1 -- start start 2023-10-11T16:24:16.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.494+0000 7f2455623700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 0x7f2450196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:16.498 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.494+0000 7f2455623700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2450196590 con 0x7f24500fe640 2023-10-11T16:24:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f244effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 0x7f2450196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f244effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 0x7f2450196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.150:50610/0 (socket says 172.21.15.150:50610) 2023-10-11T16:24:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f244effd700 1 -- 172.21.15.150:0/3609176402 learned_addr learned my addr 172.21.15.150:0/3609176402 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:24:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f244effd700 1 -- 172.21.15.150:0/3609176402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24440095f0 con 0x7f24500fe640 2023-10-11T16:24:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f244effd700 1 --2- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 0x7f2450196050 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2444006010 tx=0x7f2444004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f2437fff700 1 -- 172.21.15.150:0/3609176402 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f244401d070 con 0x7f24500fe640 2023-10-11T16:24:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2450108820 con 0x7f24500fe640 2023-10-11T16:24:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f2437fff700 1 -- 172.21.15.150:0/3609176402 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f244400bc10 con 0x7f24500fe640 2023-10-11T16:24:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f2437fff700 1 -- 172.21.15.150:0/3609176402 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2444003910 con 0x7f24500fe640 2023-10-11T16:24:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.495+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2450108690 con 0x7f24500fe640 2023-10-11T16:24:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.496+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24501089b0 con 0x7f24500fe640 2023-10-11T16:24:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.496+0000 7f2437fff700 1 -- 172.21.15.150:0/3609176402 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f244400bd80 con 0x7f24500fe640 2023-10-11T16:24:16.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.497+0000 7f2437fff700 1 --2- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2438038850 0x7f243803ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:16.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.497+0000 7f2437fff700 1 -- 172.21.15.150:0/3609176402 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f244404c600 con 0x7f24500fe640 2023-10-11T16:24:16.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.497+0000 7f244e7fc700 1 --2- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2438038850 0x7f243803ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:16.503 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.497+0000 7f244e7fc700 1 --2- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2438038850 0x7f243803ad00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2440006fd0 tx=0x7f2440006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.502+0000 7f2437fff700 1 -- 172.21.15.150:0/3609176402 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2444029360 con 0x7f24500fe640 2023-10-11T16:24:16.697 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.695+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f24501006e0 con 0x7f2438038850 2023-10-11T16:24:16.704 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.703+0000 7f2437fff700 1 -- 172.21.15.150:0/3609176402 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f24501006e0 con 0x7f2438038850 2023-10-11T16:24:16.704 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled mon update... 2023-10-11T16:24:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.704+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2438038850 msgr2=0x7f243803ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:16.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.704+0000 7f2455623700 1 --2- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2438038850 0x7f243803ad00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2440006fd0 tx=0x7f2440006e40 comp rx=0 tx=0).stop 2023-10-11T16:24:16.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.704+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 msgr2=0x7f2450196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:16.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.704+0000 7f2455623700 1 --2- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 0x7f2450196050 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2444006010 tx=0x7f2444004ff0 comp rx=0 tx=0).stop 2023-10-11T16:24:16.707 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.704+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 shutdown_connections 2023-10-11T16:24:16.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.704+0000 7f2455623700 1 --2- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2438038850 0x7f243803ad00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:16.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.704+0000 7f2455623700 1 --2- 172.21.15.150:0/3609176402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24500fe640 0x7f2450196050 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:16.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.704+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 >> 172.21.15.150:0/3609176402 conn(0x7f24500763c0 msgr2=0x7f2450106120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:16.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.705+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 shutdown_connections 2023-10-11T16:24:16.708 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:16.705+0000 7f2455623700 1 -- 172.21.15.150:0/3609176402 wait complete. 2023-10-11T16:24:17.165 DEBUG:teuthology.orchestra.run.smithi130:mon.b> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.b.service 2023-10-11T16:24:17.169 DEBUG:teuthology.orchestra.run.smithi150:mon.c> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.c.service 2023-10-11T16:24:17.171 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-11T16:24:17.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-11T16:24:17.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:24:17.206 INFO:journalctl@ceph.mon.b.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:24:17.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:17 smithi111 ceph-mon[98527]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:17.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:17.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:17.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:18.219 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:18.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:18 smithi111 ceph-mon[98527]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:18.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:18 smithi111 ceph-mon[98527]: Saving service mon spec with placement smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c;count:3 2023-10-11T16:24:18.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:18 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:18.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:18 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:18.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:18 smithi111 ceph-mon[98527]: Deploying daemon mon.c on smithi150 2023-10-11T16:24:19.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:19 smithi111 ceph-mon[98527]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:19.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:19 smithi150 systemd[1]: Starting Ceph mon.c for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:24:20.038 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.035+0000 7f73ca535700 1 -- 172.21.15.150:0/2249192099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 msgr2=0x7f73c41011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:20.039 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.036+0000 7f73ca535700 1 --2- 172.21.15.150:0/2249192099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 0x7f73c41011d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009910 tx=0x7f73ac009c20 comp rx=0 tx=0).stop 2023-10-11T16:24:20.039 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.036+0000 7f73ca535700 1 -- 172.21.15.150:0/2249192099 shutdown_connections 2023-10-11T16:24:20.039 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.036+0000 7f73ca535700 1 --2- 172.21.15.150:0/2249192099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 0x7f73c41011d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:20.040 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.036+0000 7f73ca535700 1 -- 172.21.15.150:0/2249192099 >> 172.21.15.150:0/2249192099 conn(0x7f73c4074940 msgr2=0x7f73c4074d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:20.040 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.036+0000 7f73ca535700 1 -- 172.21.15.150:0/2249192099 shutdown_connections 2023-10-11T16:24:20.040 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.036+0000 7f73ca535700 1 -- 172.21.15.150:0/2249192099 wait complete. 2023-10-11T16:24:20.040 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.037+0000 7f73ca535700 1 Processor -- start 2023-10-11T16:24:20.042 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.037+0000 7f73ca535700 1 -- start start 2023-10-11T16:24:20.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.037+0000 7f73ca535700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 0x7f73c41951e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:20.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.037+0000 7f73ca535700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c4195720 con 0x7f73c4100e00 2023-10-11T16:24:20.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.038+0000 7f73c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 0x7f73c41951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:20.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.038+0000 7f73c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 0x7f73c41951e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.150:50616/0 (socket says 172.21.15.150:50616) 2023-10-11T16:24:20.043 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.038+0000 7f73c3fff700 1 -- 172.21.15.150:0/712448166 learned_addr learned my addr 172.21.15.150:0/712448166 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:24:20.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.038+0000 7f73c3fff700 1 -- 172.21.15.150:0/712448166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0095f0 con 0x7f73c4100e00 2023-10-11T16:24:20.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.038+0000 7f73c3fff700 1 --2- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 0x7f73c41951e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f73ac005850 tx=0x7f73ac004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:20.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.039+0000 7f73c17fa700 1 -- 172.21.15.150:0/712448166 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73ac01d070 con 0x7f73c4100e00 2023-10-11T16:24:20.044 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.039+0000 7f73c17fa700 1 -- 172.21.15.150:0/712448166 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73ac004370 con 0x7f73c4100e00 2023-10-11T16:24:20.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.039+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c410a1a0 con 0x7f73c4100e00 2023-10-11T16:24:20.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.039+0000 7f73c17fa700 1 -- 172.21.15.150:0/712448166 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73ac003ab0 con 0x7f73c4100e00 2023-10-11T16:24:20.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.039+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c410a010 con 0x7f73c4100e00 2023-10-11T16:24:20.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.040+0000 7f73c17fa700 1 -- 172.21.15.150:0/712448166 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f73ac003d20 con 0x7f73c4100e00 2023-10-11T16:24:20.045 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.040+0000 7f73c17fa700 1 --2- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73b00388c0 0x7f73b003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:20.046 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.040+0000 7f73c17fa700 1 -- 172.21.15.150:0/712448166 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f73ac04c520 con 0x7f73c4100e00 2023-10-11T16:24:20.046 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.040+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c410a330 con 0x7f73c4100e00 2023-10-11T16:24:20.046 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.040+0000 7f73c37fe700 1 --2- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73b00388c0 0x7f73b003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:20.046 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.041+0000 7f73c37fe700 1 --2- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73b00388c0 0x7f73b003ad70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f73b4006fd0 tx=0x7f73b4006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:20.047 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.046+0000 7f73c17fa700 1 -- 172.21.15.150:0/712448166 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f73ac026070 con 0x7f73c4100e00 2023-10-11T16:24:20.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.330+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73c4062120 con 0x7f73c4100e00 2023-10-11T16:24:20.333 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.331+0000 7f73c17fa700 1 -- 172.21.15.150:0/712448166 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+740 (secure 0 0 0) 0x7f73ac02b440 con 0x7f73c4100e00 2023-10-11T16:24:20.334 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:24:20.334 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","modified":"2023-10-11T16:22:41.058052Z","created":"2023-10-11T16:22:41.058052Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-11T16:24:20.335 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73b00388c0 msgr2=0x7f73b003ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:20.335 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 --2- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73b00388c0 0x7f73b003ad70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f73b4006fd0 tx=0x7f73b4006e40 comp rx=0 tx=0).stop 2023-10-11T16:24:20.335 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 msgr2=0x7f73c41951e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:20.335 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 --2- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 0x7f73c41951e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f73ac005850 tx=0x7f73ac004ff0 comp rx=0 tx=0).stop 2023-10-11T16:24:20.336 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 shutdown_connections 2023-10-11T16:24:20.336 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 --2- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73b00388c0 0x7f73b003ad70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:20.336 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 --2- 172.21.15.150:0/712448166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73c4100e00 0x7f73c41951e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:20.336 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 >> 172.21.15.150:0/712448166 conn(0x7f73c4074940 msgr2=0x7f73c4104620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:20.336 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 shutdown_connections 2023-10-11T16:24:20.337 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:20.333+0000 7f73ca535700 1 -- 172.21.15.150:0/712448166 wait complete. 2023-10-11T16:24:20.337 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2023-10-11T16:24:20.410 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 podman[132154]: 2023-10-11T16:24:20.410 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: set uid:gid to 167:167 (ceph:ceph) 2023-10-11T16:24:20.411 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev), process ceph-mon, pid 7 2023-10-11T16:24:20.411 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: pidfile_write: ignore empty --pid-file 2023-10-11T16:24:20.411 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: load: jerasure load: lrc 2023-10-11T16:24:20.411 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: RocksDB version: 7.9.2 2023-10-11T16:24:20.411 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Git sha 0 2023-10-11T16:24:20.412 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Compile date 2023-09-28 19:19:12 2023-10-11T16:24:20.412 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: DB SUMMARY 2023-10-11T16:24:20.412 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: DB Session ID: 2L32LZ20I3C51G9QFUQ6 2023-10-11T16:24:20.412 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: CURRENT file: CURRENT 2023-10-11T16:24:20.412 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: IDENTITY file: IDENTITY 2023-10-11T16:24:20.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-10-11T16:24:20.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-10-11T16:24:20.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-10-11T16:24:20.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.error_if_exists: 0 2023-10-11T16:24:20.413 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.create_if_missing: 0 2023-10-11T16:24:20.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.paranoid_checks: 1 2023-10-11T16:24:20.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-11T16:24:20.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-11T16:24:20.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-11T16:24:20.414 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.env: 0x56393fd84720 2023-10-11T16:24:20.415 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.fs: PosixFileSystem 2023-10-11T16:24:20.415 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.info_log: 0x5639416113c0 2023-10-11T16:24:20.415 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_file_opening_threads: 16 2023-10-11T16:24:20.415 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.statistics: (nil) 2023-10-11T16:24:20.415 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.use_fsync: 0 2023-10-11T16:24:20.415 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_log_file_size: 0 2023-10-11T16:24:20.416 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-11T16:24:20.416 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-11T16:24:20.416 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.keep_log_file_num: 1000 2023-10-11T16:24:20.416 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.recycle_log_file_num: 0 2023-10-11T16:24:20.416 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.allow_fallocate: 1 2023-10-11T16:24:20.417 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.allow_mmap_reads: 0 2023-10-11T16:24:20.417 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.allow_mmap_writes: 0 2023-10-11T16:24:20.417 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.use_direct_reads: 0 2023-10-11T16:24:20.417 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-11T16:24:20.417 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.create_missing_column_families: 0 2023-10-11T16:24:20.418 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.db_log_dir: 2023-10-11T16:24:20.418 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.wal_dir: 2023-10-11T16:24:20.418 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-11T16:24:20.418 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-11T16:24:20.418 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-11T16:24:20.419 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-11T16:24:20.419 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-11T16:24:20.419 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-11T16:24:20.419 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.advise_random_on_open: 1 2023-10-11T16:24:20.419 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.db_write_buffer_size: 0 2023-10-11T16:24:20.420 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.write_buffer_manager: 0x5639408823c0 2023-10-11T16:24:20.420 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-11T16:24:20.420 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-11T16:24:20.420 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-11T16:24:20.420 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.rate_limiter: (nil) 2023-10-11T16:24:20.421 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-11T16:24:20.421 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.wal_recovery_mode: 2 2023-10-11T16:24:20.421 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.enable_thread_tracking: 0 2023-10-11T16:24:20.421 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.enable_pipelined_write: 0 2023-10-11T16:24:20.421 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.unordered_write: 0 2023-10-11T16:24:20.421 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-11T16:24:20.422 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-11T16:24:20.422 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-11T16:24:20.422 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-11T16:24:20.422 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.row_cache: None 2023-10-11T16:24:20.422 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.wal_filter: None 2023-10-11T16:24:20.423 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-11T16:24:20.423 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.allow_ingest_behind: 0 2023-10-11T16:24:20.423 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.two_write_queues: 0 2023-10-11T16:24:20.423 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.manual_wal_flush: 0 2023-10-11T16:24:20.423 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.wal_compression: 0 2023-10-11T16:24:20.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.atomic_flush: 0 2023-10-11T16:24:20.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-11T16:24:20.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-11T16:24:20.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-11T16:24:20.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.log_readahead_size: 0 2023-10-11T16:24:20.424 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-11T16:24:20.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.best_efforts_recovery: 0 2023-10-11T16:24:20.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-11T16:24:20.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-11T16:24:20.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.allow_data_in_errors: 0 2023-10-11T16:24:20.425 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.db_host_id: __hostname__ 2023-10-11T16:24:20.426 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-11T16:24:20.426 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_background_jobs: 2 2023-10-11T16:24:20.426 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_background_compactions: -1 2023-10-11T16:24:20.426 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_subcompactions: 1 2023-10-11T16:24:20.426 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-11T16:24:20.427 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-11T16:24:20.427 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-11T16:24:20.427 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_total_wal_size: 0 2023-10-11T16:24:20.427 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-11T16:24:20.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-11T16:24:20.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-11T16:24:20.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-11T16:24:20.428 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_open_files: -1 2023-10-11T16:24:20.429 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bytes_per_sync: 0 2023-10-11T16:24:20.429 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-11T16:24:20.429 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-11T16:24:20.429 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_readahead_size: 0 2023-10-11T16:24:20.429 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_background_flushes: -1 2023-10-11T16:24:20.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Compression algorithms supported: 2023-10-11T16:24:20.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-11T16:24:20.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: kZSTD supported: 0 2023-10-11T16:24:20.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: kXpressCompression supported: 0 2023-10-11T16:24:20.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: kLZ4HCCompression supported: 1 2023-10-11T16:24:20.431 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: kZlibCompression supported: 1 2023-10-11T16:24:20.431 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: kSnappyCompression supported: 1 2023-10-11T16:24:20.431 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: kLZ4Compression supported: 1 2023-10-11T16:24:20.431 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: kBZip2Compression supported: 0 2023-10-11T16:24:20.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-11T16:24:20.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-11T16:24:20.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-10-11T16:24:20.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-11T16:24:20.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-11T16:24:20.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.merge_operator: 2023-10-11T16:24:20.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_filter: None 2023-10-11T16:24:20.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_filter_factory: None 2023-10-11T16:24:20.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.sst_partitioner_factory: None 2023-10-11T16:24:20.433 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-11T16:24:20.434 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-11T16:24:20.434 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5639416114e0) 2023-10-11T16:24:20.434 INFO:journalctl@ceph.mon.c.smithi150.stdout: cache_index_and_filter_blocks: 1 2023-10-11T16:24:20.434 INFO:journalctl@ceph.mon.c.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-11T16:24:20.435 INFO:journalctl@ceph.mon.c.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-11T16:24:20.435 INFO:journalctl@ceph.mon.c.smithi150.stdout: pin_top_level_index_and_filter: 1 2023-10-11T16:24:20.435 INFO:journalctl@ceph.mon.c.smithi150.stdout: index_type: 0 2023-10-11T16:24:20.435 INFO:journalctl@ceph.mon.c.smithi150.stdout: data_block_index_type: 0 2023-10-11T16:24:20.435 INFO:journalctl@ceph.mon.c.smithi150.stdout: index_shortening: 1 2023-10-11T16:24:20.435 INFO:journalctl@ceph.mon.c.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-11T16:24:20.436 INFO:journalctl@ceph.mon.c.smithi150.stdout: checksum: 4 2023-10-11T16:24:20.436 INFO:journalctl@ceph.mon.c.smithi150.stdout: no_block_cache: 0 2023-10-11T16:24:20.436 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_cache: 0x563940923350 2023-10-11T16:24:20.436 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_cache_name: BinnedLRUCache 2023-10-11T16:24:20.436 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_cache_options: 2023-10-11T16:24:20.437 INFO:journalctl@ceph.mon.c.smithi150.stdout: capacity : 536870912 2023-10-11T16:24:20.437 INFO:journalctl@ceph.mon.c.smithi150.stdout: num_shard_bits : 4 2023-10-11T16:24:20.437 INFO:journalctl@ceph.mon.c.smithi150.stdout: strict_capacity_limit : 0 2023-10-11T16:24:20.437 INFO:journalctl@ceph.mon.c.smithi150.stdout: high_pri_pool_ratio: 0.000 2023-10-11T16:24:20.437 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_cache_compressed: (nil) 2023-10-11T16:24:20.437 INFO:journalctl@ceph.mon.c.smithi150.stdout: persistent_cache: (nil) 2023-10-11T16:24:20.438 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_size: 4096 2023-10-11T16:24:20.438 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_size_deviation: 10 2023-10-11T16:24:20.438 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_restart_interval: 16 2023-10-11T16:24:20.438 INFO:journalctl@ceph.mon.c.smithi150.stdout: index_block_restart_interval: 1 2023-10-11T16:24:20.438 INFO:journalctl@ceph.mon.c.smithi150.stdout: metadata_block_size: 4096 2023-10-11T16:24:20.439 INFO:journalctl@ceph.mon.c.smithi150.stdout: partition_filters: 0 2023-10-11T16:24:20.439 INFO:journalctl@ceph.mon.c.smithi150.stdout: use_delta_encoding: 1 2023-10-11T16:24:20.439 INFO:journalctl@ceph.mon.c.smithi150.stdout: filter_policy: bloomfilter 2023-10-11T16:24:20.439 INFO:journalctl@ceph.mon.c.smithi150.stdout: whole_key_filtering: 1 2023-10-11T16:24:20.439 INFO:journalctl@ceph.mon.c.smithi150.stdout: verify_compression: 0 2023-10-11T16:24:20.440 INFO:journalctl@ceph.mon.c.smithi150.stdout: read_amp_bytes_per_bit: 0 2023-10-11T16:24:20.440 INFO:journalctl@ceph.mon.c.smithi150.stdout: format_version: 5 2023-10-11T16:24:20.440 INFO:journalctl@ceph.mon.c.smithi150.stdout: enable_index_compression: 1 2023-10-11T16:24:20.440 INFO:journalctl@ceph.mon.c.smithi150.stdout: block_align: 0 2023-10-11T16:24:20.440 INFO:journalctl@ceph.mon.c.smithi150.stdout: max_auto_readahead_size: 262144 2023-10-11T16:24:20.440 INFO:journalctl@ceph.mon.c.smithi150.stdout: prepopulate_block_cache: 0 2023-10-11T16:24:20.441 INFO:journalctl@ceph.mon.c.smithi150.stdout: initial_auto_readahead_size: 8192 2023-10-11T16:24:20.441 INFO:journalctl@ceph.mon.c.smithi150.stdout: num_file_reads_for_auto_readahead: 2 2023-10-11T16:24:20.441 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.write_buffer_size: 33554432 2023-10-11T16:24:20.441 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_write_buffer_number: 2 2023-10-11T16:24:20.441 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression: NoCompression 2023-10-11T16:24:20.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression: Disabled 2023-10-11T16:24:20.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.prefix_extractor: nullptr 2023-10-11T16:24:20.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-11T16:24:20.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.num_levels: 7 2023-10-11T16:24:20.442 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-11T16:24:20.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-11T16:24:20.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-11T16:24:20.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-11T16:24:20.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-11T16:24:20.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-11T16:24:20.443 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-11T16:24:20.444 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-11T16:24:20.444 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-11T16:24:20.444 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-11T16:24:20.444 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-11T16:24:20.444 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-11T16:24:20.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-11T16:24:20.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.level: 32767 2023-10-11T16:24:20.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.strategy: 0 2023-10-11T16:24:20.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-11T16:24:20.445 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-11T16:24:20.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-11T16:24:20.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-11T16:24:20.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.enabled: false 2023-10-11T16:24:20.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-11T16:24:20.446 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-11T16:24:20.447 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-11T16:24:20.447 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-11T16:24:20.447 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 bash[132154]: f9148ff0570ed254b64a8d83480fe21767927e35a4ffc08af8f67490965a6eaf 2023-10-11T16:24:20.447 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.target_file_size_base: 67108864 2023-10-11T16:24:20.447 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-11T16:24:20.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-11T16:24:20.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-11T16:24:20.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-11T16:24:20.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-11T16:24:20.448 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-11T16:24:20.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-11T16:24:20.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-11T16:24:20.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-11T16:24:20.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-11T16:24:20.449 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-11T16:24:20.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-11T16:24:20.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-11T16:24:20.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-11T16:24:20.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.arena_block_size: 1048576 2023-10-11T16:24:20.450 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-11T16:24:20.451 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-11T16:24:20.451 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.disable_auto_compactions: 0 2023-10-11T16:24:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-11T16:24:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-11T16:24:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-11T16:24:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-11T16:24:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-11T16:24:20.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-11T16:24:20.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-11T16:24:20.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-11T16:24:20.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-11T16:24:20.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-11T16:24:20.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.table_properties_collectors: 2023-10-11T16:24:20.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.inplace_update_support: 0 2023-10-11T16:24:20.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-11T16:24:20.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-11T16:24:20.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-11T16:24:20.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-11T16:24:20.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.bloom_locality: 0 2023-10-11T16:24:20.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.max_successive_merges: 0 2023-10-11T16:24:20.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-11T16:24:20.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.paranoid_file_checks: 0 2023-10-11T16:24:20.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.force_consistency_checks: 1 2023-10-11T16:24:20.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.report_bg_io_stats: 0 2023-10-11T16:24:20.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.ttl: 2592000 2023-10-11T16:24:20.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-11T16:24:20.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-11T16:24:20.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-11T16:24:20.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.enable_blob_files: false 2023-10-11T16:24:20.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.min_blob_size: 0 2023-10-11T16:24:20.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.blob_file_size: 268435456 2023-10-11T16:24:20.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-11T16:24:20.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-11T16:24:20.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-11T16:24:20.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-11T16:24:20.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-11T16:24:20.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.blob_file_starting_level: 0 2023-10-11T16:24:20.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-11T16:24:20.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-10-11T16:24:20.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-10-11T16:24:20.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 941e4c3d-4c92-4366-8821-44220b9eaa24 2023-10-11T16:24:20.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041460401076, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-10-11T16:24:20.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-10-11T16:24:20.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041460401536, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1697041460, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "941e4c3d-4c92-4366-8821-44220b9eaa24", "db_session_id": "2L32LZ20I3C51G9QFUQ6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-10-11T16:24:20.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041460401619, "job": 1, "event": "recovery_finished"} 2023-10-11T16:24:20.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-10-11T16:24:20.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-11T16:24:20.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5639409c0000 2023-10-11T16:24:20.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: DB pointer 0x5639409ac000 2023-10-11T16:24:20.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-11T16:24:20.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-11T16:24:20.461 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** DB Stats ** 2023-10-11T16:24:20.461 INFO:journalctl@ceph.mon.c.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-11T16:24:20.462 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-11T16:24:20.462 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-11T16:24:20.462 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-11T16:24:20.462 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-11T16:24:20.462 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-11T16:24:20.463 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-11T16:24:20.463 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2023-10-11T16:24:20.463 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** Compaction Stats [default] ** 2023-10-11T16:24:20.463 INFO:journalctl@ceph.mon.c.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-11T16:24:20.463 INFO:journalctl@ceph.mon.c.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-11T16:24:20.464 INFO:journalctl@ceph.mon.c.smithi150.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-11T16:24:20.464 INFO:journalctl@ceph.mon.c.smithi150.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-11T16:24:20.464 INFO:journalctl@ceph.mon.c.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-11T16:24:20.464 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2023-10-11T16:24:20.464 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** Compaction Stats [default] ** 2023-10-11T16:24:20.464 INFO:journalctl@ceph.mon.c.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-11T16:24:20.465 INFO:journalctl@ceph.mon.c.smithi150.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-11T16:24:20.465 INFO:journalctl@ceph.mon.c.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-11T16:24:20.465 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2023-10-11T16:24:20.465 INFO:journalctl@ceph.mon.c.smithi150.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-11T16:24:20.465 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2023-10-11T16:24:20.466 INFO:journalctl@ceph.mon.c.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-11T16:24:20.466 INFO:journalctl@ceph.mon.c.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-11T16:24:20.466 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-11T16:24:20.466 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-11T16:24:20.466 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-11T16:24:20.467 INFO:journalctl@ceph.mon.c.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-11T16:24:20.467 INFO:journalctl@ceph.mon.c.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-11T16:24:20.467 INFO:journalctl@ceph.mon.c.smithi150.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-11T16:24:20.467 INFO:journalctl@ceph.mon.c.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-11T16:24:20.467 INFO:journalctl@ceph.mon.c.smithi150.stdout: Block cache BinnedLRUCache@0x563940923350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-10-11T16:24:20.467 INFO:journalctl@ceph.mon.c.smithi150.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-10-11T16:24:20.468 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2023-10-11T16:24:20.468 INFO:journalctl@ceph.mon.c.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-11T16:24:20.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-10-11T16:24:20.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: using public_addr v2:172.21.15.150:0/0 -> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-11T16:24:20.468 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: starting mon.c rank -1 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:24:20.469 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(???) e0 preinit fsid 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:24:20.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).mds e1 new map 2023-10-11T16:24:20.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).mds e1 print_map 2023-10-11T16:24:20.703 INFO:journalctl@ceph.mon.c.smithi150.stdout: e1 2023-10-11T16:24:20.703 INFO:journalctl@ceph.mon.c.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-11T16:24:20.703 INFO:journalctl@ceph.mon.c.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-11T16:24:20.704 INFO:journalctl@ceph.mon.c.smithi150.stdout: legacy client fscid: -1 2023-10-11T16:24:20.704 INFO:journalctl@ceph.mon.c.smithi150.stdout: 2023-10-11T16:24:20.704 INFO:journalctl@ceph.mon.c.smithi150.stdout: No filesystems configured 2023-10-11T16:24:20.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-10-11T16:24:20.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-10-11T16:24:20.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-10-11T16:24:20.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-10-11T16:24:20.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-10-11T16:24:20.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-10-11T16:24:20.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-11T16:24:20.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:24:20.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:24:20.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:24:20.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:20.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:24:20.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:20.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:20.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:24:20.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:20.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Deploying cephadm binary to smithi150 2023-10-11T16:24:20.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:20.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:20.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Added host smithi150 2023-10-11T16:24:20.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:20.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:20.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:24:20.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:20.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:24:20.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:20.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:20.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:20.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:24:20.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:20.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:20.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2640446437' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-11T16:24:20.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:24:20.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:20.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2640446437' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-11T16:24:20.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:20.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:20.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:20.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:20.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:20.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:20.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:20.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:20.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:20.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Saving service mon spec with placement smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c;count:3 2023-10-11T16:24:20.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:20.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:20.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: Deploying daemon mon.c on smithi150 2023-10-11T16:24:20.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:20.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-10-11T16:24:20.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-10-11T16:24:20.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta expand map: {default=false} 2023-10-11T16:24:20.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta from 'false' to 'false' 2023-10-11T16:24:20.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta expanded map: {default=false} 2023-10-11T16:24:20.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta expand map: {default=info} 2023-10-11T16:24:20.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta from 'info' to 'info' 2023-10-11T16:24:20.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta expanded map: {default=info} 2023-10-11T16:24:20.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta expand map: {default=daemon} 2023-10-11T16:24:20.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-11T16:24:20.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta expanded map: {default=daemon} 2023-10-11T16:24:20.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta expand map: {default=debug} 2023-10-11T16:24:20.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta from 'debug' to 'debug' 2023-10-11T16:24:20.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 ceph-mon[132217]: expand_channel_meta expanded map: {default=debug} 2023-10-11T16:24:20.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:20 smithi150 systemd[1]: Started Ceph mon.c for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:24:21.901 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-11T16:24:21.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-11T16:24:22.552 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:24:25.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:25.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:24:25.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: mon.a calling monitor election 2023-10-11T16:24:25.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:25.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: mon.c calling monitor election 2023-10-11T16:24:25.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:25.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-11T16:24:25.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: monmap e2: 2 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:25.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: fsmap 2023-10-11T16:24:25.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:25.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: mgrmap e13: x(active, since 45s) 2023-10-11T16:24:25.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: overall HEALTH_OK 2023-10-11T16:24:25.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:25.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:25.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:25.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:25.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:24:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: mon.a calling monitor election 2023-10-11T16:24:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: mon.c calling monitor election 2023-10-11T16:24:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-11T16:24:25.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: monmap e2: 2 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:25.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: fsmap 2023-10-11T16:24:25.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:25.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: mgrmap e13: x(active, since 45s) 2023-10-11T16:24:25.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: overall HEALTH_OK 2023-10-11T16:24:25.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:25.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:25.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:25.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:25.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:26.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.749+0000 7f93943d2700 1 -- 172.21.15.150:0/3100362326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 msgr2=0x7f9378007c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:26.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.749+0000 7f93943d2700 1 --2- 172.21.15.150:0/3100362326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 0x7f9378007c50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f938c0fec70 tx=0x7f937c00feb0 comp rx=0 tx=0).stop 2023-10-11T16:24:26.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.749+0000 7f93943d2700 1 -- 172.21.15.150:0/3100362326 shutdown_connections 2023-10-11T16:24:26.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.749+0000 7f93943d2700 1 --2- 172.21.15.150:0/3100362326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 0x7f9378007c50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:26.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.749+0000 7f93943d2700 1 -- 172.21.15.150:0/3100362326 >> 172.21.15.150:0/3100362326 conn(0x7f938c0fa860 msgr2=0x7f938c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:26.752 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.749+0000 7f93943d2700 1 -- 172.21.15.150:0/3100362326 shutdown_connections 2023-10-11T16:24:26.753 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.749+0000 7f93943d2700 1 -- 172.21.15.150:0/3100362326 wait complete. 2023-10-11T16:24:26.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.750+0000 7f93943d2700 1 Processor -- start 2023-10-11T16:24:26.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.750+0000 7f93943d2700 1 -- start start 2023-10-11T16:24:26.755 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f93943d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 0x7f938c193110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:26.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f93943d2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 0x7f938c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:26.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f93943d2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c193bd0 con 0x7f93780057d0 2023-10-11T16:24:26.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f93943d2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c193d40 con 0x7f938c193650 2023-10-11T16:24:26.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939196d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 0x7f938c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:26.756 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939196d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 0x7f938c197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42322/0 (socket says 172.21.15.150:42322) 2023-10-11T16:24:26.757 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939196d700 1 -- 172.21.15.150:0/4132816848 learned_addr learned my addr 172.21.15.150:0/4132816848 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:24:26.757 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939196d700 1 -- 172.21.15.150:0/4132816848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 msgr2=0x7f938c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-10-11T16:24:26.757 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939216e700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 0x7f938c193110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:26.757 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939196d700 1 -- 172.21.15.150:0/4132816848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 msgr2=0x7f938c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-11T16:24:26.757 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939196d700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 0x7f938c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-11T16:24:26.758 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939196d700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 0x7f938c197ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:24:26.758 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939216e700 1 -- 172.21.15.150:0/4132816848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 msgr2=0x7f938c197ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:24:26.758 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939216e700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 0x7f938c197ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:26.758 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.751+0000 7f939216e700 1 -- 172.21.15.150:0/4132816848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c00f900 con 0x7f93780057d0 2023-10-11T16:24:26.758 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.752+0000 7f939216e700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 0x7f938c193110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f937c010010 tx=0x7f937c0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:26.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.752+0000 7f93837fe700 1 -- 172.21.15.150:0/4132816848 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f937c013960 con 0x7f93780057d0 2023-10-11T16:24:26.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.752+0000 7f93837fe700 1 -- 172.21.15.150:0/4132816848 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f937c004030 con 0x7f93780057d0 2023-10-11T16:24:26.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.752+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938c105b90 con 0x7f93780057d0 2023-10-11T16:24:26.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.752+0000 7f93837fe700 1 -- 172.21.15.150:0/4132816848 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f937c003a80 con 0x7f93780057d0 2023-10-11T16:24:26.759 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.752+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938c069950 con 0x7f93780057d0 2023-10-11T16:24:26.760 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.753+0000 7f93837fe700 1 -- 172.21.15.150:0/4132816848 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f937c003cf0 con 0x7f93780057d0 2023-10-11T16:24:26.760 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.753+0000 7f93837fe700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f93700386b0 0x7f937003ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:26.760 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.753+0000 7f93837fe700 1 -- 172.21.15.150:0/4132816848 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f937c000e40 con 0x7f93780057d0 2023-10-11T16:24:26.760 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.753+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c105d20 con 0x7f93780057d0 2023-10-11T16:24:26.760 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.753+0000 7f939196d700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f93700386b0 0x7f937003ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:26.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.754+0000 7f939196d700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f93700386b0 0x7f937003ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9388009180 tx=0x7f9388008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:26.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:26.758+0000 7f93837fe700 1 -- 172.21.15.150:0/4132816848 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f937c011540 con 0x7f93780057d0 2023-10-11T16:24:26.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:26 smithi111 ceph-mon[98527]: Deploying daemon mon.b on smithi130 2023-10-11T16:24:26.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:26 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:26.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:26 smithi150 ceph-mon[132217]: Deploying daemon mon.b on smithi130 2023-10-11T16:24:26.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:26 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:27.024 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.023+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f938c1945f0 con 0x7f93780057d0 2023-10-11T16:24:27.025 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.023+0000 7f93837fe700 1 -- 172.21.15.150:0/4132816848 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1010 (secure 0 0 0) 0x7f937c01a390 con 0x7f93780057d0 2023-10-11T16:24:27.025 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:24:27.026 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","modified":"2023-10-11T16:24:20.426930Z","created":"2023-10-11T16:22:41.058052Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-10-11T16:24:27.027 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f93700386b0 msgr2=0x7f937003ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f93700386b0 0x7f937003ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9388009180 tx=0x7f9388008040 comp rx=0 tx=0).stop 2023-10-11T16:24:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 msgr2=0x7f938c193110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 0x7f938c193110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f937c010010 tx=0x7f937c0047a0 comp rx=0 tx=0).stop 2023-10-11T16:24:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 shutdown_connections 2023-10-11T16:24:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f93700386b0 0x7f937003ab60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93780057d0 0x7f938c193110 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 --2- 172.21.15.150:0/4132816848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c193650 0x7f938c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 >> 172.21.15.150:0/4132816848 conn(0x7f938c0fa860 msgr2=0x7f938c1008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.025+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 shutdown_connections 2023-10-11T16:24:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:27.026+0000 7f93943d2700 1 -- 172.21.15.150:0/4132816848 wait complete. 2023-10-11T16:24:27.030 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 2 2023-10-11T16:24:27.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:27 smithi111 ceph-mon[98527]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:27.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:27 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/4132816848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-11T16:24:27.886 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:24:27 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:24:27.426+0000 7eff7d923700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-11T16:24:27.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:27 smithi150 ceph-mon[132217]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:27.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:27 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/4132816848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-11T16:24:28.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 systemd[1]: Starting Ceph mon.b for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:24:28.507 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-11T16:24:28.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-11T16:24:28.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 podman[131818]: 2023-10-11T16:24:28.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 bash[131818]: 4fdf13d1827a31b6df3eb4a6da8b6028e246342a262287384b72b551b7f45277 2023-10-11T16:24:28.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: set uid:gid to 167:167 (ceph:ceph) 2023-10-11T16:24:28.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev), process ceph-mon, pid 7 2023-10-11T16:24:28.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pidfile_write: ignore empty --pid-file 2023-10-11T16:24:28.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: load: jerasure load: lrc 2023-10-11T16:24:28.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: RocksDB version: 7.9.2 2023-10-11T16:24:28.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Git sha 0 2023-10-11T16:24:28.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Compile date 2023-09-28 19:19:12 2023-10-11T16:24:28.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: DB SUMMARY 2023-10-11T16:24:28.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: DB Session ID: 96BCTSGSQFO6KKYFSSGW 2023-10-11T16:24:28.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: CURRENT file: CURRENT 2023-10-11T16:24:28.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: IDENTITY file: IDENTITY 2023-10-11T16:24:28.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-10-11T16:24:28.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-10-11T16:24:28.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-10-11T16:24:28.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.error_if_exists: 0 2023-10-11T16:24:28.588 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.create_if_missing: 0 2023-10-11T16:24:28.588 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.paranoid_checks: 1 2023-10-11T16:24:28.588 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-11T16:24:28.588 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-11T16:24:28.588 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-11T16:24:28.589 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.env: 0x55dea0d86720 2023-10-11T16:24:28.589 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.fs: PosixFileSystem 2023-10-11T16:24:28.589 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.info_log: 0x55dea3f493c0 2023-10-11T16:24:28.589 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_file_opening_threads: 16 2023-10-11T16:24:28.589 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.statistics: (nil) 2023-10-11T16:24:28.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.use_fsync: 0 2023-10-11T16:24:28.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_log_file_size: 0 2023-10-11T16:24:28.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-11T16:24:28.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-11T16:24:28.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.keep_log_file_num: 1000 2023-10-11T16:24:28.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.recycle_log_file_num: 0 2023-10-11T16:24:28.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.allow_fallocate: 1 2023-10-11T16:24:28.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.allow_mmap_reads: 0 2023-10-11T16:24:28.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.allow_mmap_writes: 0 2023-10-11T16:24:28.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.use_direct_reads: 0 2023-10-11T16:24:28.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-11T16:24:28.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.create_missing_column_families: 0 2023-10-11T16:24:28.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.db_log_dir: 2023-10-11T16:24:28.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.wal_dir: 2023-10-11T16:24:28.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-11T16:24:28.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-11T16:24:28.593 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-11T16:24:28.593 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-11T16:24:28.593 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-11T16:24:28.593 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-11T16:24:28.593 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.advise_random_on_open: 1 2023-10-11T16:24:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.db_write_buffer_size: 0 2023-10-11T16:24:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.write_buffer_manager: 0x55dea31ba3c0 2023-10-11T16:24:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-11T16:24:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-11T16:24:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-11T16:24:28.594 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.rate_limiter: (nil) 2023-10-11T16:24:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-11T16:24:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.wal_recovery_mode: 2 2023-10-11T16:24:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.enable_thread_tracking: 0 2023-10-11T16:24:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.enable_pipelined_write: 0 2023-10-11T16:24:28.595 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.unordered_write: 0 2023-10-11T16:24:28.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-11T16:24:28.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-11T16:24:28.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-11T16:24:28.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-11T16:24:28.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.row_cache: None 2023-10-11T16:24:28.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.wal_filter: None 2023-10-11T16:24:28.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-11T16:24:28.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.allow_ingest_behind: 0 2023-10-11T16:24:28.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.two_write_queues: 0 2023-10-11T16:24:28.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.manual_wal_flush: 0 2023-10-11T16:24:28.597 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.wal_compression: 0 2023-10-11T16:24:28.598 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.atomic_flush: 0 2023-10-11T16:24:28.598 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-11T16:24:28.598 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-11T16:24:28.598 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-11T16:24:28.598 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.log_readahead_size: 0 2023-10-11T16:24:28.599 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-11T16:24:28.599 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.best_efforts_recovery: 0 2023-10-11T16:24:28.599 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-11T16:24:28.599 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-11T16:24:28.599 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.allow_data_in_errors: 0 2023-10-11T16:24:28.600 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.db_host_id: __hostname__ 2023-10-11T16:24:28.600 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-11T16:24:28.600 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_background_jobs: 2 2023-10-11T16:24:28.600 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_background_compactions: -1 2023-10-11T16:24:28.600 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_subcompactions: 1 2023-10-11T16:24:28.601 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-11T16:24:28.601 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-11T16:24:28.601 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-11T16:24:28.601 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_total_wal_size: 0 2023-10-11T16:24:28.601 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-11T16:24:28.602 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-11T16:24:28.602 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-11T16:24:28.602 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-11T16:24:28.602 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_open_files: -1 2023-10-11T16:24:28.602 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bytes_per_sync: 0 2023-10-11T16:24:28.602 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-11T16:24:28.603 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-11T16:24:28.603 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_readahead_size: 0 2023-10-11T16:24:28.603 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_background_flushes: -1 2023-10-11T16:24:28.603 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Compression algorithms supported: 2023-10-11T16:24:28.603 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-11T16:24:28.603 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: kZSTD supported: 0 2023-10-11T16:24:28.604 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: kXpressCompression supported: 0 2023-10-11T16:24:28.604 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: kLZ4HCCompression supported: 1 2023-10-11T16:24:28.604 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: kZlibCompression supported: 1 2023-10-11T16:24:28.604 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: kSnappyCompression supported: 1 2023-10-11T16:24:28.604 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: kLZ4Compression supported: 1 2023-10-11T16:24:28.605 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: kBZip2Compression supported: 0 2023-10-11T16:24:28.605 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-11T16:24:28.605 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-11T16:24:28.605 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-10-11T16:24:28.606 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-11T16:24:28.606 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-11T16:24:28.606 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.merge_operator: 2023-10-11T16:24:28.606 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_filter: None 2023-10-11T16:24:28.606 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_filter_factory: None 2023-10-11T16:24:28.606 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.sst_partitioner_factory: None 2023-10-11T16:24:28.607 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-11T16:24:28.607 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-11T16:24:28.607 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dea3f494e0) 2023-10-11T16:24:28.607 INFO:journalctl@ceph.mon.b.smithi130.stdout: cache_index_and_filter_blocks: 1 2023-10-11T16:24:28.607 INFO:journalctl@ceph.mon.b.smithi130.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-11T16:24:28.608 INFO:journalctl@ceph.mon.b.smithi130.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-11T16:24:28.608 INFO:journalctl@ceph.mon.b.smithi130.stdout: pin_top_level_index_and_filter: 1 2023-10-11T16:24:28.608 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_type: 0 2023-10-11T16:24:28.608 INFO:journalctl@ceph.mon.b.smithi130.stdout: data_block_index_type: 0 2023-10-11T16:24:28.608 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_shortening: 1 2023-10-11T16:24:28.609 INFO:journalctl@ceph.mon.b.smithi130.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-11T16:24:28.609 INFO:journalctl@ceph.mon.b.smithi130.stdout: checksum: 4 2023-10-11T16:24:28.609 INFO:journalctl@ceph.mon.b.smithi130.stdout: no_block_cache: 0 2023-10-11T16:24:28.609 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache: 0x55dea325b1f0 2023-10-11T16:24:28.609 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_name: BinnedLRUCache 2023-10-11T16:24:28.609 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_options: 2023-10-11T16:24:28.610 INFO:journalctl@ceph.mon.b.smithi130.stdout: capacity : 536870912 2023-10-11T16:24:28.610 INFO:journalctl@ceph.mon.b.smithi130.stdout: num_shard_bits : 4 2023-10-11T16:24:28.610 INFO:journalctl@ceph.mon.b.smithi130.stdout: strict_capacity_limit : 0 2023-10-11T16:24:28.610 INFO:journalctl@ceph.mon.b.smithi130.stdout: high_pri_pool_ratio: 0.000 2023-10-11T16:24:28.610 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_compressed: (nil) 2023-10-11T16:24:28.611 INFO:journalctl@ceph.mon.b.smithi130.stdout: persistent_cache: (nil) 2023-10-11T16:24:28.611 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_size: 4096 2023-10-11T16:24:28.611 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_size_deviation: 10 2023-10-11T16:24:28.611 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_restart_interval: 16 2023-10-11T16:24:28.611 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_block_restart_interval: 1 2023-10-11T16:24:28.611 INFO:journalctl@ceph.mon.b.smithi130.stdout: metadata_block_size: 4096 2023-10-11T16:24:28.612 INFO:journalctl@ceph.mon.b.smithi130.stdout: partition_filters: 0 2023-10-11T16:24:28.612 INFO:journalctl@ceph.mon.b.smithi130.stdout: use_delta_encoding: 1 2023-10-11T16:24:28.612 INFO:journalctl@ceph.mon.b.smithi130.stdout: filter_policy: bloomfilter 2023-10-11T16:24:28.612 INFO:journalctl@ceph.mon.b.smithi130.stdout: whole_key_filtering: 1 2023-10-11T16:24:28.612 INFO:journalctl@ceph.mon.b.smithi130.stdout: verify_compression: 0 2023-10-11T16:24:28.612 INFO:journalctl@ceph.mon.b.smithi130.stdout: read_amp_bytes_per_bit: 0 2023-10-11T16:24:28.613 INFO:journalctl@ceph.mon.b.smithi130.stdout: format_version: 5 2023-10-11T16:24:28.613 INFO:journalctl@ceph.mon.b.smithi130.stdout: enable_index_compression: 1 2023-10-11T16:24:28.613 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_align: 0 2023-10-11T16:24:28.613 INFO:journalctl@ceph.mon.b.smithi130.stdout: max_auto_readahead_size: 262144 2023-10-11T16:24:28.613 INFO:journalctl@ceph.mon.b.smithi130.stdout: prepopulate_block_cache: 0 2023-10-11T16:24:28.613 INFO:journalctl@ceph.mon.b.smithi130.stdout: initial_auto_readahead_size: 8192 2023-10-11T16:24:28.614 INFO:journalctl@ceph.mon.b.smithi130.stdout: num_file_reads_for_auto_readahead: 2 2023-10-11T16:24:28.614 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.write_buffer_size: 33554432 2023-10-11T16:24:28.614 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_write_buffer_number: 2 2023-10-11T16:24:28.614 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression: NoCompression 2023-10-11T16:24:28.614 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression: Disabled 2023-10-11T16:24:28.615 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.prefix_extractor: nullptr 2023-10-11T16:24:28.615 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-11T16:24:28.615 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.num_levels: 7 2023-10-11T16:24:28.615 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-11T16:24:28.615 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-11T16:24:28.616 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-11T16:24:28.616 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-11T16:24:28.616 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-11T16:24:28.616 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-11T16:24:28.616 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-11T16:24:28.616 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-11T16:24:28.617 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-11T16:24:28.617 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-11T16:24:28.617 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-11T16:24:28.617 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-11T16:24:28.617 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-11T16:24:28.618 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.level: 32767 2023-10-11T16:24:28.618 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.strategy: 0 2023-10-11T16:24:28.618 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-11T16:24:28.618 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-11T16:24:28.618 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-11T16:24:28.618 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-11T16:24:28.619 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.enabled: false 2023-10-11T16:24:28.619 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-11T16:24:28.619 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-11T16:24:28.619 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-11T16:24:28.619 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-11T16:24:28.620 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.target_file_size_base: 67108864 2023-10-11T16:24:28.620 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-11T16:24:28.620 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-11T16:24:28.620 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-11T16:24:28.621 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-11T16:24:28.621 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-11T16:24:28.621 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-11T16:24:28.621 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-11T16:24:28.621 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-11T16:24:28.621 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-11T16:24:28.622 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-11T16:24:28.622 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-11T16:24:28.622 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-11T16:24:28.622 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-11T16:24:28.622 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-11T16:24:28.623 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.arena_block_size: 1048576 2023-10-11T16:24:28.623 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-11T16:24:28.623 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-11T16:24:28.623 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.disable_auto_compactions: 0 2023-10-11T16:24:28.623 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-11T16:24:28.624 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-11T16:24:28.624 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-11T16:24:28.624 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-11T16:24:28.624 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-11T16:24:28.624 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-11T16:24:28.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-11T16:24:28.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-11T16:24:28.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-11T16:24:28.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-11T16:24:28.625 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.table_properties_collectors: 2023-10-11T16:24:28.626 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.inplace_update_support: 0 2023-10-11T16:24:28.626 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-11T16:24:28.626 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-11T16:24:28.626 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-11T16:24:28.627 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-11T16:24:28.627 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.bloom_locality: 0 2023-10-11T16:24:28.627 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.max_successive_merges: 0 2023-10-11T16:24:28.627 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-11T16:24:28.627 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.paranoid_file_checks: 0 2023-10-11T16:24:28.627 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.force_consistency_checks: 1 2023-10-11T16:24:28.628 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.report_bg_io_stats: 0 2023-10-11T16:24:28.628 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.ttl: 2592000 2023-10-11T16:24:28.628 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-11T16:24:28.628 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-11T16:24:28.628 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-11T16:24:28.629 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.enable_blob_files: false 2023-10-11T16:24:28.629 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.min_blob_size: 0 2023-10-11T16:24:28.629 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.blob_file_size: 268435456 2023-10-11T16:24:28.629 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-11T16:24:28.630 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-11T16:24:28.630 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-11T16:24:28.630 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-11T16:24:28.630 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-11T16:24:28.631 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.blob_file_starting_level: 0 2023-10-11T16:24:28.631 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-11T16:24:28.631 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-10-11T16:24:28.631 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-10-11T16:24:28.631 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c40ac87e-fbf8-4726-a5c7-285ea9a7213b 2023-10-11T16:24:28.632 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041468498348, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-10-11T16:24:28.632 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-10-11T16:24:28.632 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041468498778, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1697041468, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c40ac87e-fbf8-4726-a5c7-285ea9a7213b", "db_session_id": "96BCTSGSQFO6KKYFSSGW", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-10-11T16:24:28.632 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1697041468498846, "job": 1, "event": "recovery_finished"} 2023-10-11T16:24:28.632 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-10-11T16:24:28.633 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 systemd[1]: Started Ceph mon.b for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:24:28.633 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-11T16:24:28.633 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dea32f8000 2023-10-11T16:24:28.633 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: DB pointer 0x55dea32e4000 2023-10-11T16:24:28.633 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-11T16:24:28.634 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-11T16:24:28.634 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** DB Stats ** 2023-10-11T16:24:28.634 INFO:journalctl@ceph.mon.b.smithi130.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-11T16:24:28.634 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-11T16:24:28.634 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-11T16:24:28.635 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-11T16:24:28.635 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-11T16:24:28.635 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-11T16:24:28.635 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-11T16:24:28.635 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-10-11T16:24:28.636 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** Compaction Stats [default] ** 2023-10-11T16:24:28.636 INFO:journalctl@ceph.mon.b.smithi130.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-11T16:24:28.636 INFO:journalctl@ceph.mon.b.smithi130.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-11T16:24:28.636 INFO:journalctl@ceph.mon.b.smithi130.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-11T16:24:28.636 INFO:journalctl@ceph.mon.b.smithi130.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-11T16:24:28.637 INFO:journalctl@ceph.mon.b.smithi130.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-11T16:24:28.637 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-10-11T16:24:28.637 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** Compaction Stats [default] ** 2023-10-11T16:24:28.637 INFO:journalctl@ceph.mon.b.smithi130.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-11T16:24:28.638 INFO:journalctl@ceph.mon.b.smithi130.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-11T16:24:28.638 INFO:journalctl@ceph.mon.b.smithi130.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-11T16:24:28.638 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-10-11T16:24:28.638 INFO:journalctl@ceph.mon.b.smithi130.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-11T16:24:28.638 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-10-11T16:24:28.639 INFO:journalctl@ceph.mon.b.smithi130.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-11T16:24:28.639 INFO:journalctl@ceph.mon.b.smithi130.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-11T16:24:28.639 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-11T16:24:28.639 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-11T16:24:28.640 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-11T16:24:28.640 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-11T16:24:28.640 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-11T16:24:28.640 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-11T16:24:28.640 INFO:journalctl@ceph.mon.b.smithi130.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-11T16:24:28.641 INFO:journalctl@ceph.mon.b.smithi130.stdout: Block cache BinnedLRUCache@0x55dea325b1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-10-11T16:24:28.641 INFO:journalctl@ceph.mon.b.smithi130.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-10-11T16:24:28.641 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-10-11T16:24:28.641 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-11T16:24:28.641 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-10-11T16:24:28.642 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: using public_addr v2:172.21.15.130:0/0 -> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] 2023-10-11T16:24:28.642 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: starting mon.b rank -1 at public addrs [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] at bind addrs [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:24:28.642 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(???) e0 preinit fsid 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:24:28.642 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).mds e1 new map 2023-10-11T16:24:28.643 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).mds e1 print_map 2023-10-11T16:24:28.643 INFO:journalctl@ceph.mon.b.smithi130.stdout: e1 2023-10-11T16:24:28.643 INFO:journalctl@ceph.mon.b.smithi130.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-11T16:24:28.643 INFO:journalctl@ceph.mon.b.smithi130.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-11T16:24:28.643 INFO:journalctl@ceph.mon.b.smithi130.stdout: legacy client fscid: -1 2023-10-11T16:24:28.643 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-10-11T16:24:28.644 INFO:journalctl@ceph.mon.b.smithi130.stdout: No filesystems configured 2023-10-11T16:24:28.644 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-10-11T16:24:28.644 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-10-11T16:24:28.644 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-10-11T16:24:28.645 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-10-11T16:24:28.645 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-10-11T16:24:28.645 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-10-11T16:24:28.645 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-11T16:24:28.645 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:24:28.646 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:24:28.646 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-11T16:24:28.646 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:28.646 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:24:28.646 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:28.647 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:28.647 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:24:28.647 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:28.647 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.647 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Deploying cephadm binary to smithi150 2023-10-11T16:24:28.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.648 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:28.649 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Added host smithi150 2023-10-11T16:24:28.649 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.649 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.649 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.649 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.650 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.650 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:24:28.650 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.650 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.650 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.650 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:24:28.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:28.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:28.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:28.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:24:28.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:28.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:28.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2640446437' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-11T16:24:28.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.653 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.653 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:24:28.653 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.653 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2640446437' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-11T16:24:28.654 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:28.654 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.654 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.654 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:28.654 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:28.655 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:28.655 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.655 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:28.655 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:28.655 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:28.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Saving service mon spec with placement smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c;count:3 2023-10-11T16:24:28.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:28.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:28.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Deploying daemon mon.c on smithi150 2023-10-11T16:24:28.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:24:28.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.a calling monitor election 2023-10-11T16:24:28.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:28.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:28.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:28.658 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.c calling monitor election 2023-10-11T16:24:28.658 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:28.658 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.658 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:28.659 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:28.659 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-11T16:24:28.659 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: monmap e2: 2 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:28.659 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: fsmap 2023-10-11T16:24:28.659 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:28.660 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mgrmap e13: x(active, since 45s) 2023-10-11T16:24:28.660 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: overall HEALTH_OK 2023-10-11T16:24:28.660 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.660 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.660 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:28.661 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:28.661 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:28.661 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: Deploying daemon mon.b on smithi130 2023-10-11T16:24:28.661 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:28.661 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:28.662 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/4132816848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-11T16:24:28.662 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-10-11T16:24:28.662 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta expand map: {default=false} 2023-10-11T16:24:28.662 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta from 'false' to 'false' 2023-10-11T16:24:28.662 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta expanded map: {default=false} 2023-10-11T16:24:28.662 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta expand map: {default=info} 2023-10-11T16:24:28.663 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta from 'info' to 'info' 2023-10-11T16:24:28.663 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta expanded map: {default=info} 2023-10-11T16:24:28.663 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta expand map: {default=daemon} 2023-10-11T16:24:28.663 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-11T16:24:28.663 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta expanded map: {default=daemon} 2023-10-11T16:24:28.663 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta expand map: {default=debug} 2023-10-11T16:24:28.664 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta from 'debug' to 'debug' 2023-10-11T16:24:28.664 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: expand_channel_meta expanded map: {default=debug} 2023-10-11T16:24:28.664 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:28 smithi130 ceph-mon[131859]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-10-11T16:24:29.156 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:24:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.562+0000 7fec834de700 1 -- 172.21.15.150:0/2954983505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c104de0 msgr2=0x7fec68007820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.562+0000 7fec834de700 1 --2- 172.21.15.150:0/2954983505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c104de0 0x7fec68007820 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fec6c013c60 tx=0x7fec6c013f70 comp rx=0 tx=0).stop 2023-10-11T16:24:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.562+0000 7fec834de700 1 -- 172.21.15.150:0/2954983505 shutdown_connections 2023-10-11T16:24:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.562+0000 7fec834de700 1 --2- 172.21.15.150:0/2954983505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c104de0 0x7fec68007820 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.562+0000 7fec834de700 1 -- 172.21.15.150:0/2954983505 >> 172.21.15.150:0/2954983505 conn(0x7fec7c074c40 msgr2=0x7fec7c075040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.562+0000 7fec834de700 1 -- 172.21.15.150:0/2954983505 shutdown_connections 2023-10-11T16:24:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.562+0000 7fec834de700 1 -- 172.21.15.150:0/2954983505 wait complete. 2023-10-11T16:24:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.563+0000 7fec834de700 1 Processor -- start 2023-10-11T16:24:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.563+0000 7fec834de700 1 -- start start 2023-10-11T16:24:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.563+0000 7fec834de700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 0x7fec7c193190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.563+0000 7fec834de700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c1936d0 0x7fec7c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.563+0000 7fec834de700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 0x7fec7c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.563+0000 7fec834de700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c193da0 con 0x7fec7c1936d0 2023-10-11T16:24:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.563+0000 7fec834de700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c19fdf0 con 0x7fec7c104de0 2023-10-11T16:24:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.563+0000 7fec834de700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c19ff30 con 0x7fec7c198080 2023-10-11T16:24:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec81a7b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 0x7fec7c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec81a7b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 0x7fec7c198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42344/0 (socket says 172.21.15.150:42344) 2023-10-11T16:24:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec81a7b700 1 -- 172.21.15.150:0/712301004 learned_addr learned my addr 172.21.15.150:0/712301004 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:24:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec80a79700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c1936d0 0x7fec7c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec8127a700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 0x7fec7c193190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec81a7b700 1 -- 172.21.15.150:0/712301004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 msgr2=0x7fec7c198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-10-11T16:24:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec81a7b700 1 -- 172.21.15.150:0/712301004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 msgr2=0x7fec7c198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-11T16:24:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec81a7b700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 0x7fec7c198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-11T16:24:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec81a7b700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 0x7fec7c198520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:24:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec8127a700 1 -- 172.21.15.150:0/712301004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 msgr2=0x7fec7c193190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-10-11T16:24:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec8127a700 1 -- 172.21.15.150:0/712301004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 msgr2=0x7fec7c193190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-11T16:24:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec8127a700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 0x7fec7c193190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-11T16:24:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec8127a700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 0x7fec7c193190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:24:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec80a79700 1 -- 172.21.15.150:0/712301004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 msgr2=0x7fec7c198520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:24:33.573 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec80a79700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 0x7fec7c198520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:33.573 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec80a79700 1 -- 172.21.15.150:0/712301004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 msgr2=0x7fec7c193190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:24:33.573 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec80a79700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 0x7fec7c193190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:33.573 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec80a79700 1 -- 172.21.15.150:0/712301004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c013940 con 0x7fec7c1936d0 2023-10-11T16:24:33.573 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec80a79700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c1936d0 0x7fec7c197b40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec7800b580 tx=0x7fec7800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:33.574 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.564+0000 7fec727fc700 1 -- 172.21.15.150:0/712301004 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78004230 con 0x7fec7c1936d0 2023-10-11T16:24:33.574 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.565+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7c105bf0 con 0x7fec7c1936d0 2023-10-11T16:24:33.574 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.565+0000 7fec727fc700 1 -- 172.21.15.150:0/712301004 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec78004390 con 0x7fec7c1936d0 2023-10-11T16:24:33.574 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.565+0000 7fec727fc700 1 -- 172.21.15.150:0/712301004 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78011a70 con 0x7fec7c1936d0 2023-10-11T16:24:33.574 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.565+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7c0690c0 con 0x7fec7c1936d0 2023-10-11T16:24:33.575 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.566+0000 7fec727fc700 1 -- 172.21.15.150:0/712301004 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fec78011c10 con 0x7fec7c1936d0 2023-10-11T16:24:33.576 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.566+0000 7fec727fc700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fec60038d40 0x7fec6003b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:33.576 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.566+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7c105d80 con 0x7fec7c1936d0 2023-10-11T16:24:33.576 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.566+0000 7fec727fc700 1 -- 172.21.15.150:0/712301004 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec7804add0 con 0x7fec7c1936d0 2023-10-11T16:24:33.576 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.566+0000 7fec8127a700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fec60038d40 0x7fec6003b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:33.576 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.567+0000 7fec8127a700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fec60038d40 0x7fec6003b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec6c0059d0 tx=0x7fec6c0058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:33.577 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.572+0000 7fec727fc700 1 -- 172.21.15.150:0/712301004 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fec7801a400 con 0x7fec7c1936d0 2023-10-11T16:24:33.843 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.842+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec7c062120 con 0x7fec7c1936d0 2023-10-11T16:24:33.844 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.842+0000 7fec727fc700 1 -- 172.21.15.150:0/712301004 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1278 (secure 0 0 0) 0x7fec78021900 con 0x7fec7c1936d0 2023-10-11T16:24:33.844 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-10-11T16:24:33.844 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":3,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","modified":"2023-10-11T16:24:28.523209Z","created":"2023-10-11T16:22:41.058052Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-10-11T16:24:33.846 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fec60038d40 msgr2=0x7fec6003b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:33.846 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fec60038d40 0x7fec6003b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec6c0059d0 tx=0x7fec6c0058e0 comp rx=0 tx=0).stop 2023-10-11T16:24:33.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c1936d0 msgr2=0x7fec7c197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:33.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c1936d0 0x7fec7c197b40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec7800b580 tx=0x7fec7800b940 comp rx=0 tx=0).stop 2023-10-11T16:24:33.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 shutdown_connections 2023-10-11T16:24:33.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fec60038d40 0x7fec6003b1f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:33.847 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec7c198080 0x7fec7c198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:33.848 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec7c104de0 0x7fec7c193190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:33.848 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 --2- 172.21.15.150:0/712301004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec7c1936d0 0x7fec7c197b40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:33.848 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 >> 172.21.15.150:0/712301004 conn(0x7fec7c074c40 msgr2=0x7fec7c18ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:33.848 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 shutdown_connections 2023-10-11T16:24:33.848 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:33.844+0000 7fec834de700 1 -- 172.21.15.150:0/712301004 wait complete. 2023-10-11T16:24:33.849 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 3 2023-10-11T16:24:33.867 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:33.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:24:33.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: mon.a calling monitor election 2023-10-11T16:24:33.868 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: mon.c calling monitor election 2023-10-11T16:24:33.869 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:33.869 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.869 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:33.869 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.869 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.870 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:33.870 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.870 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.870 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-11T16:24:33.871 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: monmap e3: 3 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:33.871 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: fsmap 2023-10-11T16:24:33.871 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:33.871 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: mgrmap e13: x(active, since 53s) 2023-10-11T16:24:33.871 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-10-11T16:24:33.871 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-10-11T16:24:33.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-10-11T16:24:33.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2023-10-11T16:24:33.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:33.872 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:33.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:33.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:24:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: mon.a calling monitor election 2023-10-11T16:24:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: mon.c calling monitor election 2023-10-11T16:24:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-11T16:24:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: monmap e3: 3 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: fsmap 2023-10-11T16:24:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: mgrmap e13: x(active, since 53s) 2023-10-11T16:24:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-10-11T16:24:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-10-11T16:24:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-10-11T16:24:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2023-10-11T16:24:33.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:33.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:34.417 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-10-11T16:24:34.418 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph config generate-minimal-conf 2023-10-11T16:24:34.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:34 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/712301004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-11T16:24:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:34.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:34.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:34.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:34.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:34 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/712301004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-11T16:24:34.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:35.272 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:24:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:24:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: mon.a calling monitor election 2023-10-11T16:24:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: mon.c calling monitor election 2023-10-11T16:24:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:24:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-11T16:24:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: monmap e3: 3 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: fsmap 2023-10-11T16:24:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: mgrmap e13: x(active, since 53s) 2023-10-11T16:24:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-10-11T16:24:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-10-11T16:24:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-10-11T16:24:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2023-10-11T16:24:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/712301004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-11T16:24:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:36.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: mon.b calling monitor election 2023-10-11T16:24:36.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:36.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: Updating smithi111:/etc/ceph/ceph.conf 2023-10-11T16:24:36.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:24:36.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:24:36.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: mon.b calling monitor election 2023-10-11T16:24:36.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: mon.c calling monitor election 2023-10-11T16:24:36.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: mon.a calling monitor election 2023-10-11T16:24:36.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-11T16:24:36.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: monmap e3: 3 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:36.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: fsmap 2023-10-11T16:24:36.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:36.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: mgrmap e13: x(active, since 55s) 2023-10-11T16:24:36.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-10-11T16:24:36.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: Cluster is now healthy 2023-10-11T16:24:36.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: overall HEALTH_OK 2023-10-11T16:24:36.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:36.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:36.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.806+0000 7f0b85465700 1 -- 172.21.15.111:0/2111867653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b80072e90 msgr2=0x7f0b80073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:36.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.806+0000 7f0b85465700 1 --2- 172.21.15.111:0/2111867653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b80072e90 0x7f0b80073260 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009910 tx=0x7f0b70009c20 comp rx=0 tx=0).stop 2023-10-11T16:24:36.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.806+0000 7f0b85465700 1 -- 172.21.15.111:0/2111867653 shutdown_connections 2023-10-11T16:24:36.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.806+0000 7f0b85465700 1 --2- 172.21.15.111:0/2111867653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b80072e90 0x7f0b80073260 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:36.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.806+0000 7f0b85465700 1 -- 172.21.15.111:0/2111867653 >> 172.21.15.111:0/2111867653 conn(0x7f0b800fb870 msgr2=0x7f0b800fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:36.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.806+0000 7f0b85465700 1 -- 172.21.15.111:0/2111867653 shutdown_connections 2023-10-11T16:24:36.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.806+0000 7f0b85465700 1 -- 172.21.15.111:0/2111867653 wait complete. 2023-10-11T16:24:36.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.807+0000 7f0b85465700 1 Processor -- start 2023-10-11T16:24:36.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b85465700 1 -- start start 2023-10-11T16:24:36.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b85465700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b80072e90 0x7f0b801a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b85465700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b801a1be0 0x7f0b8019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b85465700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b8019bc80 0x7f0b8019c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b85465700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b801a2240 con 0x7f0b801a1be0 2023-10-11T16:24:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b85465700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b801a8500 con 0x7f0b80072e90 2023-10-11T16:24:36.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b85465700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b801a8640 con 0x7f0b8019bc80 2023-10-11T16:24:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b7e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b801a1be0 0x7f0b8019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b7e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b801a1be0 0x7f0b8019b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45356/0 (socket says 172.21.15.111:45356) 2023-10-11T16:24:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/4009616082 learned_addr learned my addr 172.21.15.111:0/4009616082 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:24:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.808+0000 7f0b7f7fe700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b8019bc80 0x7f0b8019c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:36.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b7effd700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b80072e90 0x7f0b801a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/4009616082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b8019bc80 msgr2=0x7f0b8019c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b7e7fc700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b8019bc80 0x7f0b8019c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/4009616082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b80072e90 msgr2=0x7f0b801a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b7e7fc700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b80072e90 0x7f0b801a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:36.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b7e7fc700 1 -- 172.21.15.111:0/4009616082 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700095f0 con 0x7f0b801a1be0 2023-10-11T16:24:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b7effd700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b80072e90 0x7f0b801a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:24:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b7e7fc700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b801a1be0 0x7f0b8019b740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400b780 tx=0x7f0b7400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b67fff700 1 -- 172.21.15.111:0/4009616082 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74004230 con 0x7f0b801a1be0 2023-10-11T16:24:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b80110780 con 0x7f0b801a1be0 2023-10-11T16:24:36.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b67fff700 1 -- 172.21.15.111:0/4009616082 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b74004390 con 0x7f0b801a1be0 2023-10-11T16:24:36.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.809+0000 7f0b67fff700 1 -- 172.21.15.111:0/4009616082 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74011ad0 con 0x7f0b801a1be0 2023-10-11T16:24:36.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.810+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b801105f0 con 0x7f0b801a1be0 2023-10-11T16:24:36.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.811+0000 7f0b67fff700 1 -- 172.21.15.111:0/4009616082 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f0b74011c30 con 0x7f0b801a1be0 2023-10-11T16:24:36.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.811+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b80110910 con 0x7f0b801a1be0 2023-10-11T16:24:36.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.811+0000 7f0b67fff700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b68038d40 0x7f0b6803b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:36.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.811+0000 7f0b67fff700 1 -- 172.21.15.111:0/4009616082 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0b7404af70 con 0x7f0b801a1be0 2023-10-11T16:24:36.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.811+0000 7f0b7effd700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b68038d40 0x7f0b6803b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:36.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.812+0000 7f0b7effd700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b68038d40 0x7f0b6803b1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000b5c0 tx=0x7f0b700058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:36.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:36.816+0000 7f0b67fff700 1 -- 172.21.15.111:0/4009616082 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0b7401a400 con 0x7f0b801a1be0 2023-10-11T16:24:36.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: mon.b calling monitor election 2023-10-11T16:24:36.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:36.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: Updating smithi111:/etc/ceph/ceph.conf 2023-10-11T16:24:36.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:24:36.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:24:36.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: mon.b calling monitor election 2023-10-11T16:24:36.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: mon.c calling monitor election 2023-10-11T16:24:36.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: mon.a calling monitor election 2023-10-11T16:24:36.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-11T16:24:36.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: monmap e3: 3 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:36.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: fsmap 2023-10-11T16:24:36.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:36.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: mgrmap e13: x(active, since 55s) 2023-10-11T16:24:36.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-10-11T16:24:36.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: Cluster is now healthy 2023-10-11T16:24:36.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: overall HEALTH_OK 2023-10-11T16:24:36.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:36.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:36.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: mon.b calling monitor election 2023-10-11T16:24:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: Updating smithi111:/etc/ceph/ceph.conf 2023-10-11T16:24:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:24:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:24:36.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: mon.b calling monitor election 2023-10-11T16:24:36.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: mon.c calling monitor election 2023-10-11T16:24:36.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: mon.a calling monitor election 2023-10-11T16:24:36.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-11T16:24:36.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: monmap e3: 3 mons at {a=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} 2023-10-11T16:24:36.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: fsmap 2023-10-11T16:24:36.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: osdmap e4: 0 total, 0 up, 0 in 2023-10-11T16:24:36.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: mgrmap e13: x(active, since 55s) 2023-10-11T16:24:36.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-10-11T16:24:36.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: Cluster is now healthy 2023-10-11T16:24:36.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: overall HEALTH_OK 2023-10-11T16:24:36.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:36.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:36.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:36.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.025+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0b801a91a0 con 0x7f0b801a1be0 2023-10-11T16:24:37.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.025+0000 7f0b67fff700 1 -- 172.21.15.111:0/4009616082 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f0b74004570 con 0x7f0b801a1be0 2023-10-11T16:24:37.028 INFO:teuthology.orchestra.run.smithi111.stdout:# minimal ceph.conf for 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:24:37.028 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2023-10-11T16:24:37.029 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:24:37.029 INFO:teuthology.orchestra.run.smithi111.stdout: mon_host = [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2023-10-11T16:24:37.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.027+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b68038d40 msgr2=0x7f0b6803b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:37.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.027+0000 7f0b85465700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b68038d40 0x7f0b6803b1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000b5c0 tx=0x7f0b700058e0 comp rx=0 tx=0).stop 2023-10-11T16:24:37.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.027+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b801a1be0 msgr2=0x7f0b8019b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:37.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.027+0000 7f0b85465700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b801a1be0 0x7f0b8019b740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400b780 tx=0x7f0b7400ba90 comp rx=0 tx=0).stop 2023-10-11T16:24:37.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.028+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 shutdown_connections 2023-10-11T16:24:37.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.028+0000 7f0b85465700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b68038d40 0x7f0b6803b1f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:37.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.028+0000 7f0b85465700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b8019bc80 0x7f0b8019c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:37.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.028+0000 7f0b85465700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b80072e90 0x7f0b801a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:37.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.028+0000 7f0b85465700 1 --2- 172.21.15.111:0/4009616082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b801a1be0 0x7f0b8019b740 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:37.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.028+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 >> 172.21.15.111:0/4009616082 conn(0x7f0b800fb870 msgr2=0x7f0b800fbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:37.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.028+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 shutdown_connections 2023-10-11T16:24:37.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:37.028+0000 7f0b85465700 1 -- 172.21.15.111:0/4009616082 wait complete. 2023-10-11T16:24:37.571 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-10-11T16:24:37.572 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:24:37.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2023-10-11T16:24:37.651 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:24:37.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:37.676 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.677 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:37.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:37.678 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:37.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:37.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:37.679 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:37 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4009616082' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:37.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:37.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:37.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:37.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:37.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:37 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4009616082' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:37.749 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:24:37.749 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.conf 2023-10-11T16:24:37.781 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:24:37.782 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:37.782 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.782 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.782 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:37.783 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:37.783 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:24:37.783 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:37.783 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:37.784 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:37.784 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:37 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4009616082' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:37.786 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:24:37.787 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:37.857 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:24:37.858 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2023-10-11T16:24:37.895 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:24:37.895 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:24:37.967 INFO:tasks.cephadm:Adding mgr.x on smithi111 2023-10-11T16:24:37.967 INFO:tasks.cephadm:Adding mgr.y on smithi130 2023-10-11T16:24:37.968 INFO:tasks.cephadm:Adding mgr.z on smithi150 2023-10-11T16:24:37.968 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch apply mgr '3;smithi111=x;smithi130=y;smithi150=z' 2023-10-11T16:24:38.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:38 smithi111 ceph-mon[98527]: Reconfiguring mon.a (unknown last config time)... 2023-10-11T16:24:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:38 smithi111 ceph-mon[98527]: Reconfiguring daemon mon.a on smithi111 2023-10-11T16:24:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:38.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:38.647 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:24:38.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:38 smithi150 ceph-mon[132217]: Reconfiguring mon.a (unknown last config time)... 2023-10-11T16:24:38.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:38 smithi150 ceph-mon[132217]: Reconfiguring daemon mon.a on smithi111 2023-10-11T16:24:38.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:38.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:38.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:38.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:38.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:38.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:38 smithi130 ceph-mon[131859]: Reconfiguring mon.a (unknown last config time)... 2023-10-11T16:24:38.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:38 smithi130 ceph-mon[131859]: Reconfiguring daemon mon.a on smithi111 2023-10-11T16:24:38.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:38.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:38.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:38.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:38.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:39 smithi150 ceph-mon[132217]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:39 smithi150 ceph-mon[132217]: Reconfiguring mon.b (monmap changed)... 2023-10-11T16:24:39.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:39 smithi150 ceph-mon[132217]: Reconfiguring daemon mon.b on smithi130 2023-10-11T16:24:39.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:39 smithi130 ceph-mon[131859]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:39.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:39 smithi130 ceph-mon[131859]: Reconfiguring mon.b (monmap changed)... 2023-10-11T16:24:39.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:39 smithi130 ceph-mon[131859]: Reconfiguring daemon mon.b on smithi130 2023-10-11T16:24:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 -- 172.21.15.150:0/2643452505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c101430 msgr2=0x7fa77c101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:39.873 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 --2- 172.21.15.150:0/2643452505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c101430 0x7fa77c101840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa778009910 tx=0x7fa778009c20 comp rx=0 tx=0).stop 2023-10-11T16:24:39.873 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 -- 172.21.15.150:0/2643452505 shutdown_connections 2023-10-11T16:24:39.873 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 --2- 172.21.15.150:0/2643452505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c101430 0x7fa77c101840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:39.874 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 -- 172.21.15.150:0/2643452505 >> 172.21.15.150:0/2643452505 conn(0x7fa77c0f9a60 msgr2=0x7fa77c0fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:39.874 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 -- 172.21.15.150:0/2643452505 shutdown_connections 2023-10-11T16:24:39.874 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 -- 172.21.15.150:0/2643452505 wait complete. 2023-10-11T16:24:39.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 Processor -- start 2023-10-11T16:24:39.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.872+0000 7fa7834f8700 1 -- start start 2023-10-11T16:24:39.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa7834f8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa77c101430 0x7fa77c195090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:39.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa7834f8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa77c1955d0 0x7fa77c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:39.877 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa7834f8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c195a40 0x7fa77c19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:39.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa7834f8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c19d350 con 0x7fa77c195a40 2023-10-11T16:24:39.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa7834f8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c19d4c0 con 0x7fa77c101430 2023-10-11T16:24:39.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa7834f8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c19d680 con 0x7fa77c1955d0 2023-10-11T16:24:39.878 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa77c1955d0 0x7fa77c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa77c1955d0 0x7fa77c19a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42352/0 (socket says 172.21.15.150:42352) 2023-10-11T16:24:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 -- 172.21.15.150:0/2559241007 learned_addr learned my addr 172.21.15.150:0/2559241007 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:24:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa781294700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa77c101430 0x7fa77c195090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 -- 172.21.15.150:0/2559241007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa77c101430 msgr2=0x7fa77c195090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa77c101430 0x7fa77c195090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 -- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c195a40 msgr2=0x7fa77c19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa781a95700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c195a40 0x7fa77c19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c195a40 0x7fa77c19cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 -- 172.21.15.150:0/2559241007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7780095f0 con 0x7fa77c1955d0 2023-10-11T16:24:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa780a93700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa77c1955d0 0x7fa77c19a640 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00b780 tx=0x7fa76c00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:39.880 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.873+0000 7fa7727fc700 1 -- 172.21.15.150:0/2559241007 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c004230 con 0x7fa77c1955d0 2023-10-11T16:24:39.881 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.874+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c105cb0 con 0x7fa77c1955d0 2023-10-11T16:24:39.881 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.874+0000 7fa7727fc700 1 -- 172.21.15.150:0/2559241007 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa76c004390 con 0x7fa77c1955d0 2023-10-11T16:24:39.881 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.874+0000 7fa7727fc700 1 -- 172.21.15.150:0/2559241007 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c011ad0 con 0x7fa77c1955d0 2023-10-11T16:24:39.881 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.874+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c105b20 con 0x7fa77c1955d0 2023-10-11T16:24:39.881 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.874+0000 7fa7727fc700 1 -- 172.21.15.150:0/2559241007 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7fa76c011c30 con 0x7fa77c1955d0 2023-10-11T16:24:39.881 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.875+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c18e9e0 con 0x7fa77c1955d0 2023-10-11T16:24:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.875+0000 7fa7727fc700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa768038cf0 0x7fa76803b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.875+0000 7fa7727fc700 1 -- 172.21.15.150:0/2559241007 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa76c04ac60 con 0x7fa77c1955d0 2023-10-11T16:24:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.875+0000 7fa781294700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa768038cf0 0x7fa76803b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:39.882 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.876+0000 7fa781294700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa768038cf0 0x7fa76803b1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa778005130 tx=0x7fa778005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:39.883 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:39.880+0000 7fa7727fc700 1 -- 172.21.15.150:0/2559241007 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa76c01a400 con 0x7fa77c1955d0 2023-10-11T16:24:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:39 smithi111 ceph-mon[98527]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:39 smithi111 ceph-mon[98527]: Reconfiguring mon.b (monmap changed)... 2023-10-11T16:24:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:39 smithi111 ceph-mon[98527]: Reconfiguring daemon mon.b on smithi130 2023-10-11T16:24:40.094 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.093+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi111=x;smithi130=y;smithi150=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fa77c061b00 con 0x7fa768038cf0 2023-10-11T16:24:40.101 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.100+0000 7fa7727fc700 1 -- 172.21.15.150:0/2559241007 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa77c061b00 con 0x7fa768038cf0 2023-10-11T16:24:40.101 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled mgr update... 2023-10-11T16:24:40.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa768038cf0 msgr2=0x7fa76803b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:40.103 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa768038cf0 0x7fa76803b1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa778005130 tx=0x7fa778005fd0 comp rx=0 tx=0).stop 2023-10-11T16:24:40.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa77c1955d0 msgr2=0x7fa77c19a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:40.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa77c1955d0 0x7fa77c19a640 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00b780 tx=0x7fa76c00ba90 comp rx=0 tx=0).stop 2023-10-11T16:24:40.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 shutdown_connections 2023-10-11T16:24:40.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa768038cf0 0x7fa76803b1a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:40.104 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c195a40 0x7fa77c19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:40.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa77c101430 0x7fa77c195090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:40.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 --2- 172.21.15.150:0/2559241007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa77c1955d0 0x7fa77c19a640 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:40.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 >> 172.21.15.150:0/2559241007 conn(0x7fa77c0f9a60 msgr2=0x7fa77c0fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:40.105 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 shutdown_connections 2023-10-11T16:24:40.106 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:24:40.102+0000 7fa7834f8700 1 -- 172.21.15.150:0/2559241007 wait complete. 2023-10-11T16:24:40.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.640 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:40 smithi150 ceph-mon[132217]: Reconfiguring mon.c (monmap changed)... 2023-10-11T16:24:40.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:40.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:40.641 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:40.642 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:40 smithi150 ceph-mon[132217]: Reconfiguring daemon mon.c on smithi150 2023-10-11T16:24:40.642 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:40 smithi130 ceph-mon[131859]: Reconfiguring mon.c (monmap changed)... 2023-10-11T16:24:40.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:40.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:40.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:40.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:40 smithi130 ceph-mon[131859]: Reconfiguring daemon mon.c on smithi150 2023-10-11T16:24:40.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:40 smithi111 ceph-mon[98527]: Reconfiguring mon.c (monmap changed)... 2023-10-11T16:24:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-11T16:24:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-11T16:24:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:40 smithi111 ceph-mon[98527]: Reconfiguring daemon mon.c on smithi150 2023-10-11T16:24:40.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:40.903 DEBUG:teuthology.orchestra.run.smithi130:mgr.y> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.y.service 2023-10-11T16:24:40.906 DEBUG:teuthology.orchestra.run.smithi150:mgr.z> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.z.service 2023-10-11T16:24:40.908 INFO:tasks.cephadm:Deploying OSDs... 2023-10-11T16:24:40.909 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:24:40.909 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2023-10-11T16:24:40.936 INFO:journalctl@ceph.mgr.z.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:24:40.939 INFO:journalctl@ceph.mgr.y.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:24:40.959 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-11T16:24:40.959 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2023-10-11T16:24:41.007 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-11T16:24:41.007 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:41.007 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 452 Links: 1 2023-10-11T16:24:41.007 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:41.007 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:41.008 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 16:23:49.060345518 +0000 2023-10-11T16:24:41.008 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 16:17:56.766454740 +0000 2023-10-11T16:24:41.008 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 16:17:56.766454740 +0000 2023-10-11T16:24:41.008 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2023-10-11 16:17:56.737455221 +0000 2023-10-11T16:24:41.008 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-11T16:24:41.077 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-11T16:24:41.078 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-11T16:24:41.079 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000313856 s, 1.6 MB/s 2023-10-11T16:24:41.080 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-11T16:24:41.126 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2023-10-11T16:24:41.211 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-11T16:24:41.211 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:41.211 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 458 Links: 1 2023-10-11T16:24:41.211 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:41.212 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:41.212 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 16:23:49.060345518 +0000 2023-10-11T16:24:41.212 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 16:17:57.585441178 +0000 2023-10-11T16:24:41.212 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 16:17:57.585441178 +0000 2023-10-11T16:24:41.213 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2023-10-11 16:17:57.559441609 +0000 2023-10-11T16:24:41.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-11T16:24:41.317 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-11T16:24:41.317 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-11T16:24:41.317 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000317083 s, 1.6 MB/s 2023-10-11T16:24:41.319 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-11T16:24:41.365 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2023-10-11T16:24:41.449 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-11T16:24:41.450 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:41.450 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 464 Links: 1 2023-10-11T16:24:41.450 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:41.451 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:41.451 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 16:23:49.060345518 +0000 2023-10-11T16:24:41.451 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 16:17:58.420427351 +0000 2023-10-11T16:24:41.451 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 16:17:58.420427351 +0000 2023-10-11T16:24:41.451 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2023-10-11 16:17:58.394427782 +0000 2023-10-11T16:24:41.452 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-11T16:24:41.555 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-11T16:24:41.555 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-11T16:24:41.555 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000303823 s, 1.7 MB/s 2023-10-11T16:24:41.557 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-11T16:24:41.603 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2023-10-11T16:24:41.688 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-11T16:24:41.688 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:41.688 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 470 Links: 1 2023-10-11T16:24:41.688 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:41.688 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:41.689 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 16:23:49.060345518 +0000 2023-10-11T16:24:41.689 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 16:17:59.229413955 +0000 2023-10-11T16:24:41.689 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 16:17:59.229413955 +0000 2023-10-11T16:24:41.689 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2023-10-11 16:17:59.204414369 +0000 2023-10-11T16:24:41.690 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-11T16:24:41.794 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2023-10-11T16:24:41.794 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2023-10-11T16:24:41.794 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000303168 s, 1.7 MB/s 2023-10-11T16:24:41.796 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-11T16:24:41.842 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-10-11T16:24:41.843 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2023-10-11T16:24:41.864 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-11T16:24:41.865 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2023-10-11T16:24:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi111=x;smithi130=y;smithi150=z", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:41.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: Saving service mgr spec with placement smithi111=x;smithi130=y;smithi150=z;count:3 2023-10-11T16:24:41.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:41.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:41.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:41.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:41.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-11T16:24:41.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:41.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:41.925 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-11T16:24:41.926 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:41.926 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 393636 Links: 1 2023-10-11T16:24:41.926 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:41.926 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:41.926 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2023-10-11 16:24:27.965273958 +0000 2023-10-11T16:24:41.927 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2023-10-11 16:15:14.185670140 +0000 2023-10-11T16:24:41.927 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2023-10-11 16:15:14.185670140 +0000 2023-10-11T16:24:41.927 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2023-10-11T16:24:41.927 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-11T16:24:41.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:41.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi111=x;smithi130=y;smithi150=z", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:41.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: Saving service mgr spec with placement smithi111=x;smithi130=y;smithi150=z;count:3 2023-10-11T16:24:41.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:41.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:41.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:41.948 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.948 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:41.948 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-11T16:24:41.948 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:41.948 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi111=x;smithi130=y;smithi150=z", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: Saving service mgr spec with placement smithi111=x;smithi130=y;smithi150=z;count:3 2023-10-11T16:24:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:41.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:41.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:41.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:41.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:41.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-11T16:24:41.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:41.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:41.996 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2023-10-11T16:24:41.997 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2023-10-11T16:24:41.997 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000251584 s, 2.0 MB/s 2023-10-11T16:24:41.997 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-11T16:24:42.057 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2023-10-11T16:24:42.115 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-11T16:24:42.115 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:42.116 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 386726 Links: 1 2023-10-11T16:24:42.116 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:42.116 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:42.116 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2023-10-11 16:24:27.965273958 +0000 2023-10-11T16:24:42.116 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2023-10-11 16:15:14.708660569 +0000 2023-10-11T16:24:42.117 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2023-10-11 16:15:14.708660569 +0000 2023-10-11T16:24:42.117 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2023-10-11T16:24:42.117 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-11T16:24:42.183 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2023-10-11T16:24:42.184 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2023-10-11T16:24:42.184 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000215619 s, 2.4 MB/s 2023-10-11T16:24:42.185 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-11T16:24:42.243 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2023-10-11T16:24:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-11T16:24:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:42.302 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 396458 Links: 1 2023-10-11T16:24:42.302 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:42.303 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:42.303 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2023-10-11 16:24:27.965273958 +0000 2023-10-11T16:24:42.303 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2023-10-11 16:15:15.209651400 +0000 2023-10-11T16:24:42.303 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2023-10-11 16:15:15.209651400 +0000 2023-10-11T16:24:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2023-10-11T16:24:42.304 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-11T16:24:42.368 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2023-10-11T16:24:42.368 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2023-10-11T16:24:42.369 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000229624 s, 2.2 MB/s 2023-10-11T16:24:42.370 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-11T16:24:42.429 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2023-10-11T16:24:42.487 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-11T16:24:42.487 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:42.488 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 394908 Links: 1 2023-10-11T16:24:42.488 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:42.488 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:42.488 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2023-10-11 16:24:27.965273958 +0000 2023-10-11T16:24:42.489 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2023-10-11 16:15:15.719642066 +0000 2023-10-11T16:24:42.489 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2023-10-11 16:15:15.719642066 +0000 2023-10-11T16:24:42.489 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2023-10-11T16:24:42.489 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-11T16:24:42.553 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2023-10-11T16:24:42.554 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2023-10-11T16:24:42.554 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000196408 s, 2.6 MB/s 2023-10-11T16:24:42.555 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-11T16:24:42.588 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:42 smithi150 systemd[1]: Starting Ceph mgr.z for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:24:42.613 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-10-11T16:24:42.614 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2023-10-11T16:24:42.635 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-11T16:24:42.635 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2023-10-11T16:24:42.692 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-11T16:24:42.693 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:42.693 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 389765 Links: 1 2023-10-11T16:24:42.693 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:42.693 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:42.693 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-11 16:24:42.407256169 +0000 2023-10-11T16:24:42.694 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-11 16:15:14.201328891 +0000 2023-10-11T16:24:42.694 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-11 16:15:14.201328891 +0000 2023-10-11T16:24:42.694 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-11T16:24:42.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-11T16:24:42.761 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-11T16:24:42.761 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-11T16:24:42.761 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000324771 s, 1.6 MB/s 2023-10-11T16:24:42.762 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-11T16:24:42.820 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2023-10-11T16:24:42.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:42 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:42.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:42 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:24:42.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:42 smithi130 ceph-mon[131859]: Deploying daemon mgr.z on smithi150 2023-10-11T16:24:42.862 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:42 smithi150 podman[134077]: 2023-10-11T16:24:42.862 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:42 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:42.862 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:42 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:24:42.862 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:42 smithi150 ceph-mon[132217]: Deploying daemon mgr.z on smithi150 2023-10-11T16:24:42.878 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-11T16:24:42.878 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:42.878 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 388812 Links: 1 2023-10-11T16:24:42.878 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:42.879 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:42.879 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-11 16:24:42.407256169 +0000 2023-10-11T16:24:42.879 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-11 16:15:14.701319214 +0000 2023-10-11T16:24:42.879 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-11 16:15:14.701319214 +0000 2023-10-11T16:24:42.879 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-11T16:24:42.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-11T16:24:42.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:42 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:42.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:42 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:24:42.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:42 smithi111 ceph-mon[98527]: Deploying daemon mgr.z on smithi150 2023-10-11T16:24:42.944 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-11T16:24:42.944 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-11T16:24:42.945 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000220485 s, 2.3 MB/s 2023-10-11T16:24:42.946 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-11T16:24:43.003 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2023-10-11T16:24:43.061 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-11T16:24:43.062 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:43.062 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 388913 Links: 1 2023-10-11T16:24:43.062 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:43.062 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:43.062 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-11 16:24:42.407256169 +0000 2023-10-11T16:24:43.063 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-11 16:15:15.205309460 +0000 2023-10-11T16:24:43.063 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-11 16:15:15.205309460 +0000 2023-10-11T16:24:43.063 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-11T16:24:43.063 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-11T16:24:43.128 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-11T16:24:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-11T16:24:43.129 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000246878 s, 2.1 MB/s 2023-10-11T16:24:43.130 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-11T16:24:43.188 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2023-10-11T16:24:43.202 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 bash[134077]: 8e80464a50b8a79d7a6230ef56dc8094121eac903ee4793969b8585384439022 2023-10-11T16:24:43.202 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mgr[134250]: -- 172.21.15.150:0/2755793932 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eac0b35520 con 0x55eac0b0ac00 2023-10-11T16:24:43.203 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 systemd[1]: Started Ceph mgr.z for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:24:43.203 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:43.146+0000 7fbd2f873200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T16:24:43.219 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-11T16:24:43.219 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T16:24:43.220 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 392797 Links: 1 2023-10-11T16:24:43.220 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:24:43.220 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T16:24:43.220 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2023-10-11 16:24:42.407256169 +0000 2023-10-11T16:24:43.220 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2023-10-11 16:15:15.708299726 +0000 2023-10-11T16:24:43.221 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2023-10-11 16:15:15.708299726 +0000 2023-10-11T16:24:43.221 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2023-10-11T16:24:43.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-11T16:24:43.285 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2023-10-11T16:24:43.285 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2023-10-11T16:24:43.286 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000243135 s, 2.1 MB/s 2023-10-11T16:24:43.287 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-11T16:24:43.345 INFO:tasks.cephadm:Deploying osd.0 on smithi111 with /dev/vg_nvme/lv_4... 2023-10-11T16:24:43.346 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-11T16:24:43.586 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:43.230+0000 7fbd2f873200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T16:24:43.587 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:43.322+0000 7fbd2f873200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T16:24:43.587 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:43.418+0000 7fbd2f873200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T16:24:43.587 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:43.585+0000 7fbd2f873200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T16:24:43.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:43 smithi130 ceph-mon[131859]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:43.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:43.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-11T16:24:43.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:43.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:43 smithi111 ceph-mon[98527]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:43.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-11T16:24:43.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:43.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:43.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mon[132217]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:43.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.891 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:43.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:43.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-11T16:24:43.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:43.892 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:44.194 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:24:44.202 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:43.890+0000 7fbd2f873200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T16:24:44.202 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:43.979+0000 7fbd2f873200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T16:24:44.202 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:44 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:44.114+0000 7fbd2f873200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T16:24:44.598 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:44 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:44.209+0000 7fbd2f873200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T16:24:44.807 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:44 smithi111 ceph-mon[98527]: Deploying daemon mgr.y on smithi130 2023-10-11T16:24:44.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:44 smithi130 ceph-mon[131859]: Deploying daemon mgr.y on smithi130 2023-10-11T16:24:44.832 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:44 smithi130 systemd[1]: Starting Ceph mgr.y for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:24:44.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:44 smithi150 ceph-mon[132217]: Deploying daemon mgr.y on smithi130 2023-10-11T16:24:44.952 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:44 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:44.642+0000 7fbd2f873200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T16:24:45.264 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 podman[133299]: 2023-10-11T16:24:45.578 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 bash[133299]: ba9e7fe172a635a663cc9b40a596bab037477157dcca493f0533ded43733ee53 2023-10-11T16:24:45.578 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-mgr[133340]: -- 172.21.15.130:0/1685573377 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e66fafb520 con 0x55e66fad0c00 2023-10-11T16:24:45.579 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 systemd[1]: Started Ceph mgr.y for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:24:45.579 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:45.405+0000 7f5f24676200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T16:24:45.579 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:45.484+0000 7f5f24676200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T16:24:45.579 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:45.575+0000 7f5f24676200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T16:24:45.599 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:45 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:45.226+0000 7fbd2f873200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T16:24:45.600 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:45 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:45.397+0000 7fbd2f873200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T16:24:45.831 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:45.671+0000 7f5f24676200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T16:24:45.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-mon[131859]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:45.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:45 smithi111 ceph-mon[98527]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:45 smithi150 ceph-mon[132217]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:46.158 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-11T16:24:46.158 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-11T16:24:46.159 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2023-10-11T16:24:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2023-10-11T16:24:46.159 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0304052 s, 345 MB/s 2023-10-11T16:24:46.159 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2023-10-11T16:24:46.173 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:45.849+0000 7f5f24676200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T16:24:46.173 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:46 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:46.171+0000 7f5f24676200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T16:24:46.452 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:46.197+0000 7fbd2f873200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:24:46.452 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:46.292+0000 7fbd2f873200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T16:24:46.506 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:46 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:46.265+0000 7f5f24676200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T16:24:46.507 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:46 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:46.403+0000 7f5f24676200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T16:24:46.671 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2023-10-11T16:24:46.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:46 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:46.763 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:46 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:46.763 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:46.470+0000 7fbd2f873200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:24:46.764 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:46.761+0000 7fbd2f873200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T16:24:46.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:46 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:46.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:46 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:46.832 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:46 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:46.505+0000 7f5f24676200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T16:24:46.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:46 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:46.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:46 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:47.154 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:47.000+0000 7fbd2f873200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T16:24:47.331 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:46 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:46.961+0000 7f5f24676200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T16:24:47.418 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:47.153+0000 7fbd2f873200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T16:24:47.518 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:24:47.702 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:47.418+0000 7fbd2f873200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T16:24:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:47.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:47.831 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:47.550+0000 7f5f24676200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T16:24:47.832 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:47.721+0000 7f5f24676200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T16:24:47.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:47.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:47.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:47.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:47.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:47.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:47.855 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:47.855 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.856 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.856 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.856 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.856 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:47.857 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:47.857 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:47.857 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-11T16:24:47.857 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-11T16:24:47.857 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:48.436 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:48.119+0000 7fbd2f873200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T16:24:48.436 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:48.247+0000 7fbd2f873200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T16:24:48.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:48.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:24:48.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:48.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-mon[132217]: Reconfiguring mgr.x (unknown last config time)... 2023-10-11T16:24:48.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-mon[132217]: Reconfiguring daemon mgr.x on smithi111 2023-10-11T16:24:48.703 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:48.435+0000 7fbd2f873200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T16:24:48.704 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:48.550+0000 7fbd2f873200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T16:24:48.743 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:48 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:48.480+0000 7f5f24676200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:24:48.743 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:48 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:48.571+0000 7f5f24676200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T16:24:48.743 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:48 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:48.741+0000 7f5f24676200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:24:48.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:48 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:48.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:48 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:24:48.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:48 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:48.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:48 smithi130 ceph-mon[131859]: Reconfiguring mgr.x (unknown last config time)... 2023-10-11T16:24:48.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:48 smithi130 ceph-mon[131859]: Reconfiguring daemon mgr.x on smithi111 2023-10-11T16:24:48.808 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:48 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:48.808 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:48 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:24:48.808 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:48 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:48.808 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:48 smithi111 ceph-mon[98527]: Reconfiguring mgr.x (unknown last config time)... 2023-10-11T16:24:48.809 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:48 smithi111 ceph-mon[98527]: Reconfiguring daemon mgr.x on smithi111 2023-10-11T16:24:49.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.080+0000 7f95dc73f700 1 -- 172.21.15.111:0/2375619064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95d41002f0 msgr2=0x7f95d4100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:49.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.080+0000 7f95dc73f700 1 --2- 172.21.15.111:0/2375619064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95d41002f0 0x7f95d4100700 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f95d00098f0 tx=0x7f95d0009c00 comp rx=0 tx=0).stop 2023-10-11T16:24:49.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.080+0000 7f95dc73f700 1 -- 172.21.15.111:0/2375619064 shutdown_connections 2023-10-11T16:24:49.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.080+0000 7f95dc73f700 1 --2- 172.21.15.111:0/2375619064 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95d4101e80 0x7f95d4108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.080+0000 7f95dc73f700 1 --2- 172.21.15.111:0/2375619064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95d41002f0 0x7f95d4100700 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.080+0000 7f95dc73f700 1 --2- 172.21.15.111:0/2375619064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d41014f0 0x7f95d4101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.080+0000 7f95dc73f700 1 -- 172.21.15.111:0/2375619064 >> 172.21.15.111:0/2375619064 conn(0x7f95d40fb900 msgr2=0x7f95d40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:24:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.080+0000 7f95dc73f700 1 -- 172.21.15.111:0/2375619064 shutdown_connections 2023-10-11T16:24:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.081+0000 7f95dc73f700 1 -- 172.21.15.111:0/2375619064 wait complete. 2023-10-11T16:24:49.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.082+0000 7f95dc73f700 1 Processor -- start 2023-10-11T16:24:49.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.082+0000 7f95dc73f700 1 -- start start 2023-10-11T16:24:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95dc73f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d41002f0 0x7f95d4074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95dc73f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95d41014f0 0x7f95d4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95dc73f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95d4101e80 0x7f95d4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95dc73f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4073ba0 con 0x7f95d41014f0 2023-10-11T16:24:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95dc73f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4073d10 con 0x7f95d4101e80 2023-10-11T16:24:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95dc73f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d41a0160 con 0x7f95d41002f0 2023-10-11T16:24:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95da4db700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d41002f0 0x7f95d4074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95da4db700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d41002f0 0x7f95d4074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:45414/0 (socket says 172.21.15.111:45414) 2023-10-11T16:24:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95da4db700 1 -- 172.21.15.111:0/1573571498 learned_addr learned my addr 172.21.15.111:0/1573571498 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:24:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95dacdc700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95d4101e80 0x7f95d4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:49.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95d9cda700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95d41014f0 0x7f95d4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:49.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95da4db700 1 -- 172.21.15.111:0/1573571498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95d4101e80 msgr2=0x7f95d4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:49.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95da4db700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95d4101e80 0x7f95d4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:49.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95da4db700 1 -- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95d41014f0 msgr2=0x7f95d4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:24:49.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95da4db700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95d41014f0 0x7f95d4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:24:49.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.083+0000 7f95da4db700 1 -- 172.21.15.111:0/1573571498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d0009520 con 0x7f95d41002f0 2023-10-11T16:24:49.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.084+0000 7f95da4db700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d41002f0 0x7f95d4074890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009fd0 tx=0x7f95d000b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:49.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.084+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d001c070 con 0x7f95d41002f0 2023-10-11T16:24:49.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.084+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95d000f7d0 con 0x7f95d41002f0 2023-10-11T16:24:49.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.084+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d410bcb0 con 0x7f95d41002f0 2023-10-11T16:24:49.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.084+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0003a40 con 0x7f95d41002f0 2023-10-11T16:24:49.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.084+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d410bb20 con 0x7f95d41002f0 2023-10-11T16:24:49.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.085+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 13) v1 ==== 45540+0+0 (secure 0 0 0) 0x7f95d0020030 con 0x7f95d41002f0 2023-10-11T16:24:49.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.085+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d4107980 con 0x7f95d41002f0 2023-10-11T16:24:49.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.085+0000 7f95cb7fe700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95c00389e0 0x7f95c003ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:24:49.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.085+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f95d004c340 con 0x7f95d41002f0 2023-10-11T16:24:49.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.086+0000 7f95d9cda700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95c00389e0 0x7f95c003ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:24:49.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.086+0000 7f95d9cda700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95c00389e0 0x7f95c003ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009c60 tx=0x7f95c4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:24:49.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.092+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f95d000e9b0 con 0x7f95d41002f0 2023-10-11T16:24:49.202 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:48.766+0000 7fbd2f873200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T16:24:49.332 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:49.016+0000 7f5f24676200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T16:24:49.332 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:49.242+0000 7f5f24676200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T16:24:49.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:49.336+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f95d4061c00 con 0x7f95c00389e0 2023-10-11T16:24:49.654 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:49.388+0000 7f5f24676200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T16:24:49.655 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:49.655 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:49.655 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:49.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:24:49.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:24:49.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:49.656 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:49.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:49.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:49.657 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:49.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:24:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:24:49.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:49.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:49.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:49.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:49.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:49.952 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:24:49.636+0000 7fbd2f873200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T16:24:49.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:49.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:49.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:49.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:24:49.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:24:49.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:49.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:24:49.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:24:49.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:24:49.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:50.081 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:49 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:49.652+0000 7f5f24676200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T16:24:50.623 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:50.349+0000 7f5f24676200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T16:24:50.624 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:50.479+0000 7f5f24676200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T16:24:50.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:50.626+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 14) v1 ==== 90600+0+0 (secure 0 0 0) 0x7f95d002c440 con 0x7f95d41002f0 2023-10-11T16:24:50.791 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:50.792 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:50.792 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:24:50.792 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:24:50.792 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: Standby manager daemon z started 2023-10-11T16:24:50.792 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-11T16:24:50.793 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:24:50.793 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-11T16:24:50.793 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:24:50.793 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:50.793 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:50.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:24:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:24:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: Standby manager daemon z started 2023-10-11T16:24:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-11T16:24:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:24:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-11T16:24:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:24:50.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:50.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:50.997 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:50.668+0000 7f5f24676200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T16:24:50.997 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:50.781+0000 7f5f24676200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T16:24:50.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:24:50.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:50.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:24:50.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:24:50.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: Standby manager daemon z started 2023-10-11T16:24:50.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-11T16:24:50.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:24:50.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-11T16:24:50.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.150:0/3345559754' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:24:50.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:51.000 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:24:51.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:51 smithi111 ceph-mon[98527]: mgrmap e14: x(active, since 70s), standbys: z 2023-10-11T16:24:51.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-11T16:24:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:51 smithi150 ceph-mon[132217]: mgrmap e14: x(active, since 70s), standbys: z 2023-10-11T16:24:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-11T16:24:52.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:51 smithi130 ceph-mon[131859]: mgrmap e14: x(active, since 70s), standbys: z 2023-10-11T16:24:52.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-11T16:24:52.082 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:50 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:50.995+0000 7f5f24676200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T16:24:52.082 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:24:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:24:51.848+0000 7f5f24676200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T16:24:52.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:24:52.636+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mgrmap(e 15) v1 ==== 135660+0+0 (secure 0 0 0) 0x7f95d000ebe0 con 0x7f95d41002f0 2023-10-11T16:24:52.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: Standby manager daemon y started 2023-10-11T16:24:52.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-11T16:24:52.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:24:52.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-11T16:24:52.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:24:52.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:52.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2312668454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4f58cc5-3bfa-4fdb-a127-1926aadd513d"}]: dispatch 2023-10-11T16:24:52.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2312668454' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4f58cc5-3bfa-4fdb-a127-1926aadd513d"}]': finished 2023-10-11T16:24:52.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: osdmap e5: 1 total, 0 up, 1 in 2023-10-11T16:24:52.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:52 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:24:52.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: Standby manager daemon y started 2023-10-11T16:24:52.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-11T16:24:52.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:24:52.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-11T16:24:52.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:24:52.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:52.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2312668454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4f58cc5-3bfa-4fdb-a127-1926aadd513d"}]: dispatch 2023-10-11T16:24:52.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2312668454' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4f58cc5-3bfa-4fdb-a127-1926aadd513d"}]': finished 2023-10-11T16:24:52.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: osdmap e5: 1 total, 0 up, 1 in 2023-10-11T16:24:52.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:52 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:24:53.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: Standby manager daemon y started 2023-10-11T16:24:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-11T16:24:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:24:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-11T16:24:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.130:0/2266458519' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:24:53.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:53.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2312668454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4f58cc5-3bfa-4fdb-a127-1926aadd513d"}]: dispatch 2023-10-11T16:24:53.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2312668454' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4f58cc5-3bfa-4fdb-a127-1926aadd513d"}]': finished 2023-10-11T16:24:53.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: osdmap e5: 1 total, 0 up, 1 in 2023-10-11T16:24:53.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:52 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:24:53.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:53 smithi111 ceph-mon[98527]: mgrmap e15: x(active, since 72s), standbys: z, y 2023-10-11T16:24:53.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-11T16:24:53.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:53 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1041816578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:24:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:53 smithi150 ceph-mon[132217]: mgrmap e15: x(active, since 72s), standbys: z, y 2023-10-11T16:24:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-11T16:24:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:53 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1041816578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:24:54.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:53 smithi130 ceph-mon[131859]: mgrmap e15: x(active, since 72s), standbys: z, y 2023-10-11T16:24:54.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-11T16:24:54.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:53 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1041816578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:24:55.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:55 smithi130 ceph-mon[131859]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:55.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:55 smithi111 ceph-mon[98527]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:55 smithi150 ceph-mon[132217]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:57.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:57 smithi111 ceph-mon[98527]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:57 smithi150 ceph-mon[132217]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:58.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:57 smithi130 ceph-mon[131859]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:59.878 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:24:59 smithi111 ceph-mon[98527]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:24:59.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:24:59 smithi150 ceph-mon[132217]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:00.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:24:59 smithi130 ceph-mon[131859]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:00.855 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-11T16:25:00.855 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:00.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:00 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-11T16:25:00.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:00 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:01.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:00 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-11T16:25:01.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:00 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:01.846 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:01 smithi111 ceph-mon[98527]: Deploying daemon osd.0 on smithi111 2023-10-11T16:25:01.847 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:01 smithi111 ceph-mon[98527]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:01 smithi150 ceph-mon[132217]: Deploying daemon osd.0 on smithi111 2023-10-11T16:25:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:01 smithi150 ceph-mon[132217]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:02.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:01 smithi130 ceph-mon[131859]: Deploying daemon osd.0 on smithi111 2023-10-11T16:25:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:01 smithi130 ceph-mon[131859]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:03.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:03 smithi111 ceph-mon[98527]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:03 smithi150 ceph-mon[132217]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:04.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:03 smithi130 ceph-mon[131859]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:05 smithi111 ceph-mon[98527]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:05.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:05 smithi150 ceph-mon[132217]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:06.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:05 smithi130 ceph-mon[131859]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:07.319 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:07 smithi111 ceph-mon[98527]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:07.320 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:07 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:07.320 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:07 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:07.320 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:07 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:07.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:07 smithi130 ceph-mon[131859]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:07.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:07 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:07.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:07 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:07.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:07 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:07 smithi150 ceph-mon[132217]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:07 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:07.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:07 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:07.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:07 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:09.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.648+0000 7f95cb7fe700 1 -- 172.21.15.111:0/1573571498 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f95d4061c00 con 0x7f95c00389e0 2023-10-11T16:25:09.650 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 0 on host 'smithi111' 2023-10-11T16:25:09.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.650+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95c00389e0 msgr2=0x7f95c003ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:09.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.650+0000 7f95dc73f700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95c00389e0 0x7f95c003ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009c60 tx=0x7f95c4009260 comp rx=0 tx=0).stop 2023-10-11T16:25:09.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.650+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d41002f0 msgr2=0x7f95d4074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:09.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.650+0000 7f95dc73f700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d41002f0 0x7f95d4074890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f95d0009fd0 tx=0x7f95d000b790 comp rx=0 tx=0).stop 2023-10-11T16:25:09.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.651+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 shutdown_connections 2023-10-11T16:25:09.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.651+0000 7f95dc73f700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95c00389e0 0x7f95c003ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:09.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.651+0000 7f95dc73f700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95d4101e80 0x7f95d4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:09.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.651+0000 7f95dc73f700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95d41002f0 0x7f95d4074890 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:09.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.651+0000 7f95dc73f700 1 --2- 172.21.15.111:0/1573571498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95d41014f0 0x7f95d4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:09.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.651+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 >> 172.21.15.111:0/1573571498 conn(0x7f95d40fb900 msgr2=0x7f95d4104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:25:09.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.651+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 shutdown_connections 2023-10-11T16:25:09.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:09.651+0000 7f95dc73f700 1 -- 172.21.15.111:0/1573571498 wait complete. 2023-10-11T16:25:09.811 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:09.811 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.811 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.812 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:09.812 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:09.812 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:09.812 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.813 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:25:09.813 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:09 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:09.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:09.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:09.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:09.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:09.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:25:09.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:09 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:09.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:09.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:09.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:09.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:25:09.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:09 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:10.136 DEBUG:teuthology.orchestra.run.smithi111:osd.0> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.0.service 2023-10-11T16:25:10.141 INFO:tasks.cephadm:Deploying osd.1 on smithi130 with /dev/vg_nvme/lv_4... 2023-10-11T16:25:10.142 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-11T16:25:10.211 INFO:journalctl@ceph.osd.0.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:25:10.788 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:25:10.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:10.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:10 smithi111 ceph-mon[98527]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:10 smithi111 ceph-mon[98527]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-11T16:25:10.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:10.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:10.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:10.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:10 smithi150 ceph-mon[132217]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:10.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:10 smithi150 ceph-mon[132217]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-11T16:25:11.010 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:11.010 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:11.010 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:11.010 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:10 smithi130 ceph-mon[131859]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:11.011 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:10 smithi130 ceph-mon[131859]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-11T16:25:11.868 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:11 smithi111 ceph-mon[98527]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-11T16:25:11.869 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:11 smithi111 ceph-mon[98527]: osdmap e6: 1 total, 0 up, 1 in 2023-10-11T16:25:11.869 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:11.869 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:11 smithi111 ceph-mon[98527]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:25:11.870 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:11 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-0[110751]: 2023-10-11T16:25:11.638+0000 7fd574bf1700 -1 osd.0 0 waiting for initial osdmap 2023-10-11T16:25:11.870 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:11 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-0[110751]: 2023-10-11T16:25:11.643+0000 7fd56e9e2700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:25:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:11 smithi150 ceph-mon[132217]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-11T16:25:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:11 smithi150 ceph-mon[132217]: osdmap e6: 1 total, 0 up, 1 in 2023-10-11T16:25:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:11 smithi150 ceph-mon[132217]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:25:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:11 smithi130 ceph-mon[131859]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-11T16:25:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:11 smithi130 ceph-mon[131859]: osdmap e6: 1 total, 0 up, 1 in 2023-10-11T16:25:12.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:12.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:11 smithi130 ceph-mon[131859]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:25:12.266 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-11T16:25:12.266 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-11T16:25:12.266 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2023-10-11T16:25:12.266 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2023-10-11T16:25:12.267 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108098 s, 970 MB/s 2023-10-11T16:25:12.267 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2023-10-11T16:25:12.640 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi130:vg_nvme/lv_4 2023-10-11T16:25:12.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:12 smithi111 ceph-mon[98527]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:25:12.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:12 smithi111 ceph-mon[98527]: osdmap e7: 1 total, 0 up, 1 in 2023-10-11T16:25:12.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:12 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:12.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:12 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:12.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:12 smithi111 ceph-mon[98527]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:12.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:12 smithi150 ceph-mon[132217]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:25:12.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:12 smithi150 ceph-mon[132217]: osdmap e7: 1 total, 0 up, 1 in 2023-10-11T16:25:12.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:12 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:12.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:12 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:12.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:12 smithi150 ceph-mon[132217]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:12 smithi130 ceph-mon[131859]: from='osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:25:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:12 smithi130 ceph-mon[131859]: osdmap e7: 1 total, 0 up, 1 in 2023-10-11T16:25:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:12 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:13.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:12 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:13.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:12 smithi130 ceph-mon[131859]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-11T16:25:13.277 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:25:13.795 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:25:13.795 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:25:13.795 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:13.796 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] boot 2023-10-11T16:25:13.796 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: osdmap e8: 1 total, 1 up, 1 in 2023-10-11T16:25:13.796 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:13.796 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi111 2023-10-11T16:25:13.797 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:13.797 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:13.797 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:13.797 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi111 to 17271M 2023-10-11T16:25:13.797 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:13.798 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:13.798 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:13.798 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:13.798 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:13.798 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:25:13.799 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:13 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:13.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:25:13.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:25:13.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:13.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] boot 2023-10-11T16:25:13.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: osdmap e8: 1 total, 1 up, 1 in 2023-10-11T16:25:13.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:13.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi111 2023-10-11T16:25:13.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:13.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:13.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:13.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi111 to 17271M 2023-10-11T16:25:13.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:13.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:13.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:13.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:13.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:13.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:25:13.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:13 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:14.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:25:14.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:25:14.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:14.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: osd.0 [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] boot 2023-10-11T16:25:14.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: osdmap e8: 1 total, 1 up, 1 in 2023-10-11T16:25:14.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:25:14.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi111 2023-10-11T16:25:14.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:14.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:14.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:14.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi111 to 17271M 2023-10-11T16:25:14.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:14.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:14.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:14.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:14.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:14.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:25:14.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:13 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:14.497 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.494+0000 7f157fb4f700 1 -- 172.21.15.130:0/2554181001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 msgr2=0x7f1578107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:14.498 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.494+0000 7f157fb4f700 1 --2- 172.21.15.130:0/2554181001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 0x7f1578107630 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f157000b0a0 tx=0x7f157000b3b0 comp rx=0 tx=0).stop 2023-10-11T16:25:14.498 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.494+0000 7f157fb4f700 1 -- 172.21.15.130:0/2554181001 shutdown_connections 2023-10-11T16:25:14.498 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.494+0000 7f157fb4f700 1 --2- 172.21.15.130:0/2554181001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 0x7f1578107630 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:14.498 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.494+0000 7f157fb4f700 1 --2- 172.21.15.130:0/2554181001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15780ff1e0 0x7f15780ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:14.498 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.494+0000 7f157fb4f700 1 --2- 172.21.15.130:0/2554181001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15781003e0 0x7f1578100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:14.499 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.494+0000 7f157fb4f700 1 -- 172.21.15.130:0/2554181001 >> 172.21.15.130:0/2554181001 conn(0x7f15780fa7f0 msgr2=0x7f15780fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:25:14.499 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.494+0000 7f157fb4f700 1 -- 172.21.15.130:0/2554181001 shutdown_connections 2023-10-11T16:25:14.499 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 -- 172.21.15.130:0/2554181001 wait complete. 2023-10-11T16:25:14.499 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 Processor -- start 2023-10-11T16:25:14.499 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 -- start start 2023-10-11T16:25:14.500 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15780ff1e0 0x7f1578069130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:14.500 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15781003e0 0x7f157819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:14.500 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 0x7f157819d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:14.500 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819de90 con 0x7f15781003e0 2023-10-11T16:25:14.500 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819e000 con 0x7f1578100d70 2023-10-11T16:25:14.501 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157fb4f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819e1c0 con 0x7f15780ff1e0 2023-10-11T16:25:14.501 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157e0ec700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 0x7f157819d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:14.501 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157e0ec700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 0x7f157819d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:43362/0 (socket says 172.21.15.130:43362) 2023-10-11T16:25:14.501 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.495+0000 7f157e0ec700 1 -- 172.21.15.130:0/3061767638 learned_addr learned my addr 172.21.15.130:0/3061767638 (peer_addr_for_me v2:172.21.15.130:0/0) 2023-10-11T16:25:14.501 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157d8eb700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15780ff1e0 0x7f1578069130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:14.502 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157e0ec700 1 -- 172.21.15.130:0/3061767638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15780ff1e0 msgr2=0x7f1578069130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:14.502 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157e0ec700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15780ff1e0 0x7f1578069130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:14.502 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157e0ec700 1 -- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15781003e0 msgr2=0x7f157819b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:14.502 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157e0ec700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15781003e0 0x7f157819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:14.502 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157e0ec700 1 -- 172.21.15.130:0/3061767638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570009d00 con 0x7f1578100d70 2023-10-11T16:25:14.503 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157d8eb700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15780ff1e0 0x7f1578069130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:25:14.503 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157e0ec700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 0x7f157819d6f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f157000b070 tx=0x7f1570004b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:25:14.503 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f156effd700 1 -- 172.21.15.130:0/3061767638 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157000e040 con 0x7f1578100d70 2023-10-11T16:25:14.503 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f156effd700 1 -- 172.21.15.130:0/3061767638 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f157001db40 con 0x7f1578100d70 2023-10-11T16:25:14.503 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157810aca0 con 0x7f1578100d70 2023-10-11T16:25:14.504 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f156effd700 1 -- 172.21.15.130:0/3061767638 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157001bcc0 con 0x7f1578100d70 2023-10-11T16:25:14.504 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.496+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157810ab10 con 0x7f1578100d70 2023-10-11T16:25:14.505 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.497+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1578106870 con 0x7f1578100d70 2023-10-11T16:25:14.505 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.498+0000 7f156effd700 1 -- 172.21.15.130:0/3061767638 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 15) v1 ==== 135660+0+0 (secure 0 0 0) 0x7f157001dcb0 con 0x7f1578100d70 2023-10-11T16:25:14.505 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.499+0000 7f156effd700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15640a10c0 0x7f15640a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:14.505 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.499+0000 7f156effd700 1 -- 172.21.15.130:0/3061767638 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f15700cd040 con 0x7f1578100d70 2023-10-11T16:25:14.506 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.499+0000 7f157d8eb700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15640a10c0 0x7f15640a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:14.506 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.500+0000 7f157d8eb700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15640a10c0 0x7f15640a3570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1578100240 tx=0x7f1568008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:25:14.506 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.503+0000 7f156effd700 1 -- 172.21.15.130:0/3061767638 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1570098290 con 0x7f1578100d70 2023-10-11T16:25:14.699 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:14.698+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1578061ae0 con 0x7f15640a10c0 2023-10-11T16:25:14.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:14 smithi150 ceph-mon[132217]: osdmap e9: 1 total, 1 up, 1 in 2023-10-11T16:25:14.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:14 smithi150 ceph-mon[132217]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:15.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:14 smithi130 ceph-mon[131859]: osdmap e9: 1 total, 1 up, 1 in 2023-10-11T16:25:15.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:14 smithi130 ceph-mon[131859]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:15.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:14 smithi111 ceph-mon[98527]: osdmap e9: 1 total, 1 up, 1 in 2023-10-11T16:25:15.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:14 smithi111 ceph-mon[98527]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:15.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:15 smithi130 ceph-mon[131859]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:15.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:15 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:15.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:15 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:15.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:15 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:15.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:15 smithi150 ceph-mon[132217]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:15.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:15 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:15 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:15 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:16.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:15 smithi111 ceph-mon[98527]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:16.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:15 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:16.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:15 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:16.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:15 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:16.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:16 smithi150 ceph-mon[132217]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:17.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:16 smithi130 ceph-mon[131859]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:17.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:16 smithi111 ceph-mon[98527]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:17.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:17 smithi150 ceph-mon[132217]: from='client.? 172.21.15.130:0/3748438149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]: dispatch 2023-10-11T16:25:17.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:17 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]: dispatch 2023-10-11T16:25:17.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:17 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]': finished 2023-10-11T16:25:17.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:17 smithi150 ceph-mon[132217]: osdmap e10: 2 total, 1 up, 2 in 2023-10-11T16:25:17.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:17 smithi130 ceph-mon[131859]: from='client.? 172.21.15.130:0/3748438149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]: dispatch 2023-10-11T16:25:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:17 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]: dispatch 2023-10-11T16:25:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:17 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]': finished 2023-10-11T16:25:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:17 smithi130 ceph-mon[131859]: osdmap e10: 2 total, 1 up, 2 in 2023-10-11T16:25:18.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:18.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:17 smithi111 ceph-mon[98527]: from='client.? 172.21.15.130:0/3748438149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]: dispatch 2023-10-11T16:25:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:17 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]: dispatch 2023-10-11T16:25:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:17 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c29c935-2362-4f82-a918-4fb7c7ce51f8"}]': finished 2023-10-11T16:25:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:17 smithi111 ceph-mon[98527]: osdmap e10: 2 total, 1 up, 2 in 2023-10-11T16:25:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:18.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:18 smithi150 ceph-mon[132217]: from='client.? 172.21.15.130:0/2513597583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:25:18.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:18 smithi150 ceph-mon[132217]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:19.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:18 smithi130 ceph-mon[131859]: from='client.? 172.21.15.130:0/2513597583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:25:19.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:18 smithi130 ceph-mon[131859]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:19.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:18 smithi111 ceph-mon[98527]: from='client.? 172.21.15.130:0/2513597583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:25:19.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:18 smithi111 ceph-mon[98527]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:21.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:21 smithi111 ceph-mon[98527]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:21.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:21 smithi150 ceph-mon[132217]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:22.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:21 smithi130 ceph-mon[131859]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:23.713 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:23 smithi130 ceph-mon[131859]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:23.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:23 smithi111 ceph-mon[98527]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:23 smithi150 ceph-mon[132217]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:24.734 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:24 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-11T16:25:24.735 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:24 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:24.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:24 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-11T16:25:24.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:24 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:24.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:24 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-11T16:25:24.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:24 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:25.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:25 smithi111 ceph-mon[98527]: Deploying daemon osd.1 on smithi130 2023-10-11T16:25:25.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:25 smithi111 ceph-mon[98527]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:25.918 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:25 smithi130 ceph-mon[131859]: Deploying daemon osd.1 on smithi130 2023-10-11T16:25:25.918 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:25 smithi130 ceph-mon[131859]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:25 smithi150 ceph-mon[132217]: Deploying daemon osd.1 on smithi130 2023-10-11T16:25:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:25 smithi150 ceph-mon[132217]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:27.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:27 smithi130 ceph-mon[131859]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:27.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:27 smithi111 ceph-mon[98527]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:27.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:27 smithi150 ceph-mon[132217]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:29.042 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:29.043 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:29.043 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:29.043 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:28 smithi130 ceph-mon[131859]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:29.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:28 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:29.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:28 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:29.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:28 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:29.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:28 smithi150 ceph-mon[132217]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:29.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:28 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:29.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:28 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:29.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:28 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:29.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:28 smithi111 ceph-mon[98527]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:30.764 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.763+0000 7f156effd700 1 -- 172.21.15.130:0/3061767638 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1578061ae0 con 0x7f15640a10c0 2023-10-11T16:25:30.764 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 1 on host 'smithi130' 2023-10-11T16:25:30.766 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15640a10c0 msgr2=0x7f15640a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:30.767 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15640a10c0 0x7f15640a3570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1578100240 tx=0x7f1568008040 comp rx=0 tx=0).stop 2023-10-11T16:25:30.767 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 msgr2=0x7f157819d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:30.767 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 0x7f157819d6f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f157000b070 tx=0x7f1570004b00 comp rx=0 tx=0).stop 2023-10-11T16:25:30.767 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 shutdown_connections 2023-10-11T16:25:30.767 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15640a10c0 0x7f15640a3570 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:30.768 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1578100d70 0x7f157819d6f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:30.768 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15780ff1e0 0x7f1578069130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:30.768 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 --2- 172.21.15.130:0/3061767638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15781003e0 0x7f157819b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:30.768 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 >> 172.21.15.130:0/3061767638 conn(0x7f15780fa7f0 msgr2=0x7f15781039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:25:30.768 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 shutdown_connections 2023-10-11T16:25:30.769 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:25:30.765+0000 7f157fb4f700 1 -- 172.21.15.130:0/3061767638 wait complete. 2023-10-11T16:25:31.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:31.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:31.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:31.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:25:31.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:31.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:31.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:31.170 DEBUG:teuthology.orchestra.run.smithi130:osd.1> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.1.service 2023-10-11T16:25:31.173 INFO:tasks.cephadm:Deploying osd.2 on smithi150 with /dev/vg_nvme/lv_4... 2023-10-11T16:25:31.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-11T16:25:31.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:31.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:31.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:31.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:25:31.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:31.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:31.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:31.205 INFO:journalctl@ceph.osd.1.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:25:31.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:31.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:31.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:31.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:25:31.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:31.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:31.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:31.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:31.818 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:25:32.191 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:32 smithi130 ceph-mon[131859]: from='osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-11T16:25:32.192 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:32 smithi130 ceph-mon[131859]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-11T16:25:32.192 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-1[136889]: 2023-10-11T16:25:31.957+0000 7f1fa9aef640 -1 osd.1 0 log_to_monitors true 2023-10-11T16:25:32.314 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:32 smithi150 ceph-mon[132217]: from='osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-11T16:25:32.315 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:32 smithi150 ceph-mon[132217]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-11T16:25:32.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:32 smithi111 ceph-mon[98527]: from='osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-11T16:25:32.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:32 smithi111 ceph-mon[98527]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-11T16:25:33.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:33 smithi150 ceph-mon[132217]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:33.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:33 smithi150 ceph-mon[132217]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-11T16:25:33.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:33 smithi150 ceph-mon[132217]: osdmap e11: 2 total, 1 up, 2 in 2023-10-11T16:25:33.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:33.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:33 smithi150 ceph-mon[132217]: from='osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:25:33.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:33 smithi150 ceph-mon[132217]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:25:33.245 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:33 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-1[136889]: 2023-10-11T16:25:33.029+0000 7f1f9f166700 -1 osd.1 0 waiting for initial osdmap 2023-10-11T16:25:33.245 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:33 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-1[136889]: 2023-10-11T16:25:33.033+0000 7f1f9af5b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:25:33.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:33 smithi130 ceph-mon[131859]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:33.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:33 smithi130 ceph-mon[131859]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-11T16:25:33.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:33 smithi130 ceph-mon[131859]: osdmap e11: 2 total, 1 up, 2 in 2023-10-11T16:25:33.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:33 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:33.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:33 smithi130 ceph-mon[131859]: from='osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:25:33.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:33 smithi130 ceph-mon[131859]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:25:33.287 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-11T16:25:33.287 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-11T16:25:33.287 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-10-11T16:25:33.287 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-10-11T16:25:33.287 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011011 s, 952 MB/s 2023-10-11T16:25:33.288 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 2023-10-11T16:25:33.288 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-10-11T16:25:33.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:33 smithi111 ceph-mon[98527]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:33.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:33 smithi111 ceph-mon[98527]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-11T16:25:33.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:33 smithi111 ceph-mon[98527]: osdmap e11: 2 total, 1 up, 2 in 2023-10-11T16:25:33.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:33.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:33 smithi111 ceph-mon[98527]: from='osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:25:33.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:33 smithi111 ceph-mon[98527]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:25:33.770 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2023-10-11T16:25:34.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:25:34.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: osdmap e12: 2 total, 1 up, 2 in 2023-10-11T16:25:34.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:34.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:34.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi130 2023-10-11T16:25:34.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:34.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi130 to 17168M 2023-10-11T16:25:34.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:34.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:34.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:34.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:25:34.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:34 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:34.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:25:34.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: osdmap e12: 2 total, 1 up, 2 in 2023-10-11T16:25:34.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:34.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:34.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi130 2023-10-11T16:25:34.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:34.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi130 to 17168M 2023-10-11T16:25:34.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:34.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:34.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:34.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:25:34.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:34 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:34.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:25:34.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: osdmap e12: 2 total, 1 up, 2 in 2023-10-11T16:25:34.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:34.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:34.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi130 2023-10-11T16:25:34.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:34.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi130 to 17168M 2023-10-11T16:25:34.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:34.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:34.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:34.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:34.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:25:34.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:34 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:34.468 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:25:35.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:35 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:25:35.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:35 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:25:35.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:35 smithi150 ceph-mon[132217]: pgmap v59: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:35.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:35 smithi150 ceph-mon[132217]: osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] boot 2023-10-11T16:25:35.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:35 smithi150 ceph-mon[132217]: osdmap e13: 2 total, 2 up, 2 in 2023-10-11T16:25:35.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:35 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:35.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:35 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:25:35.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:35 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:25:35.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:35 smithi130 ceph-mon[131859]: pgmap v59: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:35.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:35 smithi130 ceph-mon[131859]: osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] boot 2023-10-11T16:25:35.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:35 smithi130 ceph-mon[131859]: osdmap e13: 2 total, 2 up, 2 in 2023-10-11T16:25:35.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:35.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:35 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:25:35.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:35 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:25:35.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:35 smithi111 ceph-mon[98527]: pgmap v59: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-11T16:25:35.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:35 smithi111 ceph-mon[98527]: osd.1 [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] boot 2023-10-11T16:25:35.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:35 smithi111 ceph-mon[98527]: osdmap e13: 2 total, 2 up, 2 in 2023-10-11T16:25:35.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:35 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:25:35.761 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 -- 172.21.15.150:0/269454483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594105fc0 msgr2=0x7fa594108420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:35.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/269454483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594105fc0 0x7fa594108420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa58c00b0a0 tx=0x7fa58c00b3b0 comp rx=0 tx=0).stop 2023-10-11T16:25:35.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 -- 172.21.15.150:0/269454483 shutdown_connections 2023-10-11T16:25:35.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/269454483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594105fc0 0x7fa594108420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:35.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/269454483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa594100d80 0x7fa594103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:35.762 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/269454483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5941036a0 0x7fa594105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:35.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 -- 172.21.15.150:0/269454483 >> 172.21.15.150:0/269454483 conn(0x7fa5940fa820 msgr2=0x7fa5940fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:25:35.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 -- 172.21.15.150:0/269454483 shutdown_connections 2023-10-11T16:25:35.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.758+0000 7fa59ac0f700 1 -- 172.21.15.150:0/269454483 wait complete. 2023-10-11T16:25:35.763 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa59ac0f700 1 Processor -- start 2023-10-11T16:25:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa59ac0f700 1 -- start start 2023-10-11T16:25:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa59ac0f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594100d80 0x7fa594196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa59ac0f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5941036a0 0x7fa594196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa59ac0f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa594105fc0 0x7fa59419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa59ac0f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59419dbd0 con 0x7fa594105fc0 2023-10-11T16:25:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa59ac0f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59419dd10 con 0x7fa5941036a0 2023-10-11T16:25:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa59ac0f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59419dea0 con 0x7fa594100d80 2023-10-11T16:25:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa5989ab700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594100d80 0x7fa594196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa5989ab700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594100d80 0x7fa594196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42392/0 (socket says 172.21.15.150:42392) 2023-10-11T16:25:35.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.759+0000 7fa5989ab700 1 -- 172.21.15.150:0/3529739854 learned_addr learned my addr 172.21.15.150:0/3529739854 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:25:35.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa593fff700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5941036a0 0x7fa594196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:35.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa5989ab700 1 -- 172.21.15.150:0/3529739854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5941036a0 msgr2=0x7fa594196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:35.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa5989ab700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5941036a0 0x7fa594196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:35.766 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa5989ab700 1 -- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa594105fc0 msgr2=0x7fa59419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:35.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa5991ac700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa594105fc0 0x7fa59419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:35.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa5989ab700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa594105fc0 0x7fa59419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:35.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa5989ab700 1 -- 172.21.15.150:0/3529739854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c009d00 con 0x7fa594100d80 2023-10-11T16:25:35.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa593fff700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5941036a0 0x7fa594196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:25:35.767 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa5991ac700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa594105fc0 0x7fa59419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:25:35.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa5989ab700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594100d80 0x7fa594196040 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa58800c950 tx=0x7fa58800cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:25:35.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa591ffb700 1 -- 172.21.15.150:0/3529739854 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588007ac0 con 0x7fa594100d80 2023-10-11T16:25:35.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59410bcc0 con 0x7fa594100d80 2023-10-11T16:25:35.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa591ffb700 1 -- 172.21.15.150:0/3529739854 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa588004cf0 con 0x7fa594100d80 2023-10-11T16:25:35.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa591ffb700 1 -- 172.21.15.150:0/3529739854 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588005580 con 0x7fa594100d80 2023-10-11T16:25:35.768 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.760+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59410bb30 con 0x7fa594100d80 2023-10-11T16:25:35.769 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.762+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5940fc390 con 0x7fa594100d80 2023-10-11T16:25:35.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.763+0000 7fa591ffb700 1 -- 172.21.15.150:0/3529739854 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 15) v1 ==== 135660+0+0 (secure 0 0 0) 0x7fa588007c20 con 0x7fa594100d80 2023-10-11T16:25:35.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.763+0000 7fa591ffb700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5840a10c0 0x7fa5840a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:35.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.763+0000 7fa591ffb700 1 -- 172.21.15.150:0/3529739854 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fa5880cace0 con 0x7fa594100d80 2023-10-11T16:25:35.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.764+0000 7fa593fff700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5840a10c0 0x7fa5840a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:35.770 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.764+0000 7fa593fff700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5840a10c0 0x7fa5840a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa580005fd0 tx=0x7fa580005e30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:25:35.771 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.770+0000 7fa591ffb700 1 -- 172.21.15.150:0/3529739854 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa588095c80 con 0x7fa594100d80 2023-10-11T16:25:35.975 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:35.974+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa594061aa0 con 0x7fa5840a10c0 2023-10-11T16:25:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:36 smithi111 ceph-mon[98527]: osdmap e14: 2 total, 2 up, 2 in 2023-10-11T16:25:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:36.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:36.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:36 smithi150 ceph-mon[132217]: osdmap e14: 2 total, 2 up, 2 in 2023-10-11T16:25:36.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:36.946 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:36.947 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:37.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:36 smithi130 ceph-mon[131859]: osdmap e14: 2 total, 2 up, 2 in 2023-10-11T16:25:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:37.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:37 smithi111 ceph-mon[98527]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:37.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:37 smithi111 ceph-mon[98527]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:37 smithi150 ceph-mon[132217]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:37 smithi150 ceph-mon[132217]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:38.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:37 smithi130 ceph-mon[131859]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:37 smithi130 ceph-mon[131859]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:38.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:38 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/1356018200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]: dispatch 2023-10-11T16:25:38.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:38 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]: dispatch 2023-10-11T16:25:38.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:38 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]': finished 2023-10-11T16:25:38.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:38 smithi111 ceph-mon[98527]: osdmap e15: 3 total, 2 up, 3 in 2023-10-11T16:25:38.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:38 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/1356018200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]: dispatch 2023-10-11T16:25:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:38 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]: dispatch 2023-10-11T16:25:38.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:38 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]': finished 2023-10-11T16:25:38.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:38 smithi150 ceph-mon[132217]: osdmap e15: 3 total, 2 up, 3 in 2023-10-11T16:25:38.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:39.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:38 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/1356018200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]: dispatch 2023-10-11T16:25:39.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:38 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]: dispatch 2023-10-11T16:25:39.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:38 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "614135a6-6a84-40f7-8b31-58459c019526"}]': finished 2023-10-11T16:25:39.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:38 smithi130 ceph-mon[131859]: osdmap e15: 3 total, 2 up, 3 in 2023-10-11T16:25:39.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:39.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:39 smithi150 ceph-mon[132217]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:39.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:39 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/854526657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:25:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:39 smithi111 ceph-mon[98527]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:39.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:39 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/854526657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:25:40.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:39 smithi130 ceph-mon[131859]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:39 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/854526657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:25:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:41 smithi111 ceph-mon[98527]: pgmap v65: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:41 smithi150 ceph-mon[132217]: pgmap v65: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:42.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:41 smithi130 ceph-mon[131859]: pgmap v65: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:43.813 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:43 smithi150 ceph-mon[132217]: pgmap v66: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:43 smithi111 ceph-mon[98527]: pgmap v66: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:44.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:43 smithi130 ceph-mon[131859]: pgmap v66: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:45 smithi111 ceph-mon[98527]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-11T16:25:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:45.912 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:45 smithi150 ceph-mon[132217]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:45.912 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-11T16:25:45.913 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:46.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:45 smithi130 ceph-mon[131859]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-11T16:25:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:46.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:46 smithi111 ceph-mon[98527]: Deploying daemon osd.2 on smithi150 2023-10-11T16:25:46.901 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:46 smithi150 ceph-mon[132217]: Deploying daemon osd.2 on smithi150 2023-10-11T16:25:47.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:46 smithi130 ceph-mon[131859]: Deploying daemon osd.2 on smithi150 2023-10-11T16:25:47.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:47 smithi111 ceph-mon[98527]: pgmap v68: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:47.935 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:47 smithi150 ceph-mon[132217]: pgmap v68: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:48.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:47 smithi130 ceph-mon[131859]: pgmap v68: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:48.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:48 smithi150 ceph-mon[132217]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:48.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:48 smithi111 ceph-mon[98527]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:49.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:48 smithi130 ceph-mon[131859]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:50.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:50.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:50.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:50.645 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:50.645 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:50.645 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:50.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:50.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:50.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:51 smithi150 ceph-mon[132217]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:51.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:51.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:51.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:51.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:51.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:51 smithi111 ceph-mon[98527]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:51.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:51.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:51.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:51.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:51.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:52.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:51 smithi130 ceph-mon[131859]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:52.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:52.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:52.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:52.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:52.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:52.292 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.290+0000 7fa591ffb700 1 -- 172.21.15.150:0/3529739854 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa594061aa0 con 0x7fa5840a10c0 2023-10-11T16:25:52.293 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 2 on host 'smithi150' 2023-10-11T16:25:52.294 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5840a10c0 msgr2=0x7fa5840a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:52.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5840a10c0 0x7fa5840a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa580005fd0 tx=0x7fa580005e30 comp rx=0 tx=0).stop 2023-10-11T16:25:52.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594100d80 msgr2=0x7fa594196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:52.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594100d80 0x7fa594196040 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa58800c950 tx=0x7fa58800cd10 comp rx=0 tx=0).stop 2023-10-11T16:25:52.295 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 shutdown_connections 2023-10-11T16:25:52.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5840a10c0 0x7fa5840a3570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:52.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa594105fc0 0x7fa59419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:52.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa594100d80 0x7fa594196040 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:52.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 --2- 172.21.15.150:0/3529739854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5941036a0 0x7fa594196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:52.296 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 >> 172.21.15.150:0/3529739854 conn(0x7fa5940fa820 msgr2=0x7fa5941019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:25:52.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 shutdown_connections 2023-10-11T16:25:52.297 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:25:52.292+0000 7fa59ac0f700 1 -- 172.21.15.150:0/3529739854 wait complete. 2023-10-11T16:25:52.604 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:52 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:52.605 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:52 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:25:52.605 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:52 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:52.605 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:52 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:52.605 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:52 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:52.605 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:52 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:52.781 DEBUG:teuthology.orchestra.run.smithi150:osd.2> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.2.service 2023-10-11T16:25:52.784 INFO:tasks.cephadm:Deploying osd.3 on smithi111 with /dev/vg_nvme/lv_3... 2023-10-11T16:25:52.784 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-11T16:25:52.813 INFO:journalctl@ceph.osd.2.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:25:52.853 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:52 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:52.854 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:52 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:25:52.854 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:52 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:52.854 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:52 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:52.854 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:52 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:52.855 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:52 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:52 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:52 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:25:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:52 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:53.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:52 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:53.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:52 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:53.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:52 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:25:53.561 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:53 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-2[138166]: 2023-10-11T16:25:53.354+0000 7f487c1e9640 -1 osd.2 0 log_to_monitors true 2023-10-11T16:25:53.636 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:25:53.871 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:53 smithi111 ceph-mon[98527]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:53.871 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:53 smithi111 ceph-mon[98527]: from='osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-11T16:25:53.872 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:53 smithi111 ceph-mon[98527]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-11T16:25:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:53 smithi150 ceph-mon[132217]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:53 smithi150 ceph-mon[132217]: from='osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-11T16:25:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:53 smithi150 ceph-mon[132217]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-11T16:25:54.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:53 smithi130 ceph-mon[131859]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:54.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:53 smithi130 ceph-mon[131859]: from='osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-11T16:25:54.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:53 smithi130 ceph-mon[131859]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-11T16:25:54.702 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:54 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-2[138166]: 2023-10-11T16:25:54.623+0000 7f4871860700 -1 osd.2 0 waiting for initial osdmap 2023-10-11T16:25:54.703 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:54 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-2[138166]: 2023-10-11T16:25:54.627+0000 7f4869e4e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:25:54.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:54 smithi150 ceph-mon[132217]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-11T16:25:54.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:54 smithi150 ceph-mon[132217]: osdmap e16: 3 total, 2 up, 3 in 2023-10-11T16:25:54.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:54 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:54.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:54 smithi150 ceph-mon[132217]: from='osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:25:54.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:54 smithi150 ceph-mon[132217]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:25:54.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:54 smithi111 ceph-mon[98527]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-11T16:25:54.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:54 smithi111 ceph-mon[98527]: osdmap e16: 3 total, 2 up, 3 in 2023-10-11T16:25:54.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:54 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:54.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:54 smithi111 ceph-mon[98527]: from='osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:25:54.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:54 smithi111 ceph-mon[98527]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:25:55.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:54 smithi130 ceph-mon[131859]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-11T16:25:55.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:54 smithi130 ceph-mon[131859]: osdmap e16: 3 total, 2 up, 3 in 2023-10-11T16:25:55.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:54 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:55.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:54 smithi130 ceph-mon[131859]: from='osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:25:55.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:54 smithi130 ceph-mon[131859]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:25:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-11T16:25:55.483 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-11T16:25:55.483 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2023-10-11T16:25:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2023-10-11T16:25:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0319896 s, 328 MB/s 2023-10-11T16:25:55.484 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 2023-10-11T16:25:55.484 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2023-10-11T16:25:55.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:55.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:25:55.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: osdmap e17: 3 total, 2 up, 3 in 2023-10-11T16:25:55.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:55.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:55.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:55.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:55.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:55.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:55.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:55.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:55.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:55.911 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2023-10-11T16:25:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:25:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: osdmap e17: 3 total, 2 up, 3 in 2023-10-11T16:25:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:55.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:55.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:55.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:55.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:55.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:56.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:56.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:25:56.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: osdmap e17: 3 total, 2 up, 3 in 2023-10-11T16:25:56.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:56.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:56.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:56.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:56.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:25:56.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:56.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:56.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:25:56.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:56.765 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:25:56.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:25:56.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:25:56.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi150 2023-10-11T16:25:56.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi150 to 17168M 2023-10-11T16:25:56.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:56.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:25:56.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:25:56.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] boot 2023-10-11T16:25:56.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: osdmap e18: 3 total, 3 up, 3 in 2023-10-11T16:25:56.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:56.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: pgmap v76: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:56.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:25:57.037 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:25:57.037 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:25:57.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi150 2023-10-11T16:25:57.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi150 to 17168M 2023-10-11T16:25:57.038 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:57.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:25:57.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:25:57.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] boot 2023-10-11T16:25:57.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: osdmap e18: 3 total, 3 up, 3 in 2023-10-11T16:25:57.039 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:57.040 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: pgmap v76: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:57.040 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:25:57.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:25:57.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:25:57.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi150 2023-10-11T16:25:57.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi150 to 17168M 2023-10-11T16:25:57.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:57.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:25:57.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:25:57.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: osd.2 [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] boot 2023-10-11T16:25:57.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: osdmap e18: 3 total, 3 up, 3 in 2023-10-11T16:25:57.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:25:57.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: pgmap v76: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-11T16:25:57.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:25:57.923 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:57 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-11T16:25:57.923 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:57 smithi111 ceph-mon[98527]: osdmap e19: 3 total, 3 up, 3 in 2023-10-11T16:25:57.923 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:57 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:25:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:57 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-11T16:25:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:57 smithi150 ceph-mon[132217]: osdmap e19: 3 total, 3 up, 3 in 2023-10-11T16:25:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:57 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:25:58.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:57 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-11T16:25:58.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:57 smithi130 ceph-mon[131859]: osdmap e19: 3 total, 3 up, 3 in 2023-10-11T16:25:58.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:57 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:25:58.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.177+0000 7fac6b9e7700 1 -- 172.21.15.111:0/140141108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac64074b20 msgr2=0x7fac64072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:58.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.177+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/140141108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac64074b20 0x7fac64072f80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fac600098f0 tx=0x7fac60009c00 comp rx=0 tx=0).stop 2023-10-11T16:25:58.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.177+0000 7fac6b9e7700 1 -- 172.21.15.111:0/140141108 shutdown_connections 2023-10-11T16:25:58.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.177+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/140141108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6410c8d0 0x7fac6410ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:58.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.177+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/140141108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac64074b20 0x7fac64072f80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:58.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.177+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/140141108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac64073550 0x7fac640739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:58.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.177+0000 7fac6b9e7700 1 -- 172.21.15.111:0/140141108 >> 172.21.15.111:0/140141108 conn(0x7fac640fb900 msgr2=0x7fac640fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:25:58.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.178+0000 7fac6b9e7700 1 -- 172.21.15.111:0/140141108 shutdown_connections 2023-10-11T16:25:58.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.178+0000 7fac6b9e7700 1 -- 172.21.15.111:0/140141108 wait complete. 2023-10-11T16:25:58.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.178+0000 7fac6b9e7700 1 Processor -- start 2023-10-11T16:25:58.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac6b9e7700 1 -- start start 2023-10-11T16:25:58.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac6b9e7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac64073550 0x7fac6419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:58.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac6b9e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac64074b20 0x7fac6419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:58.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac6b9e7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6410c8d0 0x7fac641a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:58.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac6b9e7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac641a41e0 con 0x7fac64074b20 2023-10-11T16:25:58.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac6b9e7700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac641a4320 con 0x7fac6410c8d0 2023-10-11T16:25:58.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac6b9e7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac641a44b0 con 0x7fac64073550 2023-10-11T16:25:58.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac69783700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac64073550 0x7fac6419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:58.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac69783700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac64073550 0x7fac6419c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:33770/0 (socket says 172.21.15.111:33770) 2023-10-11T16:25:58.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac69783700 1 -- 172.21.15.111:0/3217572869 learned_addr learned my addr 172.21.15.111:0/3217572869 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:25:58.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac68f82700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac64074b20 0x7fac6419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:58.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.179+0000 7fac69f84700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6410c8d0 0x7fac641a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:58.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac69783700 1 -- 172.21.15.111:0/3217572869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6410c8d0 msgr2=0x7fac641a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:58.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac69783700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6410c8d0 0x7fac641a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:58.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac69783700 1 -- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac64074b20 msgr2=0x7fac6419cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:25:58.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac69783700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac64074b20 0x7fac6419cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:25:58.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac69783700 1 -- 172.21.15.111:0/3217572869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540095f0 con 0x7fac64073550 2023-10-11T16:25:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac69783700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac64073550 0x7fac6419c650 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac600059c0 tx=0x7fac6000b560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:25:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac5a7fc700 1 -- 172.21.15.111:0/3217572869 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6001c070 con 0x7fac64073550 2023-10-11T16:25:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac5a7fc700 1 -- 172.21.15.111:0/3217572869 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac60005500 con 0x7fac64073550 2023-10-11T16:25:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.180+0000 7fac5a7fc700 1 -- 172.21.15.111:0/3217572869 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60004c00 con 0x7fac64073550 2023-10-11T16:25:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.181+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac60009520 con 0x7fac64073550 2023-10-11T16:25:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.181+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac64112250 con 0x7fac64073550 2023-10-11T16:25:58.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.182+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac64102420 con 0x7fac64073550 2023-10-11T16:25:58.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.184+0000 7fac5a7fc700 1 -- 172.21.15.111:0/3217572869 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 15) v1 ==== 135660+0+0 (secure 0 0 0) 0x7fac6000b9a0 con 0x7fac64073550 2023-10-11T16:25:58.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.185+0000 7fac5a7fc700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac500a10c0 0x7fac500a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:25:58.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.185+0000 7fac5a7fc700 1 -- 172.21.15.111:0/3217572869 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3237+0+0 (secure 0 0 0) 0x7fac600cc820 con 0x7fac64073550 2023-10-11T16:25:58.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.185+0000 7fac68f82700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac500a10c0 0x7fac500a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:25:58.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.186+0000 7fac68f82700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac500a10c0 0x7fac500a3570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fac54005fd0 tx=0x7fac54009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:25:58.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.187+0000 7fac5a7fc700 1 -- 172.21.15.111:0/3217572869 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fac60097530 con 0x7fac64073550 2023-10-11T16:25:58.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.401+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fac640619a0 con 0x7fac500a10c0 2023-10-11T16:25:58.635 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:58 smithi111 sudo[113766]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-11T16:25:58.636 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:58 smithi111 sudo[113766]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:25:58.636 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:58 smithi111 sudo[113766]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:25:58.636 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:58 smithi111 sudo[113766]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:25:58.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:25:58.654+0000 7fac5a7fc700 1 -- 172.21.15.111:0/3217572869 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fac60097900 con 0x7fac64073550 2023-10-11T16:25:58.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-11T16:25:58.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:58 smithi150 ceph-mon[132217]: osdmap e20: 3 total, 3 up, 3 in 2023-10-11T16:25:58.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:58 smithi150 ceph-mon[132217]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:25:58.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:58 smithi150 ceph-mon[132217]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:58.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:58.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:58.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:59.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-11T16:25:59.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:58 smithi130 ceph-mon[131859]: osdmap e20: 3 total, 3 up, 3 in 2023-10-11T16:25:59.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:58 smithi130 ceph-mon[131859]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:25:59.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:58 smithi130 ceph-mon[131859]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:59.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:59.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:59.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:59.135 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:58 smithi111 sudo[113835]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-11T16:25:59.135 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:58 smithi111 sudo[113835]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:25:59.136 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:58 smithi111 sudo[113835]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:25:59.136 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:25:58 smithi111 sudo[113835]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:25:59.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-11T16:25:59.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:58 smithi111 ceph-mon[98527]: osdmap e20: 3 total, 3 up, 3 in 2023-10-11T16:25:59.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:58 smithi111 ceph-mon[98527]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:25:59.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:58 smithi111 ceph-mon[98527]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:25:59.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:25:59.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:25:59.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:25:59.385 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:59 smithi130 sudo[139166]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-11T16:25:59.385 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:59 smithi130 sudo[139166]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:25:59.386 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:59 smithi130 sudo[139166]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:25:59.386 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:59 smithi130 sudo[139166]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:25:59.651 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:59 smithi130 sudo[139169]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-11T16:25:59.652 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:59 smithi130 sudo[139169]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:25:59.652 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:59 smithi130 sudo[139169]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:25:59.652 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:25:59 smithi130 sudo[139169]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:25:59.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:59 smithi111 ceph-mon[98527]: osdmap e21: 3 total, 3 up, 3 in 2023-10-11T16:25:59.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:59 smithi111 ceph-mon[98527]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-10-11T16:25:59.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:59.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:25:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:59.922 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:59 smithi150 ceph-mon[132217]: osdmap e21: 3 total, 3 up, 3 in 2023-10-11T16:25:59.922 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:59 smithi150 ceph-mon[132217]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-10-11T16:25:59.922 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:59.923 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:25:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:25:59.923 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:59 smithi150 sudo[140320]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-11T16:25:59.923 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:59 smithi150 sudo[140320]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:25:59.923 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:59 smithi150 sudo[140320]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:25:59.924 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:59 smithi150 sudo[140320]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:26:00.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:59 smithi130 ceph-mon[131859]: osdmap e21: 3 total, 3 up, 3 in 2023-10-11T16:26:00.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:59 smithi130 ceph-mon[131859]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-10-11T16:26:00.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:00.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:25:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:00.202 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:59 smithi150 sudo[140323]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-11T16:26:00.202 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:59 smithi150 sudo[140323]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:26:00.203 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:59 smithi150 sudo[140323]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:26:00.203 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:25:59 smithi150 sudo[140323]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:26:00.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:00 smithi111 sudo[113964]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-11T16:26:00.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:00 smithi111 sudo[113964]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:26:00.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:00 smithi111 sudo[113964]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:26:00.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:00 smithi111 sudo[113964]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:26:01.300 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:01.301 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 sudo[140326]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-11T16:26:01.301 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 sudo[140326]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:26:01.301 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 sudo[140326]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:26:01.301 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 sudo[140326]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:26:01.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:01.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:01 smithi111 sudo[114001]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-11T16:26:01.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:01 smithi111 sudo[114001]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:26:01.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:01 smithi111 sudo[114001]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:26:01.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:01 smithi111 sudo[114001]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:26:01.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:01.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 sudo[140329]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-11T16:26:01.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 sudo[140329]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:26:01.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 sudo[140329]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:26:01.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:01 smithi150 sudo[140329]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:26:02.038 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:01 smithi130 sudo[139172]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-11T16:26:02.039 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:01 smithi130 sudo[139172]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:26:02.039 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:01 smithi130 sudo[139172]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:26:02.039 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:01 smithi130 sudo[139172]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:26:02.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 sudo[139175]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-11T16:26:02.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 sudo[139175]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-11T16:26:02.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 sudo[139175]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-11T16:26:02.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 sudo[139175]: pam_unix(sudo:session): session closed for user root 2023-10-11T16:26:02.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:02.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:02.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:02.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:02.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:02.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3442954917' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]: dispatch 2023-10-11T16:26:02.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:02 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]: dispatch 2023-10-11T16:26:02.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:02.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:02.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:02.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:02.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:02.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3442954917' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]: dispatch 2023-10-11T16:26:02.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:02 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]: dispatch 2023-10-11T16:26:02.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:02.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:02.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:02.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:02.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:02.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:26:02.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:26:02.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:26:02.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3442954917' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]: dispatch 2023-10-11T16:26:02.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:02 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]: dispatch 2023-10-11T16:26:03.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:03 smithi130 ceph-mon[131859]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:03.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:03 smithi130 ceph-mon[131859]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:03.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:03 smithi130 ceph-mon[131859]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:03.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:03 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]': finished 2023-10-11T16:26:03.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:03 smithi130 ceph-mon[131859]: osdmap e22: 4 total, 3 up, 4 in 2023-10-11T16:26:03.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:03 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:03.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:03 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2638440739' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:03.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:03 smithi111 ceph-mon[98527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:03.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:03 smithi111 ceph-mon[98527]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:03.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:03 smithi111 ceph-mon[98527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:03.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:03 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]': finished 2023-10-11T16:26:03.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:03 smithi111 ceph-mon[98527]: osdmap e22: 4 total, 3 up, 4 in 2023-10-11T16:26:03.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:03 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:03.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:03 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2638440739' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:03.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:03 smithi150 ceph-mon[132217]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-11T16:26:03.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:03 smithi150 ceph-mon[132217]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:03 smithi150 ceph-mon[132217]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-11T16:26:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:03 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f2ce623-b8ba-448b-b9b3-f0970bc8be1d"}]': finished 2023-10-11T16:26:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:03 smithi150 ceph-mon[132217]: osdmap e22: 4 total, 3 up, 4 in 2023-10-11T16:26:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:03 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:03 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2638440739' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:05.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:05 smithi111 ceph-mon[98527]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:05.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:05 smithi150 ceph-mon[132217]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:06.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:05 smithi130 ceph-mon[131859]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:07.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:07 smithi111 ceph-mon[98527]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-10-11T16:26:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:07 smithi150 ceph-mon[132217]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-10-11T16:26:08.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:07 smithi130 ceph-mon[131859]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 218 KiB/s wr, 15 op/s 2023-10-11T16:26:09.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:09 smithi111 ceph-mon[98527]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-10-11T16:26:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:09 smithi150 ceph-mon[132217]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-10-11T16:26:10.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:09 smithi130 ceph-mon[131859]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-10-11T16:26:10.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-11T16:26:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:10.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-11T16:26:10.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:11.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-11T16:26:11.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:11.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:11 smithi111 ceph-mon[98527]: Deploying daemon osd.3 on smithi111 2023-10-11T16:26:11.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:11 smithi111 ceph-mon[98527]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-11T16:26:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:11 smithi150 ceph-mon[132217]: Deploying daemon osd.3 on smithi111 2023-10-11T16:26:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:11 smithi150 ceph-mon[132217]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-11T16:26:12.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:11 smithi130 ceph-mon[131859]: Deploying daemon osd.3 on smithi111 2023-10-11T16:26:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:11 smithi130 ceph-mon[131859]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-11T16:26:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:13 smithi111 ceph-mon[98527]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-11T16:26:13.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:13 smithi150 ceph-mon[132217]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-11T16:26:14.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:13 smithi130 ceph-mon[131859]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-11T16:26:15.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:15 smithi111 ceph-mon[98527]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-10-11T16:26:15.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:15 smithi150 ceph-mon[132217]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-10-11T16:26:16.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:15 smithi130 ceph-mon[131859]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-10-11T16:26:17.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:17 smithi130 ceph-mon[131859]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-10-11T16:26:17.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:17.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:17.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:17.360 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:17 smithi111 ceph-mon[98527]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-10-11T16:26:17.360 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:17.360 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:17.361 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:17 smithi150 ceph-mon[132217]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-10-11T16:26:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:17.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:17.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:19.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:19 smithi130 ceph-mon[131859]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:19.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:19 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:19 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:19 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:19.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:19 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:19.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:19 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:19 smithi111 ceph-mon[98527]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:19.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:19 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:19 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:19 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:19.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:19 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:19.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:19 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:19 smithi150 ceph-mon[132217]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:19 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:19 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:19 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:19.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:19 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:19.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:19 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:19.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.666+0000 7fac5a7fc700 1 -- 172.21.15.111:0/3217572869 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fac640619a0 con 0x7fac500a10c0 2023-10-11T16:26:19.668 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 3 on host 'smithi111' 2023-10-11T16:26:19.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac500a10c0 msgr2=0x7fac500a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:19.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac500a10c0 0x7fac500a3570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fac54005fd0 tx=0x7fac54009310 comp rx=0 tx=0).stop 2023-10-11T16:26:19.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac64073550 msgr2=0x7fac6419c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:19.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac64073550 0x7fac6419c650 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac600059c0 tx=0x7fac6000b560 comp rx=0 tx=0).stop 2023-10-11T16:26:19.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 shutdown_connections 2023-10-11T16:26:19.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac500a10c0 0x7fac500a3570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:19.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6410c8d0 0x7fac641a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:19.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac64073550 0x7fac6419c650 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:19.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 --2- 172.21.15.111:0/3217572869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac64074b20 0x7fac6419cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:19.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 >> 172.21.15.111:0/3217572869 conn(0x7fac640fb900 msgr2=0x7fac640fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:26:19.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 shutdown_connections 2023-10-11T16:26:19.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:26:19.669+0000 7fac6b9e7700 1 -- 172.21.15.111:0/3217572869 wait complete. 2023-10-11T16:26:20.102 DEBUG:teuthology.orchestra.run.smithi111:osd.3> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.3.service 2023-10-11T16:26:20.108 INFO:tasks.cephadm:Deploying osd.4 on smithi130 with /dev/vg_nvme/lv_3... 2023-10-11T16:26:20.108 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-11T16:26:20.183 INFO:journalctl@ceph.osd.3.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:26:20.757 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:26:20.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:20 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:26:20.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:20 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:26:20.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:20 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:26:20.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:20 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:20.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:20 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:20.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:20 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:20.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:20 smithi111 ceph-mon[98527]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-11T16:26:20.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:20 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:26:20.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:20 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:26:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:20 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:26:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:20 smithi150 ceph-mon[132217]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-11T16:26:20.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:20 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:26:20.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:20 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:26:20.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:20 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:26:20.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:20 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:20.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:20 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:20.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:20 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:20.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:20 smithi130 ceph-mon[131859]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-11T16:26:21.666 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:21 smithi111 ceph-mon[98527]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:21.667 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:21 smithi111 ceph-mon[98527]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-11T16:26:21.667 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:21 smithi111 ceph-mon[98527]: osdmap e23: 4 total, 3 up, 4 in 2023-10-11T16:26:21.667 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:21 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:21.668 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:21 smithi111 ceph-mon[98527]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:26:21.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:21 smithi150 ceph-mon[132217]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:21.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:21 smithi150 ceph-mon[132217]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-11T16:26:21.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:21 smithi150 ceph-mon[132217]: osdmap e23: 4 total, 3 up, 4 in 2023-10-11T16:26:21.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:21 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:21.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:21 smithi150 ceph-mon[132217]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:26:22.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:21 smithi130 ceph-mon[131859]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:22.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:21 smithi130 ceph-mon[131859]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-11T16:26:22.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:21 smithi130 ceph-mon[131859]: osdmap e23: 4 total, 3 up, 4 in 2023-10-11T16:26:22.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:21 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:22.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:21 smithi130 ceph-mon[131859]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:26:22.137 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:26:21 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3[116284]: 2023-10-11T16:26:21.663+0000 7fd1e3141700 -1 osd.3 0 waiting for initial osdmap 2023-10-11T16:26:22.138 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:26:21 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3[116284]: 2023-10-11T16:26:21.671+0000 7fd1def36700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:26:22.255 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-11T16:26:22.256 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-11T16:26:22.256 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2023-10-11T16:26:22.256 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2023-10-11T16:26:22.256 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130604 s, 803 MB/s 2023-10-11T16:26:22.257 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2023-10-11T16:26:22.611 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi130:vg_nvme/lv_3 2023-10-11T16:26:22.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:22 smithi130 ceph-mon[131859]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:26:22.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:22 smithi130 ceph-mon[131859]: osdmap e24: 4 total, 3 up, 4 in 2023-10-11T16:26:22.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:22.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:22.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:22 smithi130 ceph-mon[131859]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:22.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:22 smithi111 ceph-mon[98527]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:26:22.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:22 smithi111 ceph-mon[98527]: osdmap e24: 4 total, 3 up, 4 in 2023-10-11T16:26:22.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:22.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:22.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:22 smithi111 ceph-mon[98527]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:22.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:22 smithi150 ceph-mon[132217]: from='osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:26:22.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:22 smithi150 ceph-mon[132217]: osdmap e24: 4 total, 3 up, 4 in 2023-10-11T16:26:22.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:22.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:22.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:22 smithi150 ceph-mon[132217]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-11T16:26:23.256 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:26:23.726 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:26:23.726 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:26:23.726 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] boot 2023-10-11T16:26:23.726 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: osdmap e25: 4 total, 4 up, 4 in 2023-10-11T16:26:23.727 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:23.727 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi111 2023-10-11T16:26:23.727 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:23.727 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:23.727 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:23.727 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:23.728 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi111 to 8635M 2023-10-11T16:26:23.728 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:23.728 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:23.728 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:23.728 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:26:23.729 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:23.729 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:26:23.729 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:23 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:26:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:26:23.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:26:23.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] boot 2023-10-11T16:26:23.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: osdmap e25: 4 total, 4 up, 4 in 2023-10-11T16:26:23.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:23.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi111 2023-10-11T16:26:23.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:23.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:23.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:23.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:23.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi111 to 8635M 2023-10-11T16:26:23.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:23.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:23.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:23.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:26:23.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:23.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:26:23.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:23 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:26:24.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:26:24.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:26:24.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: osd.3 [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] boot 2023-10-11T16:26:24.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: osdmap e25: 4 total, 4 up, 4 in 2023-10-11T16:26:24.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:26:24.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi111 2023-10-11T16:26:24.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:24.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:24.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:24.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:24.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi111 to 8635M 2023-10-11T16:26:24.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:24.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:24.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:24.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:26:24.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:24.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:26:24.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:23 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:26:24.340 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 -- 172.21.15.130:0/1551551817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 msgr2=0x7fb000107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:24.341 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 --2- 172.21.15.130:0/1551551817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 0x7fb000107630 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faff800b0a0 tx=0x7faff800b3b0 comp rx=0 tx=0).stop 2023-10-11T16:26:24.341 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 -- 172.21.15.130:0/1551551817 shutdown_connections 2023-10-11T16:26:24.341 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 --2- 172.21.15.130:0/1551551817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 0x7fb000107630 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:24.341 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 --2- 172.21.15.130:0/1551551817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000ff1e0 0x7fb0000ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:24.342 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 --2- 172.21.15.130:0/1551551817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0001003e0 0x7fb000100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:24.342 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 -- 172.21.15.130:0/1551551817 >> 172.21.15.130:0/1551551817 conn(0x7fb0000fa7f0 msgr2=0x7fb0000fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:26:24.342 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 -- 172.21.15.130:0/1551551817 shutdown_connections 2023-10-11T16:26:24.342 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 -- 172.21.15.130:0/1551551817 wait complete. 2023-10-11T16:26:24.342 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 Processor -- start 2023-10-11T16:26:24.343 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.338+0000 7fb0083c5700 1 -- start start 2023-10-11T16:26:24.343 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb0083c5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0000ff1e0 0x7fb000196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:26:24.343 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb0083c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0001003e0 0x7fb000196750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:26:24.343 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb0083c5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 0x7fb00019d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:26:24.344 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb0083c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00019dda0 con 0x7fb0001003e0 2023-10-11T16:26:24.344 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb0083c5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00019dee0 con 0x7fb000100d70 2023-10-11T16:26:24.344 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb0083c5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00019e070 con 0x7fb0000ff1e0 2023-10-11T16:26:24.344 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 0x7fb00019d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:26:24.344 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 0x7fb00019d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:43438/0 (socket says 172.21.15.130:43438) 2023-10-11T16:26:24.345 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 -- 172.21.15.130:0/2786009949 learned_addr learned my addr 172.21.15.130:0/2786009949 (peer_addr_for_me v2:172.21.15.130:0/0) 2023-10-11T16:26:24.345 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006161700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0000ff1e0 0x7fb000196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:26:24.345 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 -- 172.21.15.130:0/2786009949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0000ff1e0 msgr2=0x7fb000196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:24.345 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0000ff1e0 0x7fb000196210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:24.345 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 -- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0001003e0 msgr2=0x7fb000196750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:24.346 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0001003e0 0x7fb000196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:24.346 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 -- 172.21.15.130:0/2786009949 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff8009d00 con 0x7fb000100d70 2023-10-11T16:26:24.346 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb006962700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 0x7fb00019d7d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faff800b070 tx=0x7faff8004b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:26:24.346 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7faff77fe700 1 -- 172.21.15.130:0/2786009949 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff800e040 con 0x7fb000100d70 2023-10-11T16:26:24.347 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00010ac30 con 0x7fb000100d70 2023-10-11T16:26:24.347 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7faff77fe700 1 -- 172.21.15.130:0/2786009949 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faff801db40 con 0x7fb000100d70 2023-10-11T16:26:24.347 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7faff77fe700 1 -- 172.21.15.130:0/2786009949 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff801bcb0 con 0x7fb000100d70 2023-10-11T16:26:24.347 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.339+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00010aaa0 con 0x7fb000100d70 2023-10-11T16:26:24.349 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.340+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb000066ea0 con 0x7fb000100d70 2023-10-11T16:26:24.349 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.341+0000 7faff77fe700 1 -- 172.21.15.130:0/2786009949 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7faff801dcb0 con 0x7fb000100d70 2023-10-11T16:26:24.349 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.342+0000 7faff77fe700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafec0a11d0 0x7fafec0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:26:24.349 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.342+0000 7faff77fe700 1 -- 172.21.15.130:0/2786009949 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7faff80cdd10 con 0x7fb000100d70 2023-10-11T16:26:24.349 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.342+0000 7fb006161700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafec0a11d0 0x7fafec0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:26:24.350 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.343+0000 7fb006161700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafec0a11d0 0x7fafec0a3680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faffc006fd0 tx=0x7faffc008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:26:24.350 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.346+0000 7faff77fe700 1 -- 172.21.15.130:0/2786009949 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faff805f730 con 0x7fb000100d70 2023-10-11T16:26:24.542 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:24.541+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb00019eb60 con 0x7fafec0a11d0 2023-10-11T16:26:24.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:24 smithi150 ceph-mon[132217]: osdmap e26: 4 total, 4 up, 4 in 2023-10-11T16:26:24.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:24 smithi150 ceph-mon[132217]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:24 smithi150 ceph-mon[132217]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:26:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:24 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:26:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:24 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:26:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:24 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:24 smithi150 ceph-mon[132217]: osdmap e27: 4 total, 4 up, 4 in 2023-10-11T16:26:25.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:24 smithi130 ceph-mon[131859]: osdmap e26: 4 total, 4 up, 4 in 2023-10-11T16:26:25.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:24 smithi130 ceph-mon[131859]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:25.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:24 smithi130 ceph-mon[131859]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:26:25.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:24 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:26:25.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:24 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:26:25.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:24 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:25.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:24 smithi130 ceph-mon[131859]: osdmap e27: 4 total, 4 up, 4 in 2023-10-11T16:26:25.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:24 smithi111 ceph-mon[98527]: osdmap e26: 4 total, 4 up, 4 in 2023-10-11T16:26:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:24 smithi111 ceph-mon[98527]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:24 smithi111 ceph-mon[98527]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:26:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:24 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:26:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:24 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:26:25.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:24 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:25.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:24 smithi111 ceph-mon[98527]: osdmap e27: 4 total, 4 up, 4 in 2023-10-11T16:26:26.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:26 smithi150 ceph-mon[132217]: osdmap e28: 4 total, 4 up, 4 in 2023-10-11T16:26:26.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:26 smithi150 ceph-mon[132217]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:27.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:26 smithi130 ceph-mon[131859]: osdmap e28: 4 total, 4 up, 4 in 2023-10-11T16:26:27.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:26 smithi130 ceph-mon[131859]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:27.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:26 smithi111 ceph-mon[98527]: osdmap e28: 4 total, 4 up, 4 in 2023-10-11T16:26:27.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:26 smithi111 ceph-mon[98527]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:27.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:27 smithi150 ceph-mon[132217]: osdmap e29: 4 total, 4 up, 4 in 2023-10-11T16:26:27.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:27 smithi150 ceph-mon[132217]: from='client.? 172.21.15.130:0/1850595724' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]: dispatch 2023-10-11T16:26:27.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:27 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]: dispatch 2023-10-11T16:26:27.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:27 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]': finished 2023-10-11T16:26:27.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:27 smithi150 ceph-mon[132217]: osdmap e30: 5 total, 4 up, 5 in 2023-10-11T16:26:27.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:27 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:27.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:27 smithi150 ceph-mon[132217]: from='client.? 172.21.15.130:0/3843127292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:28.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:27 smithi130 ceph-mon[131859]: osdmap e29: 4 total, 4 up, 4 in 2023-10-11T16:26:28.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:27 smithi130 ceph-mon[131859]: from='client.? 172.21.15.130:0/1850595724' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]: dispatch 2023-10-11T16:26:28.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:27 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]: dispatch 2023-10-11T16:26:28.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:27 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]': finished 2023-10-11T16:26:28.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:27 smithi130 ceph-mon[131859]: osdmap e30: 5 total, 4 up, 5 in 2023-10-11T16:26:28.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:27 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:28.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:27 smithi130 ceph-mon[131859]: from='client.? 172.21.15.130:0/3843127292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:28.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:27 smithi111 ceph-mon[98527]: osdmap e29: 4 total, 4 up, 4 in 2023-10-11T16:26:28.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:27 smithi111 ceph-mon[98527]: from='client.? 172.21.15.130:0/1850595724' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]: dispatch 2023-10-11T16:26:28.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:27 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]: dispatch 2023-10-11T16:26:28.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:27 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c35c66a-dcd9-48ed-9b46-d62ea7add46b"}]': finished 2023-10-11T16:26:28.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:27 smithi111 ceph-mon[98527]: osdmap e30: 5 total, 4 up, 5 in 2023-10-11T16:26:28.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:27 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:28.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:27 smithi111 ceph-mon[98527]: from='client.? 172.21.15.130:0/3843127292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:28.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:28 smithi150 ceph-mon[132217]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:29.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:28 smithi130 ceph-mon[131859]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:29.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:28 smithi111 ceph-mon[98527]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:31.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:31 smithi130 ceph-mon[131859]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-10-11T16:26:31.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:31 smithi111 ceph-mon[98527]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-10-11T16:26:31.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:31 smithi150 ceph-mon[132217]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2023-10-11T16:26:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:33 smithi130 ceph-mon[131859]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-10-11T16:26:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:33 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-11T16:26:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:33 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:33.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:33 smithi111 ceph-mon[98527]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-10-11T16:26:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-11T16:26:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:33 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:33.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:33 smithi150 ceph-mon[132217]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-10-11T16:26:33.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-11T16:26:33.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:33 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:34.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:34 smithi130 ceph-mon[131859]: Deploying daemon osd.4 on smithi130 2023-10-11T16:26:34.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:34 smithi111 ceph-mon[98527]: Deploying daemon osd.4 on smithi130 2023-10-11T16:26:34.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:34 smithi150 ceph-mon[132217]: Deploying daemon osd.4 on smithi130 2023-10-11T16:26:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:35 smithi130 ceph-mon[131859]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-11T16:26:35.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:35 smithi111 ceph-mon[98527]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-11T16:26:35.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:35 smithi150 ceph-mon[132217]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-11T16:26:37.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:37 smithi130 ceph-mon[131859]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-10-11T16:26:37.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:37 smithi111 ceph-mon[98527]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-10-11T16:26:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:37 smithi150 ceph-mon[132217]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-10-11T16:26:38.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:38.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:38.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:38.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:38 smithi130 ceph-mon[131859]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-10-11T16:26:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:38.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:38.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:38 smithi150 ceph-mon[132217]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-10-11T16:26:39.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:39.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:39.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:39.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:38 smithi111 ceph-mon[98527]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-10-11T16:26:40.557 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 4 on host 'smithi130' 2023-10-11T16:26:40.558 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.555+0000 7faff77fe700 1 -- 172.21.15.130:0/2786009949 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb00019eb60 con 0x7fafec0a11d0 2023-10-11T16:26:40.559 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.557+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafec0a11d0 msgr2=0x7fafec0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:40.560 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.557+0000 7fb0083c5700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafec0a11d0 0x7fafec0a3680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faffc006fd0 tx=0x7faffc008040 comp rx=0 tx=0).stop 2023-10-11T16:26:40.560 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.557+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 msgr2=0x7fb00019d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:40.560 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.557+0000 7fb0083c5700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 0x7fb00019d7d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faff800b070 tx=0x7faff8004b00 comp rx=0 tx=0).stop 2023-10-11T16:26:40.560 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.558+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 shutdown_connections 2023-10-11T16:26:40.560 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.558+0000 7fb0083c5700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafec0a11d0 0x7fafec0a3680 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:40.561 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.558+0000 7fb0083c5700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb000100d70 0x7fb00019d7d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:40.561 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.558+0000 7fb0083c5700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0000ff1e0 0x7fb000196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:40.561 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.558+0000 7fb0083c5700 1 --2- 172.21.15.130:0/2786009949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0001003e0 0x7fb000196750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:40.561 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.558+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 >> 172.21.15.130:0/2786009949 conn(0x7fb0000fa7f0 msgr2=0x7fb000068630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:26:40.561 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.558+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 shutdown_connections 2023-10-11T16:26:40.562 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:26:40.558+0000 7fb0083c5700 1 -- 172.21.15.130:0/2786009949 wait complete. 2023-10-11T16:26:40.951 DEBUG:teuthology.orchestra.run.smithi130:osd.4> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.4.service 2023-10-11T16:26:40.953 INFO:tasks.cephadm:Deploying osd.5 on smithi150 with /dev/vg_nvme/lv_3... 2023-10-11T16:26:40.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-11T16:26:40.986 INFO:journalctl@ceph.osd.4.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:26:41.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:41.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:41.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:26:41.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:26:41.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:26:41.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-11T16:26:41.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:41.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:41.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:41.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:26:41.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:26:41.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:26:41.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-11T16:26:41.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:41.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:41.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:41.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:26:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:26:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:26:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-11T16:26:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:41.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:41.603 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:26:41.781 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:26:41 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4[142394]: 2023-10-11T16:26:41.712+0000 7f77537ff640 -1 osd.4 0 log_to_monitors true 2023-10-11T16:26:42.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:41 smithi130 ceph-mon[131859]: from='osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-11T16:26:42.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:41 smithi130 ceph-mon[131859]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-11T16:26:42.147 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:41 smithi150 ceph-mon[132217]: from='osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-11T16:26:42.148 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:41 smithi150 ceph-mon[132217]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-11T16:26:42.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:41 smithi111 ceph-mon[98527]: from='osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-11T16:26:42.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:41 smithi111 ceph-mon[98527]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-11T16:26:43.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:42 smithi130 ceph-mon[131859]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-11T16:26:43.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:42 smithi130 ceph-mon[131859]: osdmap e31: 5 total, 4 up, 5 in 2023-10-11T16:26:43.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:43.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:42 smithi130 ceph-mon[131859]: from='osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:26:43.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:42 smithi130 ceph-mon[131859]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:26:43.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:42 smithi130 ceph-mon[131859]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:43.084 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:26:42 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4[142394]: 2023-10-11T16:26:42.922+0000 7f7748e76700 -1 osd.4 0 waiting for initial osdmap 2023-10-11T16:26:43.084 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:26:42 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4[142394]: 2023-10-11T16:26:42.928+0000 7f7744c6b700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:26:43.115 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-11T16:26:43.116 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-11T16:26:43.116 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-10-11T16:26:43.116 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-10-11T16:26:43.116 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109952 s, 954 MB/s 2023-10-11T16:26:43.116 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-10-11T16:26:43.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:42 smithi150 ceph-mon[132217]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-11T16:26:43.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:42 smithi150 ceph-mon[132217]: osdmap e31: 5 total, 4 up, 5 in 2023-10-11T16:26:43.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:43.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:42 smithi150 ceph-mon[132217]: from='osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:26:43.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:42 smithi150 ceph-mon[132217]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:26:43.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:42 smithi150 ceph-mon[132217]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:43.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:42 smithi111 ceph-mon[98527]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-11T16:26:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:42 smithi111 ceph-mon[98527]: osdmap e31: 5 total, 4 up, 5 in 2023-10-11T16:26:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:42 smithi111 ceph-mon[98527]: from='osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:26:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:42 smithi111 ceph-mon[98527]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:26:43.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:42 smithi111 ceph-mon[98527]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:43.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2023-10-11T16:26:44.189 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:26:44.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:26:44.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: osdmap e32: 5 total, 4 up, 5 in 2023-10-11T16:26:44.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:44.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:44.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi130 2023-10-11T16:26:44.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:44.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:44.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi130 to 8584M 2023-10-11T16:26:44.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:44.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:44.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:26:44.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:26:44.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:43 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:26:44.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:26:44.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: osdmap e32: 5 total, 4 up, 5 in 2023-10-11T16:26:44.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:44.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:44.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi130 2023-10-11T16:26:44.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:44.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:44.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi130 to 8584M 2023-10-11T16:26:44.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:44.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:44.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:26:44.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:26:44.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:43 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:26:44.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:26:44.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: osdmap e32: 5 total, 4 up, 5 in 2023-10-11T16:26:44.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:44.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:44.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi130 2023-10-11T16:26:44.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:44.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:26:44.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi130 to 8584M 2023-10-11T16:26:44.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:44.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:26:44.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:26:44.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:44.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:26:44.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:43 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:26:45.178 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:44 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:26:45.179 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:44 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:26:45.179 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:44 smithi150 ceph-mon[132217]: osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] boot 2023-10-11T16:26:45.179 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:44 smithi150 ceph-mon[132217]: osdmap e33: 5 total, 5 up, 5 in 2023-10-11T16:26:45.179 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:44 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:45.179 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:44 smithi150 ceph-mon[132217]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:45.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:44 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:26:45.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:44 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:26:45.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:44 smithi130 ceph-mon[131859]: osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] boot 2023-10-11T16:26:45.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:44 smithi130 ceph-mon[131859]: osdmap e33: 5 total, 5 up, 5 in 2023-10-11T16:26:45.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:44 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:45.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:44 smithi130 ceph-mon[131859]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:45.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 -- 172.21.15.150:0/1255973588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc0ff1e0 msgr2=0x7f3cfc0ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:45.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 --2- 172.21.15.150:0/1255973588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc0ff1e0 0x7f3cfc0ff5f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009990 tx=0x7f3cf0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:26:45.376 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 -- 172.21.15.150:0/1255973588 shutdown_connections 2023-10-11T16:26:45.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 --2- 172.21.15.150:0/1255973588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cfc100d70 0x7f3cfc107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:45.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 --2- 172.21.15.150:0/1255973588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc0ff1e0 0x7f3cfc0ff5f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:45.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 --2- 172.21.15.150:0/1255973588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cfc1003e0 0x7f3cfc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:45.377 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 -- 172.21.15.150:0/1255973588 >> 172.21.15.150:0/1255973588 conn(0x7f3cfc0fa7f0 msgr2=0x7f3cfc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:26:45.378 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 -- 172.21.15.150:0/1255973588 shutdown_connections 2023-10-11T16:26:45.378 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.374+0000 7f3d0208f700 1 -- 172.21.15.150:0/1255973588 wait complete. 2023-10-11T16:26:45.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3d0208f700 1 Processor -- start 2023-10-11T16:26:45.380 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3d0208f700 1 -- start start 2023-10-11T16:26:45.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3d0208f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cfc0ff1e0 0x7f3cfc069130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:26:45.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3d0208f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc1003e0 0x7f3cfc19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:26:45.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3d0208f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cfc100d70 0x7f3cfc19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:26:45.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3d0208f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc19dc00 con 0x7f3cfc0ff1e0 2023-10-11T16:26:45.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3d0208f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc19dd70 con 0x7f3cfc100d70 2023-10-11T16:26:45.381 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3d0208f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc19df30 con 0x7f3cfc1003e0 2023-10-11T16:26:45.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3cfaffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc1003e0 0x7f3cfc19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:26:45.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3cfaffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc1003e0 0x7f3cfc19ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42478/0 (socket says 172.21.15.150:42478) 2023-10-11T16:26:45.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3cfaffd700 1 -- 172.21.15.150:0/108914816 learned_addr learned my addr 172.21.15.150:0/108914816 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:26:45.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.375+0000 7f3cfb7fe700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cfc0ff1e0 0x7f3cfc069130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:26:45.382 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cfbfff700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cfc100d70 0x7f3cfc19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:26:45.383 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cfaffd700 1 -- 172.21.15.150:0/108914816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cfc100d70 msgr2=0x7f3cfc19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:45.383 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cfaffd700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cfc100d70 0x7f3cfc19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:45.383 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cfaffd700 1 -- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cfc0ff1e0 msgr2=0x7f3cfc069130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:26:45.383 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cfaffd700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cfc0ff1e0 0x7f3cfc069130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:26:45.383 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cfaffd700 1 -- 172.21.15.150:0/108914816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf00095f0 con 0x7f3cfc1003e0 2023-10-11T16:26:45.383 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cfaffd700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc1003e0 0x7f3cfc19ae10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00c920 tx=0x7f3cec00cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:26:45.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cf8ff9700 1 -- 172.21.15.150:0/108914816 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec007b00 con 0x7f3cfc1003e0 2023-10-11T16:26:45.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cf8ff9700 1 -- 172.21.15.150:0/108914816 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3cec004cf0 con 0x7f3cfc1003e0 2023-10-11T16:26:45.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3cf8ff9700 1 -- 172.21.15.150:0/108914816 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec005610 con 0x7f3cfc1003e0 2023-10-11T16:26:45.384 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc10ab30 con 0x7f3cfc1003e0 2023-10-11T16:26:45.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.376+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc10a9a0 con 0x7f3cfc1003e0 2023-10-11T16:26:45.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.377+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc19e620 con 0x7f3cfc1003e0 2023-10-11T16:26:45.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.378+0000 7f3cf8ff9700 1 -- 172.21.15.150:0/108914816 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f3cec004750 con 0x7f3cfc1003e0 2023-10-11T16:26:45.385 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.378+0000 7f3cf8ff9700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ce40a1150 0x7f3ce40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:26:45.386 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.379+0000 7f3cf8ff9700 1 -- 172.21.15.150:0/108914816 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f3cec0cb4f0 con 0x7f3cfc1003e0 2023-10-11T16:26:45.386 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.379+0000 7f3cfb7fe700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ce40a1150 0x7f3ce40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:26:45.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:44 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:26:45.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:44 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:26:45.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:44 smithi111 ceph-mon[98527]: osd.4 [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] boot 2023-10-11T16:26:45.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:44 smithi111 ceph-mon[98527]: osdmap e33: 5 total, 5 up, 5 in 2023-10-11T16:26:45.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:26:45.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:44 smithi111 ceph-mon[98527]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-11T16:26:45.388 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.379+0000 7f3cfb7fe700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ce40a1150 0x7f3ce40a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b5c0 tx=0x7f3cf0005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:26:45.388 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.383+0000 7f3cf8ff9700 1 -- 172.21.15.150:0/108914816 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3cec095cb0 con 0x7f3cfc1003e0 2023-10-11T16:26:45.580 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:26:45.578+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cfc19ebd0 con 0x7f3ce40a1150 2023-10-11T16:26:46.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:45 smithi150 ceph-mon[132217]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:26:46.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:26:46.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:26:46.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:46.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:45 smithi150 ceph-mon[132217]: osdmap e34: 5 total, 5 up, 5 in 2023-10-11T16:26:46.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:45 smithi130 ceph-mon[131859]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:26:46.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:26:46.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:26:46.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:46.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:45 smithi130 ceph-mon[131859]: osdmap e34: 5 total, 5 up, 5 in 2023-10-11T16:26:46.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:45 smithi111 ceph-mon[98527]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:26:46.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:26:46.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:26:46.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:46.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:45 smithi111 ceph-mon[98527]: osdmap e34: 5 total, 5 up, 5 in 2023-10-11T16:26:47.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:46 smithi150 ceph-mon[132217]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:47.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:46 smithi130 ceph-mon[131859]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:47.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:46 smithi111 ceph-mon[98527]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:48.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:48 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/309518024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]: dispatch 2023-10-11T16:26:48.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:48 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]: dispatch 2023-10-11T16:26:48.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:48 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]': finished 2023-10-11T16:26:48.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:48 smithi130 ceph-mon[131859]: osdmap e35: 6 total, 5 up, 6 in 2023-10-11T16:26:48.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:48 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:26:48.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:48 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/309518024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]: dispatch 2023-10-11T16:26:48.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:48 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]: dispatch 2023-10-11T16:26:48.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:48 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]': finished 2023-10-11T16:26:48.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:48 smithi111 ceph-mon[98527]: osdmap e35: 6 total, 5 up, 6 in 2023-10-11T16:26:48.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:48 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:26:48.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:48 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/309518024' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]: dispatch 2023-10-11T16:26:48.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:48 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]: dispatch 2023-10-11T16:26:48.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:48 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cbf73bf-6335-4f23-946e-b1b50c55d94f"}]': finished 2023-10-11T16:26:48.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:48 smithi150 ceph-mon[132217]: osdmap e35: 6 total, 5 up, 6 in 2023-10-11T16:26:48.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:48 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:26:49.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:49 smithi130 ceph-mon[131859]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:49.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:49 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/3756874006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:49.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:49 smithi111 ceph-mon[98527]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:49.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:49 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/3756874006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:49.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:49 smithi150 ceph-mon[132217]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:49.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:49 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/3756874006' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:26:51.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:51 smithi111 ceph-mon[98527]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:51 smithi150 ceph-mon[132217]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:52.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:51 smithi130 ceph-mon[131859]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:53.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:53 smithi111 ceph-mon[98527]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:53 smithi150 ceph-mon[132217]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:54.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:53 smithi130 ceph-mon[131859]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:54.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:54 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-11T16:26:54.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:54 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:54.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:54 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-11T16:26:54.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:54 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:55.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:54 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-11T16:26:55.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:54 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:26:55.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:55 smithi111 ceph-mon[98527]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:55.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:55 smithi111 ceph-mon[98527]: Deploying daemon osd.5 on smithi150 2023-10-11T16:26:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:55 smithi150 ceph-mon[132217]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:55 smithi150 ceph-mon[132217]: Deploying daemon osd.5 on smithi150 2023-10-11T16:26:56.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:55 smithi130 ceph-mon[131859]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:56.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:55 smithi130 ceph-mon[131859]: Deploying daemon osd.5 on smithi150 2023-10-11T16:26:57.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:57 smithi150 ceph-mon[132217]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:57.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:57 smithi111 ceph-mon[98527]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:58.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:57 smithi130 ceph-mon[131859]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:59.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:59 smithi111 ceph-mon[98527]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:59.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:59.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:59.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:26:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:26:59.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:59 smithi150 ceph-mon[132217]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:26:59.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:59.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:26:59.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:26:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:00.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:59 smithi130 ceph-mon[131859]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:00.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:00.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:00.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:26:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:01.746 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 5 on host 'smithi150' 2023-10-11T16:27:01.747 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.744+0000 7f3cf8ff9700 1 -- 172.21.15.150:0/108914816 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3cfc19ebd0 con 0x7f3ce40a1150 2023-10-11T16:27:01.748 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ce40a1150 msgr2=0x7f3ce40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:01.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ce40a1150 0x7f3ce40a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b5c0 tx=0x7f3cf0005fb0 comp rx=0 tx=0).stop 2023-10-11T16:27:01.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc1003e0 msgr2=0x7f3cfc19ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:01.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc1003e0 0x7f3cfc19ae10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00c920 tx=0x7f3cec00cce0 comp rx=0 tx=0).stop 2023-10-11T16:27:01.749 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 shutdown_connections 2023-10-11T16:27:01.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ce40a1150 0x7f3ce40a3600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:01.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3cfc100d70 0x7f3cfc19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:01.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cfc0ff1e0 0x7f3cfc069130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:01.750 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 --2- 172.21.15.150:0/108914816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cfc1003e0 0x7f3cfc19ae10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:01.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 >> 172.21.15.150:0/108914816 conn(0x7f3cfc0fa7f0 msgr2=0x7f3cfc0fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:27:01.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 shutdown_connections 2023-10-11T16:27:01.751 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:01.747+0000 7f3d0208f700 1 -- 172.21.15.150:0/108914816 wait complete. 2023-10-11T16:27:01.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:01 smithi111 ceph-mon[98527]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:01.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:01.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:01.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:01.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:01.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:01.894 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:01 smithi150 ceph-mon[132217]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:01.895 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:01.895 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:01.895 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:01.895 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:01.896 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:01 smithi130 ceph-mon[131859]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:02.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:02.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:02.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:02.180 DEBUG:teuthology.orchestra.run.smithi150:osd.5> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.5.service 2023-10-11T16:27:02.184 INFO:tasks.cephadm:Deploying osd.6 on smithi111 with /dev/vg_nvme/lv_2... 2023-10-11T16:27:02.184 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-10-11T16:27:02.224 INFO:journalctl@ceph.osd.5.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:27:02.845 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:02 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:02.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:02 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:27:02.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:02 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:02.846 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:02.847 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:02.847 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:02.847 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:27:02 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:27:02.828+0000 7f2f7787c640 -1 osd.5 0 log_to_monitors true 2023-10-11T16:27:02.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:02 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:02.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:02 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:27:02.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:02 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:02.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:02.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:02.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:03.053 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:27:03.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:02 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:02 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:27:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:02 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:03.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:03 smithi111 ceph-mon[98527]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:03.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:03 smithi111 ceph-mon[98527]: from='osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-11T16:27:03.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:03 smithi111 ceph-mon[98527]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-11T16:27:03.886 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:03 smithi150 ceph-mon[132217]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:03.887 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:03 smithi150 ceph-mon[132217]: from='osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-11T16:27:03.887 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:03 smithi150 ceph-mon[132217]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-11T16:27:04.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:03 smithi130 ceph-mon[131859]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:03 smithi130 ceph-mon[131859]: from='osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-11T16:27:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:03 smithi130 ceph-mon[131859]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-11T16:27:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-10-11T16:27:05.075 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-10-11T16:27:05.076 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2023-10-11T16:27:05.076 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2023-10-11T16:27:05.076 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0305714 s, 343 MB/s 2023-10-11T16:27:05.076 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 2023-10-11T16:27:05.077 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2023-10-11T16:27:05.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-11T16:27:05.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: osdmap e36: 6 total, 5 up, 6 in 2023-10-11T16:27:05.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:05.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:27:05.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:27:05.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:05.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi150 2023-10-11T16:27:05.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:05.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:05.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi150 to 8584M 2023-10-11T16:27:05.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:05.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:05.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:05.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:27:05.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:04 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:05.112 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-11T16:27:05.112 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: osdmap e36: 6 total, 5 up, 6 in 2023-10-11T16:27:05.112 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:05.113 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:27:05.113 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:27:05.113 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:05.113 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi150 2023-10-11T16:27:05.113 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.114 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.114 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:05.114 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:05.114 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi150 to 8584M 2023-10-11T16:27:05.114 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:05.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:05.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:05.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.115 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:27:05.116 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:04 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:05.202 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:27:04.721+0000 7f2f6cef3700 -1 osd.5 0 waiting for initial osdmap 2023-10-11T16:27:05.203 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:27:04.729+0000 7f2f674e5700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:27:05.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-11T16:27:05.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: osdmap e36: 6 total, 5 up, 6 in 2023-10-11T16:27:05.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:05.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:27:05.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:27:05.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-11T16:27:05.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi150 2023-10-11T16:27:05.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:05.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:05.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi150 to 8584M 2023-10-11T16:27:05.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:05.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:05.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:05.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:05.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:27:05.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:04 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:05.533 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2023-10-11T16:27:05.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:05 smithi111 ceph-mon[98527]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:27:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:05 smithi111 ceph-mon[98527]: osdmap e37: 6 total, 5 up, 6 in 2023-10-11T16:27:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:05.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:05 smithi111 ceph-mon[98527]: osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] boot 2023-10-11T16:27:05.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:05 smithi111 ceph-mon[98527]: osdmap e38: 6 total, 6 up, 6 in 2023-10-11T16:27:05.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:06.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:05 smithi130 ceph-mon[131859]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:27:06.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:05 smithi130 ceph-mon[131859]: osdmap e37: 6 total, 5 up, 6 in 2023-10-11T16:27:06.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:05 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:06.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:05 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:06.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:05 smithi130 ceph-mon[131859]: osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] boot 2023-10-11T16:27:06.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:05 smithi130 ceph-mon[131859]: osdmap e38: 6 total, 6 up, 6 in 2023-10-11T16:27:06.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:05 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:06.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:05 smithi150 ceph-mon[132217]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:27:06.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:05 smithi150 ceph-mon[132217]: osdmap e37: 6 total, 5 up, 6 in 2023-10-11T16:27:06.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:05 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:06.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:05 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:06.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:05 smithi150 ceph-mon[132217]: osd.5 [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] boot 2023-10-11T16:27:06.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:05 smithi150 ceph-mon[132217]: osdmap e38: 6 total, 6 up, 6 in 2023-10-11T16:27:06.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:05 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:27:06.380 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:27:06.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:06 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:27:06.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:06 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:27:06.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:06 smithi111 ceph-mon[98527]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:06.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:06 smithi111 ceph-mon[98527]: osdmap e39: 6 total, 6 up, 6 in 2023-10-11T16:27:07.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:06 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:27:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:06 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:27:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:06 smithi130 ceph-mon[131859]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:06 smithi130 ceph-mon[131859]: osdmap e39: 6 total, 6 up, 6 in 2023-10-11T16:27:07.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:06 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:27:07.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:06 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:27:07.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:06 smithi150 ceph-mon[132217]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:07.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:06 smithi150 ceph-mon[132217]: osdmap e39: 6 total, 6 up, 6 in 2023-10-11T16:27:07.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.840+0000 7ff8a3be5700 1 -- 172.21.15.111:0/1182424700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 msgr2=0x7ff89c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.840+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/1182424700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 0x7ff89c0739c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009990 tx=0x7ff88c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:27:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.841+0000 7ff8a3be5700 1 -- 172.21.15.111:0/1182424700 shutdown_connections 2023-10-11T16:27:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.841+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/1182424700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff89c1081c0 0x7ff89c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.841+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/1182424700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89c074b20 0x7ff89c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.841+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/1182424700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 0x7ff89c0739c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.841+0000 7ff8a3be5700 1 -- 172.21.15.111:0/1182424700 >> 172.21.15.111:0/1182424700 conn(0x7ff89c0fb900 msgr2=0x7ff89c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:27:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.841+0000 7ff8a3be5700 1 -- 172.21.15.111:0/1182424700 shutdown_connections 2023-10-11T16:27:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.841+0000 7ff8a3be5700 1 -- 172.21.15.111:0/1182424700 wait complete. 2023-10-11T16:27:07.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a3be5700 1 Processor -- start 2023-10-11T16:27:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a3be5700 1 -- start start 2023-10-11T16:27:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a3be5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 0x7ff89c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a3be5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff89c074b20 0x7ff89c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a3be5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89c1081c0 0x7ff89c19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a3be5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c19fdc0 con 0x7ff89c073550 2023-10-11T16:27:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a3be5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c19ff00 con 0x7ff89c1081c0 2023-10-11T16:27:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a3be5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c1a0090 con 0x7ff89c074b20 2023-10-11T16:27:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a1981700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 0x7ff89c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a1981700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 0x7ff89c198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47114/0 (socket says 172.21.15.111:47114) 2023-10-11T16:27:07.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.842+0000 7ff8a1981700 1 -- 172.21.15.111:0/2583587726 learned_addr learned my addr 172.21.15.111:0/2583587726 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:27:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a2182700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89c1081c0 0x7ff89c19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a1180700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff89c074b20 0x7ff89c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a1981700 1 -- 172.21.15.111:0/2583587726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff89c074b20 msgr2=0x7ff89c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a1981700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff89c074b20 0x7ff89c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:07.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a1981700 1 -- 172.21.15.111:0/2583587726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89c1081c0 msgr2=0x7ff89c19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a1981700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89c1081c0 0x7ff89c19f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a1981700 1 -- 172.21.15.111:0/2583587726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0095f0 con 0x7ff89c073550 2023-10-11T16:27:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a2182700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89c1081c0 0x7ff89c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:27:07.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a1981700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 0x7ff89c198230 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff89800d980 tx=0x7ff89800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:27:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff892ffd700 1 -- 172.21.15.111:0/2583587726 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898009b40 con 0x7ff89c073550 2023-10-11T16:27:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff892ffd700 1 -- 172.21.15.111:0/2583587726 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff89800b6e0 con 0x7ff89c073550 2023-10-11T16:27:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89c10de90 con 0x7ff89c073550 2023-10-11T16:27:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff892ffd700 1 -- 172.21.15.111:0/2583587726 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89800f620 con 0x7ff89c073550 2023-10-11T16:27:07.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.843+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89c10dd00 con 0x7ff89c073550 2023-10-11T16:27:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.845+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89c106870 con 0x7ff89c073550 2023-10-11T16:27:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.846+0000 7ff892ffd700 1 -- 172.21.15.111:0/2583587726 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ff898010460 con 0x7ff89c073550 2023-10-11T16:27:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.848+0000 7ff892ffd700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8880a1150 0x7ff8880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.848+0000 7ff892ffd700 1 -- 172.21.15.111:0/2583587726 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7ff8980cbd80 con 0x7ff89c073550 2023-10-11T16:27:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.848+0000 7ff8a1180700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8880a1150 0x7ff8880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.848+0000 7ff8a1180700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8880a1150 0x7ff8880a3600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00b5c0 tx=0x7ff88c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:27:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:07.853+0000 7ff892ffd700 1 -- 172.21.15.111:0/2583587726 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff8980962d0 con 0x7ff89c073550 2023-10-11T16:27:08.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:08.063+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff89c061ac0 con 0x7ff8880a1150 2023-10-11T16:27:09.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:08 smithi130 ceph-mon[131859]: osdmap e40: 6 total, 6 up, 6 in 2023-10-11T16:27:09.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:08 smithi130 ceph-mon[131859]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:09.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:08 smithi130 ceph-mon[131859]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:09.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:08 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:09.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:08 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:09.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:08 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:09.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:08 smithi111 ceph-mon[98527]: osdmap e40: 6 total, 6 up, 6 in 2023-10-11T16:27:09.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:08 smithi111 ceph-mon[98527]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:09.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:08 smithi111 ceph-mon[98527]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:09.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:08 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:09.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:08 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:09.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:08 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:09.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:08 smithi150 ceph-mon[132217]: osdmap e40: 6 total, 6 up, 6 in 2023-10-11T16:27:09.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:08 smithi150 ceph-mon[132217]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:09.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:08 smithi150 ceph-mon[132217]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:09.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:08 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:09.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:08 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:09.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:08 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:09.880 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:09 smithi111 ceph-mon[98527]: osdmap e41: 6 total, 6 up, 6 in 2023-10-11T16:27:10.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:09 smithi130 ceph-mon[131859]: osdmap e41: 6 total, 6 up, 6 in 2023-10-11T16:27:10.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:09 smithi150 ceph-mon[132217]: osdmap e41: 6 total, 6 up, 6 in 2023-10-11T16:27:11.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:10 smithi130 ceph-mon[131859]: osdmap e42: 6 total, 6 up, 6 in 2023-10-11T16:27:11.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:10 smithi130 ceph-mon[131859]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-11T16:27:11.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:10 smithi111 ceph-mon[98527]: osdmap e42: 6 total, 6 up, 6 in 2023-10-11T16:27:11.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:10 smithi111 ceph-mon[98527]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-11T16:27:11.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:10 smithi150 ceph-mon[132217]: osdmap e42: 6 total, 6 up, 6 in 2023-10-11T16:27:11.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:10 smithi150 ceph-mon[132217]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-11T16:27:12.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:11 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/887559146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40e6958d-a7e1-4cd8-abcb-dd770747d086"}]: dispatch 2023-10-11T16:27:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:11 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/887559146' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40e6958d-a7e1-4cd8-abcb-dd770747d086"}]': finished 2023-10-11T16:27:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:11 smithi130 ceph-mon[131859]: osdmap e43: 7 total, 6 up, 7 in 2023-10-11T16:27:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:12.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:11 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/887559146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40e6958d-a7e1-4cd8-abcb-dd770747d086"}]: dispatch 2023-10-11T16:27:12.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:11 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/887559146' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40e6958d-a7e1-4cd8-abcb-dd770747d086"}]': finished 2023-10-11T16:27:12.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:11 smithi111 ceph-mon[98527]: osdmap e43: 7 total, 6 up, 7 in 2023-10-11T16:27:12.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:12.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:11 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/887559146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40e6958d-a7e1-4cd8-abcb-dd770747d086"}]: dispatch 2023-10-11T16:27:12.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:11 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/887559146' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40e6958d-a7e1-4cd8-abcb-dd770747d086"}]': finished 2023-10-11T16:27:12.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:11 smithi150 ceph-mon[132217]: osdmap e43: 7 total, 6 up, 7 in 2023-10-11T16:27:12.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:13.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:12 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3181330768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:27:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:12 smithi130 ceph-mon[131859]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2023-10-11T16:27:13.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:12 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3181330768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:27:13.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:12 smithi111 ceph-mon[98527]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2023-10-11T16:27:13.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:12 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3181330768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:27:13.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:12 smithi150 ceph-mon[132217]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2023-10-11T16:27:15.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:15 smithi111 ceph-mon[98527]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-11T16:27:15.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:15 smithi150 ceph-mon[132217]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-11T16:27:16.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:15 smithi130 ceph-mon[131859]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-11T16:27:17.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:17 smithi111 ceph-mon[98527]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-10-11T16:27:17.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:17 smithi150 ceph-mon[132217]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-10-11T16:27:18.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:17 smithi130 ceph-mon[131859]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-10-11T16:27:19.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:19 smithi111 ceph-mon[98527]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:19.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:19 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-11T16:27:19.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:19 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:19 smithi150 ceph-mon[132217]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:19 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-11T16:27:19.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:19 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:20.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:19 smithi130 ceph-mon[131859]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:20.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:19 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-11T16:27:20.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:19 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:20.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:20 smithi111 ceph-mon[98527]: Deploying daemon osd.6 on smithi111 2023-10-11T16:27:20.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:20 smithi150 ceph-mon[132217]: Deploying daemon osd.6 on smithi111 2023-10-11T16:27:21.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:20 smithi130 ceph-mon[131859]: Deploying daemon osd.6 on smithi111 2023-10-11T16:27:21.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:21 smithi111 ceph-mon[98527]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:21.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:21 smithi150 ceph-mon[132217]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:22.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:21 smithi130 ceph-mon[131859]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:23.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:23 smithi111 ceph-mon[98527]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:23 smithi150 ceph-mon[132217]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:24.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:23 smithi130 ceph-mon[131859]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:25 smithi150 ceph-mon[132217]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:25 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:26.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:25 smithi130 ceph-mon[131859]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:26.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:25 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:26.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:25 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:26.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:25 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:26.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:25 smithi111 ceph-mon[98527]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:26.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:26.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:26.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:25 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:26.861 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:26 smithi111 ceph-mon[98527]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:26.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:26 smithi150 ceph-mon[132217]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:27.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:26 smithi130 ceph-mon[131859]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.626+0000 7ff892ffd700 1 -- 172.21.15.111:0/2583587726 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff89c061ac0 con 0x7ff8880a1150 2023-10-11T16:27:28.628 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 6 on host 'smithi111' 2023-10-11T16:27:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8880a1150 msgr2=0x7ff8880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8880a1150 0x7ff8880a3600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00b5c0 tx=0x7ff88c005fd0 comp rx=0 tx=0).stop 2023-10-11T16:27:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 msgr2=0x7ff89c198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 0x7ff89c198230 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff89800d980 tx=0x7ff89800dc90 comp rx=0 tx=0).stop 2023-10-11T16:27:28.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 shutdown_connections 2023-10-11T16:27:28.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8880a1150 0x7ff8880a3600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:28.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff89c1081c0 0x7ff89c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:28.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff89c073550 0x7ff89c198230 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:28.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 --2- 172.21.15.111:0/2583587726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff89c074b20 0x7ff89c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:28.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 >> 172.21.15.111:0/2583587726 conn(0x7ff89c0fb900 msgr2=0x7ff89c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:27:28.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 shutdown_connections 2023-10-11T16:27:28.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:27:28.629+0000 7ff8a3be5700 1 -- 172.21.15.111:0/2583587726 wait complete. 2023-10-11T16:27:29.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:29.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:29.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:29.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:29.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:27:29.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:29.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:29.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:29 smithi111 ceph-mon[98527]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-11T16:27:29.201 DEBUG:teuthology.orchestra.run.smithi111:osd.6> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.6.service 2023-10-11T16:27:29.206 INFO:tasks.cephadm:Deploying osd.7 on smithi130 with /dev/vg_nvme/lv_2... 2023-10-11T16:27:29.206 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-10-11T16:27:29.280 INFO:journalctl@ceph.osd.6.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:27:29.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:29.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:29.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:29.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:29.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:27:29.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:29.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:29.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:29 smithi130 ceph-mon[131859]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-11T16:27:29.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:29.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:29.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:29.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:29.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:27:29.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:29.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:29.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:29.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:29 smithi150 ceph-mon[132217]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-11T16:27:29.870 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:27:30.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:30 smithi130 ceph-mon[131859]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-11T16:27:30.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:30 smithi130 ceph-mon[131859]: osdmap e44: 7 total, 6 up, 7 in 2023-10-11T16:27:30.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:30 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:30.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:30 smithi130 ceph-mon[131859]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:27:30.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:30 smithi111 ceph-mon[98527]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-11T16:27:30.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:30 smithi111 ceph-mon[98527]: osdmap e44: 7 total, 6 up, 7 in 2023-10-11T16:27:30.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:30 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:30.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:30 smithi111 ceph-mon[98527]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:27:30.887 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:27:30 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:27:30.604+0000 7f7441438700 -1 osd.6 0 waiting for initial osdmap 2023-10-11T16:27:30.887 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:27:30 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:27:30.616+0000 7f743ba2a700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:27:30.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:30 smithi150 ceph-mon[132217]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-11T16:27:30.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:30 smithi150 ceph-mon[132217]: osdmap e44: 7 total, 6 up, 7 in 2023-10-11T16:27:30.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:30 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:30.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:30 smithi150 ceph-mon[132217]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:27:31.342 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-10-11T16:27:31.342 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-10-11T16:27:31.342 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2023-10-11T16:27:31.342 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2023-10-11T16:27:31.343 INFO:teuthology.orchestra.run.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148039 s, 708 MB/s 2023-10-11T16:27:31.343 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2023-10-11T16:27:31.690 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:31 smithi130 ceph-mon[131859]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:31.690 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:31 smithi130 ceph-mon[131859]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:27:31.691 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:31 smithi130 ceph-mon[131859]: osdmap e45: 7 total, 6 up, 7 in 2023-10-11T16:27:31.691 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:31 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:31.691 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:31 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:31.724 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi130:vg_nvme/lv_2 2023-10-11T16:27:31.865 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:31 smithi111 ceph-mon[98527]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:31.866 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:31 smithi111 ceph-mon[98527]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:27:31.866 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:31 smithi111 ceph-mon[98527]: osdmap e45: 7 total, 6 up, 7 in 2023-10-11T16:27:31.866 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:31 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:31.866 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:31 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:31.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:31 smithi150 ceph-mon[132217]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:31.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:31 smithi150 ceph-mon[132217]: from='osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:27:31.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:31 smithi150 ceph-mon[132217]: osdmap e45: 7 total, 6 up, 7 in 2023-10-11T16:27:31.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:31 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:31.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:31 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:32.368 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:27:32.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:27:32.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:27:32.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] boot 2023-10-11T16:27:32.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: osdmap e46: 7 total, 7 up, 7 in 2023-10-11T16:27:32.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:32.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:32.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:32.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:32 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:27:32.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:27:32.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] boot 2023-10-11T16:27:32.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: osdmap e46: 7 total, 7 up, 7 in 2023-10-11T16:27:32.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:32.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:32.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:32.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:32 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:27:32.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:27:32.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: osd.6 [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] boot 2023-10-11T16:27:32.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: osdmap e46: 7 total, 7 up, 7 in 2023-10-11T16:27:32.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:27:32.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:32.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:32.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:32.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:32.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:32 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:33.632 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 -- 172.21.15.130:0/1095704040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c100620 msgr2=0x7f1d9c100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:33.632 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 --2- 172.21.15.130:0/1095704040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c100620 0x7f1d9c100a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1d90009990 tx=0x7f1d90009ca0 comp rx=0 tx=0).stop 2023-10-11T16:27:33.632 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 -- 172.21.15.130:0/1095704040 shutdown_connections 2023-10-11T16:27:33.633 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 --2- 172.21.15.130:0/1095704040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d9c100fb0 0x7f1d9c107870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:33.633 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 --2- 172.21.15.130:0/1095704040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c0ff420 0x7f1d9c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:33.633 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 --2- 172.21.15.130:0/1095704040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c100620 0x7f1d9c100a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:33.633 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 -- 172.21.15.130:0/1095704040 >> 172.21.15.130:0/1095704040 conn(0x7f1d9c0faa50 msgr2=0x7f1d9c0fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:27:33.633 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 -- 172.21.15.130:0/1095704040 shutdown_connections 2023-10-11T16:27:33.634 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.628+0000 7f1da0941700 1 -- 172.21.15.130:0/1095704040 wait complete. 2023-10-11T16:27:33.634 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1da0941700 1 Processor -- start 2023-10-11T16:27:33.634 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1da0941700 1 -- start start 2023-10-11T16:27:33.634 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1da0941700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c0ff420 0x7f1d9c1962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:33.635 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1da0941700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c100620 0x7f1d9c196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:33.635 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1da0941700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d9c100fb0 0x7f1d9c19d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:33.635 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1da0941700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c19de50 con 0x7f1d9c100fb0 2023-10-11T16:27:33.635 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1da0941700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c19df90 con 0x7f1d9c0ff420 2023-10-11T16:27:33.635 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1da0941700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c19e120 con 0x7f1d9c100620 2023-10-11T16:27:33.636 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d9a59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c0ff420 0x7f1d9c1962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:33.636 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d9a59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c0ff420 0x7f1d9c1962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:43544/0 (socket says 172.21.15.130:43544) 2023-10-11T16:27:33.636 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d9a59c700 1 -- 172.21.15.130:0/1466263827 learned_addr learned my addr 172.21.15.130:0/1466263827 (peer_addr_for_me v2:172.21.15.130:0/0) 2023-10-11T16:27:33.636 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d99d9b700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c100620 0x7f1d9c196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:33.636 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d9a59c700 1 -- 172.21.15.130:0/1466263827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c100620 msgr2=0x7f1d9c196800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:33.636 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d9a59c700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c100620 0x7f1d9c196800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:33.637 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d9a59c700 1 -- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d9c100fb0 msgr2=0x7f1d9c19d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:33.637 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d9a59c700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d9c100fb0 0x7f1d9c19d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:33.637 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.629+0000 7f1d9a59c700 1 -- 172.21.15.130:0/1466263827 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d900095f0 con 0x7f1d9c0ff420 2023-10-11T16:27:33.637 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.630+0000 7f1d9a59c700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c0ff420 0x7f1d9c1962c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00c970 tx=0x7f1d8c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:27:33.638 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.630+0000 7f1d8b7fe700 1 -- 172.21.15.130:0/1466263827 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c004220 con 0x7f1d9c0ff420 2023-10-11T16:27:33.638 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.630+0000 7f1d8b7fe700 1 -- 172.21.15.130:0/1466263827 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d8c007d60 con 0x7f1d9c0ff420 2023-10-11T16:27:33.638 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.630+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c10adf0 con 0x7f1d9c0ff420 2023-10-11T16:27:33.638 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.630+0000 7f1d8b7fe700 1 -- 172.21.15.130:0/1466263827 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c003d00 con 0x7f1d9c0ff420 2023-10-11T16:27:33.638 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.630+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c10ac60 con 0x7f1d9c0ff420 2023-10-11T16:27:33.639 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.631+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c067b00 con 0x7f1d9c0ff420 2023-10-11T16:27:33.640 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.633+0000 7f1d8b7fe700 1 -- 172.21.15.130:0/1466263827 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f1d8c004380 con 0x7f1d9c0ff420 2023-10-11T16:27:33.640 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.633+0000 7f1d8b7fe700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d840a11d0 0x7f1d840a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:33.640 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.634+0000 7f1d8b7fe700 1 -- 172.21.15.130:0/1466263827 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f1d8c0cbab0 con 0x7f1d9c0ff420 2023-10-11T16:27:33.640 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.634+0000 7f1d99d9b700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d840a11d0 0x7f1d840a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:33.641 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.634+0000 7f1d99d9b700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d840a11d0 0x7f1d840a3680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000b5c0 tx=0x7f1d90005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:27:33.641 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.636+0000 7f1d8b7fe700 1 -- 172.21.15.130:0/1466263827 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1d8c095f90 con 0x7f1d9c0ff420 2023-10-11T16:27:33.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:33 smithi130 ceph-mon[131859]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:33 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi111 2023-10-11T16:27:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:33 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi111 to 5757M 2023-10-11T16:27:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:33 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:33 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:27:33.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:33 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:33.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:33 smithi130 ceph-mon[131859]: osdmap e47: 7 total, 7 up, 7 in 2023-10-11T16:27:33.833 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:33.832+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d9c19ed30 con 0x7f1d840a11d0 2023-10-11T16:27:33.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:33 smithi111 ceph-mon[98527]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:33 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi111 2023-10-11T16:27:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:33 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi111 to 5757M 2023-10-11T16:27:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:33 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:33 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:27:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:33 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:33 smithi111 ceph-mon[98527]: osdmap e47: 7 total, 7 up, 7 in 2023-10-11T16:27:33.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:33 smithi150 ceph-mon[132217]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-11T16:27:33.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:33 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi111 2023-10-11T16:27:33.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:33 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi111 to 5757M 2023-10-11T16:27:33.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:33 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:33.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:33 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:27:33.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:33 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:33.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:33 smithi150 ceph-mon[132217]: osdmap e47: 7 total, 7 up, 7 in 2023-10-11T16:27:34.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:34.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:34.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:34.913 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:34.913 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:34.913 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:34.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:34.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:34.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:35.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:35 smithi111 ceph-mon[98527]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:35.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:35 smithi111 ceph-mon[98527]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:35.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:35 smithi150 ceph-mon[132217]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:35.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:35 smithi150 ceph-mon[132217]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:36.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:35 smithi130 ceph-mon[131859]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:36.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:35 smithi130 ceph-mon[131859]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:36.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:36 smithi111 ceph-mon[98527]: from='client.? 172.21.15.130:0/1074723677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]: dispatch 2023-10-11T16:27:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:36 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]: dispatch 2023-10-11T16:27:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:36 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]': finished 2023-10-11T16:27:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:36 smithi111 ceph-mon[98527]: osdmap e48: 8 total, 7 up, 8 in 2023-10-11T16:27:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:36 smithi150 ceph-mon[132217]: from='client.? 172.21.15.130:0/1074723677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]: dispatch 2023-10-11T16:27:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:36 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]: dispatch 2023-10-11T16:27:36.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:36 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]': finished 2023-10-11T16:27:36.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:36 smithi150 ceph-mon[132217]: osdmap e48: 8 total, 7 up, 8 in 2023-10-11T16:27:36.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:37.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:36 smithi130 ceph-mon[131859]: from='client.? 172.21.15.130:0/1074723677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]: dispatch 2023-10-11T16:27:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:36 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]: dispatch 2023-10-11T16:27:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:36 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e9ae755-b404-4fbd-ad30-08dee80f0a95"}]': finished 2023-10-11T16:27:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:36 smithi130 ceph-mon[131859]: osdmap e48: 8 total, 7 up, 8 in 2023-10-11T16:27:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:37.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:37 smithi111 ceph-mon[98527]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:37.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:37 smithi111 ceph-mon[98527]: from='client.? 172.21.15.130:0/1088061183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:27:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:37 smithi150 ceph-mon[132217]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:37 smithi150 ceph-mon[132217]: from='client.? 172.21.15.130:0/1088061183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:27:38.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:37 smithi130 ceph-mon[131859]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:37 smithi130 ceph-mon[131859]: from='client.? 172.21.15.130:0/1088061183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:27:39.893 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:39 smithi130 ceph-mon[131859]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:39 smithi150 ceph-mon[132217]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:40.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:39 smithi111 ceph-mon[98527]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:41.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:41 smithi130 ceph-mon[131859]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:41 smithi150 ceph-mon[132217]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:42.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:41 smithi111 ceph-mon[98527]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:42.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:42 smithi150 ceph-mon[132217]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:43.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:42 smithi130 ceph-mon[131859]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:43.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:42 smithi111 ceph-mon[98527]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:43.788 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-11T16:27:43.788 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:43 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:43.788 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:43 smithi130 ceph-mon[131859]: Deploying daemon osd.7 on smithi130 2023-10-11T16:27:43.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-11T16:27:43.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:43 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:43.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:43 smithi150 ceph-mon[132217]: Deploying daemon osd.7 on smithi130 2023-10-11T16:27:44.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-11T16:27:44.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:43 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:44.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:43 smithi111 ceph-mon[98527]: Deploying daemon osd.7 on smithi130 2023-10-11T16:27:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:45 smithi111 ceph-mon[98527]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:45 smithi150 ceph-mon[132217]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:45 smithi130 ceph-mon[131859]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:47.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:47 smithi111 ceph-mon[98527]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:47.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:47.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:47.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:47 smithi150 ceph-mon[132217]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:48.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:47 smithi130 ceph-mon[131859]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:48.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:48.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:48.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:49.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:49 smithi111 ceph-mon[98527]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:49.937 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:49 smithi130 ceph-mon[131859]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:49.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:49 smithi150 ceph-mon[132217]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:50.250 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.249+0000 7f1d8b7fe700 1 -- 172.21.15.130:0/1466263827 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1d9c19ed30 con 0x7f1d840a11d0 2023-10-11T16:27:50.251 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 7 on host 'smithi130' 2023-10-11T16:27:50.253 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d840a11d0 msgr2=0x7f1d840a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:50.253 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d840a11d0 0x7f1d840a3680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000b5c0 tx=0x7f1d90005fd0 comp rx=0 tx=0).stop 2023-10-11T16:27:50.253 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c0ff420 msgr2=0x7f1d9c1962c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:50.253 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c0ff420 0x7f1d9c1962c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00c970 tx=0x7f1d8c00cc80 comp rx=0 tx=0).stop 2023-10-11T16:27:50.254 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 shutdown_connections 2023-10-11T16:27:50.254 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d840a11d0 0x7f1d840a3680 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:50.254 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d9c100fb0 0x7f1d9c19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:50.254 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d9c0ff420 0x7f1d9c1962c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:50.255 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 --2- 172.21.15.130:0/1466263827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d9c100620 0x7f1d9c196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:50.255 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 >> 172.21.15.130:0/1466263827 conn(0x7f1d9c0faa50 msgr2=0x7f1d9c0690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:27:50.255 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 shutdown_connections 2023-10-11T16:27:50.255 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:27:50.252+0000 7f1da0941700 1 -- 172.21.15.130:0/1466263827 wait complete. 2023-10-11T16:27:50.648 DEBUG:teuthology.orchestra.run.smithi130:osd.7> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.7.service 2023-10-11T16:27:50.650 INFO:tasks.cephadm:Deploying osd.8 on smithi150 with /dev/vg_nvme/lv_2... 2023-10-11T16:27:50.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-10-11T16:27:50.679 INFO:journalctl@ceph.osd.7.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:27:50.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:27:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:50.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:50.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.193 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.193 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.194 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:51.194 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:51.195 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:51.197 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.198 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:27:51.198 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:51.199 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:51.199 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.199 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:51.200 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.304 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:27:51.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:51.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:51.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:51.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:27:51.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:51.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:51.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:27:51.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:51.486 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:27:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:27:51.315+0000 7f4f3110f640 -1 osd.7 0 log_to_monitors true 2023-10-11T16:27:52.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:51 smithi150 ceph-mon[132217]: from='osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-11T16:27:52.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:51 smithi150 ceph-mon[132217]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-11T16:27:52.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:51 smithi130 ceph-mon[131859]: from='osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-11T16:27:52.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:51 smithi130 ceph-mon[131859]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-11T16:27:52.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:51 smithi111 ceph-mon[98527]: from='osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-11T16:27:52.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:51 smithi111 ceph-mon[98527]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-11T16:27:52.804 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-10-11T16:27:52.805 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-10-11T16:27:52.805 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-10-11T16:27:52.805 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-10-11T16:27:52.805 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110461 s, 949 MB/s 2023-10-11T16:27:52.806 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-10-11T16:27:53.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:52 smithi150 ceph-mon[132217]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-11T16:27:53.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:52 smithi150 ceph-mon[132217]: osdmap e49: 8 total, 7 up, 8 in 2023-10-11T16:27:53.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:52 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:53.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:52 smithi150 ceph-mon[132217]: from='osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:27:53.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:52 smithi150 ceph-mon[132217]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:27:53.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:52 smithi150 ceph-mon[132217]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:53.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2023-10-11T16:27:53.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:27:52 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:27:52.945+0000 7f4f26f87700 -1 osd.7 0 waiting for initial osdmap 2023-10-11T16:27:53.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:27:52 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:27:52.954+0000 7f4f20d78700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:27:53.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:52 smithi130 ceph-mon[131859]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-11T16:27:53.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:52 smithi130 ceph-mon[131859]: osdmap e49: 8 total, 7 up, 8 in 2023-10-11T16:27:53.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:52 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:53.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:52 smithi130 ceph-mon[131859]: from='osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:27:53.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:52 smithi130 ceph-mon[131859]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:27:53.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:52 smithi130 ceph-mon[131859]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:53.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:52 smithi111 ceph-mon[98527]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-11T16:27:53.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:52 smithi111 ceph-mon[98527]: osdmap e49: 8 total, 7 up, 8 in 2023-10-11T16:27:53.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:52 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:53.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:52 smithi111 ceph-mon[98527]: from='osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:27:53.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:52 smithi111 ceph-mon[98527]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:27:53.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:52 smithi111 ceph-mon[98527]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:53.885 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:27:54.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:27:54.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: osdmap e50: 8 total, 7 up, 8 in 2023-10-11T16:27:54.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:54.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:54.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi130 2023-10-11T16:27:54.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi130 to 5722M 2023-10-11T16:27:54.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:54.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:54.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:54.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:27:54.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:53 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:27:54.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:27:54.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: osdmap e50: 8 total, 7 up, 8 in 2023-10-11T16:27:54.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:54.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:54.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi130 2023-10-11T16:27:54.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi130 to 5722M 2023-10-11T16:27:54.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:54.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:54.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:54.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:27:54.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:53 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:27:54.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:27:54.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: osdmap e50: 8 total, 7 up, 8 in 2023-10-11T16:27:54.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:54.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:54.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi130 2023-10-11T16:27:54.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:27:54.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi130 to 5722M 2023-10-11T16:27:54.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:54.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:27:54.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:54.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:27:54.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:27:54.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:53 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:27:55.084 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 -- 172.21.15.150:0/824122758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c00ff1e0 msgr2=0x7f81c00ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:55.085 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 --2- 172.21.15.150:0/824122758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c00ff1e0 0x7f81c00ff5f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009990 tx=0x7f81b0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:27:55.085 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 -- 172.21.15.150:0/824122758 shutdown_connections 2023-10-11T16:27:55.085 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 --2- 172.21.15.150:0/824122758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81c0100d70 0x7f81c0107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:55.085 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 --2- 172.21.15.150:0/824122758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c00ff1e0 0x7f81c00ff5f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:55.086 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 --2- 172.21.15.150:0/824122758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c01003e0 0x7f81c0100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:55.086 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 -- 172.21.15.150:0/824122758 >> 172.21.15.150:0/824122758 conn(0x7f81c00fa7f0 msgr2=0x7f81c00fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:27:55.086 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 -- 172.21.15.150:0/824122758 shutdown_connections 2023-10-11T16:27:55.086 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.082+0000 7f81c67e5700 1 -- 172.21.15.150:0/824122758 wait complete. 2023-10-11T16:27:55.087 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c67e5700 1 Processor -- start 2023-10-11T16:27:55.087 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c67e5700 1 -- start start 2023-10-11T16:27:55.089 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c67e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00ff1e0 0x7f81c0196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:55.089 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c67e5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81c01003e0 0x7f81c01965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:55.090 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c67e5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c0100d70 0x7f81c019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:55.090 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c67e5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c019dbf0 con 0x7f81c00ff1e0 2023-10-11T16:27:55.090 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c67e5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c019dd30 con 0x7f81c01003e0 2023-10-11T16:27:55.090 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c67e5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c019dec0 con 0x7f81c0100d70 2023-10-11T16:27:55.090 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c4d82700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c0100d70 0x7f81c019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:55.091 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c4d82700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c0100d70 0x7f81c019d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42598/0 (socket says 172.21.15.150:42598) 2023-10-11T16:27:55.091 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.083+0000 7f81c4d82700 1 -- 172.21.15.150:0/3383658466 learned_addr learned my addr 172.21.15.150:0/3383658466 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:27:55.091 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81bf7fe700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81c01003e0 0x7f81c01965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:55.091 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81c4d82700 1 -- 172.21.15.150:0/3383658466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81c01003e0 msgr2=0x7f81c01965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:55.091 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81c4d82700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81c01003e0 0x7f81c01965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:55.092 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81c4d82700 1 -- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00ff1e0 msgr2=0x7f81c0196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:27:55.092 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81c4d82700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00ff1e0 0x7f81c0196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:27:55.092 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81c4d82700 1 -- 172.21.15.150:0/3383658466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b00095f0 con 0x7f81c0100d70 2023-10-11T16:27:55.092 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81bf7fe700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81c01003e0 0x7f81c01965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:27:55.092 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81c4d82700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c0100d70 0x7f81c019d620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f81b800add0 tx=0x7f81b800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:27:55.093 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81bd7fa700 1 -- 172.21.15.150:0/3383658466 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b800f070 con 0x7f81c0100d70 2023-10-11T16:27:55.093 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c010ab30 con 0x7f81c0100d70 2023-10-11T16:27:55.093 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81bd7fa700 1 -- 172.21.15.150:0/3383658466 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81b8007b30 con 0x7f81c0100d70 2023-10-11T16:27:55.093 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81bd7fa700 1 -- 172.21.15.150:0/3383658466 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b80053e0 con 0x7f81c0100d70 2023-10-11T16:27:55.093 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.084+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c010a9a0 con 0x7f81c0100d70 2023-10-11T16:27:55.094 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.085+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c019e550 con 0x7f81c0100d70 2023-10-11T16:27:55.094 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.086+0000 7f81bd7fa700 1 -- 172.21.15.150:0/3383658466 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f81b800c940 con 0x7f81c0100d70 2023-10-11T16:27:55.094 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.087+0000 7f81bd7fa700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81ac0a1150 0x7f81ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:27:55.094 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.087+0000 7f81bd7fa700 1 -- 172.21.15.150:0/3383658466 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f81b80cb5a0 con 0x7f81c0100d70 2023-10-11T16:27:55.094 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.087+0000 7f81bffff700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81ac0a1150 0x7f81ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:27:55.095 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.087+0000 7f81bffff700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81ac0a1150 0x7f81ac0a3600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f81b00098e0 tx=0x7f81b0003680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:27:55.095 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.091+0000 7f81bd7fa700 1 -- 172.21.15.150:0/3383658466 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f81b8095820 con 0x7f81c0100d70 2023-10-11T16:27:55.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:54 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:27:55.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:54 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:27:55.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:54 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:55.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:54 smithi150 ceph-mon[132217]: osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] boot 2023-10-11T16:27:55.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:54 smithi150 ceph-mon[132217]: osdmap e51: 8 total, 8 up, 8 in 2023-10-11T16:27:55.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:54 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:55.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:54 smithi150 ceph-mon[132217]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:55.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:54 smithi150 ceph-mon[132217]: osdmap e52: 8 total, 8 up, 8 in 2023-10-11T16:27:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:27:55.285+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f81c00fb840 con 0x7f81ac0a1150 2023-10-11T16:27:55.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:54 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:27:55.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:54 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:27:55.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:54 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:55.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:54 smithi130 ceph-mon[131859]: osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] boot 2023-10-11T16:27:55.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:54 smithi130 ceph-mon[131859]: osdmap e51: 8 total, 8 up, 8 in 2023-10-11T16:27:55.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:54 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:55.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:54 smithi130 ceph-mon[131859]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:55.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:54 smithi130 ceph-mon[131859]: osdmap e52: 8 total, 8 up, 8 in 2023-10-11T16:27:55.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:54 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:27:55.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:54 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:27:55.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:54 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:55.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:54 smithi111 ceph-mon[98527]: osd.7 [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] boot 2023-10-11T16:27:55.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:54 smithi111 ceph-mon[98527]: osdmap e51: 8 total, 8 up, 8 in 2023-10-11T16:27:55.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:54 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:27:55.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:54 smithi111 ceph-mon[98527]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-11T16:27:55.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:54 smithi111 ceph-mon[98527]: osdmap e52: 8 total, 8 up, 8 in 2023-10-11T16:27:56.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:55 smithi150 ceph-mon[132217]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:56.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:56.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:56.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:55 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:56.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:55 smithi150 ceph-mon[132217]: osdmap e53: 8 total, 8 up, 8 in 2023-10-11T16:27:56.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:55 smithi130 ceph-mon[131859]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:56.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:56.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:56.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:55 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:56.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:55 smithi130 ceph-mon[131859]: osdmap e53: 8 total, 8 up, 8 in 2023-10-11T16:27:56.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:55 smithi111 ceph-mon[98527]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:27:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:27:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:27:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:55 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:27:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:55 smithi111 ceph-mon[98527]: osdmap e53: 8 total, 8 up, 8 in 2023-10-11T16:27:57.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:56 smithi150 ceph-mon[132217]: pgmap v171: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2023-10-11T16:27:57.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:56 smithi130 ceph-mon[131859]: pgmap v171: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2023-10-11T16:27:57.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:56 smithi111 ceph-mon[98527]: pgmap v171: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2023-10-11T16:27:58.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:57 smithi130 ceph-mon[131859]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-10-11T16:27:58.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:57 smithi130 ceph-mon[131859]: osdmap e54: 8 total, 8 up, 8 in 2023-10-11T16:27:58.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:57 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/4147321860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]: dispatch 2023-10-11T16:27:58.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:57 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]: dispatch 2023-10-11T16:27:58.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:57 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]': finished 2023-10-11T16:27:58.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:57 smithi130 ceph-mon[131859]: osdmap e55: 9 total, 8 up, 9 in 2023-10-11T16:27:58.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:57 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:27:58.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:57 smithi111 ceph-mon[98527]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-10-11T16:27:58.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:57 smithi111 ceph-mon[98527]: osdmap e54: 8 total, 8 up, 8 in 2023-10-11T16:27:58.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:57 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/4147321860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]: dispatch 2023-10-11T16:27:58.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:57 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]: dispatch 2023-10-11T16:27:58.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:57 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]': finished 2023-10-11T16:27:58.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:57 smithi111 ceph-mon[98527]: osdmap e55: 9 total, 8 up, 9 in 2023-10-11T16:27:58.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:57 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:27:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:57 smithi150 ceph-mon[132217]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-10-11T16:27:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:57 smithi150 ceph-mon[132217]: osdmap e54: 8 total, 8 up, 8 in 2023-10-11T16:27:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:57 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/4147321860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]: dispatch 2023-10-11T16:27:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:57 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]: dispatch 2023-10-11T16:27:58.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:57 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bfb9857-3e28-481a-8a1e-9f3b9b1ea176"}]': finished 2023-10-11T16:27:58.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:57 smithi150 ceph-mon[132217]: osdmap e55: 9 total, 8 up, 9 in 2023-10-11T16:27:58.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:57 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:27:59.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:58 smithi130 ceph-mon[131859]: pgmap v174: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2023-10-11T16:27:59.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:27:58 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/2544361048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:27:59.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:58 smithi111 ceph-mon[98527]: pgmap v174: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2023-10-11T16:27:59.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:27:58 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/2544361048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:27:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:58 smithi150 ceph-mon[132217]: pgmap v174: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2023-10-11T16:27:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:27:58 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/2544361048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:28:00.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:00 smithi150 ceph-mon[132217]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:00.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:00 smithi150 ceph-mon[132217]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-10-11T16:28:00.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:00 smithi150 ceph-mon[132217]: Cluster is now healthy 2023-10-11T16:28:01.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:00 smithi130 ceph-mon[131859]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:01.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:00 smithi130 ceph-mon[131859]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-10-11T16:28:01.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:00 smithi130 ceph-mon[131859]: Cluster is now healthy 2023-10-11T16:28:01.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:00 smithi111 ceph-mon[98527]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:01.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:00 smithi111 ceph-mon[98527]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-10-11T16:28:01.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:00 smithi111 ceph-mon[98527]: Cluster is now healthy 2023-10-11T16:28:03.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:03 smithi130 ceph-mon[131859]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:03.385 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:03 smithi150 ceph-mon[132217]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:03.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:03 smithi111 ceph-mon[98527]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:05.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:05 smithi150 ceph-mon[132217]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:05.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:05 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-10-11T16:28:05.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:05 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:05.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:05 smithi150 ceph-mon[132217]: Deploying daemon osd.8 on smithi150 2023-10-11T16:28:05.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:05 smithi111 ceph-mon[98527]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:05.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-10-11T16:28:05.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:05 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:05.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:05 smithi111 ceph-mon[98527]: Deploying daemon osd.8 on smithi150 2023-10-11T16:28:05.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:05 smithi130 ceph-mon[131859]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:05.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:05 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-10-11T16:28:05.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:05 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:05.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:05 smithi130 ceph-mon[131859]: Deploying daemon osd.8 on smithi150 2023-10-11T16:28:07.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:07 smithi111 ceph-mon[98527]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:07.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:07 smithi150 ceph-mon[132217]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:07.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:07 smithi130 ceph-mon[131859]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:09.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:09 smithi150 ceph-mon[132217]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:09.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:09 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:09.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:09 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:09.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:09 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:09.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:09 smithi111 ceph-mon[98527]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:09.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:09.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:09.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:09 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:09.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:09 smithi130 ceph-mon[131859]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:09.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:09 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:09.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:09 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:09.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:09 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:11.149 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 8 on host 'smithi150' 2023-10-11T16:28:11.150 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.148+0000 7f81bd7fa700 1 -- 172.21.15.150:0/3383658466 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f81c00fb840 con 0x7f81ac0a1150 2023-10-11T16:28:11.151 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.150+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81ac0a1150 msgr2=0x7f81ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:11.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.150+0000 7f81c67e5700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81ac0a1150 0x7f81ac0a3600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f81b00098e0 tx=0x7f81b0003680 comp rx=0 tx=0).stop 2023-10-11T16:28:11.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c0100d70 msgr2=0x7f81c019d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:11.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c0100d70 0x7f81c019d620 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f81b800add0 tx=0x7f81b800c330 comp rx=0 tx=0).stop 2023-10-11T16:28:11.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 shutdown_connections 2023-10-11T16:28:11.152 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81ac0a1150 0x7f81ac0a3600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:11.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81c0100d70 0x7f81c019d620 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:11.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00ff1e0 0x7f81c0196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:11.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 --2- 172.21.15.150:0/3383658466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81c01003e0 0x7f81c01965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:11.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 >> 172.21.15.150:0/3383658466 conn(0x7f81c00fa7f0 msgr2=0x7f81c00fb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:28:11.153 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 shutdown_connections 2023-10-11T16:28:11.154 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:28:11.151+0000 7f81c67e5700 1 -- 172.21.15.150:0/3383658466 wait complete. 2023-10-11T16:28:11.582 DEBUG:teuthology.orchestra.run.smithi150:osd.8> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.8.service 2023-10-11T16:28:11.585 INFO:tasks.cephadm:Deploying osd.9 on smithi111 with /dev/vg_nvme/lv_1... 2023-10-11T16:28:11.586 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-10-11T16:28:11.617 INFO:journalctl@ceph.osd.8.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:28:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:11.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:11.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:11.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:12.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:12.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:12.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:12.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:12.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:12.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:12.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:12.439 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:28:12.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:28:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:28:12.439+0000 7f2a3d17c640 -1 osd.8 0 log_to_monitors true 2023-10-11T16:28:12.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:12 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:28:12.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:12 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:28:12.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:12 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:28:12.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:12 smithi150 ceph-mon[132217]: from='osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-11T16:28:12.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:12 smithi150 ceph-mon[132217]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-11T16:28:12.992 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:12 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:28:12.992 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:12 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:28:12.993 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:12 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:28:12.993 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:12 smithi111 ceph-mon[98527]: from='osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-11T16:28:12.993 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:12 smithi111 ceph-mon[98527]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-11T16:28:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:12 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:28:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:12 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:28:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:12 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:28:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:12 smithi130 ceph-mon[131859]: from='osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-11T16:28:13.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:12 smithi130 ceph-mon[131859]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-11T16:28:13.649 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:13 smithi150 ceph-mon[132217]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:13.650 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:13 smithi150 ceph-mon[132217]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-10-11T16:28:13.650 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:13 smithi150 ceph-mon[132217]: osdmap e56: 9 total, 8 up, 9 in 2023-10-11T16:28:13.650 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:13 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:13.650 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:13 smithi150 ceph-mon[132217]: from='osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:28:13.651 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:13 smithi150 ceph-mon[132217]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:28:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:13 smithi111 ceph-mon[98527]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:13 smithi111 ceph-mon[98527]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-10-11T16:28:13.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:13 smithi111 ceph-mon[98527]: osdmap e56: 9 total, 8 up, 9 in 2023-10-11T16:28:13.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:13 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:13.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:13 smithi111 ceph-mon[98527]: from='osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:28:13.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:13 smithi111 ceph-mon[98527]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:28:13.952 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:28:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:28:13.651+0000 7f2a327f3700 -1 osd.8 0 waiting for initial osdmap 2023-10-11T16:28:13.952 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:28:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:28:13.665+0000 7f2a2a5e0700 -1 osd.8 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:28:14.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:13 smithi130 ceph-mon[131859]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:14.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:13 smithi130 ceph-mon[131859]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-10-11T16:28:14.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:13 smithi130 ceph-mon[131859]: osdmap e56: 9 total, 8 up, 9 in 2023-10-11T16:28:14.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:13 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:14.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:13 smithi130 ceph-mon[131859]: from='osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:28:14.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:13 smithi130 ceph-mon[131859]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:28:14.309 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-10-11T16:28:14.310 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-10-11T16:28:14.310 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2023-10-11T16:28:14.310 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2023-10-11T16:28:14.310 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0322375 s, 325 MB/s 2023-10-11T16:28:14.311 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2023-10-11T16:28:14.720 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi111:vg_nvme/lv_1 2023-10-11T16:28:14.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:28:14.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: osdmap e57: 9 total, 8 up, 9 in 2023-10-11T16:28:14.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:14.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:14.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi150 2023-10-11T16:28:14.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:14.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:14.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:14.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:14.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:14.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi150 to 5722M 2023-10-11T16:28:14.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:14.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:14.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:14.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:28:14.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:14.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:28:14.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:28:14.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:14.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] boot 2023-10-11T16:28:14.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: osdmap e58: 9 total, 9 up, 9 in 2023-10-11T16:28:14.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:15.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:28:15.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: osdmap e57: 9 total, 8 up, 9 in 2023-10-11T16:28:15.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:15.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:15.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi150 2023-10-11T16:28:15.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:15.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:15.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:15.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:15.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:15.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi150 to 5722M 2023-10-11T16:28:15.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:15.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:15.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:15.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:28:15.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:15.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:28:15.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:28:15.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:15.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] boot 2023-10-11T16:28:15.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: osdmap e58: 9 total, 9 up, 9 in 2023-10-11T16:28:15.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:15.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:28:15.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: osdmap e57: 9 total, 8 up, 9 in 2023-10-11T16:28:15.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:15.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:15.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi150 2023-10-11T16:28:15.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:15.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:15.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:15.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:15.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:15.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi150 to 5722M 2023-10-11T16:28:15.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:15.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:15.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:15.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:28:15.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:15.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:28:15.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:28:15.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-11T16:28:15.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: osd.8 [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] boot 2023-10-11T16:28:15.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: osdmap e58: 9 total, 9 up, 9 in 2023-10-11T16:28:15.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:28:15.567 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:28:16.100 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:15 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:28:16.100 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:15 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:28:16.100 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:15 smithi111 ceph-mon[98527]: osdmap e59: 9 total, 9 up, 9 in 2023-10-11T16:28:16.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:15 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:28:16.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:15 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:28:16.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:15 smithi150 ceph-mon[132217]: osdmap e59: 9 total, 9 up, 9 in 2023-10-11T16:28:16.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:15 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:28:16.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:15 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:28:16.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:15 smithi130 ceph-mon[131859]: osdmap e59: 9 total, 9 up, 9 in 2023-10-11T16:28:16.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 -- 172.21.15.111:0/336104400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 msgr2=0x7f7c34100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:16.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 --2- 172.21.15.111:0/336104400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 0x7f7c34100700 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009990 tx=0x7f7c30009ca0 comp rx=0 tx=0).stop 2023-10-11T16:28:16.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 -- 172.21.15.111:0/336104400 shutdown_connections 2023-10-11T16:28:16.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 --2- 172.21.15.111:0/336104400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c34101e80 0x7f7c34108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:16.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 --2- 172.21.15.111:0/336104400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 0x7f7c34100700 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:16.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 --2- 172.21.15.111:0/336104400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c341014f0 0x7f7c34101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:16.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 -- 172.21.15.111:0/336104400 >> 172.21.15.111:0/336104400 conn(0x7f7c340fb900 msgr2=0x7f7c340fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:28:16.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 -- 172.21.15.111:0/336104400 shutdown_connections 2023-10-11T16:28:16.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.972+0000 7f7c3c043700 1 -- 172.21.15.111:0/336104400 wait complete. 2023-10-11T16:28:16.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.973+0000 7f7c3c043700 1 Processor -- start 2023-10-11T16:28:16.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.973+0000 7f7c3c043700 1 -- start start 2023-10-11T16:28:16.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c3c043700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 0x7f7c34198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:28:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c3c043700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c341014f0 0x7f7c341987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:28:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c3c043700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c34101e80 0x7f7c3419f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:28:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c3c043700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3419fe10 con 0x7f7c341002f0 2023-10-11T16:28:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c3c043700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3419ff50 con 0x7f7c341014f0 2023-10-11T16:28:16.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c3c043700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c341a00e0 con 0x7f7c34101e80 2023-10-11T16:28:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c39ddf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 0x7f7c34198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:28:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c39ddf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 0x7f7c34198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59520/0 (socket says 172.21.15.111:59520) 2023-10-11T16:28:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c39ddf700 1 -- 172.21.15.111:0/1401706451 learned_addr learned my addr 172.21.15.111:0/1401706451 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:28:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c395de700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c341014f0 0x7f7c341987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:28:16.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c3a5e0700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c34101e80 0x7f7c3419f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:28:16.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c395de700 1 -- 172.21.15.111:0/1401706451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c34101e80 msgr2=0x7f7c3419f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:16.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c395de700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c34101e80 0x7f7c3419f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:16.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c395de700 1 -- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 msgr2=0x7f7c34198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:16.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c395de700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 0x7f7c34198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:16.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c395de700 1 -- 172.21.15.111:0/1401706451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c300095f0 con 0x7f7c341014f0 2023-10-11T16:28:16.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.974+0000 7f7c39ddf700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 0x7f7c34198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:28:16.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.975+0000 7f7c395de700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c341014f0 0x7f7c341987c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400e9f0 tx=0x7f7c2400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:28:16.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.975+0000 7f7c2affd700 1 -- 172.21.15.111:0/1401706451 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2400c9c0 con 0x7f7c341014f0 2023-10-11T16:28:16.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.975+0000 7f7c2affd700 1 -- 172.21.15.111:0/1401706451 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c24010430 con 0x7f7c341014f0 2023-10-11T16:28:16.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.975+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3410bc90 con 0x7f7c341014f0 2023-10-11T16:28:16.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.975+0000 7f7c2affd700 1 -- 172.21.15.111:0/1401706451 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24018680 con 0x7f7c341014f0 2023-10-11T16:28:16.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.975+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3410bb00 con 0x7f7c341014f0 2023-10-11T16:28:16.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.976+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c34066ea0 con 0x7f7c341014f0 2023-10-11T16:28:16.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.978+0000 7f7c2affd700 1 -- 172.21.15.111:0/1401706451 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7c24010860 con 0x7f7c341014f0 2023-10-11T16:28:16.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.979+0000 7f7c2affd700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7c200a1150 0x7f7c200a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:28:16.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.979+0000 7f7c2affd700 1 -- 172.21.15.111:0/1401706451 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f7c24014070 con 0x7f7c341014f0 2023-10-11T16:28:16.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.980+0000 7f7c39ddf700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7c200a1150 0x7f7c200a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:28:16.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.980+0000 7f7c39ddf700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7c200a1150 0x7f7c200a3600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b2e0 tx=0x7f7c30005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:28:16.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:16.982+0000 7f7c2affd700 1 -- 172.21.15.111:0/1401706451 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7c240960f0 con 0x7f7c341014f0 2023-10-11T16:28:17.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:17.189+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c341a0ca0 con 0x7f7c200a1150 2023-10-11T16:28:17.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:17 smithi150 ceph-mon[132217]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:17.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:17 smithi150 ceph-mon[132217]: osdmap e60: 9 total, 9 up, 9 in 2023-10-11T16:28:17.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:28:17.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:28:17.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:18.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:17 smithi130 ceph-mon[131859]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:17 smithi130 ceph-mon[131859]: osdmap e60: 9 total, 9 up, 9 in 2023-10-11T16:28:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:28:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:28:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:18.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:17 smithi111 ceph-mon[98527]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:17 smithi111 ceph-mon[98527]: osdmap e60: 9 total, 9 up, 9 in 2023-10-11T16:28:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:28:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:28:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:18.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:18 smithi150 ceph-mon[132217]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:28:18.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:18 smithi150 ceph-mon[132217]: osdmap e61: 9 total, 9 up, 9 in 2023-10-11T16:28:19.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:18 smithi130 ceph-mon[131859]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:28:19.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:18 smithi130 ceph-mon[131859]: osdmap e61: 9 total, 9 up, 9 in 2023-10-11T16:28:19.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:18 smithi111 ceph-mon[98527]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:28:19.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:18 smithi111 ceph-mon[98527]: osdmap e61: 9 total, 9 up, 9 in 2023-10-11T16:28:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:19 smithi150 ceph-mon[132217]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:19 smithi150 ceph-mon[132217]: osdmap e62: 9 total, 9 up, 9 in 2023-10-11T16:28:20.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:19 smithi130 ceph-mon[131859]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:20.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:19 smithi130 ceph-mon[131859]: osdmap e62: 9 total, 9 up, 9 in 2023-10-11T16:28:20.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:19 smithi111 ceph-mon[98527]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:20.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:19 smithi111 ceph-mon[98527]: osdmap e62: 9 total, 9 up, 9 in 2023-10-11T16:28:20.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:20 smithi150 ceph-mon[132217]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-10-11T16:28:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:20 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1337783268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48901f71-e122-4d76-b262-e0f1d8f18ede"}]: dispatch 2023-10-11T16:28:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:20 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1337783268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48901f71-e122-4d76-b262-e0f1d8f18ede"}]': finished 2023-10-11T16:28:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:20 smithi150 ceph-mon[132217]: osdmap e63: 10 total, 9 up, 10 in 2023-10-11T16:28:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:21.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:20 smithi130 ceph-mon[131859]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-10-11T16:28:21.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:20 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1337783268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48901f71-e122-4d76-b262-e0f1d8f18ede"}]: dispatch 2023-10-11T16:28:21.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:20 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1337783268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48901f71-e122-4d76-b262-e0f1d8f18ede"}]': finished 2023-10-11T16:28:21.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:20 smithi130 ceph-mon[131859]: osdmap e63: 10 total, 9 up, 10 in 2023-10-11T16:28:21.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:20 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:21.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:20 smithi111 ceph-mon[98527]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 131 KiB/s, 0 objects/s recovering 2023-10-11T16:28:21.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:20 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1337783268' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48901f71-e122-4d76-b262-e0f1d8f18ede"}]: dispatch 2023-10-11T16:28:21.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:20 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1337783268' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48901f71-e122-4d76-b262-e0f1d8f18ede"}]': finished 2023-10-11T16:28:21.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:20 smithi111 ceph-mon[98527]: osdmap e63: 10 total, 9 up, 10 in 2023-10-11T16:28:21.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:20 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:21.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:21 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2459514579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:28:22.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:21 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2459514579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:28:22.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:21 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2459514579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:28:22.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:22 smithi150 ceph-mon[132217]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-10-11T16:28:23.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:22 smithi130 ceph-mon[131859]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-10-11T16:28:23.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:22 smithi111 ceph-mon[98527]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-10-11T16:28:25.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:25 smithi111 ceph-mon[98527]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-10-11T16:28:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:25 smithi150 ceph-mon[132217]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-10-11T16:28:26.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:25 smithi130 ceph-mon[131859]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-10-11T16:28:27.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:27 smithi150 ceph-mon[132217]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-11T16:28:28.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:27 smithi130 ceph-mon[131859]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-11T16:28:28.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:27 smithi111 ceph-mon[98527]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-11T16:28:28.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:28 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-10-11T16:28:28.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:28 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:29.014 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:28 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-10-11T16:28:29.014 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:28 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:29.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-10-11T16:28:29.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:28 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:29.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:29 smithi150 ceph-mon[132217]: Deploying daemon osd.9 on smithi111 2023-10-11T16:28:29.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:29 smithi150 ceph-mon[132217]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-10-11T16:28:30.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:29 smithi130 ceph-mon[131859]: Deploying daemon osd.9 on smithi111 2023-10-11T16:28:30.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:29 smithi130 ceph-mon[131859]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-10-11T16:28:30.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:29 smithi111 ceph-mon[98527]: Deploying daemon osd.9 on smithi111 2023-10-11T16:28:30.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:29 smithi111 ceph-mon[98527]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-10-11T16:28:30.815 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:30 smithi111 ceph-mon[98527]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:30.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:30 smithi150 ceph-mon[132217]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:31.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:30 smithi130 ceph-mon[131859]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:33.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:33 smithi111 ceph-mon[98527]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:33.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:33 smithi150 ceph-mon[132217]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:33.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:33 smithi130 ceph-mon[131859]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:34.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:34.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:34.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:34 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:34.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:34.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:34.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:34 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:34.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:34.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:34.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:34 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:35.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:35 smithi111 ceph-mon[98527]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:35.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:35 smithi150 ceph-mon[132217]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:36.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:35 smithi130 ceph-mon[131859]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:37.654 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 9 on host 'smithi111' 2023-10-11T16:28:37.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.651+0000 7f7c2affd700 1 -- 172.21.15.111:0/1401706451 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7c341a0ca0 con 0x7f7c200a1150 2023-10-11T16:28:37.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7c200a1150 msgr2=0x7f7c200a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:37.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7c200a1150 0x7f7c200a3600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000b2e0 tx=0x7f7c30005fb0 comp rx=0 tx=0).stop 2023-10-11T16:28:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c341014f0 msgr2=0x7f7c341987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c341014f0 0x7f7c341987c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400e9f0 tx=0x7f7c2400ed00 comp rx=0 tx=0).stop 2023-10-11T16:28:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 shutdown_connections 2023-10-11T16:28:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7c200a1150 0x7f7c200a3600 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:37.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c34101e80 0x7f7c3419f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:37.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c341002f0 0x7f7c34198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:37.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 --2- 172.21.15.111:0/1401706451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c341014f0 0x7f7c341987c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:37.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.654+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 >> 172.21.15.111:0/1401706451 conn(0x7f7c340fb900 msgr2=0x7f7c340fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:28:37.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.655+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 shutdown_connections 2023-10-11T16:28:37.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:28:37.655+0000 7f7c3c043700 1 -- 172.21.15.111:0/1401706451 wait complete. 2023-10-11T16:28:37.809 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:37 smithi111 ceph-mon[98527]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:37.809 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:37.809 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:37.810 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:37.810 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:37.810 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:37.810 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:37.810 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:37 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:37 smithi150 ceph-mon[132217]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:37 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:37 smithi130 ceph-mon[131859]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:37 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:38.107 DEBUG:teuthology.orchestra.run.smithi111:osd.9> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.9.service 2023-10-11T16:28:38.112 INFO:tasks.cephadm:Deploying osd.10 on smithi130 with /dev/vg_nvme/lv_1... 2023-10-11T16:28:38.112 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-10-11T16:28:38.188 INFO:journalctl@ceph.osd.9.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:28:38.760 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:28:38.929 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:38 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:28:38.947 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:38 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:28:38.947 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:38 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:28:38.947 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:38.947 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:38 smithi111 ceph-mon[98527]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-10-11T16:28:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:38 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:28:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:38 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:28:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:38 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:28:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:38.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:38 smithi150 ceph-mon[132217]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-10-11T16:28:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:38 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:28:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:38 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:28:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:38 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:28:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:38 smithi130 ceph-mon[131859]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-10-11T16:28:39.840 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:39 smithi111 ceph-mon[98527]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:39.840 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:39 smithi111 ceph-mon[98527]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-10-11T16:28:39.840 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:39 smithi111 ceph-mon[98527]: osdmap e64: 10 total, 9 up, 10 in 2023-10-11T16:28:39.841 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:39 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:39.841 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:39 smithi111 ceph-mon[98527]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:28:39.841 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:28:39 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:28:39.668+0000 7fb769245700 -1 osd.9 0 waiting for initial osdmap 2023-10-11T16:28:39.841 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:28:39 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:28:39.684+0000 7fb764839700 -1 osd.9 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:28:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:39 smithi150 ceph-mon[132217]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:39 smithi150 ceph-mon[132217]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-10-11T16:28:39.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:39 smithi150 ceph-mon[132217]: osdmap e64: 10 total, 9 up, 10 in 2023-10-11T16:28:39.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:39 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:39.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:39 smithi150 ceph-mon[132217]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:28:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:39 smithi130 ceph-mon[131859]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:39 smithi130 ceph-mon[131859]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-10-11T16:28:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:39 smithi130 ceph-mon[131859]: osdmap e64: 10 total, 9 up, 10 in 2023-10-11T16:28:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:39 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:40.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:39 smithi130 ceph-mon[131859]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2023-10-11T16:28:40.272 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-10-11T16:28:40.272 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-10-11T16:28:40.272 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2023-10-11T16:28:40.272 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2023-10-11T16:28:40.273 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114381 s, 917 MB/s 2023-10-11T16:28:40.273 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2023-10-11T16:28:40.599 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi130:vg_nvme/lv_1 2023-10-11T16:28:40.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:40 smithi130 ceph-mon[131859]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:28:40.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:40 smithi130 ceph-mon[131859]: osdmap e65: 10 total, 9 up, 10 in 2023-10-11T16:28:40.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:40.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:40.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:40 smithi130 ceph-mon[131859]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:40.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:40 smithi111 ceph-mon[98527]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:28:40.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:40 smithi111 ceph-mon[98527]: osdmap e65: 10 total, 9 up, 10 in 2023-10-11T16:28:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:40.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:40 smithi111 ceph-mon[98527]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:40.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:40 smithi150 ceph-mon[132217]: from='osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2023-10-11T16:28:40.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:40 smithi150 ceph-mon[132217]: osdmap e65: 10 total, 9 up, 10 in 2023-10-11T16:28:40.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:40.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:40.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:40 smithi150 ceph-mon[132217]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-10-11T16:28:41.242 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:28:41.754 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:28:41.755 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:28:41.755 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] boot 2023-10-11T16:28:41.755 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: osdmap e66: 10 total, 10 up, 10 in 2023-10-11T16:28:41.755 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:41.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi111 2023-10-11T16:28:41.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:41.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:41.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:41.756 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:41.757 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:41.757 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:41.757 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi111 to 4317M 2023-10-11T16:28:41.757 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:41.757 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:41.757 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:41.758 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:28:41.758 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:41.758 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:28:41.758 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:41 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:28:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:28:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:28:41.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] boot 2023-10-11T16:28:41.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: osdmap e66: 10 total, 10 up, 10 in 2023-10-11T16:28:41.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:41.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi111 2023-10-11T16:28:41.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:41.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:41.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:41.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:41.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:41.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:41.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi111 to 4317M 2023-10-11T16:28:41.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:41.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:41.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:41.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:28:41.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:41.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:28:41.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:41 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:28:42.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:28:42.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:28:42.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: osd.9 [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] boot 2023-10-11T16:28:42.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: osdmap e66: 10 total, 10 up, 10 in 2023-10-11T16:28:42.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:28:42.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi111 2023-10-11T16:28:42.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:42.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:42.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:42.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:42.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:42.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:28:42.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi111 to 4317M 2023-10-11T16:28:42.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:42.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:42.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:28:42.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:28:42.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:42.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:28:42.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:41 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:28:42.416 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 -- 172.21.15.130:0/539070622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3601003e0 msgr2=0x7ff360100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:42.417 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 --2- 172.21.15.130:0/539070622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3601003e0 0x7ff360100830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009990 tx=0x7ff35c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:28:42.417 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 -- 172.21.15.130:0/539070622 shutdown_connections 2023-10-11T16:28:42.417 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 --2- 172.21.15.130:0/539070622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff360100d70 0x7ff360107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:42.417 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 --2- 172.21.15.130:0/539070622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3600ff1e0 0x7ff3600ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:42.418 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 --2- 172.21.15.130:0/539070622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3601003e0 0x7ff360100830 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:42.418 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 -- 172.21.15.130:0/539070622 >> 172.21.15.130:0/539070622 conn(0x7ff3600fa7f0 msgr2=0x7ff3600fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:28:42.418 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 -- 172.21.15.130:0/539070622 shutdown_connections 2023-10-11T16:28:42.418 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.415+0000 7ff367acf700 1 -- 172.21.15.130:0/539070622 wait complete. 2023-10-11T16:28:42.421 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.416+0000 7ff367acf700 1 Processor -- start 2023-10-11T16:28:42.421 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.416+0000 7ff367acf700 1 -- start start 2023-10-11T16:28:42.421 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.416+0000 7ff367acf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3600ff1e0 0x7ff360069130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:28:42.422 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.416+0000 7ff367acf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3601003e0 0x7ff36019aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:28:42.422 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.416+0000 7ff367acf700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff360100d70 0x7ff36019d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:28:42.422 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.416+0000 7ff367acf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36019dca0 con 0x7ff3600ff1e0 2023-10-11T16:28:42.422 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.416+0000 7ff367acf700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36019de10 con 0x7ff360100d70 2023-10-11T16:28:42.423 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff367acf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36019dfd0 con 0x7ff3601003e0 2023-10-11T16:28:42.423 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff360100d70 0x7ff36019d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:28:42.424 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff360100d70 0x7ff36019d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:43700/0 (socket says 172.21.15.130:43700) 2023-10-11T16:28:42.424 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 -- 172.21.15.130:0/572479729 learned_addr learned my addr 172.21.15.130:0/572479729 (peer_addr_for_me v2:172.21.15.130:0/0) 2023-10-11T16:28:42.424 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36506a700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3601003e0 0x7ff36019aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:28:42.424 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36586b700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3600ff1e0 0x7ff360069130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:28:42.424 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 -- 172.21.15.130:0/572479729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3601003e0 msgr2=0x7ff36019aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:42.425 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3601003e0 0x7ff36019aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:42.425 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 -- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3600ff1e0 msgr2=0x7ff360069130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:42.425 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3600ff1e0 0x7ff360069130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:42.425 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 -- 172.21.15.130:0/572479729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0095f0 con 0x7ff360100d70 2023-10-11T16:28:42.426 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff36606c700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff360100d70 0x7ff36019d500 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff35800b970 tx=0x7ff35800bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:28:42.426 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff356ffd700 1 -- 172.21.15.130:0/572479729 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35800c9b0 con 0x7ff360100d70 2023-10-11T16:28:42.426 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff356ffd700 1 -- 172.21.15.130:0/572479729 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff358014440 con 0x7ff360100d70 2023-10-11T16:28:42.426 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36010ac50 con 0x7ff360100d70 2023-10-11T16:28:42.426 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff356ffd700 1 -- 172.21.15.130:0/572479729 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3580075d0 con 0x7ff360100d70 2023-10-11T16:28:42.427 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.417+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36010aac0 con 0x7ff360100d70 2023-10-11T16:28:42.427 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.418+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36019e740 con 0x7ff360100d70 2023-10-11T16:28:42.427 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.419+0000 7ff356ffd700 1 -- 172.21.15.130:0/572479729 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ff3580145b0 con 0x7ff360100d70 2023-10-11T16:28:42.427 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.420+0000 7ff356ffd700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff34c0a11d0 0x7ff34c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:28:42.427 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.420+0000 7ff356ffd700 1 -- 172.21.15.130:0/572479729 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6261+0+0 (secure 0 0 0) 0x7ff358006e40 con 0x7ff360100d70 2023-10-11T16:28:42.428 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.420+0000 7ff36586b700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff34c0a11d0 0x7ff34c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:28:42.428 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.421+0000 7ff36586b700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff34c0a11d0 0x7ff34c0a3680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff350006fd0 tx=0x7ff350008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:28:42.429 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.424+0000 7ff356ffd700 1 -- 172.21.15.130:0/572479729 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff358095c70 con 0x7ff360100d70 2023-10-11T16:28:42.618 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:42.618+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff36019ecf0 con 0x7ff34c0a11d0 2023-10-11T16:28:42.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:42 smithi150 ceph-mon[132217]: osdmap e67: 10 total, 10 up, 10 in 2023-10-11T16:28:42.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:42 smithi150 ceph-mon[132217]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:42.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:42 smithi150 ceph-mon[132217]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:28:42.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:28:42.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:28:42.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:43.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:42 smithi130 ceph-mon[131859]: osdmap e67: 10 total, 10 up, 10 in 2023-10-11T16:28:43.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:42 smithi130 ceph-mon[131859]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:43.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:42 smithi130 ceph-mon[131859]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:28:43.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:28:43.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:28:43.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:43.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:42 smithi111 ceph-mon[98527]: osdmap e67: 10 total, 10 up, 10 in 2023-10-11T16:28:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:42 smithi111 ceph-mon[98527]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:42 smithi111 ceph-mon[98527]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:28:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:28:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:28:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:45 smithi150 ceph-mon[132217]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:45 smithi150 ceph-mon[132217]: from='client.? 172.21.15.130:0/238791573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]: dispatch 2023-10-11T16:28:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:45 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]: dispatch 2023-10-11T16:28:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:45 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]': finished 2023-10-11T16:28:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:45 smithi150 ceph-mon[132217]: osdmap e68: 11 total, 10 up, 11 in 2023-10-11T16:28:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:45 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:28:46.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:45 smithi130 ceph-mon[131859]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:45 smithi130 ceph-mon[131859]: from='client.? 172.21.15.130:0/238791573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]: dispatch 2023-10-11T16:28:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:45 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]: dispatch 2023-10-11T16:28:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:45 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]': finished 2023-10-11T16:28:46.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:45 smithi130 ceph-mon[131859]: osdmap e68: 11 total, 10 up, 11 in 2023-10-11T16:28:46.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:45 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:28:46.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:45 smithi111 ceph-mon[98527]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:45 smithi111 ceph-mon[98527]: from='client.? 172.21.15.130:0/238791573' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]: dispatch 2023-10-11T16:28:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:45 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]: dispatch 2023-10-11T16:28:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:45 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13ec3960-3990-4c4e-a67f-cafbd49e83eb"}]': finished 2023-10-11T16:28:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:45 smithi111 ceph-mon[98527]: osdmap e68: 11 total, 10 up, 11 in 2023-10-11T16:28:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:45 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:28:46.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:46 smithi150 ceph-mon[132217]: from='client.? 172.21.15.130:0/4155262420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:28:47.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:46 smithi130 ceph-mon[131859]: from='client.? 172.21.15.130:0/4155262420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:28:47.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:46 smithi111 ceph-mon[98527]: from='client.? 172.21.15.130:0/4155262420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:28:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:47 smithi150 ceph-mon[132217]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:48.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:47 smithi130 ceph-mon[131859]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:48.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:47 smithi111 ceph-mon[98527]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:48.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:48 smithi150 ceph-mon[132217]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:49.075 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:48 smithi130 ceph-mon[131859]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:49.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:48 smithi111 ceph-mon[98527]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:51.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:51 smithi130 ceph-mon[131859]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:51.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-10-11T16:28:51.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:51 smithi150 ceph-mon[132217]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-10-11T16:28:51.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:52.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:51 smithi111 ceph-mon[98527]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:52.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-10-11T16:28:52.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:28:52.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:52 smithi130 ceph-mon[131859]: Deploying daemon osd.10 on smithi130 2023-10-11T16:28:52.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:52 smithi150 ceph-mon[132217]: Deploying daemon osd.10 on smithi130 2023-10-11T16:28:53.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:52 smithi111 ceph-mon[98527]: Deploying daemon osd.10 on smithi130 2023-10-11T16:28:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:53 smithi150 ceph-mon[132217]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:53.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:53 smithi130 ceph-mon[131859]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:54.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:53 smithi111 ceph-mon[98527]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:55.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:55 smithi130 ceph-mon[131859]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:55 smithi150 ceph-mon[132217]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:56.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:55 smithi111 ceph-mon[98527]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:57.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:57.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:57.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:57.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:56 smithi130 ceph-mon[131859]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:57.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:57.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:57.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:57.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:56 smithi150 ceph-mon[132217]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:57.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:57.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:28:57.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:57.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:56 smithi111 ceph-mon[98527]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:58.730 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 10 on host 'smithi130' 2023-10-11T16:28:58.731 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.729+0000 7ff356ffd700 1 -- 172.21.15.130:0/572479729 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ff36019ecf0 con 0x7ff34c0a11d0 2023-10-11T16:28:58.733 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff34c0a11d0 msgr2=0x7ff34c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:58.733 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff34c0a11d0 0x7ff34c0a3680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff350006fd0 tx=0x7ff350008040 comp rx=0 tx=0).stop 2023-10-11T16:28:58.733 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff360100d70 msgr2=0x7ff36019d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:28:58.733 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff360100d70 0x7ff36019d500 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff35800b970 tx=0x7ff35800bd30 comp rx=0 tx=0).stop 2023-10-11T16:28:58.734 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 shutdown_connections 2023-10-11T16:28:58.734 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff34c0a11d0 0x7ff34c0a3680 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:58.734 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff360100d70 0x7ff36019d500 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:58.734 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3600ff1e0 0x7ff360069130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:58.734 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 --2- 172.21.15.130:0/572479729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3601003e0 0x7ff36019aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:28:58.735 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 >> 172.21.15.130:0/572479729 conn(0x7ff3600fa7f0 msgr2=0x7ff3600fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:28:58.735 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 shutdown_connections 2023-10-11T16:28:58.735 INFO:teuthology.orchestra.run.smithi130.stderr:2023-10-11T16:28:58.732+0000 7ff367acf700 1 -- 172.21.15.130:0/572479729 wait complete. 2023-10-11T16:28:59.152 DEBUG:teuthology.orchestra.run.smithi130:osd.10> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.10.service 2023-10-11T16:28:59.154 INFO:tasks.cephadm:Deploying osd.11 on smithi150 with /dev/vg_nvme/lv_1... 2023-10-11T16:28:59.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-10-11T16:28:59.186 INFO:journalctl@ceph.osd.10.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:28:59.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:59 smithi130 ceph-mon[131859]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:59.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:28:59 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:59 smithi111 ceph-mon[98527]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:59.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:28:59 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:59 smithi150 ceph-mon[132217]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:28:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:28:59 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:28:59.805 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:29:00.034 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:28:59 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:28:59.873+0000 7facceaae640 -1 osd.10 0 log_to_monitors true 2023-10-11T16:29:00.918 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:00 smithi150 ceph-mon[132217]: from='osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-11T16:29:00.919 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:00 smithi150 ceph-mon[132217]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-11T16:29:01.066 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:00 smithi130 ceph-mon[131859]: from='osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-11T16:29:01.066 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:00 smithi130 ceph-mon[131859]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-11T16:29:01.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:00 smithi111 ceph-mon[98527]: from='osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-11T16:29:01.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:00 smithi111 ceph-mon[98527]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-11T16:29:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-10-11T16:29:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-10-11T16:29:01.333 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2023-10-11T16:29:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2023-10-11T16:29:01.334 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110253 s, 951 MB/s 2023-10-11T16:29:01.334 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2023-10-11T16:29:01.749 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi150:vg_nvme/lv_1 2023-10-11T16:29:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:29:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-10-11T16:29:01.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: osdmap e69: 11 total, 10 up, 11 in 2023-10-11T16:29:01.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:01.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:29:01.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:29:01.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:01.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:01.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:01.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:01.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:01.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:01.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.081 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:29:01.668+0000 7facc4125700 -1 osd.10 0 waiting for initial osdmap 2023-10-11T16:29:02.082 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:29:01.683+0000 7facbd715700 -1 osd.10 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:29:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:29:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-10-11T16:29:02.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: osdmap e69: 11 total, 10 up, 11 in 2023-10-11T16:29:02.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:02.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:29:02.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:29:02.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:02.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:02.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:02.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:02.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:02.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:02.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:29:02.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-10-11T16:29:02.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: osdmap e69: 11 total, 10 up, 11 in 2023-10-11T16:29:02.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:02.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:29:02.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-10-11T16:29:02.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:02.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:02.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:02.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:02.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:02.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:02.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:02.398 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:29:02.895 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi130 2023-10-11T16:29:02.896 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi130 to 4292M 2023-10-11T16:29:02.896 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:02.896 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:29:02.896 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:02.896 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:29:02.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: osdmap e70: 11 total, 10 up, 11 in 2023-10-11T16:29:02.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:02.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:02.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:02.897 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:02.898 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:02.898 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:02 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi130 2023-10-11T16:29:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi130 to 4292M 2023-10-11T16:29:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:29:03.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:03.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:29:03.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: osdmap e70: 11 total, 10 up, 11 in 2023-10-11T16:29:03.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:03.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:03.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:03.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:03.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:03.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:02 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi130 2023-10-11T16:29:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi130 to 4292M 2023-10-11T16:29:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:29:03.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:03.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-10-11T16:29:03.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: osdmap e70: 11 total, 10 up, 11 in 2023-10-11T16:29:03.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:03.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:03.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:03.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:03.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:03.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:02 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:03.551 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 -- 172.21.15.150:0/1285000202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940ff1e0 msgr2=0x7f7e940ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/1285000202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940ff1e0 0x7f7e940ff5f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009990 tx=0x7f7e84009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 -- 172.21.15.150:0/1285000202 shutdown_connections 2023-10-11T16:29:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/1285000202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e94100d70 0x7f7e94107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:03.552 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/1285000202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940ff1e0 0x7f7e940ff5f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/1285000202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e941003e0 0x7f7e94100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 -- 172.21.15.150:0/1285000202 >> 172.21.15.150:0/1285000202 conn(0x7f7e940fa7f0 msgr2=0x7f7e940fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 -- 172.21.15.150:0/1285000202 shutdown_connections 2023-10-11T16:29:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.549+0000 7f7e9c73b700 1 -- 172.21.15.150:0/1285000202 wait complete. 2023-10-11T16:29:03.553 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9c73b700 1 Processor -- start 2023-10-11T16:29:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9c73b700 1 -- start start 2023-10-11T16:29:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9c73b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e940ff1e0 0x7f7e94069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9c73b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e941003e0 0x7f7e9419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:03.554 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9c73b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94100d70 0x7f7e9419d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9c73b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9419da20 con 0x7f7e94100d70 2023-10-11T16:29:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9c73b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9419db90 con 0x7f7e940ff1e0 2023-10-11T16:29:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9c73b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9419dd50 con 0x7f7e941003e0 2023-10-11T16:29:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9a4d7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e940ff1e0 0x7f7e94069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9a4d7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e940ff1e0 0x7f7e94069970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.150:38686/0 (socket says 172.21.15.150:38686) 2023-10-11T16:29:03.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9a4d7700 1 -- 172.21.15.150:0/256110732 learned_addr learned my addr 172.21.15.150:0/256110732 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:29:03.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9acd8700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94100d70 0x7f7e9419d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:03.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e99cd6700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e941003e0 0x7f7e9419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:03.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9a4d7700 1 -- 172.21.15.150:0/256110732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e941003e0 msgr2=0x7f7e9419ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:03.556 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9a4d7700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e941003e0 0x7f7e9419ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:03.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9a4d7700 1 -- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94100d70 msgr2=0x7f7e9419d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:03.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9a4d7700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94100d70 0x7f7e9419d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:03.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.550+0000 7f7e9a4d7700 1 -- 172.21.15.150:0/256110732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e840095f0 con 0x7f7e940ff1e0 2023-10-11T16:29:03.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.551+0000 7f7e9a4d7700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e940ff1e0 0x7f7e94069970 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7e84000c00 tx=0x7f7e8400f6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:03.557 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.551+0000 7f7e8b7fe700 1 -- 172.21.15.150:0/256110732 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8401d070 con 0x7f7e940ff1e0 2023-10-11T16:29:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.551+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9410ab30 con 0x7f7e940ff1e0 2023-10-11T16:29:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.551+0000 7f7e8b7fe700 1 -- 172.21.15.150:0/256110732 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e84022470 con 0x7f7e940ff1e0 2023-10-11T16:29:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.551+0000 7f7e8b7fe700 1 -- 172.21.15.150:0/256110732 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e84017650 con 0x7f7e940ff1e0 2023-10-11T16:29:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.551+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9410a9a0 con 0x7f7e940ff1e0 2023-10-11T16:29:03.559 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.552+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e94106680 con 0x7f7e940ff1e0 2023-10-11T16:29:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.553+0000 7f7e8b7fe700 1 -- 172.21.15.150:0/256110732 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7e840225e0 con 0x7f7e940ff1e0 2023-10-11T16:29:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.554+0000 7f7e8b7fe700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7e800a1150 0x7f7e800a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.554+0000 7f7e8b7fe700 1 -- 172.21.15.150:0/256110732 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f7e840cd720 con 0x7f7e940ff1e0 2023-10-11T16:29:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.554+0000 7f7e99cd6700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7e800a1150 0x7f7e800a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.555+0000 7f7e99cd6700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7e800a1150 0x7f7e800a3600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000ac50 tx=0x7f7e9000a380 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.557+0000 7f7e8b7fe700 1 -- 172.21.15.150:0/256110732 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7e84097630 con 0x7f7e940ff1e0 2023-10-11T16:29:03.757 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:03.757+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e94061bc0 con 0x7f7e800a1150 2023-10-11T16:29:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:29:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:29:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:29:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:29:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] boot 2023-10-11T16:29:03.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: osdmap e71: 11 total, 11 up, 11 in 2023-10-11T16:29:03.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:03 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:29:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:29:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:29:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:29:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] boot 2023-10-11T16:29:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: osdmap e71: 11 total, 11 up, 11 in 2023-10-11T16:29:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:03 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:04.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:29:04.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:29:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:29:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-11T16:29:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: osd.10 [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] boot 2023-10-11T16:29:04.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: osdmap e71: 11 total, 11 up, 11 in 2023-10-11T16:29:04.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:03 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:29:04.911 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:04 smithi150 ceph-mon[132217]: osdmap e72: 11 total, 11 up, 11 in 2023-10-11T16:29:04.912 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:04 smithi150 ceph-mon[132217]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:29:04.912 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:29:04.912 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:29:04.912 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:04.913 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:04 smithi150 ceph-mon[132217]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:05.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:04 smithi130 ceph-mon[131859]: osdmap e72: 11 total, 11 up, 11 in 2023-10-11T16:29:05.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:04 smithi130 ceph-mon[131859]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:29:05.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:29:05.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:29:05.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:05.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:04 smithi130 ceph-mon[131859]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:05.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:04 smithi111 ceph-mon[98527]: osdmap e72: 11 total, 11 up, 11 in 2023-10-11T16:29:05.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:04 smithi111 ceph-mon[98527]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:29:05.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-11T16:29:05.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-11T16:29:05.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:05.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:04 smithi111 ceph-mon[98527]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:05.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:05 smithi150 ceph-mon[132217]: osdmap e73: 11 total, 11 up, 11 in 2023-10-11T16:29:06.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:05 smithi130 ceph-mon[131859]: osdmap e73: 11 total, 11 up, 11 in 2023-10-11T16:29:06.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:05 smithi111 ceph-mon[98527]: osdmap e73: 11 total, 11 up, 11 in 2023-10-11T16:29:06.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:06 smithi150 ceph-mon[132217]: osdmap e74: 11 total, 11 up, 11 in 2023-10-11T16:29:06.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:06 smithi150 ceph-mon[132217]: pgmap v227: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-11T16:29:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:06 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/1267870771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]: dispatch 2023-10-11T16:29:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:06 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]: dispatch 2023-10-11T16:29:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:06 smithi150 ceph-mon[132217]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]': finished 2023-10-11T16:29:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:06 smithi150 ceph-mon[132217]: osdmap e75: 12 total, 11 up, 12 in 2023-10-11T16:29:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:06 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:07.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:06 smithi130 ceph-mon[131859]: osdmap e74: 11 total, 11 up, 11 in 2023-10-11T16:29:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:06 smithi130 ceph-mon[131859]: pgmap v227: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-11T16:29:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:06 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/1267870771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]: dispatch 2023-10-11T16:29:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:06 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]: dispatch 2023-10-11T16:29:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:06 smithi130 ceph-mon[131859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]': finished 2023-10-11T16:29:07.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:06 smithi130 ceph-mon[131859]: osdmap e75: 12 total, 11 up, 12 in 2023-10-11T16:29:07.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:06 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:07.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:06 smithi111 ceph-mon[98527]: osdmap e74: 11 total, 11 up, 11 in 2023-10-11T16:29:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:06 smithi111 ceph-mon[98527]: pgmap v227: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-11T16:29:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:06 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/1267870771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]: dispatch 2023-10-11T16:29:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:06 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]: dispatch 2023-10-11T16:29:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:06 smithi111 ceph-mon[98527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb"}]': finished 2023-10-11T16:29:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:06 smithi111 ceph-mon[98527]: osdmap e75: 12 total, 11 up, 12 in 2023-10-11T16:29:07.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:07 smithi150 ceph-mon[132217]: from='client.? 172.21.15.150:0/2627704356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:29:08.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:07 smithi130 ceph-mon[131859]: from='client.? 172.21.15.150:0/2627704356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:29:08.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:07 smithi111 ceph-mon[98527]: from='client.? 172.21.15.150:0/2627704356' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-11T16:29:08.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:08 smithi150 ceph-mon[132217]: pgmap v229: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-10-11T16:29:09.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:08 smithi130 ceph-mon[131859]: pgmap v229: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-10-11T16:29:09.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:08 smithi111 ceph-mon[98527]: pgmap v229: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-10-11T16:29:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:11 smithi150 ceph-mon[132217]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-11T16:29:12.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:11 smithi130 ceph-mon[131859]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-11T16:29:12.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:11 smithi111 ceph-mon[98527]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-11T16:29:12.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:12 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-10-11T16:29:12.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:12 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:13.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:12 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-10-11T16:29:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:12 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:13.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:12 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-10-11T16:29:13.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:12 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:13.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:13 smithi150 ceph-mon[132217]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-10-11T16:29:13.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:13 smithi150 ceph-mon[132217]: Deploying daemon osd.11 on smithi150 2023-10-11T16:29:14.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:13 smithi130 ceph-mon[131859]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-10-11T16:29:14.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:13 smithi130 ceph-mon[131859]: Deploying daemon osd.11 on smithi150 2023-10-11T16:29:14.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:13 smithi111 ceph-mon[98527]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-10-11T16:29:14.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:13 smithi111 ceph-mon[98527]: Deploying daemon osd.11 on smithi150 2023-10-11T16:29:15.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:15 smithi150 ceph-mon[132217]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:16.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:15 smithi130 ceph-mon[131859]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:16.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:15 smithi111 ceph-mon[98527]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:17.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:17 smithi150 ceph-mon[132217]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:17.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:17.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:17.929 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:18.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:17 smithi130 ceph-mon[131859]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:18.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:18.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:17 smithi111 ceph-mon[98527]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:18.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:18.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:18 smithi150 ceph-mon[132217]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:19.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:18 smithi130 ceph-mon[131859]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:19.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:18 smithi111 ceph-mon[98527]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:19.521 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 11 on host 'smithi150' 2023-10-11T16:29:19.522 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.519+0000 7f7e8b7fe700 1 -- 172.21.15.150:0/256110732 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f7e94061bc0 con 0x7f7e800a1150 2023-10-11T16:29:19.523 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7e800a1150 msgr2=0x7f7e800a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:19.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7e800a1150 0x7f7e800a3600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000ac50 tx=0x7f7e9000a380 comp rx=0 tx=0).stop 2023-10-11T16:29:19.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e940ff1e0 msgr2=0x7f7e94069970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:19.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e940ff1e0 0x7f7e94069970 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7e84000c00 tx=0x7f7e8400f6e0 comp rx=0 tx=0).stop 2023-10-11T16:29:19.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 shutdown_connections 2023-10-11T16:29:19.524 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7e800a1150 0x7f7e800a3600 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:19.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e94100d70 0x7f7e9419d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:19.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e940ff1e0 0x7f7e94069970 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:19.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 --2- 172.21.15.150:0/256110732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e941003e0 0x7f7e9419ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:19.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 >> 172.21.15.150:0/256110732 conn(0x7f7e940fa7f0 msgr2=0x7f7e941037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:19.525 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 shutdown_connections 2023-10-11T16:29:19.526 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:19.522+0000 7f7e9c73b700 1 -- 172.21.15.150:0/256110732 wait complete. 2023-10-11T16:29:19.994 DEBUG:teuthology.orchestra.run.smithi150:osd.11> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.11.service 2023-10-11T16:29:19.996 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-10-11T16:29:19.997 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd stat -f json 2023-10-11T16:29:20.030 INFO:journalctl@ceph.osd.11.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:29:20.772 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.772 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.773 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.773 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:20 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:20 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:20 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:20 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:20 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.854 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:29:20.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:20 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:20 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:20 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:20.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:20 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:21.029 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:29:20 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:29:20.769+0000 7fa609f04640 -1 osd.11 0 log_to_monitors true 2023-10-11T16:29:21.696 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:21 smithi111 ceph-mon[98527]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:21.697 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:21 smithi111 ceph-mon[98527]: from='osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-11T16:29:21.697 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:21 smithi111 ceph-mon[98527]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-11T16:29:21.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:21 smithi150 ceph-mon[132217]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:21.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:21 smithi150 ceph-mon[132217]: from='osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-11T16:29:21.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:21 smithi150 ceph-mon[132217]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-11T16:29:21.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:21 smithi130 ceph-mon[131859]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:21.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:21 smithi130 ceph-mon[131859]: from='osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-11T16:29:21.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:21 smithi130 ceph-mon[131859]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-11T16:29:22.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 -- 172.21.15.111:0/151705929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 msgr2=0x7fad7c10ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 --2- 172.21.15.111:0/151705929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 0x7fad7c10ecb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fad7400b0d0 tx=0x7fad7400b3e0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 -- 172.21.15.111:0/151705929 shutdown_connections 2023-10-11T16:29:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 --2- 172.21.15.111:0/151705929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 0x7fad7c10ecb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 --2- 172.21.15.111:0/151705929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad7c074b20 0x7fad7c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 --2- 172.21.15.111:0/151705929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad7c073550 0x7fad7c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 -- 172.21.15.111:0/151705929 >> 172.21.15.111:0/151705929 conn(0x7fad7c0fb900 msgr2=0x7fad7c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 -- 172.21.15.111:0/151705929 shutdown_connections 2023-10-11T16:29:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.348+0000 7fad8272c700 1 -- 172.21.15.111:0/151705929 wait complete. 2023-10-11T16:29:22.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.349+0000 7fad8272c700 1 Processor -- start 2023-10-11T16:29:22.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.349+0000 7fad8272c700 1 -- start start 2023-10-11T16:29:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.349+0000 7fad8272c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad7c073550 0x7fad7c19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:22.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad8272c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad7c074b20 0x7fad7c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:22.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad8272c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 0x7fad7c1a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:22.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad8272c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1a42a0 con 0x7fad7c10c8d0 2023-10-11T16:29:22.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad8272c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1a43e0 con 0x7fad7c074b20 2023-10-11T16:29:22.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad8272c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1a4570 con 0x7fad7c073550 2023-10-11T16:29:22.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad7b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad7c074b20 0x7fad7c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:22.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad80cc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 0x7fad7c1a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:22.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad7b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad7c074b20 0x7fad7c19cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:35616/0 (socket says 172.21.15.111:35616) 2023-10-11T16:29:22.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad7b7fe700 1 -- 172.21.15.111:0/1155502220 learned_addr learned my addr 172.21.15.111:0/1155502220 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:22.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad7bfff700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad7c073550 0x7fad7c19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:22.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad80cc9700 1 -- 172.21.15.111:0/1155502220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad7c073550 msgr2=0x7fad7c19c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:22.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad80cc9700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad7c073550 0x7fad7c19c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad80cc9700 1 -- 172.21.15.111:0/1155502220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad7c074b20 msgr2=0x7fad7c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:22.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad80cc9700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad7c074b20 0x7fad7c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad80cc9700 1 -- 172.21.15.111:0/1155502220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad74009d00 con 0x7fad7c10c8d0 2023-10-11T16:29:22.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad7bfff700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad7c073550 0x7fad7c19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:29:22.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.350+0000 7fad7b7fe700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad7c074b20 0x7fad7c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:29:22.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.351+0000 7fad80cc9700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 0x7fad7c1a3cd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fad74009fd0 tx=0x7fad74008eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:22.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.351+0000 7fad797fa700 1 -- 172.21.15.111:0/1155502220 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7400e040 con 0x7fad7c10c8d0 2023-10-11T16:29:22.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.351+0000 7fad797fa700 1 -- 172.21.15.111:0/1155502220 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad7400bd30 con 0x7fad7c10c8d0 2023-10-11T16:29:22.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.351+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c112290 con 0x7fad7c10c8d0 2023-10-11T16:29:22.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.351+0000 7fad797fa700 1 -- 172.21.15.111:0/1155502220 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7401d470 con 0x7fad7c10c8d0 2023-10-11T16:29:22.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.351+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c112100 con 0x7fad7c10c8d0 2023-10-11T16:29:22.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.352+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7c102420 con 0x7fad7c10c8d0 2023-10-11T16:29:22.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.354+0000 7fad797fa700 1 -- 172.21.15.111:0/1155502220 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fad74004a50 con 0x7fad7c10c8d0 2023-10-11T16:29:22.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.355+0000 7fad797fa700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad640a1150 0x7fad640a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:22.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.355+0000 7fad797fa700 1 -- 172.21.15.111:0/1155502220 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fad740cd290 con 0x7fad7c10c8d0 2023-10-11T16:29:22.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.355+0000 7fad7bfff700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad640a1150 0x7fad640a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:22.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.356+0000 7fad7bfff700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad640a1150 0x7fad640a3600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad6c011490 tx=0x7fad6c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:22.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.360+0000 7fad797fa700 1 -- 172.21.15.111:0/1155502220 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fad74096fa0 con 0x7fad7c10c8d0 2023-10-11T16:29:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.581+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fad7c061950 con 0x7fad7c10c8d0 2023-10-11T16:29:22.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.581+0000 7fad797fa700 1 -- 172.21.15.111:0/1155502220 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7fad7409a5c0 con 0x7fad7c10c8d0 2023-10-11T16:29:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:22.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad640a1150 msgr2=0x7fad640a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad640a1150 0x7fad640a3600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fad6c011490 tx=0x7fad6c009260 comp rx=0 tx=0).stop 2023-10-11T16:29:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 msgr2=0x7fad7c1a3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 0x7fad7c1a3cd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fad74009fd0 tx=0x7fad74008eb0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 shutdown_connections 2023-10-11T16:29:22.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad640a1150 0x7fad640a3600 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c10c8d0 0x7fad7c1a3cd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad7c073550 0x7fad7c19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 --2- 172.21.15.111:0/1155502220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad7c074b20 0x7fad7c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:22.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 >> 172.21.15.111:0/1155502220 conn(0x7fad7c0fb900 msgr2=0x7fad7c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:22.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 shutdown_connections 2023-10-11T16:29:22.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:22.584+0000 7fad8272c700 1 -- 172.21.15.111:0/1155502220 wait complete. 2023-10-11T16:29:22.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-10-11T16:29:22.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: osdmap e76: 12 total, 11 up, 12 in 2023-10-11T16:29:22.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:22.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:29:22.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:29:22.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:22.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: Detected new or changed devices on smithi150 2023-10-11T16:29:22.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:22.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:22.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:22.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:22.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi150 to 4292M 2023-10-11T16:29:22.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:22.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:22.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:22.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:29:22.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:22.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:22.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:22.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:22.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:22.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:29:22.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:22.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:22 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1155502220' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-11T16:29:22.952 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:29:22.671+0000 7fa5ff57b700 -1 osd.11 0 waiting for initial osdmap 2023-10-11T16:29:22.953 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:29:22.688+0000 7fa5f7b69700 -1 osd.11 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-11T16:29:22.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-10-11T16:29:22.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: osdmap e76: 12 total, 11 up, 12 in 2023-10-11T16:29:22.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:22.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:29:22.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:29:22.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:22.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: Detected new or changed devices on smithi150 2023-10-11T16:29:22.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:22.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:22.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:22.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:22.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi150 to 4292M 2023-10-11T16:29:22.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:22.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:22.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:22.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:29:22.957 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:22.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:22.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:22.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:22.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:22.958 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:22.959 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:29:22.959 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:22.959 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:22 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1155502220' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-11T16:29:23.029 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":76,"num_osds":12,"num_up_osds":11,"osd_up_since":1697041742,"num_in_osds":12,"osd_in_since":1697041746,"num_remapped_pgs":0} 2023-10-11T16:29:23.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-10-11T16:29:23.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: osdmap e76: 12 total, 11 up, 12 in 2023-10-11T16:29:23.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:23.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:29:23.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2023-10-11T16:29:23.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:23.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: Detected new or changed devices on smithi150 2023-10-11T16:29:23.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:23.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:23.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:23.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:23.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:23.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:29:23.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi150 to 4292M 2023-10-11T16:29:23.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:23.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:23.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:23.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:23.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:23.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:29:23.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:23.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:23.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:23.087 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:23.087 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:23.087 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:23.087 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:29:23.087 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:23.088 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:22 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1155502220' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-11T16:29:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:23 smithi150 ceph-mon[132217]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:29:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:23 smithi150 ceph-mon[132217]: osdmap e77: 12 total, 11 up, 12 in 2023-10-11T16:29:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:23.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:23 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:24.031 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd stat -f json 2023-10-11T16:29:24.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:23 smithi130 ceph-mon[131859]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:29:24.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:23 smithi130 ceph-mon[131859]: osdmap e77: 12 total, 11 up, 12 in 2023-10-11T16:29:24.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:24.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:23 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:24.096 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:23 smithi111 ceph-mon[98527]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2023-10-11T16:29:24.097 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:23 smithi111 ceph-mon[98527]: osdmap e77: 12 total, 11 up, 12 in 2023-10-11T16:29:24.097 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:24.097 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:23 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:24.882 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:29:24.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:24 smithi150 ceph-mon[132217]: purged_snaps scrub starts 2023-10-11T16:29:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:24 smithi150 ceph-mon[132217]: purged_snaps scrub ok 2023-10-11T16:29:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:24 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:24 smithi150 ceph-mon[132217]: osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] boot 2023-10-11T16:29:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:24 smithi150 ceph-mon[132217]: osdmap e78: 12 total, 12 up, 12 in 2023-10-11T16:29:24.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:24 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:24.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:24 smithi150 ceph-mon[132217]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:25.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:24 smithi130 ceph-mon[131859]: purged_snaps scrub starts 2023-10-11T16:29:25.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:24 smithi130 ceph-mon[131859]: purged_snaps scrub ok 2023-10-11T16:29:25.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:24 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:25.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:24 smithi130 ceph-mon[131859]: osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] boot 2023-10-11T16:29:25.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:24 smithi130 ceph-mon[131859]: osdmap e78: 12 total, 12 up, 12 in 2023-10-11T16:29:25.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:24 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:25.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:24 smithi130 ceph-mon[131859]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:25.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:24 smithi111 ceph-mon[98527]: purged_snaps scrub starts 2023-10-11T16:29:25.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:24 smithi111 ceph-mon[98527]: purged_snaps scrub ok 2023-10-11T16:29:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:24 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:24 smithi111 ceph-mon[98527]: osd.11 [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] boot 2023-10-11T16:29:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:24 smithi111 ceph-mon[98527]: osdmap e78: 12 total, 12 up, 12 in 2023-10-11T16:29:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:24 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:29:25.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:24 smithi111 ceph-mon[98527]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-11T16:29:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:25 smithi150 ceph-mon[132217]: osdmap e79: 12 total, 12 up, 12 in 2023-10-11T16:29:26.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:25 smithi130 ceph-mon[131859]: osdmap e79: 12 total, 12 up, 12 in 2023-10-11T16:29:26.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:25 smithi111 ceph-mon[98527]: osdmap e79: 12 total, 12 up, 12 in 2023-10-11T16:29:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 -- 172.21.15.111:0/646342483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301002f0 msgr2=0x7fc130100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 --2- 172.21.15.111:0/646342483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301002f0 0x7fc130100700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc120009990 tx=0x7fc120009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 -- 172.21.15.111:0/646342483 shutdown_connections 2023-10-11T16:29:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 --2- 172.21.15.111:0/646342483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc130101e80 0x7fc130108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 --2- 172.21.15.111:0/646342483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301002f0 0x7fc130100700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 --2- 172.21.15.111:0/646342483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1301014f0 0x7fc130101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 -- 172.21.15.111:0/646342483 >> 172.21.15.111:0/646342483 conn(0x7fc1300fb900 msgr2=0x7fc1300fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 -- 172.21.15.111:0/646342483 shutdown_connections 2023-10-11T16:29:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.389+0000 7fc135090700 1 -- 172.21.15.111:0/646342483 wait complete. 2023-10-11T16:29:26.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.390+0000 7fc135090700 1 Processor -- start 2023-10-11T16:29:26.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.390+0000 7fc135090700 1 -- start start 2023-10-11T16:29:26.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc135090700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1301002f0 0x7fc130074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:26.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc135090700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301014f0 0x7fc130072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:26.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc135090700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc130101e80 0x7fc130073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:26.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc135090700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc130073ba0 con 0x7fc1301014f0 2023-10-11T16:29:26.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc135090700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc130073d10 con 0x7fc1301002f0 2023-10-11T16:29:26.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc135090700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1301a00f0 con 0x7fc130101e80 2023-10-11T16:29:26.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301014f0 0x7fc130072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:26.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301014f0 0x7fc130072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59404/0 (socket says 172.21.15.111:59404) 2023-10-11T16:29:26.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12e59c700 1 -- 172.21.15.111:0/3445298517 learned_addr learned my addr 172.21.15.111:0/3445298517 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:26.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12ed9d700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1301002f0 0x7fc130074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:26.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12f59e700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc130101e80 0x7fc130073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:26.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12e59c700 1 -- 172.21.15.111:0/3445298517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc130101e80 msgr2=0x7fc130073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:26.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12e59c700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc130101e80 0x7fc130073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12e59c700 1 -- 172.21.15.111:0/3445298517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1301002f0 msgr2=0x7fc130074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:26.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12e59c700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1301002f0 0x7fc130074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12e59c700 1 -- 172.21.15.111:0/3445298517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1200095f0 con 0x7fc1301014f0 2023-10-11T16:29:26.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.391+0000 7fc12ed9d700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1301002f0 0x7fc130074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:29:26.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.392+0000 7fc12e59c700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301014f0 0x7fc130072ee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc12400e9a0 tx=0x7fc12400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:26.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.392+0000 7fc117fff700 1 -- 172.21.15.111:0/3445298517 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124009790 con 0x7fc1301014f0 2023-10-11T16:29:26.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.392+0000 7fc117fff700 1 -- 172.21.15.111:0/3445298517 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc124010430 con 0x7fc1301014f0 2023-10-11T16:29:26.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.392+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13010bcc0 con 0x7fc1301014f0 2023-10-11T16:29:26.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.392+0000 7fc117fff700 1 -- 172.21.15.111:0/3445298517 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124018680 con 0x7fc1301014f0 2023-10-11T16:29:26.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.392+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13010bb30 con 0x7fc1301014f0 2023-10-11T16:29:26.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.393+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc130107900 con 0x7fc1301014f0 2023-10-11T16:29:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.395+0000 7fc117fff700 1 -- 172.21.15.111:0/3445298517 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fc1240105a0 con 0x7fc1301014f0 2023-10-11T16:29:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.396+0000 7fc117fff700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc1180a11d0 0x7fc1180a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.397+0000 7fc117fff700 1 -- 172.21.15.111:0/3445298517 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc124014070 con 0x7fc1301014f0 2023-10-11T16:29:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.397+0000 7fc12ed9d700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc1180a11d0 0x7fc1180a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:26.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.397+0000 7fc12ed9d700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc1180a11d0 0x7fc1180a3680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc12000b2e0 tx=0x7fc120005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:26.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.401+0000 7fc117fff700 1 -- 172.21.15.111:0/3445298517 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc124051730 con 0x7fc1301014f0 2023-10-11T16:29:26.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.597+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc130061b80 con 0x7fc1301014f0 2023-10-11T16:29:26.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.597+0000 7fc117fff700 1 -- 172.21.15.111:0/3445298517 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7fc1240961d0 con 0x7fc1301014f0 2023-10-11T16:29:26.599 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:26.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.599+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc1180a11d0 msgr2=0x7fc1180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:26.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.599+0000 7fc135090700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc1180a11d0 0x7fc1180a3680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc12000b2e0 tx=0x7fc120005fb0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.599+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301014f0 msgr2=0x7fc130072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.599+0000 7fc135090700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301014f0 0x7fc130072ee0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc12400e9a0 tx=0x7fc12400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.600+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 shutdown_connections 2023-10-11T16:29:26.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.600+0000 7fc135090700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc1180a11d0 0x7fc1180a3680 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.600+0000 7fc135090700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc130101e80 0x7fc130073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.600+0000 7fc135090700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1301002f0 0x7fc130074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.600+0000 7fc135090700 1 --2- 172.21.15.111:0/3445298517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1301014f0 0x7fc130072ee0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.600+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 >> 172.21.15.111:0/3445298517 conn(0x7fc1300fb900 msgr2=0x7fc130104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:26.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.600+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 shutdown_connections 2023-10-11T16:29:26.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:26.600+0000 7fc135090700 1 -- 172.21.15.111:0/3445298517 wait complete. 2023-10-11T16:29:26.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:26 smithi111 ceph-mon[98527]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:26.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:26 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3445298517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-11T16:29:26.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:26 smithi150 ceph-mon[132217]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:26.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:26 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3445298517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-11T16:29:27.061 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":79,"num_osds":12,"num_up_osds":12,"osd_up_since":1697041763,"num_in_osds":12,"osd_in_since":1697041746,"num_remapped_pgs":0} 2023-10-11T16:29:27.062 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd dump --format=json 2023-10-11T16:29:27.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:26 smithi130 ceph-mon[131859]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:27.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:26 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3445298517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-11T16:29:27.908 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:29:29.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:29 smithi111 ceph-mon[98527]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:29.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 -- 172.21.15.111:0/73442686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0bc100ee0 msgr2=0x7fc0bc104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:29.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/73442686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0bc100ee0 0x7fc0bc104eb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80099c0 tx=0x7fc0a8009cd0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 -- 172.21.15.111:0/73442686 shutdown_connections 2023-10-11T16:29:29.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/73442686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0bc1055e0 0x7fc0bc10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/73442686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0bc100540 0x7fc0bc100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/73442686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0bc100ee0 0x7fc0bc104eb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 -- 172.21.15.111:0/73442686 >> 172.21.15.111:0/73442686 conn(0x7fc0bc0fbdc0 msgr2=0x7fc0bc0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:29.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 -- 172.21.15.111:0/73442686 shutdown_connections 2023-10-11T16:29:29.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.439+0000 7fc0c2d73700 1 -- 172.21.15.111:0/73442686 wait complete. 2023-10-11T16:29:29.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.440+0000 7fc0c2d73700 1 Processor -- start 2023-10-11T16:29:29.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.440+0000 7fc0c2d73700 1 -- start start 2023-10-11T16:29:29.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c2d73700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0bc100540 0x7fc0bc1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:29.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c2d73700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0bc100ee0 0x7fc0bc198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:29.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c2d73700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0bc1055e0 0x7fc0bc19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:29.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c2d73700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc19d590 con 0x7fc0bc100540 2023-10-11T16:29:29.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c2d73700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc19d700 con 0x7fc0bc1055e0 2023-10-11T16:29:29.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c2d73700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc19d8c0 con 0x7fc0bc100ee0 2023-10-11T16:29:29.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c0b0f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0bc100540 0x7fc0bc1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:29.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c0b0f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0bc100540 0x7fc0bc1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59428/0 (socket says 172.21.15.111:59428) 2023-10-11T16:29:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c0b0f700 1 -- 172.21.15.111:0/1802649639 learned_addr learned my addr 172.21.15.111:0/1802649639 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c1310700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0bc1055e0 0x7fc0bc19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0bbfff700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0bc100ee0 0x7fc0bc198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c0b0f700 1 -- 172.21.15.111:0/1802649639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0bc100ee0 msgr2=0x7fc0bc198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c0b0f700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0bc100ee0 0x7fc0bc198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c0b0f700 1 -- 172.21.15.111:0/1802649639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0bc1055e0 msgr2=0x7fc0bc19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:29.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c0b0f700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0bc1055e0 0x7fc0bc19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.441+0000 7fc0c0b0f700 1 -- 172.21.15.111:0/1802649639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80095f0 con 0x7fc0bc100540 2023-10-11T16:29:29.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.442+0000 7fc0c0b0f700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0bc100540 0x7fc0bc1985d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000e9a0 tx=0x7fc0b000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:29.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.442+0000 7fc0b9ffb700 1 -- 172.21.15.111:0/1802649639 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0009790 con 0x7fc0bc100540 2023-10-11T16:29:29.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.442+0000 7fc0b9ffb700 1 -- 172.21.15.111:0/1802649639 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0b0010430 con 0x7fc0bc100540 2023-10-11T16:29:29.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.442+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0bc110360 con 0x7fc0bc100540 2023-10-11T16:29:29.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.442+0000 7fc0b9ffb700 1 -- 172.21.15.111:0/1802649639 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0018680 con 0x7fc0bc100540 2023-10-11T16:29:29.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.442+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0bc1101d0 con 0x7fc0bc100540 2023-10-11T16:29:29.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.443+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0bc1104f0 con 0x7fc0bc100540 2023-10-11T16:29:29.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.445+0000 7fc0b9ffb700 1 -- 172.21.15.111:0/1802649639 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fc0b00105a0 con 0x7fc0bc100540 2023-10-11T16:29:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.446+0000 7fc0b9ffb700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0ac0a11d0 0x7fc0ac0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.446+0000 7fc0b9ffb700 1 -- 172.21.15.111:0/1802649639 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc0b0014070 con 0x7fc0bc100540 2023-10-11T16:29:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.446+0000 7fc0bbfff700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0ac0a11d0 0x7fc0ac0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.447+0000 7fc0bbfff700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0ac0a11d0 0x7fc0ac0a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80098e0 tx=0x7fc0a8000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:29.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:29 smithi150 ceph-mon[132217]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:29.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.450+0000 7fc0b9ffb700 1 -- 172.21.15.111:0/1802649639 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc0b0051730 con 0x7fc0bc100540 2023-10-11T16:29:29.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:29 smithi130 ceph-mon[131859]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.650+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0bc066ea0 con 0x7fc0bc100540 2023-10-11T16:29:29.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.651+0000 7fc0b9ffb700 1 -- 172.21.15.111:0/1802649639 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18492 (secure 0 0 0) 0x7fc0b00961d0 con 0x7fc0bc100540 2023-10-11T16:29:29.653 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:29.653 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":79,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:29:24.677205+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:29:29.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0ac0a11d0 msgr2=0x7fc0ac0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:29.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0ac0a11d0 0x7fc0ac0a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80098e0 tx=0x7fc0a8000bc0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0bc100540 msgr2=0x7fc0bc1985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:29.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0bc100540 0x7fc0bc1985d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000e9a0 tx=0x7fc0b000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 shutdown_connections 2023-10-11T16:29:29.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0ac0a11d0 0x7fc0ac0a3680 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0bc1055e0 0x7fc0bc19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0bc100540 0x7fc0bc1985d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 --2- 172.21.15.111:0/1802649639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0bc100ee0 0x7fc0bc198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:29.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 >> 172.21.15.111:0/1802649639 conn(0x7fc0bc0fbdc0 msgr2=0x7fc0bc0fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:29.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 shutdown_connections 2023-10-11T16:29:29.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:29.654+0000 7fc0c2d73700 1 -- 172.21.15.111:0/1802649639 wait complete. 2023-10-11T16:29:30.186 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-11T16:29:30.187 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd pool get .mgr pg_num 2023-10-11T16:29:30.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:30 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1802649639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:29:30.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:30 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1802649639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:29:30.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:30 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1802649639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:29:31.030 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:29:31.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:31 smithi111 ceph-mon[98527]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:31.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:31 smithi150 ceph-mon[132217]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:32.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:31 smithi130 ceph-mon[131859]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:32.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.540+0000 7fd43aa1e700 1 -- 172.21.15.111:0/2665923234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4341055e0 msgr2=0x7fd43410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:32.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.540+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/2665923234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4341055e0 0x7fd43410d160 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00b0d0 tx=0x7fd42c00b3e0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.541+0000 7fd43aa1e700 1 -- 172.21.15.111:0/2665923234 shutdown_connections 2023-10-11T16:29:32.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.541+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/2665923234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4341055e0 0x7fd43410d160 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.541+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/2665923234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd434100540 0x7fd434100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.541+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/2665923234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd434100ee0 0x7fd434104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.541+0000 7fd43aa1e700 1 -- 172.21.15.111:0/2665923234 >> 172.21.15.111:0/2665923234 conn(0x7fd4340fbdc0 msgr2=0x7fd4340fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:32.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.541+0000 7fd43aa1e700 1 -- 172.21.15.111:0/2665923234 shutdown_connections 2023-10-11T16:29:32.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.541+0000 7fd43aa1e700 1 -- 172.21.15.111:0/2665923234 wait complete. 2023-10-11T16:29:32.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.542+0000 7fd43aa1e700 1 Processor -- start 2023-10-11T16:29:32.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.542+0000 7fd43aa1e700 1 -- start start 2023-10-11T16:29:32.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.542+0000 7fd43aa1e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd434100540 0x7fd434072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:32.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.542+0000 7fd43aa1e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd434100ee0 0x7fd434073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:32.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.542+0000 7fd43aa1e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4341055e0 0x7fd4341a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:32.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.542+0000 7fd43aa1e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd434073c20 con 0x7fd434100ee0 2023-10-11T16:29:32.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.542+0000 7fd43aa1e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4341a4bd0 con 0x7fd434100540 2023-10-11T16:29:32.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.542+0000 7fd43aa1e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4341a4d40 con 0x7fd4341055e0 2023-10-11T16:29:32.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd434100ee0 0x7fd434073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:32.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd434100ee0 0x7fd434073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59456/0 (socket says 172.21.15.111:59456) 2023-10-11T16:29:32.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 -- 172.21.15.111:0/1803274517 learned_addr learned my addr 172.21.15.111:0/1803274517 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd433fff700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd434100540 0x7fd434072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd438fbb700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4341055e0 0x7fd4341a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 -- 172.21.15.111:0/1803274517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4341055e0 msgr2=0x7fd4341a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4341055e0 0x7fd4341a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 -- 172.21.15.111:0/1803274517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd434100540 msgr2=0x7fd434072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd434100540 0x7fd434072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 -- 172.21.15.111:0/1803274517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c009d00 con 0x7fd434100ee0 2023-10-11T16:29:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.543+0000 7fd4337fe700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd434100ee0 0x7fd434073330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd42800d980 tx=0x7fd42800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:32.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.544+0000 7fd4317fa700 1 -- 172.21.15.111:0/1803274517 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42800f890 con 0x7fd434100ee0 2023-10-11T16:29:32.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.544+0000 7fd4317fa700 1 -- 172.21.15.111:0/1803274517 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd428010460 con 0x7fd434100ee0 2023-10-11T16:29:32.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.544+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd434110370 con 0x7fd434100ee0 2023-10-11T16:29:32.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.544+0000 7fd4317fa700 1 -- 172.21.15.111:0/1803274517 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42800e610 con 0x7fd434100ee0 2023-10-11T16:29:32.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.544+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4341101e0 con 0x7fd434100ee0 2023-10-11T16:29:32.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.545+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd434110500 con 0x7fd434100ee0 2023-10-11T16:29:32.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.547+0000 7fd4317fa700 1 -- 172.21.15.111:0/1803274517 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fd4280105d0 con 0x7fd434100ee0 2023-10-11T16:29:32.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.549+0000 7fd4317fa700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd41c0a11d0 0x7fd41c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:32.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.549+0000 7fd4317fa700 1 -- 172.21.15.111:0/1803274517 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd4280cbd30 con 0x7fd434100ee0 2023-10-11T16:29:32.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.549+0000 7fd433fff700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd41c0a11d0 0x7fd41c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:32.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.549+0000 7fd433fff700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd41c0a11d0 0x7fd41c0a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd424005950 tx=0x7fd42400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:32.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.553+0000 7fd4317fa700 1 -- 172.21.15.111:0/1803274517 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd428095920 con 0x7fd434100ee0 2023-10-11T16:29:32.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.753+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd4341a5770 con 0x7fd434100ee0 2023-10-11T16:29:32.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.753+0000 7fd4317fa700 1 -- 172.21.15.111:0/1803274517 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd428098f40 con 0x7fd434100ee0 2023-10-11T16:29:32.755 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:29:32.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd41c0a11d0 msgr2=0x7fd41c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:32.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd41c0a11d0 0x7fd41c0a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd424005950 tx=0x7fd42400b410 comp rx=0 tx=0).stop 2023-10-11T16:29:32.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd434100ee0 msgr2=0x7fd434073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:32.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd434100ee0 0x7fd434073330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd42800d980 tx=0x7fd42800dc90 comp rx=0 tx=0).stop 2023-10-11T16:29:32.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 shutdown_connections 2023-10-11T16:29:32.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd41c0a11d0 0x7fd41c0a3680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4341055e0 0x7fd4341a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd434100540 0x7fd434072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 --2- 172.21.15.111:0/1803274517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd434100ee0 0x7fd434073330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:32.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 >> 172.21.15.111:0/1803274517 conn(0x7fd4340fbdc0 msgr2=0x7fd4340fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:32.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 shutdown_connections 2023-10-11T16:29:32.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:32.756+0000 7fd43aa1e700 1 -- 172.21.15.111:0/1803274517 wait complete. 2023-10-11T16:29:32.945 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:32 smithi111 ceph-mon[98527]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:32.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:32 smithi150 ceph-mon[132217]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:33.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:32 smithi130 ceph-mon[131859]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:33.319 INFO:tasks.cephadm:Adding mds.a on smithi111 2023-10-11T16:29:33.319 INFO:tasks.cephadm:Adding mds.d on smithi111 2023-10-11T16:29:33.320 INFO:tasks.cephadm:Adding mds.g on smithi111 2023-10-11T16:29:33.320 INFO:tasks.cephadm:Adding mds.j on smithi111 2023-10-11T16:29:33.320 INFO:tasks.cephadm:Adding mds.b on smithi130 2023-10-11T16:29:33.320 INFO:tasks.cephadm:Adding mds.e on smithi130 2023-10-11T16:29:33.321 INFO:tasks.cephadm:Adding mds.h on smithi130 2023-10-11T16:29:33.321 INFO:tasks.cephadm:Adding mds.k on smithi130 2023-10-11T16:29:33.321 INFO:tasks.cephadm:Adding mds.c on smithi150 2023-10-11T16:29:33.321 INFO:tasks.cephadm:Adding mds.f on smithi150 2023-10-11T16:29:33.322 INFO:tasks.cephadm:Adding mds.i on smithi150 2023-10-11T16:29:33.322 INFO:tasks.cephadm:Adding mds.l on smithi150 2023-10-11T16:29:33.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph orch apply mds all '12;smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l' 2023-10-11T16:29:33.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:33 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1803274517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:29:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:29:34.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:33 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1803274517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:29:34.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:33 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1803274517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:29:35.125 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 -- 172.21.15.150:0/2583832052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80fe9b0 msgr2=0x7f9ff80fedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:35.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 --2- 172.21.15.150:0/2583832052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80fe9b0 0x7f9ff80fedc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009990 tx=0x7f9fec009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 -- 172.21.15.150:0/2583832052 shutdown_connections 2023-10-11T16:29:35.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 --2- 172.21.15.150:0/2583832052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ff8100540 0x7f9ff8106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.126 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 --2- 172.21.15.150:0/2583832052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80fe9b0 0x7f9ff80fedc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 --2- 172.21.15.150:0/2583832052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff80ffbb0 0x7f9ff8100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 -- 172.21.15.150:0/2583832052 >> 172.21.15.150:0/2583832052 conn(0x7f9ff80fa000 msgr2=0x7f9ff80fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:35.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 -- 172.21.15.150:0/2583832052 shutdown_connections 2023-10-11T16:29:35.127 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.119+0000 7f9ffd906700 1 -- 172.21.15.150:0/2583832052 wait complete. 2023-10-11T16:29:35.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ffd906700 1 Processor -- start 2023-10-11T16:29:35.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ffd906700 1 -- start start 2023-10-11T16:29:35.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ffd906700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff80fe9b0 0x7f9ff8108270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:35.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ffd906700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80ffbb0 0x7f9ff819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:35.128 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ffd906700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ff8100540 0x7f9ff819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ffd906700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff819db80 con 0x7f9ff80fe9b0 2023-10-11T16:29:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ffd906700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff819dcf0 con 0x7f9ff8100540 2023-10-11T16:29:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ffd906700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff819deb0 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ff67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80ffbb0 0x7f9ff819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ff67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80ffbb0 0x7f9ff819ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42870/0 (socket says 172.21.15.150:42870) 2023-10-11T16:29:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ff67fc700 1 -- 172.21.15.150:0/1225868151 learned_addr learned my addr 172.21.15.150:0/1225868151 (peer_addr_for_me v2:172.21.15.150:0/0) 2023-10-11T16:29:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ff77fe700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ff8100540 0x7f9ff819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ff67fc700 1 -- 172.21.15.150:0/1225868151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ff8100540 msgr2=0x7f9ff819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ff67fc700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ff8100540 0x7f9ff819d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ff67fc700 1 -- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff80fe9b0 msgr2=0x7f9ff8108270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.120+0000 7f9ff67fc700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff80fe9b0 0x7f9ff8108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.121+0000 7f9ff67fc700 1 -- 172.21.15.150:0/1225868151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec0095f0 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.121+0000 7f9ff67fc700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80ffbb0 0x7f9ff819ad90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800b530 tx=0x7f9fe800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.121+0000 7f9ffc904700 1 -- 172.21.15.150:0/1225868151 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8011830 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.121+0000 7f9ffc904700 1 -- 172.21.15.150:0/1225868151 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fe8011e50 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.121+0000 7f9ffc904700 1 -- 172.21.15.150:0/1225868151 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe800f2e0 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.121+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff810ac00 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.121+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff810aa70 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.122+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff81950e0 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.123+0000 7f9ffc904700 1 -- 172.21.15.150:0/1225868151 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f9fe8010580 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.123+0000 7f9ffc904700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9fe00a11d0 0x7f9fe00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.123+0000 7f9ffc904700 1 -- 172.21.15.150:0/1225868151 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9fe80cbfb0 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.124+0000 7f9ff6ffd700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9fe00a11d0 0x7f9fe00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.124+0000 7f9ff6ffd700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9fe00a11d0 0x7f9fe00a3680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0098e0 tx=0x7f9fec005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.128+0000 7f9ffc904700 1 -- 172.21.15.150:0/1225868151 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9fe8095b00 con 0x7f9ff80ffbb0 2023-10-11T16:29:35.327 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.325+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ff8195ea0 con 0x7f9fe00a11d0 2023-10-11T16:29:35.335 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.334+0000 7f9ffc904700 1 -- 172.21.15.150:0/1225868151 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f9ff8195ea0 con 0x7f9fe00a11d0 2023-10-11T16:29:35.336 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled mds.all update... 2023-10-11T16:29:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9fe00a11d0 msgr2=0x7f9fe00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9fe00a11d0 0x7f9fe00a3680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0098e0 tx=0x7f9fec005fb0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80ffbb0 msgr2=0x7f9ff819ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80ffbb0 0x7f9ff819ad90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800b530 tx=0x7f9fe800b8f0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 shutdown_connections 2023-10-11T16:29:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9fe00a11d0 0x7f9fe00a3680 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ff8100540 0x7f9ff819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff80fe9b0 0x7f9ff8108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 --2- 172.21.15.150:0/1225868151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff80ffbb0 0x7f9ff819ad90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 >> 172.21.15.150:0/1225868151 conn(0x7f9ff80fa000 msgr2=0x7f9ff80fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 shutdown_connections 2023-10-11T16:29:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2023-10-11T16:29:35.336+0000 7f9ffd906700 1 -- 172.21.15.150:0/1225868151 wait complete. 2023-10-11T16:29:35.722 DEBUG:teuthology.orchestra.run.smithi111:mds.a> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.a.service 2023-10-11T16:29:35.729 DEBUG:teuthology.orchestra.run.smithi111:mds.d> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.d.service 2023-10-11T16:29:35.775 DEBUG:teuthology.orchestra.run.smithi111:mds.g> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.g.service 2023-10-11T16:29:35.809 INFO:journalctl@ceph.mds.a.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:29:35.813 DEBUG:teuthology.orchestra.run.smithi111:mds.j> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.j.service 2023-10-11T16:29:35.848 INFO:journalctl@ceph.mds.d.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:29:35.851 DEBUG:teuthology.orchestra.run.smithi130:mds.b> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.b.service 2023-10-11T16:29:35.854 DEBUG:teuthology.orchestra.run.smithi130:mds.e> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.e.service 2023-10-11T16:29:35.885 INFO:journalctl@ceph.mds.b.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:29:35.885 INFO:journalctl@ceph.mds.g.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:29:35.886 DEBUG:teuthology.orchestra.run.smithi130:mds.h> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.h.service 2023-10-11T16:29:35.912 INFO:journalctl@ceph.mds.e.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:29:35.913 DEBUG:teuthology.orchestra.run.smithi130:mds.k> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.k.service 2023-10-11T16:29:35.918 INFO:journalctl@ceph.mds.j.smithi111.stdout:-- Logs begin at Wed 2023-10-11 16:05:25 UTC. -- 2023-10-11T16:29:35.934 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:35 smithi130 ceph-mon[131859]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:35.934 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:35.935 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:35 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:35.935 DEBUG:teuthology.orchestra.run.smithi150:mds.c> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.c.service 2023-10-11T16:29:35.938 DEBUG:teuthology.orchestra.run.smithi150:mds.f> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.f.service 2023-10-11T16:29:35.939 INFO:journalctl@ceph.mds.h.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:29:35.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:35 smithi150 ceph-mon[132217]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:35.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:35 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:35.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:35 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:35.953 DEBUG:teuthology.orchestra.run.smithi150:mds.i> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.i.service 2023-10-11T16:29:35.960 INFO:journalctl@ceph.mds.k.smithi130.stdout:-- Logs begin at Wed 2023-10-11 16:00:09 UTC. -- 2023-10-11T16:29:35.970 INFO:journalctl@ceph.mds.c.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:29:35.971 DEBUG:teuthology.orchestra.run.smithi150:mds.l> sudo journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.l.service 2023-10-11T16:29:35.980 INFO:journalctl@ceph.mds.f.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:29:35.981 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-10-11T16:29:35.982 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:29:35.997 INFO:journalctl@ceph.mds.i.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:29:36.007 INFO:journalctl@ceph.mds.l.smithi150.stdout:-- Logs begin at Wed 2023-10-11 16:00:35 UTC. -- 2023-10-11T16:29:36.043 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:35 smithi111 ceph-mon[98527]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:36.044 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:35 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:36.044 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:35 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:29:36.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.185+0000 7f7d567ff700 1 -- 172.21.15.111:0/2609604696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 msgr2=0x7f7d501006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.185+0000 7f7d567ff700 1 --2- 172.21.15.111:0/2609604696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 0x7f7d501006a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009990 tx=0x7f7d40009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.186+0000 7f7d567ff700 1 -- 172.21.15.111:0/2609604696 shutdown_connections 2023-10-11T16:29:36.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.186+0000 7f7d567ff700 1 --2- 172.21.15.111:0/2609604696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d50101e10 0x7f7d501086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.186+0000 7f7d567ff700 1 --2- 172.21.15.111:0/2609604696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 0x7f7d501006a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.186+0000 7f7d567ff700 1 --2- 172.21.15.111:0/2609604696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d50101480 0x7f7d501018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.186+0000 7f7d567ff700 1 -- 172.21.15.111:0/2609604696 >> 172.21.15.111:0/2609604696 conn(0x7f7d500fb870 msgr2=0x7f7d500fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:36.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.186+0000 7f7d567ff700 1 -- 172.21.15.111:0/2609604696 shutdown_connections 2023-10-11T16:29:36.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.186+0000 7f7d567ff700 1 -- 172.21.15.111:0/2609604696 wait complete. 2023-10-11T16:29:36.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d567ff700 1 Processor -- start 2023-10-11T16:29:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d567ff700 1 -- start start 2023-10-11T16:29:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d567ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 0x7f7d50198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d567ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d50101480 0x7f7d501986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d567ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d50101e10 0x7f7d5019f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d567ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5019fd10 con 0x7f7d50101e10 2023-10-11T16:29:36.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d567ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5019fe50 con 0x7f7d50101480 2023-10-11T16:29:36.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d567ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5019ffe0 con 0x7f7d50100290 2023-10-11T16:29:36.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d4ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 0x7f7d50198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:36.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d4ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 0x7f7d50198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:37510/0 (socket says 172.21.15.111:37510) 2023-10-11T16:29:36.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d4ffff700 1 -- 172.21.15.111:0/1643477995 learned_addr learned my addr 172.21.15.111:0/1643477995 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:36.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d54d9c700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d50101e10 0x7f7d5019f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:36.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.187+0000 7f7d4f7fe700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d50101480 0x7f7d501986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:36.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4ffff700 1 -- 172.21.15.111:0/1643477995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d50101480 msgr2=0x7f7d501986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4ffff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d50101480 0x7f7d501986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4ffff700 1 -- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d50101e10 msgr2=0x7f7d5019f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4ffff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d50101e10 0x7f7d5019f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4ffff700 1 -- 172.21.15.111:0/1643477995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d48009c40 con 0x7f7d50100290 2023-10-11T16:29:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4ffff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 0x7f7d50198190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d40005720 tx=0x7f7d40009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4d7fa700 1 -- 172.21.15.111:0/1643477995 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4001d070 con 0x7f7d50100290 2023-10-11T16:29:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4d7fa700 1 -- 172.21.15.111:0/1643477995 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d40021410 con 0x7f7d50100290 2023-10-11T16:29:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d400095f0 con 0x7f7d50100290 2023-10-11T16:29:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d4d7fa700 1 -- 172.21.15.111:0/1643477995 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40017650 con 0x7f7d50100290 2023-10-11T16:29:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.188+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5010bbe0 con 0x7f7d50100290 2023-10-11T16:29:36.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.189+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d50067a60 con 0x7f7d50100290 2023-10-11T16:29:36.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.191+0000 7f7d4d7fa700 1 -- 172.21.15.111:0/1643477995 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7d40021580 con 0x7f7d50100290 2023-10-11T16:29:36.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.192+0000 7f7d4d7fa700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7d380a11d0 0x7f7d380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:36.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.192+0000 7f7d4d7fa700 1 -- 172.21.15.111:0/1643477995 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7d400cd870 con 0x7f7d50100290 2023-10-11T16:29:36.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.193+0000 7f7d4f7fe700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7d380a11d0 0x7f7d380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.193+0000 7f7d4f7fe700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7d380a11d0 0x7f7d380a3680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009c60 tx=0x7f7d44009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:36.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.195+0000 7f7d4d7fa700 1 -- 172.21.15.111:0/1643477995 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7d40097450 con 0x7f7d50100290 2023-10-11T16:29:36.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.395+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d501a0a50 con 0x7f7d50100290 2023-10-11T16:29:36.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.397+0000 7f7d4d7fa700 1 -- 172.21.15.111:0/1643477995 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18492 (secure 0 0 0) 0x7f7d4009aa70 con 0x7f7d50100290 2023-10-11T16:29:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:36.399 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":79,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:29:24.677205+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:29:36.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7d380a11d0 msgr2=0x7f7d380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7d380a11d0 0x7f7d380a3680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009c60 tx=0x7f7d44009260 comp rx=0 tx=0).stop 2023-10-11T16:29:36.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 msgr2=0x7f7d50198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 0x7f7d50198190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7d40005720 tx=0x7f7d40009580 comp rx=0 tx=0).stop 2023-10-11T16:29:36.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 shutdown_connections 2023-10-11T16:29:36.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7d380a11d0 0x7f7d380a3680 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d50101e10 0x7f7d5019f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50100290 0x7f7d50198190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 --2- 172.21.15.111:0/1643477995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d50101480 0x7f7d501986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 >> 172.21.15.111:0/1643477995 conn(0x7f7d500fb870 msgr2=0x7f7d500fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:36.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 shutdown_connections 2023-10-11T16:29:36.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.399+0000 7f7d567ff700 1 -- 172.21.15.111:0/1643477995 wait complete. 2023-10-11T16:29:36.421 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-11T16:29:36.422 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:29:36.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/983570364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481002c0 msgr2=0x7f2c481006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/983570364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481002c0 0x7f2c481006d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009990 tx=0x7f2c38009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/983570364 shutdown_connections 2023-10-11T16:29:36.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/983570364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c48101e40 0x7f2c48108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/983570364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481002c0 0x7f2c481006d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/983570364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c481014b0 0x7f2c48101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/983570364 >> 172.21.15.111:0/983570364 conn(0x7f2c480fb8a0 msgr2=0x7f2c480fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:36.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/983570364 shutdown_connections 2023-10-11T16:29:36.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.622+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/983570364 wait complete. 2023-10-11T16:29:36.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.623+0000 7f2c4c9ff700 1 Processor -- start 2023-10-11T16:29:36.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c4c9ff700 1 -- start start 2023-10-11T16:29:36.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c4c9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c481002c0 0x7f2c480748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:36.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c4c9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481014b0 0x7f2c48072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:36.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c4c9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c48101e40 0x7f2c48073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:36.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c4c9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48073bc0 con 0x7f2c481014b0 2023-10-11T16:29:36.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c4c9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48073d30 con 0x7f2c48101e40 2023-10-11T16:29:36.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c4c9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c481a00c0 con 0x7f2c481002c0 2023-10-11T16:29:36.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c45d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481014b0 0x7f2c48072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:36.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c4659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c481002c0 0x7f2c480748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:36.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c45d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481014b0 0x7f2c48072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37282/0 (socket says 172.21.15.111:37282) 2023-10-11T16:29:36.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c45d9b700 1 -- 172.21.15.111:0/2605395028 learned_addr learned my addr 172.21.15.111:0/2605395028 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:36.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.624+0000 7f2c46d9d700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c48101e40 0x7f2c48073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:36.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c45d9b700 1 -- 172.21.15.111:0/2605395028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c481002c0 msgr2=0x7f2c480748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c45d9b700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c481002c0 0x7f2c480748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c45d9b700 1 -- 172.21.15.111:0/2605395028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c48101e40 msgr2=0x7f2c48073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c45d9b700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c48101e40 0x7f2c48073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c45d9b700 1 -- 172.21.15.111:0/2605395028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c009520 con 0x7f2c481014b0 2023-10-11T16:29:36.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c45d9b700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481014b0 0x7f2c48072ee0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00eb30 tx=0x7f2c3c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:36.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c377fe700 1 -- 172.21.15.111:0/2605395028 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c009880 con 0x7f2c481014b0 2023-10-11T16:29:36.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c380095f0 con 0x7f2c481014b0 2023-10-11T16:29:36.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c377fe700 1 -- 172.21.15.111:0/2605395028 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c3c0044e0 con 0x7f2c481014b0 2023-10-11T16:29:36.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c377fe700 1 -- 172.21.15.111:0/2605395028 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c003d80 con 0x7f2c481014b0 2023-10-11T16:29:36.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.625+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4810bc90 con 0x7f2c481014b0 2023-10-11T16:29:36.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.626+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c48067b00 con 0x7f2c481014b0 2023-10-11T16:29:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.628+0000 7f2c377fe700 1 -- 172.21.15.111:0/2605395028 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f2c3c003f00 con 0x7f2c481014b0 2023-10-11T16:29:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.629+0000 7f2c377fe700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2c300a1150 0x7f2c300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.630+0000 7f2c377fe700 1 -- 172.21.15.111:0/2605395028 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2c3c014070 con 0x7f2c481014b0 2023-10-11T16:29:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.630+0000 7f2c4659c700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2c300a1150 0x7f2c300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.630+0000 7f2c4659c700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2c300a1150 0x7f2c300a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800b2e0 tx=0x7f2c38005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.631+0000 7f2c377fe700 1 -- 172.21.15.111:0/2605395028 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2c3c096700 con 0x7f2c481014b0 2023-10-11T16:29:36.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.834+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c48107170 con 0x7f2c481014b0 2023-10-11T16:29:36.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.835+0000 7f2c377fe700 1 -- 172.21.15.111:0/2605395028 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c3c099d20 con 0x7f2c481014b0 2023-10-11T16:29:36.837 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:29:36.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2c300a1150 msgr2=0x7f2c300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2c300a1150 0x7f2c300a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800b2e0 tx=0x7f2c38005fb0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481014b0 msgr2=0x7f2c48072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:36.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481014b0 0x7f2c48072ee0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00eb30 tx=0x7f2c3c00ee40 comp rx=0 tx=0).stop 2023-10-11T16:29:36.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 shutdown_connections 2023-10-11T16:29:36.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2c300a1150 0x7f2c300a3600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2c48101e40 0x7f2c48073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c481002c0 0x7f2c480748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 --2- 172.21.15.111:0/2605395028 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c481014b0 0x7f2c48072ee0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:36.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.837+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 >> 172.21.15.111:0/2605395028 conn(0x7f2c480fb8a0 msgr2=0x7f2c480fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:36.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.838+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 shutdown_connections 2023-10-11T16:29:36.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:36.838+0000 7f2c4c9ff700 1 -- 172.21.15.111:0/2605395028 wait complete. 2023-10-11T16:29:36.858 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:29:36.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: Saving service mds.all spec with placement smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l;count:12 2023-10-11T16:29:36.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:36.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:36.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:36.859 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:36.860 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:36.860 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:36.860 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:36 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1643477995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:29:36.861 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:29:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: Saving service mds.all spec with placement smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l;count:12 2023-10-11T16:29:36.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:36.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:36.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:36.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:36.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:36.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:36.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:36 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1643477995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:29:37.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/2701136674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141002b0 msgr2=0x7f8e141006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:37.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/2701136674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141002b0 0x7f8e141006c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009990 tx=0x7f8e04009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/2701136674 shutdown_connections 2023-10-11T16:29:37.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/2701136674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e14101e30 0x7f8e141086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/2701136674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141002b0 0x7f8e141006c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/2701136674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e141014a0 0x7f8e141018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/2701136674 >> 172.21.15.111:0/2701136674 conn(0x7f8e140fb890 msgr2=0x7f8e140fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:37.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/2701136674 shutdown_connections 2023-10-11T16:29:37.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.058+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/2701136674 wait complete. 2023-10-11T16:29:37.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.059+0000 7f8e1a7ff700 1 Processor -- start 2023-10-11T16:29:37.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e1a7ff700 1 -- start start 2023-10-11T16:29:37.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e1a7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e141002b0 0x7f8e14198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:37.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e1a7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141014a0 0x7f8e141986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:37.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e1a7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e14101e30 0x7f8e1419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:37.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e1a7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1419fcf0 con 0x7f8e14101e30 2023-10-11T16:29:37.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e1a7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1419fe30 con 0x7f8e141002b0 2023-10-11T16:29:37.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e1a7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1419ffc0 con 0x7f8e141014a0 2023-10-11T16:29:37.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e13fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e141002b0 0x7f8e14198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:37.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e0bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141014a0 0x7f8e141986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:37.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e13fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e141002b0 0x7f8e14198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:37818/0 (socket says 172.21.15.111:37818) 2023-10-11T16:29:37.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e0bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141014a0 0x7f8e141986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:37554/0 (socket says 172.21.15.111:37554) 2023-10-11T16:29:37.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e13fff700 1 -- 172.21.15.111:0/3906778541 learned_addr learned my addr 172.21.15.111:0/3906778541 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:37.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e18d9c700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e14101e30 0x7f8e1419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:37.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e0bfff700 1 -- 172.21.15.111:0/3906778541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e141002b0 msgr2=0x7f8e14198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:37.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e0bfff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e141002b0 0x7f8e14198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e0bfff700 1 -- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e14101e30 msgr2=0x7f8e1419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:37.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e0bfff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e14101e30 0x7f8e1419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.060+0000 7f8e0bfff700 1 -- 172.21.15.111:0/3906778541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e040095f0 con 0x7f8e141014a0 2023-10-11T16:29:37.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.061+0000 7f8e0bfff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141014a0 0x7f8e141986b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00e9f0 tx=0x7f8dfc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:37.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.061+0000 7f8e11ffb700 1 -- 172.21.15.111:0/3906778541 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc00c9c0 con 0x7f8e141014a0 2023-10-11T16:29:37.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.061+0000 7f8e11ffb700 1 -- 172.21.15.111:0/3906778541 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dfc010430 con 0x7f8e141014a0 2023-10-11T16:29:37.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.061+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1410bc00 con 0x7f8e141014a0 2023-10-11T16:29:37.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.061+0000 7f8e11ffb700 1 -- 172.21.15.111:0/3906778541 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc018680 con 0x7f8e141014a0 2023-10-11T16:29:37.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.061+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1410ba70 con 0x7f8e141014a0 2023-10-11T16:29:37.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.062+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e141a0700 con 0x7f8e141014a0 2023-10-11T16:29:37.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.064+0000 7f8e11ffb700 1 -- 172.21.15.111:0/3906778541 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f8dfc010860 con 0x7f8e141014a0 2023-10-11T16:29:37.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.065+0000 7f8e11ffb700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e000a11d0 0x7f8e000a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:37.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.065+0000 7f8e11ffb700 1 -- 172.21.15.111:0/3906778541 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8dfc014070 con 0x7f8e141014a0 2023-10-11T16:29:37.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.065+0000 7f8e13fff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e000a11d0 0x7f8e000a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:37.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.066+0000 7f8e13fff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e000a11d0 0x7f8e000a3680 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400b2e0 tx=0x7f8e04005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:37.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.067+0000 7f8e11ffb700 1 -- 172.21.15.111:0/3906778541 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8dfc0960b0 con 0x7f8e141014a0 2023-10-11T16:29:37.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:29:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: Saving service mds.all spec with placement smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l;count:12 2023-10-11T16:29:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:29:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:37.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:37.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:37.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:36 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1643477995' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:29:37.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.310+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e141a0cb0 con 0x7f8e141014a0 2023-10-11T16:29:37.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.310+0000 7f8e11ffb700 1 -- 172.21.15.111:0/3906778541 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f8dfc0996d0 con 0x7f8e141014a0 2023-10-11T16:29:37.312 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:37.312 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-10-11T16:29:37.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e000a11d0 msgr2=0x7f8e000a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:37.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e000a11d0 0x7f8e000a3680 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400b2e0 tx=0x7f8e04005fb0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141014a0 msgr2=0x7f8e141986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:37.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141014a0 0x7f8e141986b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00e9f0 tx=0x7f8dfc00ed00 comp rx=0 tx=0).stop 2023-10-11T16:29:37.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 shutdown_connections 2023-10-11T16:29:37.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e000a11d0 0x7f8e000a3680 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e14101e30 0x7f8e1419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e141002b0 0x7f8e14198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 --2- 172.21.15.111:0/3906778541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e141014a0 0x7f8e141986b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:37.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 >> 172.21.15.111:0/3906778541 conn(0x7f8e140fb890 msgr2=0x7f8e140fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:37.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 shutdown_connections 2023-10-11T16:29:37.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:37.313+0000 7f8e1a7ff700 1 -- 172.21.15.111:0/3906778541 wait complete. 2023-10-11T16:29:37.318 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 1 2023-10-11T16:29:37.663 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:29:37 smithi150 systemd[1]: Starting Ceph mds.l for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:29:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:29:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:29:37.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mon[132217]: Deploying daemon mds.l on smithi150 2023-10-11T16:29:37.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mon[132217]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:37.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2605395028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:29:37.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3906778541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:37.955 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:29:37 smithi150 podman[157475]: 2023-10-11T16:29:37.956 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-mds[157516]: -- 172.21.15.150:0/471147130 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562d7282d520 con 0x562d72802c00 2023-10-11T16:29:37.956 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:29:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-l[157491]: starting mds.l at 2023-10-11T16:29:37.956 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:29:37 smithi150 bash[157475]: fa5a498d4653e8ebd1f68b9b27a80b583ecd019fa3a9f5dd7265347d9eed4214 2023-10-11T16:29:37.957 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:29:37 smithi150 systemd[1]: Started Ceph mds.l for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:37 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:37 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:29:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:37 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:29:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:37 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:29:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:37 smithi130 ceph-mon[131859]: Deploying daemon mds.l on smithi150 2023-10-11T16:29:38.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:37 smithi130 ceph-mon[131859]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:38.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:37 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2605395028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:29:38.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:37 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3906778541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:38.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:37 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:38.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:37 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:29:38.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:37 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:29:38.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:37 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:29:38.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:37 smithi111 ceph-mon[98527]: Deploying daemon mds.l on smithi150 2023-10-11T16:29:38.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:37 smithi111 ceph-mon[98527]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:38.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:37 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2605395028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:29:38.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:37 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3906778541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:39.020 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6832/2297521930,v1:172.21.15.150:6833/2297521930] up:boot 2023-10-11T16:29:39.020 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: fsmap 1 up:standby 2023-10-11T16:29:39.020 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-11T16:29:39.021 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.021 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.021 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.021 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:39.021 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:39.022 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:39.022 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: Deploying daemon mds.k on smithi130 2023-10-11T16:29:39.022 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:38 smithi130 ceph-mon[131859]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:39.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6832/2297521930,v1:172.21.15.150:6833/2297521930] up:boot 2023-10-11T16:29:39.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: fsmap 1 up:standby 2023-10-11T16:29:39.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-11T16:29:39.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:39.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:39.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:39.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: Deploying daemon mds.k on smithi130 2023-10-11T16:29:39.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:38 smithi150 ceph-mon[132217]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:39.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:39.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6832/2297521930,v1:172.21.15.150:6833/2297521930] up:boot 2023-10-11T16:29:39.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: fsmap 1 up:standby 2023-10-11T16:29:39.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-11T16:29:39.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:39.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:39.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:39.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:39.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: Deploying daemon mds.k on smithi130 2023-10-11T16:29:39.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:38 smithi111 ceph-mon[98527]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:39.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.540+0000 7fe4501ff700 1 -- 172.21.15.111:0/3154821778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4481002b0 msgr2=0x7fe4481006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:39.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.540+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3154821778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4481002b0 0x7fe4481006c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe444009990 tx=0x7fe444009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.541+0000 7fe4501ff700 1 -- 172.21.15.111:0/3154821778 shutdown_connections 2023-10-11T16:29:39.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.541+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3154821778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe448101e30 0x7fe4481086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.541+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3154821778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4481002b0 0x7fe4481006c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.541+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3154821778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe4481014a0 0x7fe4481018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.541+0000 7fe4501ff700 1 -- 172.21.15.111:0/3154821778 >> 172.21.15.111:0/3154821778 conn(0x7fe4480fb890 msgr2=0x7fe4480fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:39.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.541+0000 7fe4501ff700 1 -- 172.21.15.111:0/3154821778 shutdown_connections 2023-10-11T16:29:39.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.541+0000 7fe4501ff700 1 -- 172.21.15.111:0/3154821778 wait complete. 2023-10-11T16:29:39.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe4501ff700 1 Processor -- start 2023-10-11T16:29:39.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe4501ff700 1 -- start start 2023-10-11T16:29:39.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe4501ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe4481002b0 0x7fe448198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:39.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe4501ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4481014a0 0x7fe448198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:39.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe4501ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe448101e30 0x7fe44819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:39.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe4501ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44819fda0 con 0x7fe4481014a0 2023-10-11T16:29:39.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe4501ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44819fee0 con 0x7fe4481002b0 2023-10-11T16:29:39.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe4501ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4481a0070 con 0x7fe448101e30 2023-10-11T16:29:39.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe44df9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe4481002b0 0x7fe448198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:39.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe44df9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe4481002b0 0x7fe448198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:37840/0 (socket says 172.21.15.111:37840) 2023-10-11T16:29:39.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe44df9b700 1 -- 172.21.15.111:0/3649686600 learned_addr learned my addr 172.21.15.111:0/3649686600 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:39.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.542+0000 7fe44d79a700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4481014a0 0x7fe448198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:39.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe44e79c700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe448101e30 0x7fe44819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:39.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe44df9b700 1 -- 172.21.15.111:0/3649686600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe448101e30 msgr2=0x7fe44819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:39.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe44df9b700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe448101e30 0x7fe44819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe44df9b700 1 -- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4481014a0 msgr2=0x7fe448198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:39.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe44df9b700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4481014a0 0x7fe448198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe44df9b700 1 -- 172.21.15.111:0/3649686600 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4440095f0 con 0x7fe4481002b0 2023-10-11T16:29:39.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe44df9b700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe4481002b0 0x7fe448198220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe44400b620 tx=0x7fe444005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:39.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe43effd700 1 -- 172.21.15.111:0/3649686600 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44401d070 con 0x7fe4481002b0 2023-10-11T16:29:39.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44810bce0 con 0x7fe4481002b0 2023-10-11T16:29:39.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe43effd700 1 -- 172.21.15.111:0/3649686600 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe444021410 con 0x7fe4481002b0 2023-10-11T16:29:39.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe43effd700 1 -- 172.21.15.111:0/3649686600 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444017650 con 0x7fe4481002b0 2023-10-11T16:29:39.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.543+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44810bb50 con 0x7fe4481002b0 2023-10-11T16:29:39.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.544+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe448192970 con 0x7fe4481002b0 2023-10-11T16:29:39.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.546+0000 7fe43effd700 1 -- 172.21.15.111:0/3649686600 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fe444021580 con 0x7fe4481002b0 2023-10-11T16:29:39.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.548+0000 7fe43effd700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe4340a1150 0x7fe4340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:39.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.548+0000 7fe43effd700 1 -- 172.21.15.111:0/3649686600 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe4440cdab0 con 0x7fe4481002b0 2023-10-11T16:29:39.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.548+0000 7fe44d79a700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe4340a1150 0x7fe4340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:39.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.548+0000 7fe44d79a700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe4340a1150 0x7fe4340a3600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe438009a10 tx=0x7fe438009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:39.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.550+0000 7fe43effd700 1 -- 172.21.15.111:0/3649686600 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe444097690 con 0x7fe4481002b0 2023-10-11T16:29:39.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.795+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe448061700 con 0x7fe4481002b0 2023-10-11T16:29:39.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.796+0000 7fe43effd700 1 -- 172.21.15.111:0/3649686600 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7fe44409acb0 con 0x7fe4481002b0 2023-10-11T16:29:39.797 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:39.798 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-10-11T16:29:39.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe4340a1150 msgr2=0x7fe4340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:39.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe4340a1150 0x7fe4340a3600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe438009a10 tx=0x7fe438009190 comp rx=0 tx=0).stop 2023-10-11T16:29:39.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe4481002b0 msgr2=0x7fe448198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:39.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe4481002b0 0x7fe448198220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe44400b620 tx=0x7fe444005740 comp rx=0 tx=0).stop 2023-10-11T16:29:39.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 shutdown_connections 2023-10-11T16:29:39.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe4340a1150 0x7fe4340a3600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe448101e30 0x7fe44819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe4481002b0 0x7fe448198220 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 --2- 172.21.15.111:0/3649686600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4481014a0 0x7fe448198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:39.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 >> 172.21.15.111:0/3649686600 conn(0x7fe4480fb890 msgr2=0x7fe448106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:39.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 shutdown_connections 2023-10-11T16:29:39.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:39.798+0000 7fe4501ff700 1 -- 172.21.15.111:0/3649686600 wait complete. 2023-10-11T16:29:39.803 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 2 2023-10-11T16:29:39.804 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:29:39 smithi130 systemd[1]: Starting Ceph mds.k for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:40.081 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:29:39 smithi130 podman[156046]: 2023-10-11T16:29:40.082 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:29:40 smithi130 bash[156046]: 906aeca6e4fe5f7e316e6f51491389d8b0a510200f54973d3cbb673bb9101f2d 2023-10-11T16:29:40.082 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-mds[156086]: -- 172.21.15.130:0/2736665210 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5629edec3520 con 0x5629ede98c00 2023-10-11T16:29:40.082 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-k[156062]: starting mds.k at 2023-10-11T16:29:40.581 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:29:40 smithi130 systemd[1]: Started Ceph mds.k for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:40.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:40 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3649686600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:40.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:40.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:40.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:40.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:40.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:40.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:41.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3649686600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:41.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:41.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:41.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:41.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:41.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:41.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:41.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:40 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3649686600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:41.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:41.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:41.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:41.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:41.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:41.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:41.822 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:41 smithi111 ceph-mon[98527]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:41 smithi111 ceph-mon[98527]: Deploying daemon mds.j on smithi111 2023-10-11T16:29:41.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:41 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.130:6832/447441834,v1:172.21.15.130:6833/447441834] up:boot 2023-10-11T16:29:41.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:41 smithi111 ceph-mon[98527]: fsmap 2 up:standby 2023-10-11T16:29:41.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-11T16:29:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:41 smithi150 ceph-mon[132217]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:41 smithi150 ceph-mon[132217]: Deploying daemon mds.j on smithi111 2023-10-11T16:29:41.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:41 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.130:6832/447441834,v1:172.21.15.130:6833/447441834] up:boot 2023-10-11T16:29:41.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:41 smithi150 ceph-mon[132217]: fsmap 2 up:standby 2023-10-11T16:29:41.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-11T16:29:42.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:41 smithi130 ceph-mon[131859]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:42.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:41 smithi130 ceph-mon[131859]: Deploying daemon mds.j on smithi111 2023-10-11T16:29:42.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:41 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.130:6832/447441834,v1:172.21.15.130:6833/447441834] up:boot 2023-10-11T16:29:42.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:41 smithi130 ceph-mon[131859]: fsmap 2 up:standby 2023-10-11T16:29:42.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-11T16:29:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 -- 172.21.15.111:0/3651916127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 msgr2=0x7fd870100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 --2- 172.21.15.111:0/3651916127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 0x7fd870100fa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0098f0 tx=0x7fd86c009c00 comp rx=0 tx=0).stop 2023-10-11T16:29:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 -- 172.21.15.111:0/3651916127 shutdown_connections 2023-10-11T16:29:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 --2- 172.21.15.111:0/3651916127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd870103ea0 0x7fd870104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 --2- 172.21.15.111:0/3651916127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 0x7fd870100fa0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 --2- 172.21.15.111:0/3651916127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8701014e0 0x7fd870103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 -- 172.21.15.111:0/3651916127 >> 172.21.15.111:0/3651916127 conn(0x7fd8700fa750 msgr2=0x7fd8700fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:42.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 -- 172.21.15.111:0/3651916127 shutdown_connections 2023-10-11T16:29:42.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.269+0000 7fd877fff700 1 -- 172.21.15.111:0/3651916127 wait complete. 2023-10-11T16:29:42.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.270+0000 7fd877fff700 1 Processor -- start 2023-10-11T16:29:42.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.270+0000 7fd877fff700 1 -- start start 2023-10-11T16:29:42.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd877fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 0x7fd87019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd877fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8701014e0 0x7fd87019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd877fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd870103ea0 0x7fd8701a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd877fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701a41c0 con 0x7fd8700feb80 2023-10-11T16:29:42.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd877fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701a4300 con 0x7fd870103ea0 2023-10-11T16:29:42.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd877fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701a4490 con 0x7fd8701014e0 2023-10-11T16:29:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd875d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 0x7fd87019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd87659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd870103ea0 0x7fd8701a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd875d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 0x7fd87019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37354/0 (socket says 172.21.15.111:37354) 2023-10-11T16:29:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd87659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd870103ea0 0x7fd8701a3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:37860/0 (socket says 172.21.15.111:37860) 2023-10-11T16:29:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd875d9b700 1 -- 172.21.15.111:0/3610752658 learned_addr learned my addr 172.21.15.111:0/3610752658 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd87559a700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8701014e0 0x7fd87019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd875d9b700 1 -- 172.21.15.111:0/3610752658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8701014e0 msgr2=0x7fd87019cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd875d9b700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8701014e0 0x7fd87019cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd875d9b700 1 -- 172.21.15.111:0/3610752658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd870103ea0 msgr2=0x7fd8701a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd875d9b700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd870103ea0 0x7fd8701a3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.271+0000 7fd875d9b700 1 -- 172.21.15.111:0/3610752658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd868009d00 con 0x7fd8700feb80 2023-10-11T16:29:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.272+0000 7fd875d9b700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 0x7fd87019c640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009fd0 tx=0x7fd86c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.272+0000 7fd866ffd700 1 -- 172.21.15.111:0/3610752658 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c01c070 con 0x7fd8700feb80 2023-10-11T16:29:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.272+0000 7fd866ffd700 1 -- 172.21.15.111:0/3610752658 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd86c00f7d0 con 0x7fd8700feb80 2023-10-11T16:29:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.272+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c009520 con 0x7fd8700feb80 2023-10-11T16:29:42.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.272+0000 7fd866ffd700 1 -- 172.21.15.111:0/3610752658 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c003a40 con 0x7fd8700feb80 2023-10-11T16:29:42.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.272+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd870112230 con 0x7fd8700feb80 2023-10-11T16:29:42.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.273+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd870067a60 con 0x7fd8700feb80 2023-10-11T16:29:42.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.275+0000 7fd866ffd700 1 -- 172.21.15.111:0/3610752658 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fd86c020030 con 0x7fd8700feb80 2023-10-11T16:29:42.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.276+0000 7fd866ffd700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd85c0a1150 0x7fd85c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:42.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.276+0000 7fd866ffd700 1 -- 172.21.15.111:0/3610752658 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd86c0cd460 con 0x7fd8700feb80 2023-10-11T16:29:42.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.277+0000 7fd87559a700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd85c0a1150 0x7fd85c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:42.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.277+0000 7fd87559a700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd85c0a1150 0x7fd85c0a3600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd860009c60 tx=0x7fd860009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:42.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.280+0000 7fd866ffd700 1 -- 172.21.15.111:0/3610752658 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd86c097040 con 0x7fd8700feb80 2023-10-11T16:29:42.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.535+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8701a4e50 con 0x7fd8700feb80 2023-10-11T16:29:42.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.535+0000 7fd866ffd700 1 -- 172.21.15.111:0/3610752658 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2017 (secure 0 0 0) 0x7fd86c09a660 con 0x7fd8700feb80 2023-10-11T16:29:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:42.538 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-10-11T16:29:42.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd85c0a1150 msgr2=0x7fd85c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:42.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd85c0a1150 0x7fd85c0a3600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd860009c60 tx=0x7fd860009260 comp rx=0 tx=0).stop 2023-10-11T16:29:42.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 msgr2=0x7fd87019c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:42.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 0x7fd87019c640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009fd0 tx=0x7fd86c00b790 comp rx=0 tx=0).stop 2023-10-11T16:29:42.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 shutdown_connections 2023-10-11T16:29:42.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd85c0a1150 0x7fd85c0a3600 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd870103ea0 0x7fd8701a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8700feb80 0x7fd87019c640 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 --2- 172.21.15.111:0/3610752658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8701014e0 0x7fd87019cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:42.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 >> 172.21.15.111:0/3610752658 conn(0x7fd8700fa750 msgr2=0x7fd870102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:42.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 shutdown_connections 2023-10-11T16:29:42.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:42.538+0000 7fd877fff700 1 -- 172.21.15.111:0/3610752658 wait complete. 2023-10-11T16:29:42.543 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 3 2023-10-11T16:29:42.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:42 smithi111 ceph-mon[98527]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:42.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:42 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3610752658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:42.886 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:29:42 smithi111 systemd[1]: Starting Ceph mds.j for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:42.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:42 smithi150 ceph-mon[132217]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:42.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:42 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3610752658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:43.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:42 smithi130 ceph-mon[131859]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:43.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:42 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3610752658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:43.216 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:29:42 smithi111 podman[131272]: 2023-10-11T16:29:43.217 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:29:43 smithi111 bash[131272]: 51ccc626f0d708048d2b38b08c0d2f120010078d056634ce9a5a9a3d30b40144 2023-10-11T16:29:43.217 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:29:43 smithi111 ceph-mds[131312]: -- 172.21.15.111:0/3787881986 <== mon.0 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560aef795520 con 0x560aef76cc00 2023-10-11T16:29:43.217 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:29:43 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-j[131288]: starting mds.j at 2023-10-11T16:29:43.635 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:29:43 smithi111 systemd[1]: Started Ceph mds.j for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:44.561 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:44.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:44.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:44.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:44.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: Deploying daemon mds.d on smithi111 2023-10-11T16:29:44.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.111:6834/1910869239,v1:172.21.15.111:6835/1910869239] up:boot 2023-10-11T16:29:44.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: fsmap 3 up:standby 2023-10-11T16:29:44.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:44 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-11T16:29:44.630 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:44.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: Deploying daemon mds.d on smithi111 2023-10-11T16:29:44.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.111:6834/1910869239,v1:172.21.15.111:6835/1910869239] up:boot 2023-10-11T16:29:44.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: fsmap 3 up:standby 2023-10-11T16:29:44.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:44 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-11T16:29:44.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:44.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:44.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:44.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:44.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: Deploying daemon mds.d on smithi111 2023-10-11T16:29:44.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.111:6834/1910869239,v1:172.21.15.111:6835/1910869239] up:boot 2023-10-11T16:29:44.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: fsmap 3 up:standby 2023-10-11T16:29:44.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:44 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-11T16:29:44.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 -- 172.21.15.111:0/2277611819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 msgr2=0x7fbae01006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:44.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 --2- 172.21.15.111:0/2277611819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae01006a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbad400b0a0 tx=0x7fbad400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:29:44.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 -- 172.21.15.111:0/2277611819 shutdown_connections 2023-10-11T16:29:44.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 --2- 172.21.15.111:0/2277611819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101e10 0x7fbae01086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:44.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 --2- 172.21.15.111:0/2277611819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae01006a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:44.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 --2- 172.21.15.111:0/2277611819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101480 0x7fbae01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:44.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 -- 172.21.15.111:0/2277611819 >> 172.21.15.111:0/2277611819 conn(0x7fbae00fb870 msgr2=0x7fbae00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:44.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 -- 172.21.15.111:0/2277611819 shutdown_connections 2023-10-11T16:29:44.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.773+0000 7fbae6bff700 1 -- 172.21.15.111:0/2277611819 wait complete. 2023-10-11T16:29:44.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.774+0000 7fbae6bff700 1 Processor -- start 2023-10-11T16:29:44.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.774+0000 7fbae6bff700 1 -- start start 2023-10-11T16:29:44.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae6bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae0196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:44.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae6bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101480 0x7fbae0196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:44.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae6bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101e10 0x7fbae019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:44.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae6bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae019dcb0 con 0x7fbae0100290 2023-10-11T16:29:44.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae6bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae019ddf0 con 0x7fbae0101e10 2023-10-11T16:29:44.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae6bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae019df80 con 0x7fbae0101480 2023-10-11T16:29:44.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae499b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae0196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:44.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbadffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101480 0x7fbae0196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:44.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae499b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae0196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37400/0 (socket says 172.21.15.111:37400) 2023-10-11T16:29:44.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae499b700 1 -- 172.21.15.111:0/1348973951 learned_addr learned my addr 172.21.15.111:0/1348973951 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:44.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae519c700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101e10 0x7fbae019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:44.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae499b700 1 -- 172.21.15.111:0/1348973951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101480 msgr2=0x7fbae0196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:44.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae499b700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101480 0x7fbae0196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:44.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae499b700 1 -- 172.21.15.111:0/1348973951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101e10 msgr2=0x7fbae019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:44.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae499b700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101e10 0x7fbae019d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:44.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbae499b700 1 -- 172.21.15.111:0/1348973951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad8009520 con 0x7fbae0100290 2023-10-11T16:29:44.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.775+0000 7fbadffff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101480 0x7fbae0196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:29:44.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.776+0000 7fbae499b700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae0196130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbad400b070 tx=0x7fbad400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.776+0000 7fbaddffb700 1 -- 172.21.15.111:0/1348973951 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad400e040 con 0x7fbae0100290 2023-10-11T16:29:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.776+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad4009d00 con 0x7fbae0100290 2023-10-11T16:29:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.776+0000 7fbaddffb700 1 -- 172.21.15.111:0/1348973951 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbad4003c00 con 0x7fbae0100290 2023-10-11T16:29:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.776+0000 7fbaddffb700 1 -- 172.21.15.111:0/1348973951 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad40052a0 con 0x7fbae0100290 2023-10-11T16:29:44.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.776+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae010bc40 con 0x7fbae0100290 2023-10-11T16:29:44.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.777+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae0067a60 con 0x7fbae0100290 2023-10-11T16:29:44.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.779+0000 7fbaddffb700 1 -- 172.21.15.111:0/1348973951 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fbad401d480 con 0x7fbae0100290 2023-10-11T16:29:44.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.780+0000 7fbaddffb700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbad00a1150 0x7fbad00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:44.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.780+0000 7fbaddffb700 1 -- 172.21.15.111:0/1348973951 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbad40cd430 con 0x7fbae0100290 2023-10-11T16:29:44.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.780+0000 7fbadffff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbad00a1150 0x7fbad00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:44.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.781+0000 7fbadffff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbad00a1150 0x7fbad00a3600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbacc011490 tx=0x7fbacc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:44.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:44.785+0000 7fbaddffb700 1 -- 172.21.15.111:0/1348973951 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbad4097010 con 0x7fbae0100290 2023-10-11T16:29:45.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.093+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbae0107210 con 0x7fbae0100290 2023-10-11T16:29:45.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.094+0000 7fbaddffb700 1 -- 172.21.15.111:0/1348973951 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7fbad409a630 con 0x7fbae0100290 2023-10-11T16:29:45.095 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:45.096 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-10-11T16:29:45.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbad00a1150 msgr2=0x7fbad00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:45.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbad00a1150 0x7fbad00a3600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbacc011490 tx=0x7fbacc009260 comp rx=0 tx=0).stop 2023-10-11T16:29:45.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 msgr2=0x7fbae0196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:45.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae0196130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbad400b070 tx=0x7fbad400ba50 comp rx=0 tx=0).stop 2023-10-11T16:29:45.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 shutdown_connections 2023-10-11T16:29:45.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbad00a1150 0x7fbad00a3600 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:45.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101e10 0x7fbae019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:45.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae0196130 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:45.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 --2- 172.21.15.111:0/1348973951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101480 0x7fbae0196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:45.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 >> 172.21.15.111:0/1348973951 conn(0x7fbae00fb870 msgr2=0x7fbae00fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:45.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 shutdown_connections 2023-10-11T16:29:45.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:45.096+0000 7fbae6bff700 1 -- 172.21.15.111:0/1348973951 wait complete. 2023-10-11T16:29:45.100 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 4 2023-10-11T16:29:45.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:45 smithi130 ceph-mon[131859]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:45.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:45 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1348973951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:45.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:45 smithi111 ceph-mon[98527]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:45.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:45 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1348973951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:45.637 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:29:45 smithi111 systemd[1]: Starting Ceph mds.d for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:45 smithi150 ceph-mon[132217]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:45 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1348973951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:46.210 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:29:45 smithi111 podman[131662]: 2023-10-11T16:29:46.210 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:29:46 smithi111 bash[131662]: 05dd4edef92d92fcc2d617eb08196f1af080dd37c950ec5fabf0a8d8cf12277f 2023-10-11T16:29:46.635 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:29:46 smithi111 ceph-mds[131702]: -- 172.21.15.111:0/2801142146 <== mon.0 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e6e3f5520 con 0x557e6e3cc800 2023-10-11T16:29:46.636 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:29:46 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-d[131678]: starting mds.d at 2023-10-11T16:29:46.636 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:29:46 smithi111 systemd[1]: Started Ceph mds.d for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:47.118 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:47.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.326+0000 7f926afff700 1 -- 172.21.15.111:0/2151916148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264074b30 msgr2=0x7f9264072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:47.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.326+0000 7f926afff700 1 --2- 172.21.15.111:0/2151916148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264074b30 0x7f9264072f80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9258009990 tx=0x7f9258009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.326+0000 7f926afff700 1 -- 172.21.15.111:0/2151916148 shutdown_connections 2023-10-11T16:29:47.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.326+0000 7f926afff700 1 --2- 172.21.15.111:0/2151916148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9264108150 0x7f926410a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.326+0000 7f926afff700 1 --2- 172.21.15.111:0/2151916148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264074b30 0x7f9264072f80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.326+0000 7f926afff700 1 --2- 172.21.15.111:0/2151916148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9264073550 0x7f92640739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.326+0000 7f926afff700 1 -- 172.21.15.111:0/2151916148 >> 172.21.15.111:0/2151916148 conn(0x7f92640fb870 msgr2=0x7f92640fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:47.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.326+0000 7f926afff700 1 -- 172.21.15.111:0/2151916148 shutdown_connections 2023-10-11T16:29:47.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.327+0000 7f926afff700 1 -- 172.21.15.111:0/2151916148 wait complete. 2023-10-11T16:29:47.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.327+0000 7f926afff700 1 Processor -- start 2023-10-11T16:29:47.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f926afff700 1 -- start start 2023-10-11T16:29:47.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f926afff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264073550 0x7f9264198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:47.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f926afff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9264074b30 0x7f9264198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:47.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f926afff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9264108150 0x7f926419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:47.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f926afff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926419fda0 con 0x7f9264073550 2023-10-11T16:29:47.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f926afff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926419fee0 con 0x7f9264074b30 2023-10-11T16:29:47.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f926afff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92641a0070 con 0x7f9264108150 2023-10-11T16:29:47.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f9263fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9264074b30 0x7f9264198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:47.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f9268d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264073550 0x7f9264198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:47.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f9263fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9264074b30 0x7f9264198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:39460/0 (socket says 172.21.15.111:39460) 2023-10-11T16:29:47.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f9263fff700 1 -- 172.21.15.111:0/1533936770 learned_addr learned my addr 172.21.15.111:0/1533936770 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:47.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.328+0000 7f926959c700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9264108150 0x7f926419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:47.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9263fff700 1 -- 172.21.15.111:0/1533936770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9264108150 msgr2=0x7f926419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:47.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9263fff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9264108150 0x7f926419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9263fff700 1 -- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264073550 msgr2=0x7f9264198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:47.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9263fff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264073550 0x7f9264198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9263fff700 1 -- 172.21.15.111:0/1533936770 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580095f0 con 0x7f9264074b30 2023-10-11T16:29:47.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9268d9b700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264073550 0x7f9264198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:29:47.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9263fff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9264074b30 0x7f9264198760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f925000e970 tx=0x7f925000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:47.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9261ffb700 1 -- 172.21.15.111:0/1533936770 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250009790 con 0x7f9264074b30 2023-10-11T16:29:47.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9261ffb700 1 -- 172.21.15.111:0/1533936770 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f925000c7e0 con 0x7f9264074b30 2023-10-11T16:29:47.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926410de50 con 0x7f9264074b30 2023-10-11T16:29:47.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f9261ffb700 1 -- 172.21.15.111:0/1533936770 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250003c70 con 0x7f9264074b30 2023-10-11T16:29:47.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.329+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926410dcc0 con 0x7f9264074b30 2023-10-11T16:29:47.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.331+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9264106800 con 0x7f9264074b30 2023-10-11T16:29:47.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.332+0000 7f9261ffb700 1 -- 172.21.15.111:0/1533936770 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f925000cc50 con 0x7f9264074b30 2023-10-11T16:29:47.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.334+0000 7f9261ffb700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f92540a1150 0x7f92540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:47.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.334+0000 7f9261ffb700 1 -- 172.21.15.111:0/1533936770 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9250014070 con 0x7f9264074b30 2023-10-11T16:29:47.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.334+0000 7f9268d9b700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f92540a1150 0x7f92540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:47.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.335+0000 7f9268d9b700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f92540a1150 0x7f92540a3600 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f925800b2e0 tx=0x7f9258005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:47.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.339+0000 7f9261ffb700 1 -- 172.21.15.111:0/1533936770 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9250096210 con 0x7f9264074b30 2023-10-11T16:29:47.410 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:47.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.111:6836/3168828680,v1:172.21.15.111:6837/3168828680] up:boot 2023-10-11T16:29:47.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: fsmap 4 up:standby 2023-10-11T16:29:47.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-11T16:29:47.411 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.412 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.412 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.412 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:47.412 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:47.412 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:47.413 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:47 smithi130 ceph-mon[131859]: Deploying daemon mds.e on smithi130 2023-10-11T16:29:47.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.577+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f926402cfe0 con 0x7f9264074b30 2023-10-11T16:29:47.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.578+0000 7f9261ffb700 1 -- 172.21.15.111:0/1533936770 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3537 (secure 0 0 0) 0x7f9250003dd0 con 0x7f9264074b30 2023-10-11T16:29:47.580 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:47.581 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-10-11T16:29:47.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f92540a1150 msgr2=0x7f92540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:47.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f92540a1150 0x7f92540a3600 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f925800b2e0 tx=0x7f9258005fb0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9264074b30 msgr2=0x7f9264198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:47.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9264074b30 0x7f9264198760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f925000e970 tx=0x7f925000ec80 comp rx=0 tx=0).stop 2023-10-11T16:29:47.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 shutdown_connections 2023-10-11T16:29:47.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f92540a1150 0x7f92540a3600 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9264108150 0x7f926419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9264073550 0x7f9264198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 --2- 172.21.15.111:0/1533936770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9264074b30 0x7f9264198760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:47.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.580+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 >> 172.21.15.111:0/1533936770 conn(0x7f92640fb870 msgr2=0x7f92640fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:47.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.581+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 shutdown_connections 2023-10-11T16:29:47.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:47.581+0000 7f926afff700 1 -- 172.21.15.111:0/1533936770 wait complete. 2023-10-11T16:29:47.587 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 5 2023-10-11T16:29:47.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:47.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.111:6836/3168828680,v1:172.21.15.111:6837/3168828680] up:boot 2023-10-11T16:29:47.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: fsmap 4 up:standby 2023-10-11T16:29:47.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-11T16:29:47.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.599 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.600 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.600 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:47.600 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:47.600 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:47.600 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:47 smithi111 ceph-mon[98527]: Deploying daemon mds.e on smithi130 2023-10-11T16:29:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.111:6836/3168828680,v1:172.21.15.111:6837/3168828680] up:boot 2023-10-11T16:29:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: fsmap 4 up:standby 2023-10-11T16:29:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-11T16:29:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:47.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:47.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:47 smithi150 ceph-mon[132217]: Deploying daemon mds.e on smithi130 2023-10-11T16:29:48.082 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:29:47 smithi130 systemd[1]: Starting Ceph mds.e for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:48.392 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:48 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1533936770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:48.392 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:29:48 smithi130 podman[156385]: 2023-10-11T16:29:48.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:48 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1533936770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:48.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:48 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1533936770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:48.831 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:29:48 smithi130 bash[156385]: aad645dd999d73b18176e3453477ac2644687ec1f7b237319507636b8a21ed67 2023-10-11T16:29:48.832 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:29:48 smithi130 ceph-mds[156425]: -- 172.21.15.130:0/420799460 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560b54091520 con 0x560b54067000 2023-10-11T16:29:48.832 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:29:48 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-e[156401]: starting mds.e at 2023-10-11T16:29:48.832 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:29:48 smithi130 systemd[1]: Started Ceph mds.e for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:49.602 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:49.792 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:49.793 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.794 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.794 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.794 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:49.794 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:49.794 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:49.795 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: Deploying daemon mds.f on smithi150 2023-10-11T16:29:49.795 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.130:6834/89303870,v1:172.21.15.130:6835/89303870] up:boot 2023-10-11T16:29:49.795 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: fsmap 5 up:standby 2023-10-11T16:29:49.795 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:49 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:29:49.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 -- 172.21.15.111:0/3653452738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5181002d0 msgr2=0x7ff5181006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:49.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 --2- 172.21.15.111:0/3653452738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5181002d0 0x7ff5181006e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff508009990 tx=0x7ff508009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:49.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 -- 172.21.15.111:0/3653452738 shutdown_connections 2023-10-11T16:29:49.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 --2- 172.21.15.111:0/3653452738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff518101f10 0x7ff5181087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:49.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 --2- 172.21.15.111:0/3653452738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5181002d0 0x7ff5181006e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:49.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 --2- 172.21.15.111:0/3653452738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff518101560 0x7ff5181019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:49.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 -- 172.21.15.111:0/3653452738 >> 172.21.15.111:0/3653452738 conn(0x7ff5180fb930 msgr2=0x7ff5180fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:49.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 -- 172.21.15.111:0/3653452738 shutdown_connections 2023-10-11T16:29:49.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.810+0000 7ff51cfff700 1 -- 172.21.15.111:0/3653452738 wait complete. 2023-10-11T16:29:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.811+0000 7ff51cfff700 1 Processor -- start 2023-10-11T16:29:49.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.811+0000 7ff51cfff700 1 -- start start 2023-10-11T16:29:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.811+0000 7ff51cfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5181002d0 0x7ff5181981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.811+0000 7ff51cfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff518101560 0x7ff5181986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51cfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff518101f10 0x7ff51819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51cfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51819fd20 con 0x7ff518101560 2023-10-11T16:29:49.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51cfff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51819fe60 con 0x7ff518101f10 2023-10-11T16:29:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51cfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51819fff0 con 0x7ff5181002d0 2023-10-11T16:29:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5181002d0 0x7ff5181981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff515d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff518101560 0x7ff5181986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5181002d0 0x7ff5181981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55586/0 (socket says 172.21.15.111:55586) 2023-10-11T16:29:49.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 -- 172.21.15.111:0/4099885102 learned_addr learned my addr 172.21.15.111:0/4099885102 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff516d9d700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff518101f10 0x7ff51819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 -- 172.21.15.111:0/4099885102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff518101f10 msgr2=0x7ff51819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff518101f10 0x7ff51819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:49.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 -- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff518101560 msgr2=0x7ff5181986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:49.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff518101560 0x7ff5181986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:49.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 -- 172.21.15.111:0/4099885102 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080095f0 con 0x7ff5181002d0 2023-10-11T16:29:49.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff516d9d700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff518101f10 0x7ff51819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:29:49.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff51659c700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5181002d0 0x7ff5181981a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff50800b680 tx=0x7ff508005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:49.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.812+0000 7ff5077fe700 1 -- 172.21.15.111:0/4099885102 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50801d070 con 0x7ff5181002d0 2023-10-11T16:29:49.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.813+0000 7ff5077fe700 1 -- 172.21.15.111:0/4099885102 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff508022470 con 0x7ff5181002d0 2023-10-11T16:29:49.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.813+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51810bcc0 con 0x7ff5181002d0 2023-10-11T16:29:49.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.813+0000 7ff5077fe700 1 -- 172.21.15.111:0/4099885102 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508017650 con 0x7ff5181002d0 2023-10-11T16:29:49.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.813+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51810bb30 con 0x7ff5181002d0 2023-10-11T16:29:49.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.814+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff518192910 con 0x7ff5181002d0 2023-10-11T16:29:49.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.815+0000 7ff5077fe700 1 -- 172.21.15.111:0/4099885102 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ff5080225e0 con 0x7ff5181002d0 2023-10-11T16:29:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.817+0000 7ff5077fe700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff5000a1150 0x7ff5000a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.817+0000 7ff5077fe700 1 -- 172.21.15.111:0/4099885102 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff5080cdbc0 con 0x7ff5181002d0 2023-10-11T16:29:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.817+0000 7ff515d9b700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff5000a1150 0x7ff5000a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.817+0000 7ff515d9b700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff5000a1150 0x7ff5000a3600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009c60 tx=0x7ff50c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:49.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:49.819+0000 7ff5077fe700 1 -- 172.21.15.111:0/4099885102 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff5080977a0 con 0x7ff5181002d0 2023-10-11T16:29:49.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:49.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:49.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:49.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:49.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: Deploying daemon mds.f on smithi150 2023-10-11T16:29:49.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.130:6834/89303870,v1:172.21.15.130:6835/89303870] up:boot 2023-10-11T16:29:49.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: fsmap 5 up:standby 2023-10-11T16:29:49.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:49 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:29:49.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:49.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:49.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: Deploying daemon mds.f on smithi150 2023-10-11T16:29:49.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.130:6834/89303870,v1:172.21.15.130:6835/89303870] up:boot 2023-10-11T16:29:49.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: fsmap 5 up:standby 2023-10-11T16:29:49.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:49 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:29:50.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.065+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff51802cfe0 con 0x7ff5181002d0 2023-10-11T16:29:50.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.066+0000 7ff5077fe700 1 -- 172.21.15.111:0/4099885102 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4291 (secure 0 0 0) 0x7ff50809adc0 con 0x7ff5181002d0 2023-10-11T16:29:50.068 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:50.068 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-10-11T16:29:50.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff5000a1150 msgr2=0x7ff5000a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:50.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff5000a1150 0x7ff5000a3600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009c60 tx=0x7ff50c009260 comp rx=0 tx=0).stop 2023-10-11T16:29:50.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5181002d0 msgr2=0x7ff5181981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:50.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5181002d0 0x7ff5181981a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff50800b680 tx=0x7ff508005740 comp rx=0 tx=0).stop 2023-10-11T16:29:50.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 shutdown_connections 2023-10-11T16:29:50.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff5000a1150 0x7ff5000a3600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:50.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff518101f10 0x7ff51819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:50.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5181002d0 0x7ff5181981a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:50.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 --2- 172.21.15.111:0/4099885102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff518101560 0x7ff5181986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:50.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 >> 172.21.15.111:0/4099885102 conn(0x7ff5180fb930 msgr2=0x7ff518106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:50.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 shutdown_connections 2023-10-11T16:29:50.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:50.069+0000 7ff51cfff700 1 -- 172.21.15.111:0/4099885102 wait complete. 2023-10-11T16:29:50.073 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 6 2023-10-11T16:29:50.202 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:29:49 smithi150 systemd[1]: Starting Ceph mds.f for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:50.462 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:29:50 smithi150 podman[157823]: 2023-10-11T16:29:50.952 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:29:50 smithi150 bash[157823]: 5102df2c7cc7438d3c1b4378a696a70fe3b64d13e124cc021d5bfffcd03aa33b 2023-10-11T16:29:50.952 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-mds[157863]: -- 172.21.15.150:0/2733823520 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d0dfae3520 con 0x55d0dfab9000 2023-10-11T16:29:50.953 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-f[157839]: starting mds.f at 2023-10-11T16:29:50.953 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:29:50 smithi150 systemd[1]: Started Ceph mds.f for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4099885102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:50.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:50.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:50 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:51.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:50 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4099885102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:51.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:51.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:51.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:51.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:51.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:51.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:50 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:51.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:50 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4099885102' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:51.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:51.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:51.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:51.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:51.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:51.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:50 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:51.857 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:51 smithi150 ceph-mon[132217]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:51.858 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:51 smithi150 ceph-mon[132217]: Deploying daemon mds.c on smithi150 2023-10-11T16:29:51.858 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:51 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6834/2357736565,v1:172.21.15.150:6835/2357736565] up:boot 2023-10-11T16:29:51.858 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:51 smithi150 ceph-mon[132217]: fsmap 6 up:standby 2023-10-11T16:29:51.858 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:51 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:29:52.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:51 smithi130 ceph-mon[131859]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:52.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:51 smithi130 ceph-mon[131859]: Deploying daemon mds.c on smithi150 2023-10-11T16:29:52.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:51 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6834/2357736565,v1:172.21.15.150:6835/2357736565] up:boot 2023-10-11T16:29:52.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:51 smithi130 ceph-mon[131859]: fsmap 6 up:standby 2023-10-11T16:29:52.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:51 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:29:52.090 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:52.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:51 smithi111 ceph-mon[98527]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:52.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:51 smithi111 ceph-mon[98527]: Deploying daemon mds.c on smithi150 2023-10-11T16:29:52.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:51 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6834/2357736565,v1:172.21.15.150:6835/2357736565] up:boot 2023-10-11T16:29:52.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:51 smithi111 ceph-mon[98527]: fsmap 6 up:standby 2023-10-11T16:29:52.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:51 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:29:52.203 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:29:52 smithi150 systemd[1]: Starting Ceph mds.c for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:52.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.296+0000 7feaf7fff700 1 -- 172.21.15.111:0/2225048670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0101e10 msgr2=0x7feaf01086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:52.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.296+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2225048670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0101e10 0x7feaf01086d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feae800b0a0 tx=0x7feae800b3b0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.296+0000 7feaf7fff700 1 -- 172.21.15.111:0/2225048670 shutdown_connections 2023-10-11T16:29:52.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.296+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2225048670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0101e10 0x7feaf01086d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.296+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2225048670 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feaf0100290 0x7feaf01006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.296+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2225048670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0101480 0x7feaf01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.296+0000 7feaf7fff700 1 -- 172.21.15.111:0/2225048670 >> 172.21.15.111:0/2225048670 conn(0x7feaf00fb870 msgr2=0x7feaf00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:52.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.296+0000 7feaf7fff700 1 -- 172.21.15.111:0/2225048670 shutdown_connections 2023-10-11T16:29:52.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.297+0000 7feaf7fff700 1 -- 172.21.15.111:0/2225048670 wait complete. 2023-10-11T16:29:52.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.297+0000 7feaf7fff700 1 Processor -- start 2023-10-11T16:29:52.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf7fff700 1 -- start start 2023-10-11T16:29:52.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0100290 0x7feaf00748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:52.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0101480 0x7feaf0072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:52.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feaf0101e10 0x7feaf0073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:52.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0073bc0 con 0x7feaf0100290 2023-10-11T16:29:52.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf7fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0073d30 con 0x7feaf0101e10 2023-10-11T16:29:52.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf7fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf01a00f0 con 0x7feaf0101480 2023-10-11T16:29:52.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0100290 0x7feaf00748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:52.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf559a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0101480 0x7feaf0072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:52.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0100290 0x7feaf00748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50996/0 (socket says 172.21.15.111:50996) 2023-10-11T16:29:52.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf5d9b700 1 -- 172.21.15.111:0/2469901659 learned_addr learned my addr 172.21.15.111:0/2469901659 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:52.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.298+0000 7feaf659c700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feaf0101e10 0x7feaf0073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:52.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feaf559a700 1 -- 172.21.15.111:0/2469901659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feaf0101e10 msgr2=0x7feaf0073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:52.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feaf559a700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feaf0101e10 0x7feaf0073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feaf559a700 1 -- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0100290 msgr2=0x7feaf00748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:52.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feaf559a700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0100290 0x7feaf00748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feaf559a700 1 -- 172.21.15.111:0/2469901659 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae8009d00 con 0x7feaf0101480 2023-10-11T16:29:52.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feaf559a700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0101480 0x7feaf0072ee0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feae000e9f0 tx=0x7feae000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:52.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feae6ffd700 1 -- 172.21.15.111:0/2469901659 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae000c9c0 con 0x7feaf0101480 2023-10-11T16:29:52.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf010bc70 con 0x7feaf0101480 2023-10-11T16:29:52.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feae6ffd700 1 -- 172.21.15.111:0/2469901659 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feae0010430 con 0x7feaf0101480 2023-10-11T16:29:52.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feae6ffd700 1 -- 172.21.15.111:0/2469901659 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae0018680 con 0x7feaf0101480 2023-10-11T16:29:52.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.299+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf010bae0 con 0x7feaf0101480 2023-10-11T16:29:52.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.300+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf01a0800 con 0x7feaf0101480 2023-10-11T16:29:52.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.302+0000 7feae6ffd700 1 -- 172.21.15.111:0/2469901659 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7feae0010860 con 0x7feaf0101480 2023-10-11T16:29:52.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.303+0000 7feae6ffd700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feadc0a11d0 0x7feadc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:52.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.303+0000 7feae6ffd700 1 -- 172.21.15.111:0/2469901659 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7feae0014070 con 0x7feaf0101480 2023-10-11T16:29:52.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.304+0000 7feaf5d9b700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feadc0a11d0 0x7feadc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:52.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.304+0000 7feaf5d9b700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feadc0a11d0 0x7feadc0a3680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feaec007cf0 tx=0x7feaec007480 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:52.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.307+0000 7feae6ffd700 1 -- 172.21.15.111:0/2469901659 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feae0051730 con 0x7feaf0101480 2023-10-11T16:29:52.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.553+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaf0107100 con 0x7feaf0101480 2023-10-11T16:29:52.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.553+0000 7feae6ffd700 1 -- 172.21.15.111:0/2469901659 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5051 (secure 0 0 0) 0x7feae0096070 con 0x7feaf0101480 2023-10-11T16:29:52.554 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:52.555 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-10-11T16:29:52.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feadc0a11d0 msgr2=0x7feadc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:52.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feadc0a11d0 0x7feadc0a3680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feaec007cf0 tx=0x7feaec007480 comp rx=0 tx=0).stop 2023-10-11T16:29:52.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0101480 msgr2=0x7feaf0072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:52.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0101480 0x7feaf0072ee0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feae000e9f0 tx=0x7feae000ed00 comp rx=0 tx=0).stop 2023-10-11T16:29:52.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 shutdown_connections 2023-10-11T16:29:52.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feadc0a11d0 0x7feadc0a3680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feaf0101e10 0x7feaf0073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaf0100290 0x7feaf00748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 --2- 172.21.15.111:0/2469901659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf0101480 0x7feaf0072ee0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:52.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 >> 172.21.15.111:0/2469901659 conn(0x7feaf00fb870 msgr2=0x7feaf00fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 shutdown_connections 2023-10-11T16:29:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:52.556+0000 7feaf7fff700 1 -- 172.21.15.111:0/2469901659 wait complete. 2023-10-11T16:29:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 7 2023-10-11T16:29:52.650 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:29:52 smithi150 podman[158165]: 2023-10-11T16:29:52.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:52 smithi111 ceph-mon[98527]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:52.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:52 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2469901659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:52.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:52 smithi150 ceph-mon[132217]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:52.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:52 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2469901659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:52.953 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:29:52 smithi150 ceph-mds[158205]: -- 172.21.15.150:0/3034402578 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e8e781520 con 0x562e8e756800 2023-10-11T16:29:52.953 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:29:52 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-c[158181]: starting mds.c at 2023-10-11T16:29:52.953 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:29:52 smithi150 bash[158165]: 7dbd832129d2f9b59f256dc335e7757e259a418da6fff76ce99b4c92739b9455 2023-10-11T16:29:52.953 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:29:52 smithi150 systemd[1]: Started Ceph mds.c for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:52 smithi130 ceph-mon[131859]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:53.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:52 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2469901659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:53.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] up:boot 2023-10-11T16:29:53.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: fsmap 7 up:standby 2023-10-11T16:29:53.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:29:53.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:53.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:53.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:53.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:53.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:53.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:53.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:53 smithi130 ceph-mon[131859]: Deploying daemon mds.h on smithi130 2023-10-11T16:29:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] up:boot 2023-10-11T16:29:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: fsmap 7 up:standby 2023-10-11T16:29:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:29:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:53.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:53.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:53.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:53.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:53.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:53 smithi150 ceph-mon[132217]: Deploying daemon mds.h on smithi130 2023-10-11T16:29:54.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] up:boot 2023-10-11T16:29:54.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: fsmap 7 up:standby 2023-10-11T16:29:54.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:29:54.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:54.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:54.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:54.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:54.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:54.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:54.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:53 smithi111 ceph-mon[98527]: Deploying daemon mds.h on smithi130 2023-10-11T16:29:54.576 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:54.582 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:29:54 smithi130 systemd[1]: Starting Ceph mds.h for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:54.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 -- 172.21.15.111:0/2938831150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe01c101560 msgr2=0x7fe01c1019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:54.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 --2- 172.21.15.111:0/2938831150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe01c101560 0x7fe01c1019d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe004009990 tx=0x7fe004009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:54.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 -- 172.21.15.111:0/2938831150 shutdown_connections 2023-10-11T16:29:54.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 --2- 172.21.15.111:0/2938831150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01c101f10 0x7fe01c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 --2- 172.21.15.111:0/2938831150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe01c1002d0 0x7fe01c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 --2- 172.21.15.111:0/2938831150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe01c101560 0x7fe01c1019d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 -- 172.21.15.111:0/2938831150 >> 172.21.15.111:0/2938831150 conn(0x7fe01c0fb930 msgr2=0x7fe01c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 -- 172.21.15.111:0/2938831150 shutdown_connections 2023-10-11T16:29:54.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.786+0000 7fe020fff700 1 -- 172.21.15.111:0/2938831150 wait complete. 2023-10-11T16:29:54.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe020fff700 1 Processor -- start 2023-10-11T16:29:54.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe020fff700 1 -- start start 2023-10-11T16:29:54.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe020fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe01c1002d0 0x7fe01c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:54.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe020fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01c101560 0x7fe01c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:54.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe020fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe01c101f10 0x7fe01c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:54.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe020fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c19fd30 con 0x7fe01c1002d0 2023-10-11T16:29:54.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe020fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c19fe70 con 0x7fe01c101f10 2023-10-11T16:29:54.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe020fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c1a0000 con 0x7fe01c101560 2023-10-11T16:29:54.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.787+0000 7fe01a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe01c1002d0 0x7fe01c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:54.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01ad9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe01c101f10 0x7fe01c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:54.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe01c1002d0 0x7fe01c1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51016/0 (socket says 172.21.15.111:51016) 2023-10-11T16:29:54.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01a59c700 1 -- 172.21.15.111:0/1944087221 learned_addr learned my addr 172.21.15.111:0/1944087221 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:54.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe013fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01c101560 0x7fe01c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:54.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01a59c700 1 -- 172.21.15.111:0/1944087221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01c101560 msgr2=0x7fe01c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:54.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01a59c700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01c101560 0x7fe01c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:54.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01a59c700 1 -- 172.21.15.111:0/1944087221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe01c101f10 msgr2=0x7fe01c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:54.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01a59c700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe01c101f10 0x7fe01c19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:54.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01a59c700 1 -- 172.21.15.111:0/1944087221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c009c40 con 0x7fe01c1002d0 2023-10-11T16:29:54.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe013fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01c101560 0x7fe01c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:29:54.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe01a59c700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe01c1002d0 0x7fe01c1981b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c370 tx=0x7fe00c00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:54.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe0137fe700 1 -- 172.21.15.111:0/1944087221 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c017070 con 0x7fe01c1002d0 2023-10-11T16:29:54.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0040095f0 con 0x7fe01c1002d0 2023-10-11T16:29:54.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe0137fe700 1 -- 172.21.15.111:0/1944087221 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe00c00f040 con 0x7fe01c1002d0 2023-10-11T16:29:54.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.788+0000 7fe0137fe700 1 -- 172.21.15.111:0/1944087221 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c00e050 con 0x7fe01c1002d0 2023-10-11T16:29:54.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.789+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c10bcc0 con 0x7fe01c1002d0 2023-10-11T16:29:54.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.790+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c192910 con 0x7fe01c1002d0 2023-10-11T16:29:54.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.791+0000 7fe0137fe700 1 -- 172.21.15.111:0/1944087221 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fe00c0044e0 con 0x7fe01c1002d0 2023-10-11T16:29:54.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.793+0000 7fe0137fe700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe0080a11d0 0x7fe0080a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:54.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.793+0000 7fe0137fe700 1 -- 172.21.15.111:0/1944087221 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe00c0cbf10 con 0x7fe01c1002d0 2023-10-11T16:29:54.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.793+0000 7fe013fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe0080a11d0 0x7fe0080a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:54.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.793+0000 7fe013fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe0080a11d0 0x7fe0080a3680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe00400b2e0 tx=0x7fe004005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:54.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:54.797+0000 7fe0137fe700 1 -- 172.21.15.111:0/1944087221 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe00c095a60 con 0x7fe01c1002d0 2023-10-11T16:29:54.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:54 smithi150 ceph-mon[132217]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:54.956 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:54 smithi130 ceph-mon[131859]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:54.956 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:29:54 smithi130 podman[156726]: 2023-10-11T16:29:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.043+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe01c02cfe0 con 0x7fe01c1002d0 2023-10-11T16:29:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.043+0000 7fe0137fe700 1 -- 172.21.15.111:0/1944087221 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5808 (secure 0 0 0) 0x7fe00c099080 con 0x7fe01c1002d0 2023-10-11T16:29:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:55.045 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-10-11T16:29:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe0080a11d0 msgr2=0x7fe0080a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe0080a11d0 0x7fe0080a3680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe00400b2e0 tx=0x7fe004005fd0 comp rx=0 tx=0).stop 2023-10-11T16:29:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe01c1002d0 msgr2=0x7fe01c1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:55.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe01c1002d0 0x7fe01c1981b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c370 tx=0x7fe00c00c680 comp rx=0 tx=0).stop 2023-10-11T16:29:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 shutdown_connections 2023-10-11T16:29:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe0080a11d0 0x7fe0080a3680 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe01c101f10 0x7fe01c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:55.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe01c1002d0 0x7fe01c1981b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:55.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 --2- 172.21.15.111:0/1944087221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe01c101560 0x7fe01c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:55.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 >> 172.21.15.111:0/1944087221 conn(0x7fe01c0fb930 msgr2=0x7fe01c106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:55.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 shutdown_connections 2023-10-11T16:29:55.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:55.046+0000 7fe020fff700 1 -- 172.21.15.111:0/1944087221 wait complete. 2023-10-11T16:29:55.050 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 8 2023-10-11T16:29:55.065 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:54 smithi111 ceph-mon[98527]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:55.331 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:29:54 smithi130 ceph-mds[156766]: -- 172.21.15.130:0/674283168 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5647d8543520 con 0x5647d8518c00 2023-10-11T16:29:55.332 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:29:54 smithi130 bash[156726]: d5f9d2bcafc9ff5dbeba9d898086d72e391f7cbbfe17353a7bdce5b50e41f84b 2023-10-11T16:29:55.332 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:29:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-h[156742]: starting mds.h at 2023-10-11T16:29:55.332 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:29:54 smithi130 systemd[1]: Started Ceph mds.h for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:56.105 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.105 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1944087221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:56.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:56.106 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:56.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:56.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: Deploying daemon mds.i on smithi150 2023-10-11T16:29:56.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.130:6836/3927432367,v1:172.21.15.130:6837/3927432367] up:boot 2023-10-11T16:29:56.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: fsmap 8 up:standby 2023-10-11T16:29:56.107 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:56 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:29:56.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1944087221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:56.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:56.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:56.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:56.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: Deploying daemon mds.i on smithi150 2023-10-11T16:29:56.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.130:6836/3927432367,v1:172.21.15.130:6837/3927432367] up:boot 2023-10-11T16:29:56.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: fsmap 8 up:standby 2023-10-11T16:29:56.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:56 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:29:56.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1944087221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:56.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:56.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:56.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: Deploying daemon mds.i on smithi150 2023-10-11T16:29:56.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.130:6836/3927432367,v1:172.21.15.130:6837/3927432367] up:boot 2023-10-11T16:29:56.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: fsmap 8 up:standby 2023-10-11T16:29:56.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:56 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:29:56.857 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:29:56 smithi150 systemd[1]: Starting Ceph mds.i for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:57.069 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:57.137 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:29:56 smithi150 podman[158507]: 2023-10-11T16:29:57.137 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:57 smithi150 ceph-mon[132217]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:57.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 -- 172.21.15.111:0/857321196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 msgr2=0x7f2680102950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:57.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 --2- 172.21.15.111:0/857321196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 0x7f2680102950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f267c009990 tx=0x7f267c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 -- 172.21.15.111:0/857321196 shutdown_connections 2023-10-11T16:29:57.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 --2- 172.21.15.111:0/857321196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26801040c0 0x7f268010a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 --2- 172.21.15.111:0/857321196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 0x7f2680102950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 --2- 172.21.15.111:0/857321196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2680103730 0x7f2680103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 -- 172.21.15.111:0/857321196 >> 172.21.15.111:0/857321196 conn(0x7f26800fdb20 msgr2=0x7f26800fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 -- 172.21.15.111:0/857321196 shutdown_connections 2023-10-11T16:29:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.286+0000 7f26873ff700 1 -- 172.21.15.111:0/857321196 wait complete. 2023-10-11T16:29:57.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.287+0000 7f26873ff700 1 Processor -- start 2023-10-11T16:29:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.287+0000 7f26873ff700 1 -- start start 2023-10-11T16:29:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.287+0000 7f26873ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 0x7f26801981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.287+0000 7f26873ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2680103730 0x7f2680198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.287+0000 7f26873ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26801040c0 0x7f268019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.287+0000 7f26873ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268019fd40 con 0x7f2680103730 2023-10-11T16:29:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.287+0000 7f26873ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268019fe80 con 0x7f2680102540 2023-10-11T16:29:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.287+0000 7f26873ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26801a0010 con 0x7f26801040c0 2023-10-11T16:29:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 0x7f26801981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 0x7f26801981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:34496/0 (socket says 172.21.15.111:34496) 2023-10-11T16:29:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 -- 172.21.15.111:0/2564377658 learned_addr learned my addr 172.21.15.111:0/2564377658 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268499a700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2680103730 0x7f2680198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268599c700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26801040c0 0x7f268019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 -- 172.21.15.111:0/2564377658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26801040c0 msgr2=0x7f268019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26801040c0 0x7f268019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 -- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2680103730 msgr2=0x7f2680198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2680103730 0x7f2680198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 -- 172.21.15.111:0/2564377658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2670009520 con 0x7f2680102540 2023-10-11T16:29:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268499a700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2680103730 0x7f2680198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:29:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268599c700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26801040c0 0x7f268019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:29:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.288+0000 7f268519b700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 0x7f26801981c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f267c005740 tx=0x7f267c009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.289+0000 7f26767fc700 1 -- 172.21.15.111:0/2564377658 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c01d070 con 0x7f2680102540 2023-10-11T16:29:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.289+0000 7f26767fc700 1 -- 172.21.15.111:0/2564377658 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f267c021410 con 0x7f2680102540 2023-10-11T16:29:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.289+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267c0095f0 con 0x7f2680102540 2023-10-11T16:29:57.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.289+0000 7f26767fc700 1 -- 172.21.15.111:0/2564377658 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c017650 con 0x7f2680102540 2023-10-11T16:29:57.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.289+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268010de80 con 0x7f2680102540 2023-10-11T16:29:57.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.290+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2680067eb0 con 0x7f2680102540 2023-10-11T16:29:57.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.291+0000 7f26767fc700 1 -- 172.21.15.111:0/2564377658 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f267c021810 con 0x7f2680102540 2023-10-11T16:29:57.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.293+0000 7f26767fc700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f266c0a11d0 0x7f266c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:57.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.293+0000 7f26767fc700 1 -- 172.21.15.111:0/2564377658 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f267c0cd980 con 0x7f2680102540 2023-10-11T16:29:57.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.293+0000 7f268499a700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f266c0a11d0 0x7f266c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:57.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.293+0000 7f268499a700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f266c0a11d0 0x7f266c0a3680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2670009e40 tx=0x7f2670009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:57.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.297+0000 7f26767fc700 1 -- 172.21.15.111:0/2564377658 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f267c097560 con 0x7f2680102540 2023-10-11T16:29:57.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:57 smithi130 ceph-mon[131859]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:57.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:57 smithi111 ceph-mon[98527]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:57.452 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:29:57 smithi150 ceph-mds[158547]: -- 172.21.15.150:0/3480884027 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558031bc7520 con 0x558031b9d000 2023-10-11T16:29:57.452 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:29:57 smithi150 bash[158507]: 02434bbfab3d59b71e47c4c663af3afa22316ef0e2ccd915c8d0af9fd3cb5c69 2023-10-11T16:29:57.452 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:29:57 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-i[158523]: starting mds.i at 2023-10-11T16:29:57.453 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:29:57 smithi150 systemd[1]: Started Ceph mds.i for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:29:57.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.550+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2680109480 con 0x7f2680102540 2023-10-11T16:29:57.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.551+0000 7f26767fc700 1 -- 172.21.15.111:0/2564377658 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6568 (secure 0 0 0) 0x7f267c09ab80 con 0x7f2680102540 2023-10-11T16:29:57.551 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:29:57.552 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-10-11T16:29:57.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f266c0a11d0 msgr2=0x7f266c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:57.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f266c0a11d0 0x7f266c0a3680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2670009e40 tx=0x7f2670009260 comp rx=0 tx=0).stop 2023-10-11T16:29:57.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 msgr2=0x7f26801981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:57.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 0x7f26801981c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f267c005740 tx=0x7f267c009510 comp rx=0 tx=0).stop 2023-10-11T16:29:57.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 shutdown_connections 2023-10-11T16:29:57.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f266c0a11d0 0x7f266c0a3680 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26801040c0 0x7f268019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2680102540 0x7f26801981c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 --2- 172.21.15.111:0/2564377658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2680103730 0x7f2680198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:57.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 >> 172.21.15.111:0/2564377658 conn(0x7f26800fdb20 msgr2=0x7f26800ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:57.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 shutdown_connections 2023-10-11T16:29:57.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:57.553+0000 7f26873ff700 1 -- 172.21.15.111:0/2564377658 wait complete. 2023-10-11T16:29:57.556 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 9 2023-10-11T16:29:58.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:58.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:58.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:58.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: Deploying daemon mds.g on smithi111 2023-10-11T16:29:58.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2564377658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:58.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] up:boot 2023-10-11T16:29:58.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: fsmap 9 up:standby 2023-10-11T16:29:58.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:58 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:29:58.568 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.568 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.568 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.568 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:58.569 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:58.569 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:58.569 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: Deploying daemon mds.g on smithi111 2023-10-11T16:29:58.569 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2564377658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:58.570 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] up:boot 2023-10-11T16:29:58.570 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: fsmap 9 up:standby 2023-10-11T16:29:58.570 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:58 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:29:58.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:29:58.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:29:58.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:29:58.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:29:58.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: Deploying daemon mds.g on smithi111 2023-10-11T16:29:58.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2564377658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:29:58.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] up:boot 2023-10-11T16:29:58.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: fsmap 9 up:standby 2023-10-11T16:29:58.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:58 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:29:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:29:59 smithi150 ceph-mon[132217]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:59.575 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:29:59.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:29:59 smithi130 ceph-mon[131859]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:59.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:29:59 smithi111 ceph-mon[98527]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:29:59.636 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:29:59 smithi111 systemd[1]: Starting Ceph mds.g for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:29:59.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1387886868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4073550 msgr2=0x7f11c40739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:59.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1387886868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4073550 0x7f11c40739c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009990 tx=0x7f11b8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:29:59.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1387886868 shutdown_connections 2023-10-11T16:29:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1387886868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c4108150 0x7f11c410a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1387886868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c4074b30 0x7f11c4072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1387886868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4073550 0x7f11c40739c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1387886868 >> 172.21.15.111:0/1387886868 conn(0x7f11c40fb870 msgr2=0x7f11c40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:29:59.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1387886868 shutdown_connections 2023-10-11T16:29:59.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.800+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1387886868 wait complete. 2023-10-11T16:29:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.801+0000 7f11ca1ff700 1 Processor -- start 2023-10-11T16:29:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11ca1ff700 1 -- start start 2023-10-11T16:29:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11ca1ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c4073550 0x7f11c41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:59.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11ca1ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4074b30 0x7f11c4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:59.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11ca1ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c4108150 0x7f11c419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:59.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11ca1ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419fd60 con 0x7f11c4074b30 2023-10-11T16:29:59.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11ca1ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c419fea0 con 0x7f11c4073550 2023-10-11T16:29:59.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11ca1ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c41a0030 con 0x7f11c4108150 2023-10-11T16:29:59.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11c2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4074b30 0x7f11c4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:59.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11c2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4074b30 0x7f11c4198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44372/0 (socket says 172.21.15.111:44372) 2023-10-11T16:29:59.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.802+0000 7f11c2ffd700 1 -- 172.21.15.111:0/1714936601 learned_addr learned my addr 172.21.15.111:0/1714936601 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:29:59.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c37fe700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c4073550 0x7f11c41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:59.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c3fff700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c4108150 0x7f11c419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:59.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c2ffd700 1 -- 172.21.15.111:0/1714936601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c4108150 msgr2=0x7f11c419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c2ffd700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c4108150 0x7f11c419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c2ffd700 1 -- 172.21.15.111:0/1714936601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c4073550 msgr2=0x7f11c41981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:29:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c2ffd700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c4073550 0x7f11c41981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:29:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c2ffd700 1 -- 172.21.15.111:0/1714936601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b80095f0 con 0x7f11c4074b30 2023-10-11T16:29:59.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c2ffd700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4074b30 0x7f11c4198720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f11b80095c0 tx=0x7f11b800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c0ff9700 1 -- 172.21.15.111:0/1714936601 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b801d070 con 0x7f11c4074b30 2023-10-11T16:29:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c0ff9700 1 -- 172.21.15.111:0/1714936601 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11b8005030 con 0x7f11c4074b30 2023-10-11T16:29:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c410de70 con 0x7f11c4074b30 2023-10-11T16:29:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11c0ff9700 1 -- 172.21.15.111:0/1714936601 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b8004af0 con 0x7f11c4074b30 2023-10-11T16:29:59.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.803+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c410dce0 con 0x7f11c4074b30 2023-10-11T16:29:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.805+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c4106800 con 0x7f11c4074b30 2023-10-11T16:29:59.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.806+0000 7f11c0ff9700 1 -- 172.21.15.111:0/1714936601 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f11b8004cd0 con 0x7f11c4074b30 2023-10-11T16:29:59.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.807+0000 7f11c0ff9700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f11ac0a11d0 0x7f11ac0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:29:59.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.808+0000 7f11c0ff9700 1 -- 172.21.15.111:0/1714936601 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f11b80cd420 con 0x7f11c4074b30 2023-10-11T16:29:59.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.808+0000 7f11c37fe700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f11ac0a11d0 0x7f11ac0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:29:59.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.808+0000 7f11c37fe700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f11ac0a11d0 0x7f11ac0a3680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11b4005950 tx=0x7f11b400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:29:59.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:29:59.812+0000 7f11c0ff9700 1 -- 172.21.15.111:0/1714936601 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f11b8097000 con 0x7f11c4074b30 2023-10-11T16:30:00.026 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:29:59 smithi111 podman[132256]: 2023-10-11T16:30:00.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.071+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11c402cfe0 con 0x7f11c4074b30 2023-10-11T16:30:00.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.071+0000 7f11c0ff9700 1 -- 172.21.15.111:0/1714936601 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7330 (secure 0 0 0) 0x7f11b809a620 con 0x7f11c4074b30 2023-10-11T16:30:00.072 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:00.073 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-10-11T16:30:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f11ac0a11d0 msgr2=0x7f11ac0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f11ac0a11d0 0x7f11ac0a3680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11b4005950 tx=0x7f11b400b410 comp rx=0 tx=0).stop 2023-10-11T16:30:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4074b30 msgr2=0x7f11c4198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4074b30 0x7f11c4198720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f11b80095c0 tx=0x7f11b800f690 comp rx=0 tx=0).stop 2023-10-11T16:30:00.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 shutdown_connections 2023-10-11T16:30:00.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f11ac0a11d0 0x7f11ac0a3680 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:00.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c4108150 0x7f11c419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:00.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c4073550 0x7f11c41981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:00.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 --2- 172.21.15.111:0/1714936601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11c4074b30 0x7f11c4198720 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:00.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 >> 172.21.15.111:0/1714936601 conn(0x7f11c40fb870 msgr2=0x7f11c40fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:00.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 shutdown_connections 2023-10-11T16:30:00.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:00.074+0000 7f11ca1ff700 1 -- 172.21.15.111:0/1714936601 wait complete. 2023-10-11T16:30:00.077 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 10 2023-10-11T16:30:00.385 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mds[132318]: -- 172.21.15.111:0/2545402887 <== mon.0 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56537e1dd520 con 0x56537e1b4800 2023-10-11T16:30:00.386 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-g[132292]: starting mds.g at 2023-10-11T16:30:00.386 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:30:00 smithi111 bash[132256]: 92ae756e07bb1047022772b51a0b5bfd2c5ecf749c9079bcbd4bf0409c622c30 2023-10-11T16:30:00.386 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:30:00 smithi111 systemd[1]: Started Ceph mds.g for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:30:00.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: overall HEALTH_OK 2023-10-11T16:30:00.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:00.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1714936601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:00.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:00.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:00.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:00.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:30:00.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:30:00.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:00.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:00 smithi150 ceph-mon[132217]: Deploying daemon mds.a on smithi111 2023-10-11T16:30:01.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: overall HEALTH_OK 2023-10-11T16:30:01.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:01.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1714936601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:01.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:01.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:01.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:01.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:30:01.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:30:01.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:01.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:00 smithi130 ceph-mon[131859]: Deploying daemon mds.a on smithi111 2023-10-11T16:30:01.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: overall HEALTH_OK 2023-10-11T16:30:01.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:01.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1714936601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:01.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:01.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:01.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:01.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:30:01.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:30:01.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:01.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:00 smithi111 ceph-mon[98527]: Deploying daemon mds.a on smithi111 2023-10-11T16:30:01.846 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:01 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.111:6838/2081616561,v1:172.21.15.111:6839/2081616561] up:boot 2023-10-11T16:30:01.847 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:01 smithi111 ceph-mon[98527]: fsmap 10 up:standby 2023-10-11T16:30:01.847 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:01 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-11T16:30:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:01 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.111:6838/2081616561,v1:172.21.15.111:6839/2081616561] up:boot 2023-10-11T16:30:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:01 smithi150 ceph-mon[132217]: fsmap 10 up:standby 2023-10-11T16:30:01.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:01 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-11T16:30:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:01 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.111:6838/2081616561,v1:172.21.15.111:6839/2081616561] up:boot 2023-10-11T16:30:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:01 smithi130 ceph-mon[131859]: fsmap 10 up:standby 2023-10-11T16:30:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:01 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-11T16:30:02.096 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:30:02.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 -- 172.21.15.111:0/3529666060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68068b10 msgr2=0x7eff68105200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:02.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 --2- 172.21.15.111:0/3529666060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68068b10 0x7eff68105200 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009990 tx=0x7eff5c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 -- 172.21.15.111:0/3529666060 shutdown_connections 2023-10-11T16:30:02.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 --2- 172.21.15.111:0/3529666060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff68108060 0x7eff6810a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 --2- 172.21.15.111:0/3529666060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68068b10 0x7eff68105200 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 --2- 172.21.15.111:0/3529666060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff68105740 0x7eff68107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 -- 172.21.15.111:0/3529666060 >> 172.21.15.111:0/3529666060 conn(0x7eff680fa750 msgr2=0x7eff680fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 -- 172.21.15.111:0/3529666060 shutdown_connections 2023-10-11T16:30:02.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.320+0000 7eff6ebff700 1 -- 172.21.15.111:0/3529666060 wait complete. 2023-10-11T16:30:02.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.321+0000 7eff6ebff700 1 Processor -- start 2023-10-11T16:30:02.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.321+0000 7eff6ebff700 1 -- start start 2023-10-11T16:30:02.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.321+0000 7eff6ebff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff68068b10 0x7eff68198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:02.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.321+0000 7eff6ebff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68105740 0x7eff68198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:02.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff6ebff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff68108060 0x7eff6819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:02.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff6ebff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6819fd90 con 0x7eff68105740 2023-10-11T16:30:02.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff6ebff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6819fed0 con 0x7eff68068b10 2023-10-11T16:30:02.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff6ebff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff681a0060 con 0x7eff68108060 2023-10-11T16:30:02.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68105740 0x7eff68198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:02.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68105740 0x7eff68198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44396/0 (socket says 172.21.15.111:44396) 2023-10-11T16:30:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 -- 172.21.15.111:0/95187787 learned_addr learned my addr 172.21.15.111:0/95187787 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff6d19c700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff68108060 0x7eff6819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff6c99b700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff68068b10 0x7eff68198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 -- 172.21.15.111:0/95187787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff68108060 msgr2=0x7eff6819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:02.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff68108060 0x7eff6819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 -- 172.21.15.111:0/95187787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff68068b10 msgr2=0x7eff68198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff68068b10 0x7eff68198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 -- 172.21.15.111:0/95187787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c0095f0 con 0x7eff68105740 2023-10-11T16:30:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.322+0000 7eff67fff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68105740 0x7eff68198750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7eff5400e9a0 tx=0x7eff5400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:02.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.323+0000 7eff65ffb700 1 -- 172.21.15.111:0/95187787 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54009790 con 0x7eff68105740 2023-10-11T16:30:02.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.323+0000 7eff65ffb700 1 -- 172.21.15.111:0/95187787 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff54010430 con 0x7eff68105740 2023-10-11T16:30:02.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.323+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6810de30 con 0x7eff68105740 2023-10-11T16:30:02.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.323+0000 7eff65ffb700 1 -- 172.21.15.111:0/95187787 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54018680 con 0x7eff68105740 2023-10-11T16:30:02.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.323+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6810dca0 con 0x7eff68105740 2023-10-11T16:30:02.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.324+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff68067a60 con 0x7eff68105740 2023-10-11T16:30:02.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.326+0000 7eff65ffb700 1 -- 172.21.15.111:0/95187787 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7eff540105a0 con 0x7eff68105740 2023-10-11T16:30:02.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.327+0000 7eff65ffb700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7eff580a1150 0x7eff580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:02.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.327+0000 7eff65ffb700 1 -- 172.21.15.111:0/95187787 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7eff54014070 con 0x7eff68105740 2023-10-11T16:30:02.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.327+0000 7eff6c99b700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7eff580a1150 0x7eff580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:02.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.328+0000 7eff6c99b700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7eff580a1150 0x7eff580a3600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00b2e0 tx=0x7eff5c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:02.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.335+0000 7eff65ffb700 1 -- 172.21.15.111:0/95187787 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7eff54051730 con 0x7eff68105740 2023-10-11T16:30:02.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.602+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff681a0b70 con 0x7eff68105740 2023-10-11T16:30:02.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.603+0000 7eff65ffb700 1 -- 172.21.15.111:0/95187787 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8091 (secure 0 0 0) 0x7eff54096220 con 0x7eff68105740 2023-10-11T16:30:02.604 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:02.605 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-10-11T16:30:02.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.605+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7eff580a1150 msgr2=0x7eff580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:02.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.605+0000 7eff6ebff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7eff580a1150 0x7eff580a3600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00b2e0 tx=0x7eff5c005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68105740 msgr2=0x7eff68198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:02.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68105740 0x7eff68198750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7eff5400e9a0 tx=0x7eff5400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 shutdown_connections 2023-10-11T16:30:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7eff580a1150 0x7eff580a3600 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff68108060 0x7eff6819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff68068b10 0x7eff68198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 --2- 172.21.15.111:0/95187787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff68105740 0x7eff68198750 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 >> 172.21.15.111:0/95187787 conn(0x7eff680fa750 msgr2=0x7eff68106bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 shutdown_connections 2023-10-11T16:30:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:02.606+0000 7eff6ebff700 1 -- 172.21.15.111:0/95187787 wait complete. 2023-10-11T16:30:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 11 2023-10-11T16:30:02.679 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:30:02 smithi111 systemd[1]: Starting Ceph mds.a for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:30:02.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:02 smithi150 ceph-mon[132217]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:02.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:02 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/95187787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:03.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:02 smithi130 ceph-mon[131859]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:02 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/95187787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:03.105 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:30:02 smithi111 podman[132670]: 2023-10-11T16:30:03.106 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:02 smithi111 ceph-mon[98527]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:03.107 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:02 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/95187787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:03.635 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:30:03 smithi111 bash[132670]: 4849ccd264a8389c433f278588714c62a78e281e10943b8c366209109153d75b 2023-10-11T16:30:03.636 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:30:03 smithi111 ceph-mds[132711]: -- 172.21.15.111:0/2523499053 <== mon.0 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e01c9a5520 con 0x55e01c97cc00 2023-10-11T16:30:03.636 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:30:03 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-a[132687]: starting mds.a at 2023-10-11T16:30:03.636 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:30:03 smithi111 systemd[1]: Started Ceph mds.a for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:30:04.476 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.477 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.477 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.477 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:30:04.478 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:30:04.478 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:04.478 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: Deploying daemon mds.b on smithi130 2023-10-11T16:30:04.479 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.111:6840/3150990523,v1:172.21.15.111:6841/3150990523] up:boot 2023-10-11T16:30:04.479 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: fsmap 11 up:standby 2023-10-11T16:30:04.479 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:04 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-11T16:30:04.627 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:30:04.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:30:04.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:30:04.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:04.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: Deploying daemon mds.b on smithi130 2023-10-11T16:30:04.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.111:6840/3150990523,v1:172.21.15.111:6841/3150990523] up:boot 2023-10-11T16:30:04.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: fsmap 11 up:standby 2023-10-11T16:30:04.639 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:04 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-11T16:30:04.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:04.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-11T16:30:04.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-11T16:30:04.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:04.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: Deploying daemon mds.b on smithi130 2023-10-11T16:30:04.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.111:6840/3150990523,v1:172.21.15.111:6841/3150990523] up:boot 2023-10-11T16:30:04.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: fsmap 11 up:standby 2023-10-11T16:30:04.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:04 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-11T16:30:04.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.845+0000 7fbae7fff700 1 -- 172.21.15.111:0/3730045500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0101480 msgr2=0x7fbae01018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.845+0000 7fbae7fff700 1 --2- 172.21.15.111:0/3730045500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0101480 0x7fbae01018d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009990 tx=0x7fbad0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.845+0000 7fbae7fff700 1 -- 172.21.15.111:0/3730045500 shutdown_connections 2023-10-11T16:30:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.845+0000 7fbae7fff700 1 --2- 172.21.15.111:0/3730045500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101e10 0x7fbae01086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.845+0000 7fbae7fff700 1 --2- 172.21.15.111:0/3730045500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0100290 0x7fbae01006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.845+0000 7fbae7fff700 1 --2- 172.21.15.111:0/3730045500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0101480 0x7fbae01018d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:04.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.845+0000 7fbae7fff700 1 -- 172.21.15.111:0/3730045500 >> 172.21.15.111:0/3730045500 conn(0x7fbae00fb870 msgr2=0x7fbae00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.845+0000 7fbae7fff700 1 -- 172.21.15.111:0/3730045500 shutdown_connections 2023-10-11T16:30:04.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.846+0000 7fbae7fff700 1 -- 172.21.15.111:0/3730045500 wait complete. 2023-10-11T16:30:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.846+0000 7fbae7fff700 1 Processor -- start 2023-10-11T16:30:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae7fff700 1 -- start start 2023-10-11T16:30:04.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae01981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101480 0x7fbae01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101e10 0x7fbae019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae019fd20 con 0x7fbae0100290 2023-10-11T16:30:04.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae7fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae019fe60 con 0x7fbae0101480 2023-10-11T16:30:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae7fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae019fff0 con 0x7fbae0101e10 2023-10-11T16:30:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae559a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101480 0x7fbae01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae01981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae559a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101480 0x7fbae01986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:51254/0 (socket says 172.21.15.111:51254) 2023-10-11T16:30:04.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae559a700 1 -- 172.21.15.111:0/1160492768 learned_addr learned my addr 172.21.15.111:0/1160492768 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae659c700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101e10 0x7fbae019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae5d9b700 1 -- 172.21.15.111:0/1160492768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101e10 msgr2=0x7fbae019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae5d9b700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101e10 0x7fbae019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.847+0000 7fbae5d9b700 1 -- 172.21.15.111:0/1160492768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101480 msgr2=0x7fbae01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbae5d9b700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101480 0x7fbae01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:04.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbae5d9b700 1 -- 172.21.15.111:0/1160492768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad00095f0 con 0x7fbae0100290 2023-10-11T16:30:04.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbae559a700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101480 0x7fbae01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:04.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbae5d9b700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae01981a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00eee0 tx=0x7fbadc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:04.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbad6ffd700 1 -- 172.21.15.111:0/1160492768 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc018070 con 0x7fbae0100290 2023-10-11T16:30:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbad6ffd700 1 -- 172.21.15.111:0/1160492768 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbadc00f040 con 0x7fbae0100290 2023-10-11T16:30:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbad6ffd700 1 -- 172.21.15.111:0/1160492768 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc004280 con 0x7fbae0100290 2023-10-11T16:30:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae010bc50 con 0x7fbae0100290 2023-10-11T16:30:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.848+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae010bac0 con 0x7fbae0100290 2023-10-11T16:30:04.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.849+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae01a0630 con 0x7fbae0100290 2023-10-11T16:30:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.851+0000 7fbad6ffd700 1 -- 172.21.15.111:0/1160492768 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fbadc003a00 con 0x7fbae0100290 2023-10-11T16:30:04.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.852+0000 7fbad6ffd700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbacc0a11d0 0x7fbacc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:04.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.852+0000 7fbad6ffd700 1 -- 172.21.15.111:0/1160492768 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbadc0cc3b0 con 0x7fbae0100290 2023-10-11T16:30:04.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.853+0000 7fbae559a700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbacc0a11d0 0x7fbacc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:04.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.853+0000 7fbae559a700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbacc0a11d0 0x7fbacc0a3680 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbad000b2e0 tx=0x7fbad0005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:04.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:04.856+0000 7fbad6ffd700 1 -- 172.21.15.111:0/1160492768 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbadc095f90 con 0x7fbae0100290 2023-10-11T16:30:05.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.100+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbae01a0be0 con 0x7fbae0100290 2023-10-11T16:30:05.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.101+0000 7fbad6ffd700 1 -- 172.21.15.111:0/1160492768 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8852 (secure 0 0 0) 0x7fbadc0995b0 con 0x7fbae0100290 2023-10-11T16:30:05.102 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:05.103 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-10-11T16:30:05.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbacc0a11d0 msgr2=0x7fbacc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:05.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbacc0a11d0 0x7fbacc0a3680 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbad000b2e0 tx=0x7fbad0005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:05.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 msgr2=0x7fbae01981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:05.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae01981a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00eee0 tx=0x7fbadc00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:05.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 shutdown_connections 2023-10-11T16:30:05.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbacc0a11d0 0x7fbacc0a3680 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:05.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbae0101e10 0x7fbae019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:05.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbae0100290 0x7fbae01981a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 --2- 172.21.15.111:0/1160492768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbae0101480 0x7fbae01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 >> 172.21.15.111:0/1160492768 conn(0x7fbae00fb870 msgr2=0x7fbae00fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 shutdown_connections 2023-10-11T16:30:05.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:05.104+0000 7fbae7fff700 1 -- 172.21.15.111:0/1160492768 wait complete. 2023-10-11T16:30:05.108 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 12 2023-10-11T16:30:05.295 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:30:05 smithi130 systemd[1]: Starting Ceph mds.b for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:30:05.555 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:30:05 smithi130 podman[157068]: 2023-10-11T16:30:05.556 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:05 smithi130 ceph-mon[131859]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:05.556 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:05 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1160492768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:05.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:05 smithi111 ceph-mon[98527]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:05.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:05 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1160492768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:05.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:05 smithi150 ceph-mon[132217]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:05.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:05 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1160492768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:05.831 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:30:05 smithi130 ceph-mds[157108]: -- 172.21.15.130:0/1777590754 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b282d23520 con 0x55b282cf9000 2023-10-11T16:30:05.832 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:30:05 smithi130 bash[157068]: f51710e3f499383562b096527013a354de306a4d9e13bd3253415a2df422037e 2023-10-11T16:30:05.832 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:30:05 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b[157084]: starting mds.b at 2023-10-11T16:30:05.832 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:30:05 smithi130 systemd[1]: Started Ceph mds.b for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:30:06.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:06.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:30:06.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] up:boot 2023-10-11T16:30:06.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: fsmap 12 up:standby 2023-10-11T16:30:06.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:06 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:30:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:30:07.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] up:boot 2023-10-11T16:30:07.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: fsmap 12 up:standby 2023-10-11T16:30:07.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:06 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:30:07.124 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:30:07.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:30:07.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] up:boot 2023-10-11T16:30:07.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: fsmap 12 up:standby 2023-10-11T16:30:07.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:06 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:30:07.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 -- 172.21.15.111:0/268383791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 msgr2=0x7f74741018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:07.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 --2- 172.21.15.111:0/268383791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 0x7f74741018d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f745c009990 tx=0x7f745c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 -- 172.21.15.111:0/268383791 shutdown_connections 2023-10-11T16:30:07.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 --2- 172.21.15.111:0/268383791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7474101e10 0x7f74741086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 --2- 172.21.15.111:0/268383791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7474100290 0x7f74741006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 --2- 172.21.15.111:0/268383791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 0x7f74741018d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 -- 172.21.15.111:0/268383791 >> 172.21.15.111:0/268383791 conn(0x7f74740fb870 msgr2=0x7f74740fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:07.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 -- 172.21.15.111:0/268383791 shutdown_connections 2023-10-11T16:30:07.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.335+0000 7f74795ff700 1 -- 172.21.15.111:0/268383791 wait complete. 2023-10-11T16:30:07.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.336+0000 7f74795ff700 1 Processor -- start 2023-10-11T16:30:07.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.336+0000 7f74795ff700 1 -- start start 2023-10-11T16:30:07.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.336+0000 7f74795ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7474100290 0x7f7474195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:07.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f74795ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 0x7f74741964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:07.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f74795ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7474101e10 0x7f747419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:07.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f74795ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747419db00 con 0x7f7474100290 2023-10-11T16:30:07.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f74795ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747419dc40 con 0x7f7474101e10 2023-10-11T16:30:07.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f74795ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747419ddd0 con 0x7f7474101480 2023-10-11T16:30:07.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f7472ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7474100290 0x7f7474195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:07.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f746bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 0x7f74741964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:07.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f74737fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7474101e10 0x7f747419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:07.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f746bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 0x7f74741964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:38862/0 (socket says 172.21.15.111:38862) 2023-10-11T16:30:07.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f746bfff700 1 -- 172.21.15.111:0/2774559514 learned_addr learned my addr 172.21.15.111:0/2774559514 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:07.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f7472ffd700 1 -- 172.21.15.111:0/2774559514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 msgr2=0x7f74741964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:07.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f7472ffd700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 0x7f74741964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f7472ffd700 1 -- 172.21.15.111:0/2774559514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7474101e10 msgr2=0x7f747419d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:07.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f7472ffd700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7474101e10 0x7f747419d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f7472ffd700 1 -- 172.21.15.111:0/2774559514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c009c40 con 0x7f7474100290 2023-10-11T16:30:07.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.337+0000 7f7472ffd700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7474100290 0x7f7474195f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f746400e9c0 tx=0x7f746400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:07.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.338+0000 7f7470ff9700 1 -- 172.21.15.111:0/2774559514 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464004220 con 0x7f7474100290 2023-10-11T16:30:07.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.338+0000 7f7470ff9700 1 -- 172.21.15.111:0/2774559514 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7464004380 con 0x7f7474100290 2023-10-11T16:30:07.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.338+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c0095f0 con 0x7f7474100290 2023-10-11T16:30:07.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.338+0000 7f7470ff9700 1 -- 172.21.15.111:0/2774559514 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464003c00 con 0x7f7474100290 2023-10-11T16:30:07.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.338+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747410bbf0 con 0x7f7474100290 2023-10-11T16:30:07.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.339+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74741078a0 con 0x7f7474100290 2023-10-11T16:30:07.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.341+0000 7f7470ff9700 1 -- 172.21.15.111:0/2774559514 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7464003d60 con 0x7f7474100290 2023-10-11T16:30:07.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.342+0000 7f7470ff9700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74600a11d0 0x7f74600a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:07.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.342+0000 7f7470ff9700 1 -- 172.21.15.111:0/2774559514 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7464014070 con 0x7f7474100290 2023-10-11T16:30:07.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.342+0000 7f746bfff700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74600a11d0 0x7f74600a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:07.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.343+0000 7f746bfff700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74600a11d0 0x7f74600a3680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f745c00b2e0 tx=0x7f745c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:07.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.346+0000 7f7470ff9700 1 -- 172.21.15.111:0/2774559514 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7464096250 con 0x7f7474100290 2023-10-11T16:30:07.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.605+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f747402cfe0 con 0x7f7474100290 2023-10-11T16:30:07.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.605+0000 7f7470ff9700 1 -- 172.21.15.111:0/2774559514 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9613 (secure 0 0 0) 0x7f7464099870 con 0x7f7474100290 2023-10-11T16:30:07.606 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:07.607 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-10-11T16:30:07.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74600a11d0 msgr2=0x7f74600a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:07.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74600a11d0 0x7f74600a3680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f745c00b2e0 tx=0x7f745c005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7474100290 msgr2=0x7f7474195f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:07.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7474100290 0x7f7474195f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f746400e9c0 tx=0x7f746400ecd0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 shutdown_connections 2023-10-11T16:30:07.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74600a11d0 0x7f74600a3680 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7474101e10 0x7f747419d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7474100290 0x7f7474195f80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 --2- 172.21.15.111:0/2774559514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7474101480 0x7f74741964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 >> 172.21.15.111:0/2774559514 conn(0x7f74740fb870 msgr2=0x7f74741049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:07.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 shutdown_connections 2023-10-11T16:30:07.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.608+0000 7f74795ff700 1 -- 172.21.15.111:0/2774559514 wait complete. 2023-10-11T16:30:07.611 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 13 2023-10-11T16:30:07.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/1941202391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf8073550 msgr2=0x7fdbf80739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/1941202391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf8073550 0x7fdbf80739c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0098f0 tx=0x7fdbec009c00 comp rx=0 tx=0).stop 2023-10-11T16:30:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/1941202391 shutdown_connections 2023-10-11T16:30:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/1941202391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf810c860 0x7fdbf810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/1941202391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbf8074b30 0x7fdbf8072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/1941202391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf8073550 0x7fdbf80739c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/1941202391 >> 172.21.15.111:0/1941202391 conn(0x7fdbf80fb870 msgr2=0x7fdbf80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:07.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/1941202391 shutdown_connections 2023-10-11T16:30:07.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.877+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/1941202391 wait complete. 2023-10-11T16:30:07.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.878+0000 7fdbfe9ff700 1 Processor -- start 2023-10-11T16:30:07.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.878+0000 7fdbfe9ff700 1 -- start start 2023-10-11T16:30:07.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfe9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8073550 0x7fdbf819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:07.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfe9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf8074b30 0x7fdbf819cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfe9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbf810c860 0x7fdbf81a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfe9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf81a41e0 con 0x7fdbf810c860 2023-10-11T16:30:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfe9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf81a4320 con 0x7fdbf8073550 2023-10-11T16:30:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfe9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf81a44b0 con 0x7fdbf8074b30 2023-10-11T16:30:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfcf9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbf810c860 0x7fdbf81a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:07.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfcf9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbf810c860 0x7fdbf81a3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32916/0 (socket says 172.21.15.111:32916) 2023-10-11T16:30:07.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfcf9c700 1 -- 172.21.15.111:0/4111065652 learned_addr learned my addr 172.21.15.111:0/4111065652 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:07.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbf7fff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8073550 0x7fdbf819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:07.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbf77fe700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf8074b30 0x7fdbf819cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:07.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfcf9c700 1 -- 172.21.15.111:0/4111065652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf8074b30 msgr2=0x7fdbf819cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfcf9c700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf8074b30 0x7fdbf819cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfcf9c700 1 -- 172.21.15.111:0/4111065652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8073550 msgr2=0x7fdbf819c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfcf9c700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8073550 0x7fdbf819c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbfcf9c700 1 -- 172.21.15.111:0/4111065652 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec009520 con 0x7fdbf810c860 2023-10-11T16:30:07.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.879+0000 7fdbf7fff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8073550 0x7fdbf819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.880+0000 7fdbfcf9c700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbf810c860 0x7fdbf81a3c10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000c110 tx=0x7fdbf000c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.880+0000 7fdbf57fa700 1 -- 172.21.15.111:0/4111065652 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0017070 con 0x7fdbf810c860 2023-10-11T16:30:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.880+0000 7fdbf57fa700 1 -- 172.21.15.111:0/4111065652 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbf000e050 con 0x7fdbf810c860 2023-10-11T16:30:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.880+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf8112270 con 0x7fdbf810c860 2023-10-11T16:30:07.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.880+0000 7fdbf57fa700 1 -- 172.21.15.111:0/4111065652 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00078f0 con 0x7fdbf810c860 2023-10-11T16:30:07.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.880+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf81120e0 con 0x7fdbf810c860 2023-10-11T16:30:07.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.881+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf81023b0 con 0x7fdbf810c860 2023-10-11T16:30:07.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.883+0000 7fdbf57fa700 1 -- 172.21.15.111:0/4111065652 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fdbf0007a50 con 0x7fdbf810c860 2023-10-11T16:30:07.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:07 smithi111 ceph-mon[98527]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:07.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:07 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2774559514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:07.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.885+0000 7fdbf57fa700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdbe00a1150 0x7fdbe00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:07.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.885+0000 7fdbf57fa700 1 -- 172.21.15.111:0/4111065652 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdbf00cbf00 con 0x7fdbf810c860 2023-10-11T16:30:07.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.885+0000 7fdbf7fff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdbe00a1150 0x7fdbe00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:07.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.886+0000 7fdbf7fff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdbe00a1150 0x7fdbe00a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8074f00 tx=0x7fdbe8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:07.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:07.889+0000 7fdbf57fa700 1 -- 172.21.15.111:0/4111065652 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdbf0095a50 con 0x7fdbf810c860 2023-10-11T16:30:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:07 smithi150 ceph-mon[132217]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:07 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2774559514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:07 smithi130 ceph-mon[131859]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:07 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2774559514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:08.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.098+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbf8067a60 con 0x7fdbf810c860 2023-10-11T16:30:08.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.099+0000 7fdbf57fa700 1 -- 172.21.15.111:0/4111065652 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18492 (secure 0 0 0) 0x7fdbf0099070 con 0x7fdbf810c860 2023-10-11T16:30:08.100 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:08.100 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":79,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:29:24.677205+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdbe00a1150 msgr2=0x7fdbe00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdbe00a1150 0x7fdbe00a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8074f00 tx=0x7fdbe8009260 comp rx=0 tx=0).stop 2023-10-11T16:30:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbf810c860 msgr2=0x7fdbf81a3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbf810c860 0x7fdbf81a3c10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000c110 tx=0x7fdbf000c420 comp rx=0 tx=0).stop 2023-10-11T16:30:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 shutdown_connections 2023-10-11T16:30:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdbe00a1150 0x7fdbe00a3600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdbf810c860 0x7fdbf81a3c10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8073550 0x7fdbf819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 --2- 172.21.15.111:0/4111065652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdbf8074b30 0x7fdbf819cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 >> 172.21.15.111:0/4111065652 conn(0x7fdbf80fb870 msgr2=0x7fdbf80fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 shutdown_connections 2023-10-11T16:30:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.102+0000 7fdbfe9ff700 1 -- 172.21.15.111:0/4111065652 wait complete. 2023-10-11T16:30:08.124 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-11T16:30:08.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:30:08.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1522883920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f94101490 msgr2=0x7f7f94103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1522883920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f94101490 0x7f7f94103910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009990 tx=0x7f7f84009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1522883920 shutdown_connections 2023-10-11T16:30:08.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1522883920 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f94103e50 0x7f7f941042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1522883920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f940feb30 0x7f7f94100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1522883920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f94101490 0x7f7f94103910 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1522883920 >> 172.21.15.111:0/1522883920 conn(0x7f7f940fa7c0 msgr2=0x7f7f940fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:08.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1522883920 shutdown_connections 2023-10-11T16:30:08.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.331+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1522883920 wait complete. 2023-10-11T16:30:08.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.332+0000 7f7f9c1ff700 1 Processor -- start 2023-10-11T16:30:08.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9c1ff700 1 -- start start 2023-10-11T16:30:08.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9c1ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f940feb30 0x7f7f9419c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:08.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9c1ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f94101490 0x7f7f9419cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:08.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9c1ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f94103e50 0x7f7f941a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:08.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9c1ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f941a41b0 con 0x7f7f940feb30 2023-10-11T16:30:08.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9c1ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f941a42f0 con 0x7f7f94103e50 2023-10-11T16:30:08.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9c1ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f941a4480 con 0x7f7f94101490 2023-10-11T16:30:08.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f99f9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f940feb30 0x7f7f9419c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:08.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f99f9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f940feb30 0x7f7f9419c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32934/0 (socket says 172.21.15.111:32934) 2023-10-11T16:30:08.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f99f9b700 1 -- 172.21.15.111:0/1129629152 learned_addr learned my addr 172.21.15.111:0/1129629152 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:08.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9a79c700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f94103e50 0x7f7f941a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:08.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.333+0000 7f7f9979a700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f94101490 0x7f7f9419cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:08.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f99f9b700 1 -- 172.21.15.111:0/1129629152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f94101490 msgr2=0x7f7f9419cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f99f9b700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f94101490 0x7f7f9419cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f99f9b700 1 -- 172.21.15.111:0/1129629152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f94103e50 msgr2=0x7f7f941a3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f99f9b700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f94103e50 0x7f7f941a3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f99f9b700 1 -- 172.21.15.111:0/1129629152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f840095f0 con 0x7f7f940feb30 2023-10-11T16:30:08.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f9a79c700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f94103e50 0x7f7f941a3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:08.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f99f9b700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f940feb30 0x7f7f9419c630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000e9a0 tx=0x7f7f9000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:08.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f8affd700 1 -- 172.21.15.111:0/1129629152 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90009790 con 0x7f7f940feb30 2023-10-11T16:30:08.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f8affd700 1 -- 172.21.15.111:0/1129629152 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f9000cbd0 con 0x7f7f940feb30 2023-10-11T16:30:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f94112260 con 0x7f7f940feb30 2023-10-11T16:30:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f8affd700 1 -- 172.21.15.111:0/1129629152 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90018680 con 0x7f7f940feb30 2023-10-11T16:30:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.334+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f941120d0 con 0x7f7f940feb30 2023-10-11T16:30:08.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.336+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f940fc330 con 0x7f7f940feb30 2023-10-11T16:30:08.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.338+0000 7f7f8affd700 1 -- 172.21.15.111:0/1129629152 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f7f9000cd40 con 0x7f7f940feb30 2023-10-11T16:30:08.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.339+0000 7f7f8affd700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7f800a11d0 0x7f7f800a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.339+0000 7f7f8affd700 1 -- 172.21.15.111:0/1129629152 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7f90014070 con 0x7f7f940feb30 2023-10-11T16:30:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.339+0000 7f7f9979a700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7f800a11d0 0x7f7f800a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.340+0000 7f7f9979a700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7f800a11d0 0x7f7f800a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400b2e0 tx=0x7f7f84005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.341+0000 7f7f8affd700 1 -- 172.21.15.111:0/1129629152 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7f900964f0 con 0x7f7f940feb30 2023-10-11T16:30:08.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.577+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f94061700 con 0x7f7f940feb30 2023-10-11T16:30:08.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.578+0000 7f7f8affd700 1 -- 172.21.15.111:0/1129629152 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f90099b10 con 0x7f7f940feb30 2023-10-11T16:30:08.579 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:30:08.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7f800a11d0 msgr2=0x7f7f800a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7f800a11d0 0x7f7f800a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400b2e0 tx=0x7f7f84005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f940feb30 msgr2=0x7f7f9419c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f940feb30 0x7f7f9419c630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000e9a0 tx=0x7f7f9000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 shutdown_connections 2023-10-11T16:30:08.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f7f800a11d0 0x7f7f800a3680 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f94103e50 0x7f7f941a3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f940feb30 0x7f7f9419c630 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 --2- 172.21.15.111:0/1129629152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f94101490 0x7f7f9419cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 >> 172.21.15.111:0/1129629152 conn(0x7f7f940fa7c0 msgr2=0x7f7f940ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:08.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 shutdown_connections 2023-10-11T16:30:08.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.580+0000 7f7f9c1ff700 1 -- 172.21.15.111:0/1129629152 wait complete. 2023-10-11T16:30:08.603 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-11T16:30:08.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 -- 172.21.15.111:0/3014794973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4106cc0 msgr2=0x7f75c41070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 --2- 172.21.15.111:0/3014794973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4106cc0 0x7f75c41070d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009990 tx=0x7f75b4009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 -- 172.21.15.111:0/3014794973 shutdown_connections 2023-10-11T16:30:08.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 --2- 172.21.15.111:0/3014794973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c4108440 0x7f75c410ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 --2- 172.21.15.111:0/3014794973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4106cc0 0x7f75c41070d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 --2- 172.21.15.111:0/3014794973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c4107a90 0x7f75c4107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 -- 172.21.15.111:0/3014794973 >> 172.21.15.111:0/3014794973 conn(0x7f75c4075920 msgr2=0x7f75c4077d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:08.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 -- 172.21.15.111:0/3014794973 shutdown_connections 2023-10-11T16:30:08.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.819+0000 7f75c9dff700 1 -- 172.21.15.111:0/3014794973 wait complete. 2023-10-11T16:30:08.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.820+0000 7f75c9dff700 1 Processor -- start 2023-10-11T16:30:08.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.820+0000 7f75c9dff700 1 -- start start 2023-10-11T16:30:08.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c9dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c4106cc0 0x7f75c419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:08.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c9dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4107a90 0x7f75c419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:08.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c9dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c4108440 0x7f75c41a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:08.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c9dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c41a40e0 con 0x7f75c4107a90 2023-10-11T16:30:08.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c9dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c41a4220 con 0x7f75c4108440 2023-10-11T16:30:08.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c9dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c41a43b0 con 0x7f75c4106cc0 2023-10-11T16:30:08.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c4106cc0 0x7f75c419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:08.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4107a90 0x7f75c419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:08.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c4106cc0 0x7f75c419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:38904/0 (socket says 172.21.15.111:38904) 2023-10-11T16:30:08.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c37fe700 1 -- 172.21.15.111:0/1318496894 learned_addr learned my addr 172.21.15.111:0/1318496894 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:08.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c3fff700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c4108440 0x7f75c41a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:08.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c37fe700 1 -- 172.21.15.111:0/1318496894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c4108440 msgr2=0x7f75c41a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c37fe700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c4108440 0x7f75c41a3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c37fe700 1 -- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4107a90 msgr2=0x7f75c419caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:08.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c37fe700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4107a90 0x7f75c419caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:08.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c37fe700 1 -- 172.21.15.111:0/1318496894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b40095f0 con 0x7f75c4106cc0 2023-10-11T16:30:08.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.821+0000 7f75c2ffd700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4107a90 0x7f75c419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:08.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.822+0000 7f75c37fe700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c4106cc0 0x7f75c419c560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75b400b680 tx=0x7f75b40057b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:08.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.822+0000 7f75c0ff9700 1 -- 172.21.15.111:0/1318496894 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b401d070 con 0x7f75c4106cc0 2023-10-11T16:30:08.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.822+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c4112290 con 0x7f75c4106cc0 2023-10-11T16:30:08.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.822+0000 7f75c0ff9700 1 -- 172.21.15.111:0/1318496894 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75b400fbc0 con 0x7f75c4106cc0 2023-10-11T16:30:08.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.822+0000 7f75c0ff9700 1 -- 172.21.15.111:0/1318496894 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4017650 con 0x7f75c4106cc0 2023-10-11T16:30:08.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.822+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c4112100 con 0x7f75c4106cc0 2023-10-11T16:30:08.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.824+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c4067eb0 con 0x7f75c4106cc0 2023-10-11T16:30:08.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.825+0000 7f75c0ff9700 1 -- 172.21.15.111:0/1318496894 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f75b4022470 con 0x7f75c4106cc0 2023-10-11T16:30:08.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.827+0000 7f75c0ff9700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f75ac0a1150 0x7f75ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:08.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.827+0000 7f75c0ff9700 1 -- 172.21.15.111:0/1318496894 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f75b40cdc20 con 0x7f75c4106cc0 2023-10-11T16:30:08.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.827+0000 7f75c2ffd700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f75ac0a1150 0x7f75ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:08.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.827+0000 7f75c2ffd700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f75ac0a1150 0x7f75ac0a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f75b8011490 tx=0x7f75b8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:08.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:08.831+0000 7f75c0ff9700 1 -- 172.21.15.111:0/1318496894 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f75b4097800 con 0x7f75c4106cc0 2023-10-11T16:30:08.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:08 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4111065652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:08.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:08 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1129629152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:08.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:08 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4111065652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:08.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:08 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1129629152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:09.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.067+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f75c410d7b0 con 0x7f75c4106cc0 2023-10-11T16:30:09.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.068+0000 7f75c0ff9700 1 -- 172.21.15.111:0/1318496894 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f75b409ae20 con 0x7f75c4106cc0 2023-10-11T16:30:09.069 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:09.069 INFO:teuthology.orchestra.run.smithi111.stdout:[ 2023-10-11T16:30:09.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:30:09.069 INFO:teuthology.orchestra.run.smithi111.stdout: "poolnum": 1, 2023-10-11T16:30:09.070 INFO:teuthology.orchestra.run.smithi111.stdout: "poolname": ".mgr" 2023-10-11T16:30:09.070 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:30:09.070 INFO:teuthology.orchestra.run.smithi111.stdout:] 2023-10-11T16:30:09.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f75ac0a1150 msgr2=0x7f75ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:09.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f75ac0a1150 0x7f75ac0a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f75b8011490 tx=0x7f75b8009260 comp rx=0 tx=0).stop 2023-10-11T16:30:09.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c4106cc0 msgr2=0x7f75c419c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:09.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c4106cc0 0x7f75c419c560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75b400b680 tx=0x7f75b40057b0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 shutdown_connections 2023-10-11T16:30:09.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f75ac0a1150 0x7f75ac0a3600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75c4108440 0x7f75c41a3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75c4106cc0 0x7f75c419c560 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 --2- 172.21.15.111:0/1318496894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75c4107a90 0x7f75c419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.070+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 >> 172.21.15.111:0/1318496894 conn(0x7f75c4075920 msgr2=0x7f75c4077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:09.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.071+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 shutdown_connections 2023-10-11T16:30:09.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.071+0000 7f75c9dff700 1 -- 172.21.15.111:0/1318496894 wait complete. 2023-10-11T16:30:09.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:08 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4111065652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:09.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:08 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1129629152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:09.090 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-11T16:30:09.091 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-11T16:30:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2005760170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc074b30 msgr2=0x7fd7cc072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2005760170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc074b30 0x7fd7cc072f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009990 tx=0x7fd7bc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2005760170 shutdown_connections 2023-10-11T16:30:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2005760170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7cc10c890 0x7fd7cc10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2005760170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc074b30 0x7fd7cc072f80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2005760170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7cc073550 0x7fd7cc0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2005760170 >> 172.21.15.111:0/2005760170 conn(0x7fd7cc0fb8a0 msgr2=0x7fd7cc0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:09.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2005760170 shutdown_connections 2023-10-11T16:30:09.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.316+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2005760170 wait complete. 2023-10-11T16:30:09.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.317+0000 7fd7d1dff700 1 Processor -- start 2023-10-11T16:30:09.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.317+0000 7fd7d1dff700 1 -- start start 2023-10-11T16:30:09.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7d1dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7cc073550 0x7fd7cc19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:09.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7d1dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7cc074b30 0x7fd7cc19cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:09.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7d1dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc10c890 0x7fd7cc1a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:09.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7d1dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc1a4220 con 0x7fd7cc10c890 2023-10-11T16:30:09.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7d1dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc1a4360 con 0x7fd7cc074b30 2023-10-11T16:30:09.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7d1dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc1a44f0 con 0x7fd7cc073550 2023-10-11T16:30:09.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc10c890 0x7fd7cc1a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:09.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc10c890 0x7fd7cc1a3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32966/0 (socket says 172.21.15.111:32966) 2023-10-11T16:30:09.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cbfff700 1 -- 172.21.15.111:0/2645696970 learned_addr learned my addr 172.21.15.111:0/2645696970 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:09.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7caffd700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7cc074b30 0x7fd7cc19cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:09.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cb7fe700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7cc073550 0x7fd7cc19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:09.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cbfff700 1 -- 172.21.15.111:0/2645696970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7cc073550 msgr2=0x7fd7cc19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:09.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cbfff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7cc073550 0x7fd7cc19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cbfff700 1 -- 172.21.15.111:0/2645696970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7cc074b30 msgr2=0x7fd7cc19cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:09.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cbfff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7cc074b30 0x7fd7cc19cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:09.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7cbfff700 1 -- 172.21.15.111:0/2645696970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c0009520 con 0x7fd7cc10c890 2023-10-11T16:30:09.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.318+0000 7fd7caffd700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7cc074b30 0x7fd7cc19cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:09.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.319+0000 7fd7cbfff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc10c890 0x7fd7cc1a3c50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4007ee0 tx=0x7fd7c400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:09.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.319+0000 7fd7c8ff9700 1 -- 172.21.15.111:0/2645696970 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4017070 con 0x7fd7cc10c890 2023-10-11T16:30:09.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.319+0000 7fd7c8ff9700 1 -- 172.21.15.111:0/2645696970 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7c400f040 con 0x7fd7cc10c890 2023-10-11T16:30:09.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.319+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7bc0095f0 con 0x7fd7cc10c890 2023-10-11T16:30:09.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.319+0000 7fd7c8ff9700 1 -- 172.21.15.111:0/2645696970 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4005580 con 0x7fd7cc10c890 2023-10-11T16:30:09.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.319+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc112280 con 0x7fd7cc10c890 2023-10-11T16:30:09.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.320+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc1023e0 con 0x7fd7cc10c890 2023-10-11T16:30:09.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.322+0000 7fd7c8ff9700 1 -- 172.21.15.111:0/2645696970 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fd7c400db20 con 0x7fd7cc10c890 2023-10-11T16:30:09.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.323+0000 7fd7c8ff9700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7b40a1150 0x7fd7b40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:09.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.323+0000 7fd7c8ff9700 1 -- 172.21.15.111:0/2645696970 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd7c40cbc20 con 0x7fd7cc10c890 2023-10-11T16:30:09.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.323+0000 7fd7cb7fe700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7b40a1150 0x7fd7b40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:09.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.324+0000 7fd7cb7fe700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7b40a1150 0x7fd7b40a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc00b2e0 tx=0x7fd7bc005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:09.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.327+0000 7fd7c8ff9700 1 -- 172.21.15.111:0/2645696970 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd7c4095770 con 0x7fd7cc10c890 2023-10-11T16:30:09.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:09.539+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd7cc02cfe0 con 0x7fd7cc10c890 2023-10-11T16:30:10.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.093+0000 7fd7c8ff9700 1 -- 172.21.15.111:0/2645696970 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd7c4098d90 con 0x7fd7cc10c890 2023-10-11T16:30:10.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.097+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7b40a1150 msgr2=0x7fd7b40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:10.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.097+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7b40a1150 0x7fd7b40a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc00b2e0 tx=0x7fd7bc005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.097+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc10c890 msgr2=0x7fd7cc1a3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:10.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.097+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc10c890 0x7fd7cc1a3c50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4007ee0 tx=0x7fd7c400d3b0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.099+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 shutdown_connections 2023-10-11T16:30:10.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.099+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7b40a1150 0x7fd7b40a3600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.099+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc10c890 0x7fd7cc1a3c50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.099+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7cc073550 0x7fd7cc19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.099+0000 7fd7d1dff700 1 --2- 172.21.15.111:0/2645696970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7cc074b30 0x7fd7cc19cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.099+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 >> 172.21.15.111:0/2645696970 conn(0x7fd7cc0fb8a0 msgr2=0x7fd7cc0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.099+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 shutdown_connections 2023-10-11T16:30:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.100+0000 7fd7d1dff700 1 -- 172.21.15.111:0/2645696970 wait complete. 2023-10-11T16:30:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' created 2023-10-11T16:30:10.120 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-10-11T16:30:10.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 -- 172.21.15.111:0/2883738909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10101310 msgr2=0x7faf10101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:10.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 --2- 172.21.15.111:0/2883738909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10101310 0x7faf10101780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf04009990 tx=0x7faf04009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 -- 172.21.15.111:0/2883738909 shutdown_connections 2023-10-11T16:30:10.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 --2- 172.21.15.111:0/2883738909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf10101cc0 0x7faf10108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 --2- 172.21.15.111:0/2883738909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf10100080 0x7faf10100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 --2- 172.21.15.111:0/2883738909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10101310 0x7faf10101780 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 -- 172.21.15.111:0/2883738909 >> 172.21.15.111:0/2883738909 conn(0x7faf100fb6a0 msgr2=0x7faf100fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:10.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 -- 172.21.15.111:0/2883738909 shutdown_connections 2023-10-11T16:30:10.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.330+0000 7faf0f59e700 1 -- 172.21.15.111:0/2883738909 wait complete. 2023-10-11T16:30:10.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:10 smithi130 ceph-mon[131859]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:10.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:10 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:10 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1318496894' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-10-11T16:30:10.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.331+0000 7faf0f59e700 1 Processor -- start 2023-10-11T16:30:10.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.331+0000 7faf0f59e700 1 -- start start 2023-10-11T16:30:10.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0f59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10100080 0x7faf10197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:10.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0f59e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf10101310 0x7faf101984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:10.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf10101cc0 0x7faf1019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:10.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1019fb10 con 0x7faf10101cc0 2023-10-11T16:30:10.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0f59e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1019fc50 con 0x7faf10101310 2023-10-11T16:30:10.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0f59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1019fde0 con 0x7faf10100080 2023-10-11T16:30:10.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10100080 0x7faf10197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:10.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10100080 0x7faf10197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:38958/0 (socket says 172.21.15.111:38958) 2023-10-11T16:30:10.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0e59c700 1 -- 172.21.15.111:0/1943999657 learned_addr learned my addr 172.21.15.111:0/1943999657 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:10.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0dd9b700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf10101310 0x7faf101984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:10.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0ed9d700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf10101cc0 0x7faf1019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:10.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0e59c700 1 -- 172.21.15.111:0/1943999657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf10101310 msgr2=0x7faf101984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:10.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0e59c700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf10101310 0x7faf101984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0e59c700 1 -- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf10101cc0 msgr2=0x7faf1019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:10.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0e59c700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf10101cc0 0x7faf1019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:10.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0e59c700 1 -- 172.21.15.111:0/1943999657 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf00009520 con 0x7faf10100080 2023-10-11T16:30:10.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.332+0000 7faf0dd9b700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf10101310 0x7faf101984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:10.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.333+0000 7faf0ed9d700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf10101cc0 0x7faf1019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:10.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.333+0000 7faf0e59c700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10100080 0x7faf10197f90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf0000eb30 tx=0x7faf0000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:10.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.333+0000 7faeff7fe700 1 -- 172.21.15.111:0/1943999657 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00009880 con 0x7faf10100080 2023-10-11T16:30:10.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.333+0000 7faeff7fe700 1 -- 172.21.15.111:0/1943999657 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf000044e0 con 0x7faf10100080 2023-10-11T16:30:10.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.333+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf040095f0 con 0x7faf10100080 2023-10-11T16:30:10.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.333+0000 7faeff7fe700 1 -- 172.21.15.111:0/1943999657 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00003d80 con 0x7faf10100080 2023-10-11T16:30:10.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.333+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1010ba20 con 0x7faf10100080 2023-10-11T16:30:10.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.335+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf10067980 con 0x7faf10100080 2023-10-11T16:30:10.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.336+0000 7faeff7fe700 1 -- 172.21.15.111:0/1943999657 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7faf00003f20 con 0x7faf10100080 2023-10-11T16:30:10.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.338+0000 7faeff7fe700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef80a1150 0x7faef80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:10.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.338+0000 7faeff7fe700 1 -- 172.21.15.111:0/1943999657 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7497+0+0 (secure 0 0 0) 0x7faf00014070 con 0x7faf10100080 2023-10-11T16:30:10.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.338+0000 7faf0dd9b700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef80a1150 0x7faef80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:10.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.339+0000 7faf0dd9b700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef80a1150 0x7faef80a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faf0400b680 tx=0x7faf04005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:10.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.342+0000 7faeff7fe700 1 -- 172.21.15.111:0/1943999657 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faf00096590 con 0x7faf10100080 2023-10-11T16:30:10.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:10 smithi111 ceph-mon[98527]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:10.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:10 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:10 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1318496894' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-10-11T16:30:10.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:10 smithi150 ceph-mon[132217]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:10.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:10 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:10.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:10 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1318496894' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-10-11T16:30:10.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:10.572+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7faf101a0950 con 0x7faf10100080 2023-10-11T16:30:11.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.104+0000 7faeff7fe700 1 -- 172.21.15.111:0/1943999657 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v81) v1 ==== 171+0+0 (secure 0 0 0) 0x7faf00094e60 con 0x7faf10100080 2023-10-11T16:30:11.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.111+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef80a1150 msgr2=0x7faef80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.111+0000 7faf0f59e700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef80a1150 0x7faef80a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faf0400b680 tx=0x7faf04005fd0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.111+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10100080 msgr2=0x7faf10197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.111+0000 7faf0f59e700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10100080 0x7faf10197f90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf0000eb30 tx=0x7faf0000ee40 comp rx=0 tx=0).stop 2023-10-11T16:30:11.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.112+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 shutdown_connections 2023-10-11T16:30:11.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.112+0000 7faf0f59e700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef80a1150 0x7faef80a3600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.112+0000 7faf0f59e700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf10101cc0 0x7faf1019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.112+0000 7faf0f59e700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf10100080 0x7faf10197f90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.112+0000 7faf0f59e700 1 --2- 172.21.15.111:0/1943999657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf10101310 0x7faf101984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.112+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 >> 172.21.15.111:0/1943999657 conn(0x7faf100fb6a0 msgr2=0x7faf100fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:11.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.113+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 shutdown_connections 2023-10-11T16:30:11.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.113+0000 7faf0f59e700 1 -- 172.21.15.111:0/1943999657 wait complete. 2023-10-11T16:30:11.119 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' created 2023-10-11T16:30:11.145 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-11T16:30:11.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 -- 172.21.15.111:0/1258656441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 msgr2=0x7faf0c1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 --2- 172.21.15.111:0/1258656441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 0x7faf0c1006e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faefc009990 tx=0x7faefc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 -- 172.21.15.111:0/1258656441 shutdown_connections 2023-10-11T16:30:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 --2- 172.21.15.111:0/1258656441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0c101e50 0x7faf0c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 --2- 172.21.15.111:0/1258656441 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 0x7faf0c1006e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 --2- 172.21.15.111:0/1258656441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf0c1014c0 0x7faf0c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 -- 172.21.15.111:0/1258656441 >> 172.21.15.111:0/1258656441 conn(0x7faf0c0fb8b0 msgr2=0x7faf0c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:11.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 -- 172.21.15.111:0/1258656441 shutdown_connections 2023-10-11T16:30:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.363+0000 7faf123ff700 1 -- 172.21.15.111:0/1258656441 wait complete. 2023-10-11T16:30:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.364+0000 7faf123ff700 1 Processor -- start 2023-10-11T16:30:11.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.364+0000 7faf123ff700 1 -- start start 2023-10-11T16:30:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf123ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 0x7faf0c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:11.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf123ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0c1014c0 0x7faf0c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:11.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf123ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf0c101e50 0x7faf0c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:11.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf123ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c19fe30 con 0x7faf0c1002d0 2023-10-11T16:30:11.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf123ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c19ff70 con 0x7faf0c101e50 2023-10-11T16:30:11.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf123ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c1a0100 con 0x7faf0c1014c0 2023-10-11T16:30:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 0x7faf0c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 0x7faf0c1982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33004/0 (socket says 172.21.15.111:33004) 2023-10-11T16:30:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0bfff700 1 -- 172.21.15.111:0/120654916 learned_addr learned my addr 172.21.15.111:0/120654916 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:11.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf1099c700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf0c101e50 0x7faf0c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:11.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0b7fe700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0c1014c0 0x7faf0c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:11.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0bfff700 1 -- 172.21.15.111:0/120654916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0c1014c0 msgr2=0x7faf0c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0bfff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0c1014c0 0x7faf0c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0bfff700 1 -- 172.21.15.111:0/120654916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf0c101e50 msgr2=0x7faf0c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0bfff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf0c101e50 0x7faf0c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0bfff700 1 -- 172.21.15.111:0/120654916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf04009c40 con 0x7faf0c1002d0 2023-10-11T16:30:11.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.365+0000 7faf0b7fe700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0c1014c0 0x7faf0c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:11.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.366+0000 7faf0bfff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 0x7faf0c1982b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faefc005720 tx=0x7faefc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:11.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.366+0000 7faf097fa700 1 -- 172.21.15.111:0/120654916 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc01d070 con 0x7faf0c1002d0 2023-10-11T16:30:11.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.366+0000 7faf097fa700 1 -- 172.21.15.111:0/120654916 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faefc021410 con 0x7faf0c1002d0 2023-10-11T16:30:11.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.366+0000 7faf097fa700 1 -- 172.21.15.111:0/120654916 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faefc017650 con 0x7faf0c1002d0 2023-10-11T16:30:11.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.366+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc0095f0 con 0x7faf0c1002d0 2023-10-11T16:30:11.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.366+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0c10df40 con 0x7faf0c1002d0 2023-10-11T16:30:11.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.367+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0c107250 con 0x7faf0c1002d0 2023-10-11T16:30:11.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.370+0000 7faf097fa700 1 -- 172.21.15.111:0/120654916 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7faefc021580 con 0x7faf0c1002d0 2023-10-11T16:30:11.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.371+0000 7faf097fa700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef40a11d0 0x7faef40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:11.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.371+0000 7faf097fa700 1 -- 172.21.15.111:0/120654916 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7879+0+0 (secure 0 0 0) 0x7faefc0ce1b0 con 0x7faf0c1002d0 2023-10-11T16:30:11.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.371+0000 7faf0b7fe700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef40a11d0 0x7faef40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:11.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.372+0000 7faf0b7fe700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef40a11d0 0x7faef40a3680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faf00009cb0 tx=0x7faf00009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:11.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.374+0000 7faf097fa700 1 -- 172.21.15.111:0/120654916 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faefc097b40 con 0x7faf0c1002d0 2023-10-11T16:30:11.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2645696970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-10-11T16:30:11.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:11.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:30:11.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2645696970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-10-11T16:30:11.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: osdmap e80: 12 total, 12 up, 12 in 2023-10-11T16:30:11.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:11 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:30:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2645696970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-10-11T16:30:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:11.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:30:11.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2645696970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-10-11T16:30:11.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: osdmap e80: 12 total, 12 up, 12 in 2023-10-11T16:30:11.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:11 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:30:11.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2645696970' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-10-11T16:30:11.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:11.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:30:11.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:11.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2645696970' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-10-11T16:30:11.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: osdmap e80: 12 total, 12 up, 12 in 2023-10-11T16:30:11.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:11 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:30:11.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.631+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faf0c02cfe0 con 0x7faf0c1002d0 2023-10-11T16:30:11.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.639+0000 7faf097fa700 1 -- 172.21.15.111:0/120654916 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7faefc09b160 con 0x7faf0c1002d0 2023-10-11T16:30:11.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef40a11d0 msgr2=0x7faef40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef40a11d0 0x7faef40a3680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7faf00009cb0 tx=0x7faf00009260 comp rx=0 tx=0).stop 2023-10-11T16:30:11.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 msgr2=0x7faf0c1982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 0x7faf0c1982b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faefc005720 tx=0x7faefc009580 comp rx=0 tx=0).stop 2023-10-11T16:30:11.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 shutdown_connections 2023-10-11T16:30:11.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faef40a11d0 0x7faef40a3680 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf0c101e50 0x7faf0c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf0c1002d0 0x7faf0c1982b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 --2- 172.21.15.111:0/120654916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0c1014c0 0x7faf0c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.643+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 >> 172.21.15.111:0/120654916 conn(0x7faf0c0fb8b0 msgr2=0x7faf0c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:11.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.645+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 shutdown_connections 2023-10-11T16:30:11.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.645+0000 7faf123ff700 1 -- 172.21.15.111:0/120654916 wait complete. 2023-10-11T16:30:11.654 INFO:teuthology.orchestra.run.smithi111.stderr:new fs with metadata pool 2 and data pool 3 2023-10-11T16:30:11.685 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-10-11T16:30:11.686 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-10-11T16:30:11.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 -- 172.21.15.111:0/217987696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e8074b30 msgr2=0x7f48e8072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 --2- 172.21.15.111:0/217987696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e8074b30 0x7f48e8072f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f48d8009990 tx=0x7f48d8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 -- 172.21.15.111:0/217987696 shutdown_connections 2023-10-11T16:30:11.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 --2- 172.21.15.111:0/217987696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e810c890 0x7f48e810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 --2- 172.21.15.111:0/217987696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e8074b30 0x7f48e8072f80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 --2- 172.21.15.111:0/217987696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48e8073550 0x7f48e80739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 -- 172.21.15.111:0/217987696 >> 172.21.15.111:0/217987696 conn(0x7f48e80fb8a0 msgr2=0x7f48e80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:11.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 -- 172.21.15.111:0/217987696 shutdown_connections 2023-10-11T16:30:11.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.901+0000 7f48ecfff700 1 -- 172.21.15.111:0/217987696 wait complete. 2023-10-11T16:30:11.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.902+0000 7f48ecfff700 1 Processor -- start 2023-10-11T16:30:11.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48ecfff700 1 -- start start 2023-10-11T16:30:11.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48ecfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e8073550 0x7f48e819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:11.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48ecfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48e8074b30 0x7f48e819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:11.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48ecfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e810c890 0x7f48e81a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:11.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48ecfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e81a4220 con 0x7f48e810c890 2023-10-11T16:30:11.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48ecfff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e81a4360 con 0x7f48e8073550 2023-10-11T16:30:11.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48ecfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e81a44f0 con 0x7f48e8074b30 2023-10-11T16:30:11.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48e6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e810c890 0x7f48e81a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:11.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48e659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e8073550 0x7f48e819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:11.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48e659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e8073550 0x7f48e819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:51374/0 (socket says 172.21.15.111:51374) 2023-10-11T16:30:11.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48e6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e810c890 0x7f48e81a3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33020/0 (socket says 172.21.15.111:33020) 2023-10-11T16:30:11.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48e659c700 1 -- 172.21.15.111:0/2155680571 learned_addr learned my addr 172.21.15.111:0/2155680571 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:11.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.903+0000 7f48e5d9b700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48e8074b30 0x7f48e819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:11.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48e659c700 1 -- 172.21.15.111:0/2155680571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48e8074b30 msgr2=0x7f48e819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48e659c700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48e8074b30 0x7f48e819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48e659c700 1 -- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e810c890 msgr2=0x7f48e81a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:11.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48e659c700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e810c890 0x7f48e81a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:11.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48e659c700 1 -- 172.21.15.111:0/2155680571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d80095f0 con 0x7f48e8073550 2023-10-11T16:30:11.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48e6d9d700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e810c890 0x7f48e81a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:11.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48e659c700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e8073550 0x7f48e819c6a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f48d8004710 tx=0x7f48d8009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:11.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48d77fe700 1 -- 172.21.15.111:0/2155680571 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d801d070 con 0x7f48e8073550 2023-10-11T16:30:11.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48d77fe700 1 -- 172.21.15.111:0/2155680571 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d8021410 con 0x7f48e8073550 2023-10-11T16:30:11.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e8112280 con 0x7f48e8073550 2023-10-11T16:30:11.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.904+0000 7f48d77fe700 1 -- 172.21.15.111:0/2155680571 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8017650 con 0x7f48e8073550 2023-10-11T16:30:11.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.905+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e81120f0 con 0x7f48e8073550 2023-10-11T16:30:11.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.906+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e81023e0 con 0x7f48e8073550 2023-10-11T16:30:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.909+0000 7f48d77fe700 1 -- 172.21.15.111:0/2155680571 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f48d8021580 con 0x7f48e8073550 2023-10-11T16:30:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.910+0000 7f48d77fe700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d00a1150 0x7f48d00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.910+0000 7f48d77fe700 1 -- 172.21.15.111:0/2155680571 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f48d80cd970 con 0x7f48e8073550 2023-10-11T16:30:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.910+0000 7f48e5d9b700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d00a1150 0x7f48d00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:11.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.911+0000 7f48e5d9b700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d00a1150 0x7f48d00a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f48e8074850 tx=0x7f48dc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:11.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:11.914+0000 7f48d77fe700 1 -- 172.21.15.111:0/2155680571 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f48d80973c0 con 0x7f48e8073550 2023-10-11T16:30:12.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.140+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f48e8061700 con 0x7f48e8073550 2023-10-11T16:30:12.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:12.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:30:12.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:30:12.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:30:12.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:30:12.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1943999657' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-11T16:30:12.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-11T16:30:12.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-10-11T16:30:12.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: osdmap e81: 12 total, 12 up, 12 in 2023-10-11T16:30:12.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/120654916' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-10-11T16:30:12.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-10-11T16:30:12.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: osdmap e82: 12 total, 12 up, 12 in 2023-10-11T16:30:12.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/120654916' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-10-11T16:30:12.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-10-11T16:30:12.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:12 smithi111 ceph-mon[98527]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-10-11T16:30:12.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:12.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:30:12.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:30:12.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:30:12.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:30:12.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1943999657' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-11T16:30:12.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-11T16:30:12.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-10-11T16:30:12.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: osdmap e81: 12 total, 12 up, 12 in 2023-10-11T16:30:12.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/120654916' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-10-11T16:30:12.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-10-11T16:30:12.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: osdmap e82: 12 total, 12 up, 12 in 2023-10-11T16:30:12.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/120654916' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-10-11T16:30:12.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-10-11T16:30:12.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:12 smithi150 ceph-mon[132217]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-10-11T16:30:12.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:30:12.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:30:12.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:30:12.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:30:12.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:30:12.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1943999657' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-11T16:30:12.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-11T16:30:12.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-10-11T16:30:12.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: osdmap e81: 12 total, 12 up, 12 in 2023-10-11T16:30:12.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/120654916' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-10-11T16:30:12.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-10-11T16:30:12.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: osdmap e82: 12 total, 12 up, 12 in 2023-10-11T16:30:12.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/120654916' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-10-11T16:30:12.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-10-11T16:30:12.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:12 smithi130 ceph-mon[131859]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-10-11T16:30:12.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.646+0000 7f48d77fe700 1 -- 172.21.15.111:0/2155680571 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v83) v1 ==== 157+0+0 (secure 0 0 0) 0x7f48d809a9e0 con 0x7f48e8073550 2023-10-11T16:30:12.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d00a1150 msgr2=0x7f48d00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:12.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d00a1150 0x7f48d00a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f48e8074850 tx=0x7f48dc009260 comp rx=0 tx=0).stop 2023-10-11T16:30:12.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e8073550 msgr2=0x7f48e819c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:12.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e8073550 0x7f48e819c6a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f48d8004710 tx=0x7f48d8009580 comp rx=0 tx=0).stop 2023-10-11T16:30:12.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 shutdown_connections 2023-10-11T16:30:12.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f48d00a1150 0x7f48d00a3600 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e810c890 0x7f48e81a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48e8073550 0x7f48e819c6a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 --2- 172.21.15.111:0/2155680571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48e8074b30 0x7f48e819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 >> 172.21.15.111:0/2155680571 conn(0x7f48e80fb8a0 msgr2=0x7f48e80fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:12.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 shutdown_connections 2023-10-11T16:30:12.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.650+0000 7f48ecfff700 1 -- 172.21.15.111:0/2155680571 wait complete. 2023-10-11T16:30:12.672 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-10-11T16:30:12.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 -- 172.21.15.111:0/1850881158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54102500 msgr2=0x7fcf54102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:12.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 --2- 172.21.15.111:0/1850881158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54102500 0x7fcf54102910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcf440098f0 tx=0x7fcf44009c00 comp rx=0 tx=0).stop 2023-10-11T16:30:12.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 -- 172.21.15.111:0/1850881158 shutdown_connections 2023-10-11T16:30:12.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 --2- 172.21.15.111:0/1850881158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf54104080 0x7fcf5410a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 --2- 172.21.15.111:0/1850881158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54102500 0x7fcf54102910 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 --2- 172.21.15.111:0/1850881158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf541036f0 0x7fcf54103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 -- 172.21.15.111:0/1850881158 >> 172.21.15.111:0/1850881158 conn(0x7fcf540fdae0 msgr2=0x7fcf540fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:12.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 -- 172.21.15.111:0/1850881158 shutdown_connections 2023-10-11T16:30:12.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.889+0000 7fcf595ff700 1 -- 172.21.15.111:0/1850881158 wait complete. 2023-10-11T16:30:12.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.890+0000 7fcf595ff700 1 Processor -- start 2023-10-11T16:30:12.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.890+0000 7fcf595ff700 1 -- start start 2023-10-11T16:30:12.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.890+0000 7fcf595ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf54102500 0x7fcf541982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:12.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.890+0000 7fcf595ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf541036f0 0x7fcf541987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:12.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.890+0000 7fcf595ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54104080 0x7fcf5419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:12.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.890+0000 7fcf595ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5419fe20 con 0x7fcf541036f0 2023-10-11T16:30:12.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf595ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5419ff60 con 0x7fcf54102500 2023-10-11T16:30:12.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf595ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf541a00f0 con 0x7fcf54104080 2023-10-11T16:30:12.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf527fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf541036f0 0x7fcf541987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:12.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf54102500 0x7fcf541982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:12.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf54102500 0x7fcf541982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:51380/0 (socket says 172.21.15.111:51380) 2023-10-11T16:30:12.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf537fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54104080 0x7fcf5419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:12.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 -- 172.21.15.111:0/4291154046 learned_addr learned my addr 172.21.15.111:0/4291154046 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:12.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 -- 172.21.15.111:0/4291154046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54104080 msgr2=0x7fcf5419f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:12.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54104080 0x7fcf5419f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 -- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf541036f0 msgr2=0x7fcf541987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:12.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf541036f0 0x7fcf541987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:12.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 -- 172.21.15.111:0/4291154046 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4c009d00 con 0x7fcf54102500 2023-10-11T16:30:12.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf527fc700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf541036f0 0x7fcf541987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:12.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.891+0000 7fcf52ffd700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf54102500 0x7fcf541982a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009fd0 tx=0x7fcf4400b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:12.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.892+0000 7fcf3bfff700 1 -- 172.21.15.111:0/4291154046 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4401c070 con 0x7fcf54102500 2023-10-11T16:30:12.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.892+0000 7fcf3bfff700 1 -- 172.21.15.111:0/4291154046 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf4400f7d0 con 0x7fcf54102500 2023-10-11T16:30:12.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.892+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf44009520 con 0x7fcf54102500 2023-10-11T16:30:12.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.892+0000 7fcf3bfff700 1 -- 172.21.15.111:0/4291154046 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44003a40 con 0x7fcf54102500 2023-10-11T16:30:12.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.892+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5410de50 con 0x7fcf54102500 2023-10-11T16:30:12.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.893+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5410dcc0 con 0x7fcf54102500 2023-10-11T16:30:12.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.894+0000 7fcf3bfff700 1 -- 172.21.15.111:0/4291154046 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fcf44020030 con 0x7fcf54102500 2023-10-11T16:30:12.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.896+0000 7fcf3bfff700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcf3c0a1150 0x7fcf3c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:12.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.896+0000 7fcf3bfff700 1 -- 172.21.15.111:0/4291154046 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8205+0+0 (secure 0 0 0) 0x7fcf440cd8e0 con 0x7fcf54102500 2023-10-11T16:30:12.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.896+0000 7fcf527fc700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcf3c0a1150 0x7fcf3c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:12.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.896+0000 7fcf527fc700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcf3c0a1150 0x7fcf3c0a3600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcf48011490 tx=0x7fcf48009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:12.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:12.899+0000 7fcf3bfff700 1 -- 172.21.15.111:0/4291154046 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcf44097080 con 0x7fcf54102500 2023-10-11T16:30:13.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:13.115+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcf5410dfe0 con 0x7fcf54102500 2023-10-11T16:30:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:13 smithi111 ceph-mon[98527]: pgmap v268: 97 pgs: 4 creating+activating, 5 creating+peering, 9 active+clean, 79 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2023-10-11T16:30:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:13 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2155680571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-11T16:30:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:13 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-11T16:30:13.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:13 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] up:active 2023-10-11T16:30:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:13 smithi111 ceph-mon[98527]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:13 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-10-11T16:30:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:13 smithi111 ceph-mon[98527]: osdmap e83: 12 total, 12 up, 12 in 2023-10-11T16:30:13.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:13 smithi111 ceph-mon[98527]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:13.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:13 smithi150 ceph-mon[132217]: pgmap v268: 97 pgs: 4 creating+activating, 5 creating+peering, 9 active+clean, 79 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2023-10-11T16:30:13.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:13 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2155680571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-11T16:30:13.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:13 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-11T16:30:13.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:13 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] up:active 2023-10-11T16:30:13.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:13 smithi150 ceph-mon[132217]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:13.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:13 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-10-11T16:30:13.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:13 smithi150 ceph-mon[132217]: osdmap e83: 12 total, 12 up, 12 in 2023-10-11T16:30:13.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:13 smithi150 ceph-mon[132217]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:13.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:13 smithi130 ceph-mon[131859]: pgmap v268: 97 pgs: 4 creating+activating, 5 creating+peering, 9 active+clean, 79 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2023-10-11T16:30:13.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:13 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2155680571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-11T16:30:13.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:13 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-11T16:30:13.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:13 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] up:active 2023-10-11T16:30:13.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:13 smithi130 ceph-mon[131859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:13.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:13 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-10-11T16:30:13.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:13 smithi130 ceph-mon[131859]: osdmap e83: 12 total, 12 up, 12 in 2023-10-11T16:30:13.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:13 smithi130 ceph-mon[131859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:14.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4291154046' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-11T16:30:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-11T16:30:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:30:14.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:14 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4291154046' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-11T16:30:14.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-11T16:30:14.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:14.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:30:14.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:14 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4291154046' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-11T16:30:14.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-11T16:30:14.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:14.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:30:14.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:30:14.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:14 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:30:15.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.104+0000 7fcf3bfff700 1 -- 172.21.15.111:0/4291154046 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v85) v1 ==== 228+0+0 (secure 0 0 0) 0x7fcf4409a6a0 con 0x7fcf54102500 2023-10-11T16:30:15.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcf3c0a1150 msgr2=0x7fcf3c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:15.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcf3c0a1150 0x7fcf3c0a3600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcf48011490 tx=0x7fcf48009260 comp rx=0 tx=0).stop 2023-10-11T16:30:15.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf54102500 msgr2=0x7fcf541982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:15.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf54102500 0x7fcf541982a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009fd0 tx=0x7fcf4400b790 comp rx=0 tx=0).stop 2023-10-11T16:30:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 shutdown_connections 2023-10-11T16:30:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcf3c0a1150 0x7fcf3c0a3600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf54104080 0x7fcf5419f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf54102500 0x7fcf541982a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 --2- 172.21.15.111:0/4291154046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf541036f0 0x7fcf541987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.110+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 >> 172.21.15.111:0/4291154046 conn(0x7fcf540fdae0 msgr2=0x7fcf540ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:15.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.112+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 shutdown_connections 2023-10-11T16:30:15.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.112+0000 7fcf595ff700 1 -- 172.21.15.111:0/4291154046 wait complete. 2023-10-11T16:30:15.117 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data_ec' created 2023-10-11T16:30:15.135 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-10-11T16:30:15.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.336+0000 7f45b0bff700 1 -- 172.21.15.111:0/3037284887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac073550 msgr2=0x7f45ac0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:15.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.336+0000 7f45b0bff700 1 --2- 172.21.15.111:0/3037284887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac073550 0x7f45ac0739c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009990 tx=0x7f45a0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.336+0000 7f45b0bff700 1 -- 172.21.15.111:0/3037284887 shutdown_connections 2023-10-11T16:30:15.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.336+0000 7f45b0bff700 1 --2- 172.21.15.111:0/3037284887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45ac10c8a0 0x7f45ac10ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.336+0000 7f45b0bff700 1 --2- 172.21.15.111:0/3037284887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45ac074b30 0x7f45ac072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.336+0000 7f45b0bff700 1 --2- 172.21.15.111:0/3037284887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac073550 0x7f45ac0739c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.336+0000 7f45b0bff700 1 -- 172.21.15.111:0/3037284887 >> 172.21.15.111:0/3037284887 conn(0x7f45ac0fb8b0 msgr2=0x7f45ac0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.337+0000 7f45b0bff700 1 -- 172.21.15.111:0/3037284887 shutdown_connections 2023-10-11T16:30:15.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.337+0000 7f45b0bff700 1 -- 172.21.15.111:0/3037284887 wait complete. 2023-10-11T16:30:15.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.337+0000 7f45b0bff700 1 Processor -- start 2023-10-11T16:30:15.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45b0bff700 1 -- start start 2023-10-11T16:30:15.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45b0bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45ac073550 0x7f45ac19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:15.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45b0bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45ac074b30 0x7f45ac19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:15.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45b0bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac10c8a0 0x7f45ac1a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:15.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45b0bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac1a4250 con 0x7f45ac074b30 2023-10-11T16:30:15.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45b0bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac1a4390 con 0x7f45ac10c8a0 2023-10-11T16:30:15.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45b0bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac1a4520 con 0x7f45ac073550 2023-10-11T16:30:15.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45a9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45ac074b30 0x7f45ac19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:15.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45a9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45ac074b30 0x7f45ac19cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47632/0 (socket says 172.21.15.111:47632) 2023-10-11T16:30:15.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45a9d9b700 1 -- 172.21.15.111:0/704549751 learned_addr learned my addr 172.21.15.111:0/704549751 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:15.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45aa59c700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45ac073550 0x7f45ac19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:15.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.338+0000 7f45aad9d700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac10c8a0 0x7f45ac1a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:15.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f45a9d9b700 1 -- 172.21.15.111:0/704549751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45ac073550 msgr2=0x7f45ac19c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:15.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f45a9d9b700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45ac073550 0x7f45ac19c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f45a9d9b700 1 -- 172.21.15.111:0/704549751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac10c8a0 msgr2=0x7f45ac1a3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:15.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f45a9d9b700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac10c8a0 0x7f45ac1a3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:15.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f45a9d9b700 1 -- 172.21.15.111:0/704549751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a4009c40 con 0x7f45ac074b30 2023-10-11T16:30:15.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f45a9d9b700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45ac074b30 0x7f45ac19cc10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45a0004710 tx=0x7f45a0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:15.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f459b7fe700 1 -- 172.21.15.111:0/704549751 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a001d070 con 0x7f45ac074b30 2023-10-11T16:30:15.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a00095f0 con 0x7f45ac074b30 2023-10-11T16:30:15.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f459b7fe700 1 -- 172.21.15.111:0/704549751 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45a0021410 con 0x7f45ac074b30 2023-10-11T16:30:15.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f459b7fe700 1 -- 172.21.15.111:0/704549751 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0017650 con 0x7f45ac074b30 2023-10-11T16:30:15.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.339+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac1122b0 con 0x7f45ac074b30 2023-10-11T16:30:15.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.340+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac1023f0 con 0x7f45ac074b30 2023-10-11T16:30:15.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.343+0000 7f459b7fe700 1 -- 172.21.15.111:0/704549751 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f45a0021580 con 0x7f45ac074b30 2023-10-11T16:30:15.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.344+0000 7f459b7fe700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45940a1150 0x7f45940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:15.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.344+0000 7f459b7fe700 1 -- 172.21.15.111:0/704549751 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f45a00ce5d0 con 0x7f45ac074b30 2023-10-11T16:30:15.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.344+0000 7f45aa59c700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45940a1150 0x7f45940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:15.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.345+0000 7f45aa59c700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45940a1150 0x7f45940a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f45ac074f00 tx=0x7f459c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:15.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.347+0000 7f459b7fe700 1 -- 172.21.15.111:0/704549751 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f45a0097b80 con 0x7f45ac074b30 2023-10-11T16:30:15.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:15 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:30:15.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:15 smithi111 ceph-mon[98527]: pgmap v270: 97 pgs: 4 creating+activating, 5 creating+peering, 43 active+clean, 45 unknown; 579 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-10-11T16:30:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:15 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:30:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:15 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:30:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:15 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:30:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:15 smithi111 ceph-mon[98527]: osdmap e84: 12 total, 12 up, 12 in 2023-10-11T16:30:15.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:15 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:30:15.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:15 smithi150 ceph-mon[132217]: pgmap v270: 97 pgs: 4 creating+activating, 5 creating+peering, 43 active+clean, 45 unknown; 579 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-10-11T16:30:15.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:15 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:30:15.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:15 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:30:15.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:15 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:30:15.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:15 smithi150 ceph-mon[132217]: osdmap e84: 12 total, 12 up, 12 in 2023-10-11T16:30:15.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:15.545+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f45ac02cfe0 con 0x7f45ac074b30 2023-10-11T16:30:15.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:15 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:30:15.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:15 smithi130 ceph-mon[131859]: pgmap v270: 97 pgs: 4 creating+activating, 5 creating+peering, 43 active+clean, 45 unknown; 579 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-10-11T16:30:15.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:15 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:30:15.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:15 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:30:15.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:15 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:30:15.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:15 smithi130 ceph-mon[131859]: osdmap e84: 12 total, 12 up, 12 in 2023-10-11T16:30:16.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.111+0000 7f459b7fe700 1 -- 172.21.15.111:0/704549751 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7f45a009b1a0 con 0x7f45ac074b30 2023-10-11T16:30:16.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.118+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45940a1150 msgr2=0x7f45940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.118+0000 7f45b0bff700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45940a1150 0x7f45940a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f45ac074f00 tx=0x7f459c009260 comp rx=0 tx=0).stop 2023-10-11T16:30:16.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.118+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45ac074b30 msgr2=0x7f45ac19cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.118+0000 7f45b0bff700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45ac074b30 0x7f45ac19cc10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45a0004710 tx=0x7f45a0009580 comp rx=0 tx=0).stop 2023-10-11T16:30:16.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.119+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 shutdown_connections 2023-10-11T16:30:16.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.119+0000 7f45b0bff700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45940a1150 0x7f45940a3600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.119+0000 7f45b0bff700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac10c8a0 0x7f45ac1a3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.119+0000 7f45b0bff700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45ac073550 0x7f45ac19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.119+0000 7f45b0bff700 1 --2- 172.21.15.111:0/704549751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45ac074b30 0x7f45ac19cc10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.119+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 >> 172.21.15.111:0/704549751 conn(0x7f45ac0fb8b0 msgr2=0x7f45ac0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:16.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.120+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 shutdown_connections 2023-10-11T16:30:16.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.120+0000 7f45b0bff700 1 -- 172.21.15.111:0/704549751 wait complete. 2023-10-11T16:30:16.124 INFO:teuthology.orchestra.run.smithi111.stderr:set pool 4 allow_ec_overwrites to true 2023-10-11T16:30:16.151 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-10-11T16:30:16.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 -- 172.21.15.111:0/474355893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 msgr2=0x7f8f0c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 --2- 172.21.15.111:0/474355893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 0x7f8f0c1006d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009990 tx=0x7f8f00009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 -- 172.21.15.111:0/474355893 shutdown_connections 2023-10-11T16:30:16.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 --2- 172.21.15.111:0/474355893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f0c101e40 0x7f8f0c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 --2- 172.21.15.111:0/474355893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 0x7f8f0c1006d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 --2- 172.21.15.111:0/474355893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c1014b0 0x7f8f0c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 -- 172.21.15.111:0/474355893 >> 172.21.15.111:0/474355893 conn(0x7f8f0c0fb8a0 msgr2=0x7f8f0c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:16.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 -- 172.21.15.111:0/474355893 shutdown_connections 2023-10-11T16:30:16.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.351+0000 7f8f131ff700 1 -- 172.21.15.111:0/474355893 wait complete. 2023-10-11T16:30:16.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.352+0000 7f8f131ff700 1 Processor -- start 2023-10-11T16:30:16.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.352+0000 7f8f131ff700 1 -- start start 2023-10-11T16:30:16.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f131ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 0x7f8f0c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:16.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f131ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c1014b0 0x7f8f0c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:16.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f131ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f0c101e40 0x7f8f0c19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:16.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f131ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c19fe00 con 0x7f8f0c1002c0 2023-10-11T16:30:16.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f131ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c19ff40 con 0x7f8f0c1014b0 2023-10-11T16:30:16.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f131ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c1a00d0 con 0x7f8f0c101e40 2023-10-11T16:30:16.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 0x7f8f0c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:16.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 0x7f8f0c198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47640/0 (socket says 172.21.15.111:47640) 2023-10-11T16:30:16.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 -- 172.21.15.111:0/2587827795 learned_addr learned my addr 172.21.15.111:0/2587827795 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:16.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f0bfff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c1014b0 0x7f8f0c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:16.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f1179c700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f0c101e40 0x7f8f0c19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:16.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 -- 172.21.15.111:0/2587827795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f0c101e40 msgr2=0x7f8f0c19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f0c101e40 0x7f8f0c19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 -- 172.21.15.111:0/2587827795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c1014b0 msgr2=0x7f8f0c1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c1014b0 0x7f8f0c1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 -- 172.21.15.111:0/2587827795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f000095f0 con 0x7f8f0c1002c0 2023-10-11T16:30:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f1179c700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f0c101e40 0x7f8f0c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.353+0000 7f8f10f9b700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 0x7f8f0c198280 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000b2e0 tx=0x7f8f00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.354+0000 7f8f09ffb700 1 -- 172.21.15.111:0/2587827795 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0001d070 con 0x7f8f0c1002c0 2023-10-11T16:30:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.354+0000 7f8f09ffb700 1 -- 172.21.15.111:0/2587827795 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f00005030 con 0x7f8f0c1002c0 2023-10-11T16:30:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.354+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0c10bc90 con 0x7f8f0c1002c0 2023-10-11T16:30:16.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.354+0000 7f8f09ffb700 1 -- 172.21.15.111:0/2587827795 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00004ca0 con 0x7f8f0c1002c0 2023-10-11T16:30:16.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.354+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c10bb00 con 0x7f8f0c1002c0 2023-10-11T16:30:16.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.355+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0c1929f0 con 0x7f8f0c1002c0 2023-10-11T16:30:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.357+0000 7f8f09ffb700 1 -- 172.21.15.111:0/2587827795 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f8f000051a0 con 0x7f8f0c1002c0 2023-10-11T16:30:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.358+0000 7f8f09ffb700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8efc0a11d0 0x7f8efc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.358+0000 7f8f09ffb700 1 -- 172.21.15.111:0/2587827795 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f8f000cdcd0 con 0x7f8f0c1002c0 2023-10-11T16:30:16.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.358+0000 7f8f0bfff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8efc0a11d0 0x7f8efc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:16.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.359+0000 7f8f0bfff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8efc0a11d0 0x7f8efc0a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8005950 tx=0x7f8ef800b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:16.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.362+0000 7f8f09ffb700 1 -- 172.21.15.111:0/2587827795 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8f0005a730 con 0x7f8f0c1002c0 2023-10-11T16:30:16.385 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:16 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-10-11T16:30:16.385 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:16 smithi130 ceph-mon[131859]: osdmap e85: 12 total, 12 up, 12 in 2023-10-11T16:30:16.385 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:16 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/704549751' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-10-11T16:30:16.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:16 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-10-11T16:30:16.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:16 smithi111 ceph-mon[98527]: osdmap e85: 12 total, 12 up, 12 in 2023-10-11T16:30:16.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:16 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/704549751' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-10-11T16:30:16.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:16 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-10-11T16:30:16.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:16 smithi150 ceph-mon[132217]: osdmap e85: 12 total, 12 up, 12 in 2023-10-11T16:30:16.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:16 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/704549751' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-10-11T16:30:16.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.606+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f8f0c02cfe0 con 0x7f8f0c1002c0 2023-10-11T16:30:16.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.612+0000 7f8f09ffb700 1 -- 172.21.15.111:0/2587827795 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f8f0009ad10 con 0x7f8f0c1002c0 2023-10-11T16:30:16.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8efc0a11d0 msgr2=0x7f8efc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8efc0a11d0 0x7f8efc0a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8005950 tx=0x7f8ef800b410 comp rx=0 tx=0).stop 2023-10-11T16:30:16.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 msgr2=0x7f8f0c198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 0x7f8f0c198280 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8f0000b2e0 tx=0x7f8f00004730 comp rx=0 tx=0).stop 2023-10-11T16:30:16.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 shutdown_connections 2023-10-11T16:30:16.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8efc0a11d0 0x7f8efc0a3680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f0c101e40 0x7f8f0c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f0c1002c0 0x7f8f0c198280 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 --2- 172.21.15.111:0/2587827795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c1014b0 0x7f8f0c1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.614+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 >> 172.21.15.111:0/2587827795 conn(0x7f8f0c0fb8a0 msgr2=0x7f8f0c106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:16.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.615+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 shutdown_connections 2023-10-11T16:30:16.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.615+0000 7f8f131ff700 1 -- 172.21.15.111:0/2587827795 wait complete. 2023-10-11T16:30:16.619 INFO:teuthology.orchestra.run.smithi111.stderr:added data pool 4 to fsmap 2023-10-11T16:30:16.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:30:16.831 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:30:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b[157084]: 2023-10-11T16:30:16.382+0000 7fe8647ed700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-11T16:30:16.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 -- 172.21.15.111:0/1945829950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420100290 msgr2=0x7f44201006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 --2- 172.21.15.111:0/1945829950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420100290 0x7f44201006a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4410009990 tx=0x7f4410009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 -- 172.21.15.111:0/1945829950 shutdown_connections 2023-10-11T16:30:16.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 --2- 172.21.15.111:0/1945829950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4420101e10 0x7f44201086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 --2- 172.21.15.111:0/1945829950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420100290 0x7f44201006a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 --2- 172.21.15.111:0/1945829950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4420101480 0x7f44201018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 -- 172.21.15.111:0/1945829950 >> 172.21.15.111:0/1945829950 conn(0x7f44200fb870 msgr2=0x7f44200fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:16.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 -- 172.21.15.111:0/1945829950 shutdown_connections 2023-10-11T16:30:16.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.831+0000 7f44263ff700 1 -- 172.21.15.111:0/1945829950 wait complete. 2023-10-11T16:30:16.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.832+0000 7f44263ff700 1 Processor -- start 2023-10-11T16:30:16.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.832+0000 7f44263ff700 1 -- start start 2023-10-11T16:30:16.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.832+0000 7f44263ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4420100290 0x7f44200748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:16.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.832+0000 7f44263ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4420101480 0x7f4420072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:16.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.832+0000 7f44263ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420101e10 0x7f4420073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:16.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.832+0000 7f44263ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420073bc0 con 0x7f4420101e10 2023-10-11T16:30:16.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.832+0000 7f44263ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420073d30 con 0x7f4420101480 2023-10-11T16:30:16.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.832+0000 7f44263ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44201a0000 con 0x7f4420100290 2023-10-11T16:30:16.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420101e10 0x7f4420073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:16.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420101e10 0x7f4420073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47658/0 (socket says 172.21.15.111:47658) 2023-10-11T16:30:16.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 -- 172.21.15.111:0/442357375 learned_addr learned my addr 172.21.15.111:0/442357375 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:16.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f441ffff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4420100290 0x7f44200748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:16.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f441f7fe700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4420101480 0x7f4420072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:16.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 -- 172.21.15.111:0/442357375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4420100290 msgr2=0x7f44200748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4420100290 0x7f44200748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 -- 172.21.15.111:0/442357375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4420101480 msgr2=0x7f4420072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:16.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4420101480 0x7f4420072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:16.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 -- 172.21.15.111:0/442357375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100095f0 con 0x7f4420101e10 2023-10-11T16:30:16.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.833+0000 7f442499c700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420101e10 0x7f4420073420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f441800ef30 tx=0x7f441800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:16.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.834+0000 7f441d7fa700 1 -- 172.21.15.111:0/442357375 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418018070 con 0x7f4420101e10 2023-10-11T16:30:16.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.834+0000 7f441d7fa700 1 -- 172.21.15.111:0/442357375 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f441800f040 con 0x7f4420101e10 2023-10-11T16:30:16.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.834+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442010bc40 con 0x7f4420101e10 2023-10-11T16:30:16.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.834+0000 7f441d7fa700 1 -- 172.21.15.111:0/442357375 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441800cd00 con 0x7f4420101e10 2023-10-11T16:30:16.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.834+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442010bab0 con 0x7f4420101e10 2023-10-11T16:30:16.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.835+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4420107210 con 0x7f4420101e10 2023-10-11T16:30:16.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.837+0000 7f441d7fa700 1 -- 172.21.15.111:0/442357375 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f4418003a00 con 0x7f4420101e10 2023-10-11T16:30:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.838+0000 7f441d7fa700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44080a1150 0x7f44080a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.838+0000 7f441d7fa700 1 -- 172.21.15.111:0/442357375 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f44180ccff0 con 0x7f4420101e10 2023-10-11T16:30:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.838+0000 7f441ffff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44080a1150 0x7f44080a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.839+0000 7f441ffff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44080a1150 0x7f44080a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f441000b2e0 tx=0x7f4410005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:16.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:16.840+0000 7f441d7fa700 1 -- 172.21.15.111:0/442357375 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4418096580 con 0x7f4420101e10 2023-10-11T16:30:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.085+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f442002cfe0 con 0x7f4420101e10 2023-10-11T16:30:17.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.086+0000 7f441d7fa700 1 -- 172.21.15.111:0/442357375 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10759 (secure 0 0 0) 0x7f4418099ba0 con 0x7f4420101e10 2023-10-11T16:30:17.088 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:17.088 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:16.608216+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-11T16:30:17.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.088+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44080a1150 msgr2=0x7f44080a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.088+0000 7f44263ff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44080a1150 0x7f44080a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f441000b2e0 tx=0x7f4410005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420101e10 msgr2=0x7f4420073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420101e10 0x7f4420073420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f441800ef30 tx=0x7f441800c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 shutdown_connections 2023-10-11T16:30:17.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44080a1150 0x7f44080a3600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4420101e10 0x7f4420073420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4420100290 0x7f44200748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 --2- 172.21.15.111:0/442357375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4420101480 0x7f4420072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 >> 172.21.15.111:0/442357375 conn(0x7f44200fb870 msgr2=0x7f44200fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:17.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 shutdown_connections 2023-10-11T16:30:17.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.089+0000 7f44263ff700 1 -- 172.21.15.111:0/442357375 wait complete. 2023-10-11T16:30:17.093 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-10-11T16:30:17.109 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:17.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 -- 172.21.15.111:0/2802994297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08100290 msgr2=0x7feb081006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2802994297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08100290 0x7feb081006a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009990 tx=0x7feaf8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 -- 172.21.15.111:0/2802994297 shutdown_connections 2023-10-11T16:30:17.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2802994297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb08101e10 0x7feb081086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2802994297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08100290 0x7feb081006a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2802994297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb08101480 0x7feb081018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 -- 172.21.15.111:0/2802994297 >> 172.21.15.111:0/2802994297 conn(0x7feb080fb870 msgr2=0x7feb080fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:17.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 -- 172.21.15.111:0/2802994297 shutdown_connections 2023-10-11T16:30:17.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.306+0000 7feb0cdff700 1 -- 172.21.15.111:0/2802994297 wait complete. 2023-10-11T16:30:17.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.307+0000 7feb0cdff700 1 Processor -- start 2023-10-11T16:30:17.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0cdff700 1 -- start start 2023-10-11T16:30:17.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0cdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb08100290 0x7feb081981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:17.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0cdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08101480 0x7feb08198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:17.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0cdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb08101e10 0x7feb0819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:17.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0cdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0819fd40 con 0x7feb08101480 2023-10-11T16:30:17.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0cdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0819fe80 con 0x7feb08100290 2023-10-11T16:30:17.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0cdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb081a0010 con 0x7feb08101e10 2023-10-11T16:30:17.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb05d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08101480 0x7feb08198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:17.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb08100290 0x7feb081981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:17.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb08100290 0x7feb081981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:60654/0 (socket says 172.21.15.111:60654) 2023-10-11T16:30:17.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb0659c700 1 -- 172.21.15.111:0/2924644510 learned_addr learned my addr 172.21.15.111:0/2924644510 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:17.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb06d9d700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb08101e10 0x7feb0819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:17.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb05d9b700 1 -- 172.21.15.111:0/2924644510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb08101e10 msgr2=0x7feb0819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb05d9b700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb08101e10 0x7feb0819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb05d9b700 1 -- 172.21.15.111:0/2924644510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb08100290 msgr2=0x7feb081981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb05d9b700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb08100290 0x7feb081981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb05d9b700 1 -- 172.21.15.111:0/2924644510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80095f0 con 0x7feb08101480 2023-10-11T16:30:17.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.308+0000 7feb06d9d700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb08101e10 0x7feb0819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:17.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.309+0000 7feb05d9b700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08101480 0x7feb08198700 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feafc00d750 tx=0x7feafc00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:17.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.309+0000 7feaf77fe700 1 -- 172.21.15.111:0/2924644510 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc004220 con 0x7feb08101480 2023-10-11T16:30:17.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.309+0000 7feaf77fe700 1 -- 172.21.15.111:0/2924644510 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feafc004cf0 con 0x7feb08101480 2023-10-11T16:30:17.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.309+0000 7feaf77fe700 1 -- 172.21.15.111:0/2924644510 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc00f690 con 0x7feb08101480 2023-10-11T16:30:17.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.309+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0810bc40 con 0x7feb08101480 2023-10-11T16:30:17.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.309+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0810bab0 con 0x7feb08101480 2023-10-11T16:30:17.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.310+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb081928d0 con 0x7feb08101480 2023-10-11T16:30:17.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.312+0000 7feaf77fe700 1 -- 172.21.15.111:0/2924644510 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7feafc004750 con 0x7feb08101480 2023-10-11T16:30:17.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.314+0000 7feaf77fe700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feaf00a1150 0x7feaf00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:17.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.314+0000 7feaf77fe700 1 -- 172.21.15.111:0/2924644510 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7feafc0ccdb0 con 0x7feb08101480 2023-10-11T16:30:17.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.314+0000 7feb0659c700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feaf00a1150 0x7feaf00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:17.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.315+0000 7feb0659c700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feaf00a1150 0x7feaf00a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feaf800b5c0 tx=0x7feaf8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:17.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.317+0000 7feaf77fe700 1 -- 172.21.15.111:0/2924644510 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feafc096310 con 0x7feb08101480 2023-10-11T16:30:17.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:17 smithi111 ceph-mon[98527]: pgmap v273: 129 pgs: 4 creating+peering, 4 creating+activating, 93 active+clean, 28 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 9 op/s 2023-10-11T16:30:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:17 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/704549751' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-10-11T16:30:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:17 smithi111 ceph-mon[98527]: osdmap e86: 12 total, 12 up, 12 in 2023-10-11T16:30:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:17 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2587827795' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-10-11T16:30:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:17 smithi111 ceph-mon[98527]: osdmap e87: 12 total, 12 up, 12 in 2023-10-11T16:30:17.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:17 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2587827795' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-10-11T16:30:17.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:17 smithi111 ceph-mon[98527]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:17.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:17 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/442357375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:17 smithi150 ceph-mon[132217]: pgmap v273: 129 pgs: 4 creating+peering, 4 creating+activating, 93 active+clean, 28 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 9 op/s 2023-10-11T16:30:17.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:17 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/704549751' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-10-11T16:30:17.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:17 smithi150 ceph-mon[132217]: osdmap e86: 12 total, 12 up, 12 in 2023-10-11T16:30:17.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:17 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2587827795' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-10-11T16:30:17.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:17 smithi150 ceph-mon[132217]: osdmap e87: 12 total, 12 up, 12 in 2023-10-11T16:30:17.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:17 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2587827795' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-10-11T16:30:17.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:17 smithi150 ceph-mon[132217]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:17.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:17 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/442357375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:17.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.519+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb0802cfe0 con 0x7feb08101480 2023-10-11T16:30:17.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.521+0000 7feaf77fe700 1 -- 172.21.15.111:0/2924644510 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7feafc016090 con 0x7feb08101480 2023-10-11T16:30:17.523 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:17.524 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:16.607979+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:17.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feaf00a1150 msgr2=0x7feaf00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feaf00a1150 0x7feaf00a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feaf800b5c0 tx=0x7feaf8005fd0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08101480 msgr2=0x7feb08198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08101480 0x7feb08198700 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feafc00d750 tx=0x7feafc00da60 comp rx=0 tx=0).stop 2023-10-11T16:30:17.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 shutdown_connections 2023-10-11T16:30:17.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feaf00a1150 0x7feaf00a3600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb08101e10 0x7feb0819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb08100290 0x7feb081981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 --2- 172.21.15.111:0/2924644510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb08101480 0x7feb08198700 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.523+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 >> 172.21.15.111:0/2924644510 conn(0x7feb080fb870 msgr2=0x7feb08106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:17.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.524+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 shutdown_connections 2023-10-11T16:30:17.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.524+0000 7feb0cdff700 1 -- 172.21.15.111:0/2924644510 wait complete. 2023-10-11T16:30:17.546 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:17 smithi130 ceph-mon[131859]: pgmap v273: 129 pgs: 4 creating+peering, 4 creating+activating, 93 active+clean, 28 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 9 op/s 2023-10-11T16:30:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:17 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/704549751' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-10-11T16:30:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:17 smithi130 ceph-mon[131859]: osdmap e86: 12 total, 12 up, 12 in 2023-10-11T16:30:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:17 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2587827795' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-10-11T16:30:17.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:17 smithi130 ceph-mon[131859]: osdmap e87: 12 total, 12 up, 12 in 2023-10-11T16:30:17.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:17 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2587827795' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-10-11T16:30:17.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:17 smithi130 ceph-mon[131859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:17.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:17 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/442357375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:17.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 -- 172.21.15.111:0/1991081824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c074b30 msgr2=0x7f692c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 --2- 172.21.15.111:0/1991081824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c074b30 0x7f692c072f80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f691c009990 tx=0x7f691c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 -- 172.21.15.111:0/1991081824 shutdown_connections 2023-10-11T16:30:17.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 --2- 172.21.15.111:0/1991081824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f692c10c860 0x7f692c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 --2- 172.21.15.111:0/1991081824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c074b30 0x7f692c072f80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 --2- 172.21.15.111:0/1991081824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f692c073550 0x7f692c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 -- 172.21.15.111:0/1991081824 >> 172.21.15.111:0/1991081824 conn(0x7f692c0fb870 msgr2=0x7f692c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:17.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 -- 172.21.15.111:0/1991081824 shutdown_connections 2023-10-11T16:30:17.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.745+0000 7f69323ff700 1 -- 172.21.15.111:0/1991081824 wait complete. 2023-10-11T16:30:17.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.746+0000 7f69323ff700 1 Processor -- start 2023-10-11T16:30:17.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.746+0000 7f69323ff700 1 -- start start 2023-10-11T16:30:17.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.746+0000 7f69323ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f692c073550 0x7f692c19c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:17.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.746+0000 7f69323ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f692c074b30 0x7f692c19cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:17.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f69323ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c10c860 0x7f692c1a3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:17.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f69323ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c1a4150 con 0x7f692c10c860 2023-10-11T16:30:17.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f69323ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c1a4290 con 0x7f692c073550 2023-10-11T16:30:17.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f69323ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c1a4420 con 0x7f692c074b30 2023-10-11T16:30:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c10c860 0x7f692c1a3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:17.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c10c860 0x7f692c1a3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47712/0 (socket says 172.21.15.111:47712) 2023-10-11T16:30:17.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 -- 172.21.15.111:0/3745701621 learned_addr learned my addr 172.21.15.111:0/3745701621 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:17.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f692b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f692c074b30 0x7f692c19cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:17.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f692bfff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f692c073550 0x7f692c19c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:17.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 -- 172.21.15.111:0/3745701621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f692c074b30 msgr2=0x7f692c19cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f692c074b30 0x7f692c19cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 -- 172.21.15.111:0/3745701621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f692c073550 msgr2=0x7f692c19c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f692c073550 0x7f692c19c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 -- 172.21.15.111:0/3745701621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0095f0 con 0x7f692c10c860 2023-10-11T16:30:17.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f692bfff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f692c073550 0x7f692c19c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:17.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.747+0000 7f693099c700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c10c860 0x7f692c1a3b80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6924007ee0 tx=0x7f692400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:17.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.748+0000 7f69297fa700 1 -- 172.21.15.111:0/3745701621 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924017070 con 0x7f692c10c860 2023-10-11T16:30:17.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.748+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c112250 con 0x7f692c10c860 2023-10-11T16:30:17.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.748+0000 7f69297fa700 1 -- 172.21.15.111:0/3745701621 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f692400f040 con 0x7f692c10c860 2023-10-11T16:30:17.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.748+0000 7f69297fa700 1 -- 172.21.15.111:0/3745701621 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924005740 con 0x7f692c10c860 2023-10-11T16:30:17.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.748+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c1120c0 con 0x7f692c10c860 2023-10-11T16:30:17.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.749+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692c1023b0 con 0x7f692c10c860 2023-10-11T16:30:17.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.750+0000 7f69297fa700 1 -- 172.21.15.111:0/3745701621 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f692400dc60 con 0x7f692c10c860 2023-10-11T16:30:17.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.752+0000 7f69297fa700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f69140a1150 0x7f69140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:17.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.752+0000 7f69297fa700 1 -- 172.21.15.111:0/3745701621 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f69240cc2f0 con 0x7f692c10c860 2023-10-11T16:30:17.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.752+0000 7f692bfff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f69140a1150 0x7f69140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:17.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.753+0000 7f692bfff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f69140a1150 0x7f69140a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f691c00b2e0 tx=0x7f691c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:17.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.756+0000 7f69297fa700 1 -- 172.21.15.111:0/3745701621 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6924095830 con 0x7f692c10c860 2023-10-11T16:30:17.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.955+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f692c067a60 con 0x7f692c10c860 2023-10-11T16:30:17.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.957+0000 7f69297fa700 1 -- 172.21.15.111:0/3745701621 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f6924098e50 con 0x7f692c10c860 2023-10-11T16:30:17.959 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:17.959 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:16.607979+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:17.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f69140a1150 msgr2=0x7f69140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f69140a1150 0x7f69140a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f691c00b2e0 tx=0x7f691c005fd0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c10c860 msgr2=0x7f692c1a3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:17.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c10c860 0x7f692c1a3b80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6924007ee0 tx=0x7f692400d3b0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 shutdown_connections 2023-10-11T16:30:17.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f69140a1150 0x7f69140a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f692c10c860 0x7f692c1a3b80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f692c073550 0x7f692c19c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 --2- 172.21.15.111:0/3745701621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f692c074b30 0x7f692c19cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:17.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 >> 172.21.15.111:0/3745701621 conn(0x7f692c0fb870 msgr2=0x7f692c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:17.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 shutdown_connections 2023-10-11T16:30:17.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:17.959+0000 7f69323ff700 1 -- 172.21.15.111:0/3745701621 wait complete. 2023-10-11T16:30:17.995 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:30:18.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 -- 172.21.15.111:0/4233522576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 msgr2=0x7fc7381018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/4233522576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 0x7fc7381018d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc720009990 tx=0x7fc720009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 -- 172.21.15.111:0/4233522576 shutdown_connections 2023-10-11T16:30:18.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/4233522576 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc738101e10 0x7fc7381086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/4233522576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc738100290 0x7fc7381006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/4233522576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 0x7fc7381018d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 -- 172.21.15.111:0/4233522576 >> 172.21.15.111:0/4233522576 conn(0x7fc7380fb870 msgr2=0x7fc7380fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:18.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 -- 172.21.15.111:0/4233522576 shutdown_connections 2023-10-11T16:30:18.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.194+0000 7fc73d9ff700 1 -- 172.21.15.111:0/4233522576 wait complete. 2023-10-11T16:30:18.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.195+0000 7fc73d9ff700 1 Processor -- start 2023-10-11T16:30:18.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.195+0000 7fc73d9ff700 1 -- start start 2023-10-11T16:30:18.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.195+0000 7fc73d9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc738100290 0x7fc7380748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:18.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.195+0000 7fc73d9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 0x7fc738072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:18.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.195+0000 7fc73d9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc738101e10 0x7fc738073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:18.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.195+0000 7fc73d9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc738073bc0 con 0x7fc738101480 2023-10-11T16:30:18.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.195+0000 7fc73d9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc738073d30 con 0x7fc738101e10 2023-10-11T16:30:18.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.195+0000 7fc73d9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7381a0000 con 0x7fc738100290 2023-10-11T16:30:18.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 0x7fc738072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:18.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 0x7fc738072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47726/0 (socket says 172.21.15.111:47726) 2023-10-11T16:30:18.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 -- 172.21.15.111:0/3799306148 learned_addr learned my addr 172.21.15.111:0/3799306148 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:18.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc736ffd700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc738100290 0x7fc7380748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:18.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc7377fe700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc738101e10 0x7fc738073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:18.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 -- 172.21.15.111:0/3799306148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc738100290 msgr2=0x7fc7380748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc738100290 0x7fc7380748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 -- 172.21.15.111:0/3799306148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc738101e10 msgr2=0x7fc738073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc738101e10 0x7fc738073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 -- 172.21.15.111:0/3799306148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200095f0 con 0x7fc738101480 2023-10-11T16:30:18.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.196+0000 7fc72ffff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 0x7fc738072ee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc72000b620 tx=0x7fc720005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:18.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.197+0000 7fc73c9fd700 1 -- 172.21.15.111:0/3799306148 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72001d070 con 0x7fc738101480 2023-10-11T16:30:18.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.197+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73810bc50 con 0x7fc738101480 2023-10-11T16:30:18.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.197+0000 7fc73c9fd700 1 -- 172.21.15.111:0/3799306148 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc720022470 con 0x7fc738101480 2023-10-11T16:30:18.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.197+0000 7fc73c9fd700 1 -- 172.21.15.111:0/3799306148 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720017650 con 0x7fc738101480 2023-10-11T16:30:18.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.197+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73810bac0 con 0x7fc738101480 2023-10-11T16:30:18.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.198+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7381071c0 con 0x7fc738101480 2023-10-11T16:30:18.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.201+0000 7fc73c9fd700 1 -- 172.21.15.111:0/3799306148 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fc7200225e0 con 0x7fc738101480 2023-10-11T16:30:18.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.203+0000 7fc73c9fd700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc7240a1150 0x7fc7240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:18.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.203+0000 7fc73c9fd700 1 -- 172.21.15.111:0/3799306148 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc7200ce750 con 0x7fc738101480 2023-10-11T16:30:18.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.203+0000 7fc736ffd700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc7240a1150 0x7fc7240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:18.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.204+0000 7fc736ffd700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc7240a1150 0x7fc7240a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc7381012e0 tx=0x7fc728009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:18.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.204+0000 7fc73c9fd700 1 -- 172.21.15.111:0/3799306148 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc720097d40 con 0x7fc738101480 2023-10-11T16:30:18.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:18 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2924644510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:18.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:18 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3745701621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:18.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.448+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc73802cfe0 con 0x7fc738101480 2023-10-11T16:30:18.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.449+0000 7fc73c9fd700 1 -- 172.21.15.111:0/3799306148 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10759 (secure 0 0 0) 0x7fc72009b360 con 0x7fc738101480 2023-10-11T16:30:18.451 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:18.451 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:16.608216+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-11T16:30:18.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:18 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2924644510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:18.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:18 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3745701621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:18.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc7240a1150 msgr2=0x7fc7240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc7240a1150 0x7fc7240a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc7381012e0 tx=0x7fc728009260 comp rx=0 tx=0).stop 2023-10-11T16:30:18.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 msgr2=0x7fc738072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 0x7fc738072ee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc72000b620 tx=0x7fc720005740 comp rx=0 tx=0).stop 2023-10-11T16:30:18.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 shutdown_connections 2023-10-11T16:30:18.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc7240a1150 0x7fc7240a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc738101e10 0x7fc738073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc738100290 0x7fc7380748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 --2- 172.21.15.111:0/3799306148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc738101480 0x7fc738072ee0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 >> 172.21.15.111:0/3799306148 conn(0x7fc7380fb870 msgr2=0x7fc7380fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:18.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 shutdown_connections 2023-10-11T16:30:18.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.452+0000 7fc73d9ff700 1 -- 172.21.15.111:0/3799306148 wait complete. 2023-10-11T16:30:18.456 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 17 2023-10-11T16:30:18.472 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:18.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:18 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2924644510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:18.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:18 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3745701621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:18.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/3958889551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e18100290 msgr2=0x7f0e181006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/3958889551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e18100290 0x7f0e181006a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009990 tx=0x7f0e08009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/3958889551 shutdown_connections 2023-10-11T16:30:18.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/3958889551 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18101e10 0x7f0e181086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/3958889551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e18100290 0x7f0e181006a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/3958889551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e18101480 0x7f0e181018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/3958889551 >> 172.21.15.111:0/3958889551 conn(0x7f0e180fb870 msgr2=0x7f0e180fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:18.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/3958889551 shutdown_connections 2023-10-11T16:30:18.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.668+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/3958889551 wait complete. 2023-10-11T16:30:18.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.669+0000 7f0e1e5ff700 1 Processor -- start 2023-10-11T16:30:18.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1e5ff700 1 -- start start 2023-10-11T16:30:18.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1e5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e18100290 0x7f0e18198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:18.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1e5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e18101480 0x7f0e181986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:18.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1e5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18101e10 0x7f0e1819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:18.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1e5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1819fd10 con 0x7f0e18101e10 2023-10-11T16:30:18.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1e5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1819fe50 con 0x7f0e18101480 2023-10-11T16:30:18.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1e5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1819ffe0 con 0x7f0e18100290 2023-10-11T16:30:18.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1cb9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18101e10 0x7f0e1819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1cb9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18101e10 0x7f0e1819f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47752/0 (socket says 172.21.15.111:47752) 2023-10-11T16:30:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1cb9c700 1 -- 172.21.15.111:0/2242048332 learned_addr learned my addr 172.21.15.111:0/2242048332 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e17fff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e18100290 0x7f0e18198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e177fe700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e18101480 0x7f0e181986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:18.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1cb9c700 1 -- 172.21.15.111:0/2242048332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e18100290 msgr2=0x7f0e18198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1cb9c700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e18100290 0x7f0e18198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1cb9c700 1 -- 172.21.15.111:0/2242048332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e18101480 msgr2=0x7f0e181986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.670+0000 7f0e1cb9c700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e18101480 0x7f0e181986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.671+0000 7f0e1cb9c700 1 -- 172.21.15.111:0/2242048332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080095f0 con 0x7f0e18101e10 2023-10-11T16:30:18.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.671+0000 7f0e1cb9c700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18101e10 0x7f0e1819f740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000ef30 tx=0x7f0e1000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:18.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.671+0000 7f0e157fa700 1 -- 172.21.15.111:0/2242048332 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10018070 con 0x7f0e18101e10 2023-10-11T16:30:18.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.671+0000 7f0e157fa700 1 -- 172.21.15.111:0/2242048332 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e1000f040 con 0x7f0e18101e10 2023-10-11T16:30:18.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.671+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1810bbe0 con 0x7f0e18101e10 2023-10-11T16:30:18.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.671+0000 7f0e157fa700 1 -- 172.21.15.111:0/2242048332 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1000cd00 con 0x7f0e18101e10 2023-10-11T16:30:18.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.671+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1810ba50 con 0x7f0e18101e10 2023-10-11T16:30:18.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.672+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e181a06c0 con 0x7f0e18101e10 2023-10-11T16:30:18.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.674+0000 7f0e157fa700 1 -- 172.21.15.111:0/2242048332 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f0e10003a00 con 0x7f0e18101e10 2023-10-11T16:30:18.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.676+0000 7f0e157fa700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e000a11d0 0x7f0e000a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:18.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.676+0000 7f0e157fa700 1 -- 172.21.15.111:0/2242048332 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0e100ccf60 con 0x7f0e18101e10 2023-10-11T16:30:18.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.676+0000 7f0e17fff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e000a11d0 0x7f0e000a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:18.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.677+0000 7f0e17fff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e000a11d0 0x7f0e000a3680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e0800b2e0 tx=0x7f0e08005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:18.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.678+0000 7f0e157fa700 1 -- 172.21.15.111:0/2242048332 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0e100964f0 con 0x7f0e18101e10 2023-10-11T16:30:18.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.880+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e181a0c70 con 0x7f0e18101e10 2023-10-11T16:30:18.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.881+0000 7f0e157fa700 1 -- 172.21.15.111:0/2242048332 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f0e10099b10 con 0x7f0e18101e10 2023-10-11T16:30:18.883 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:18.884 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:16.607979+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:18.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e000a11d0 msgr2=0x7f0e000a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e000a11d0 0x7f0e000a3680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e0800b2e0 tx=0x7f0e08005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18101e10 msgr2=0x7f0e1819f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18101e10 0x7f0e1819f740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000ef30 tx=0x7f0e1000c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 shutdown_connections 2023-10-11T16:30:18.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e000a11d0 0x7f0e000a3680 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e18101e10 0x7f0e1819f740 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e18100290 0x7f0e18198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 --2- 172.21.15.111:0/2242048332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e18101480 0x7f0e181986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:18.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 >> 172.21.15.111:0/2242048332 conn(0x7f0e180fb870 msgr2=0x7f0e180fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:18.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 shutdown_connections 2023-10-11T16:30:18.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:18.884+0000 7f0e1e5ff700 1 -- 172.21.15.111:0/2242048332 wait complete. 2023-10-11T16:30:18.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-11T16:30:11.633745+0000', 'modified': '2023-10-11T16:30:16.608216+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24569}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.130:6839/2448702777', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2448702777}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2448702777}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-10-11T16:30:18.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-11T16:30:18.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-11T16:30:19.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 -- 172.21.15.111:0/2282817943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08100240 msgr2=0x7f9a08100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/2282817943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08100240 0x7f9a08100650 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009990 tx=0x7f9a04009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 -- 172.21.15.111:0/2282817943 shutdown_connections 2023-10-11T16:30:19.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/2282817943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a08101dc0 0x7f9a08108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/2282817943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08100240 0x7f9a08100650 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/2282817943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a08101430 0x7f9a08101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 -- 172.21.15.111:0/2282817943 >> 172.21.15.111:0/2282817943 conn(0x7f9a080fb820 msgr2=0x7f9a080fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:19.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 -- 172.21.15.111:0/2282817943 shutdown_connections 2023-10-11T16:30:19.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.105+0000 7f9a0fdff700 1 -- 172.21.15.111:0/2282817943 wait complete. 2023-10-11T16:30:19.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.106+0000 7f9a0fdff700 1 Processor -- start 2023-10-11T16:30:19.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.106+0000 7f9a0fdff700 1 -- start start 2023-10-11T16:30:19.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.106+0000 7f9a0fdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a08100240 0x7f9a080748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.106+0000 7f9a0fdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a08101430 0x7f9a08072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.106+0000 7f9a0fdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08101dc0 0x7f9a08073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.106+0000 7f9a0fdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a08073bc0 con 0x7f9a08101dc0 2023-10-11T16:30:19.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.106+0000 7f9a0fdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a08073d30 con 0x7f9a08101430 2023-10-11T16:30:19.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.106+0000 7f9a0fdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a081a00a0 con 0x7f9a08100240 2023-10-11T16:30:19.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08101dc0 0x7f9a08073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08101dc0 0x7f9a08073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47760/0 (socket says 172.21.15.111:47760) 2023-10-11T16:30:19.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 -- 172.21.15.111:0/3768068200 learned_addr learned my addr 172.21.15.111:0/3768068200 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:19.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 -- 172.21.15.111:0/3768068200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a08100240 msgr2=0x7f9a080748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a08100240 0x7f9a080748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a08100240 0x7f9a080748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 -- 172.21.15.111:0/3768068200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a08101430 msgr2=0x7f9a08072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a08101430 0x7f9a08072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 -- 172.21.15.111:0/3768068200 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a040095f0 con 0x7f9a08101dc0 2023-10-11T16:30:19.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a08100240 0x7f9a080748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:30:19.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.107+0000 7f9a0e39c700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08101dc0 0x7f9a08073420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b970 tx=0x7f9a0000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:19.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.108+0000 7f99feffd700 1 -- 172.21.15.111:0/3768068200 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000c9b0 con 0x7f9a08101dc0 2023-10-11T16:30:19.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.108+0000 7f99feffd700 1 -- 172.21.15.111:0/3768068200 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a00014440 con 0x7f9a08101dc0 2023-10-11T16:30:19.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.108+0000 7f9a0fdff700 1 -- 172.21.15.111:0/3768068200 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0810bc70 con 0x7f9a08101dc0 2023-10-11T16:30:19.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.108+0000 7f99feffd700 1 -- 172.21.15.111:0/3768068200 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a000075d0 con 0x7f9a08101dc0 2023-10-11T16:30:19.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.108+0000 7f9a0fdff700 1 -- 172.21.15.111:0/3768068200 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0810bae0 con 0x7f9a08101dc0 2023-10-11T16:30:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.111+0000 7f99feffd700 1 -- 172.21.15.111:0/3768068200 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f9a000145b0 con 0x7f9a08101dc0 2023-10-11T16:30:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.112+0000 7f9a2371db80 1 Processor -- start 2023-10-11T16:30:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a2371db80 1 -- start start 2023-10-11T16:30:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b2bb0 0x55a5220b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b3500 0x55a5220b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a2371db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a521fb8010 con 0x55a5220b40c0 2023-10-11T16:30:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a2371db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52200a850 con 0x55a5220b3500 2023-10-11T16:30:19.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a2371db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a522003ce0 con 0x55a5220b2bb0 2023-10-11T16:30:19.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f99feffd700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f99f40a11d0 0x7f99f40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a0d39a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b2bb0 0x55a5220b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a0db9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b3500 0x55a5220b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a0db9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b3500 0x55a5220b3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:60732/0 (socket says 172.21.15.111:60732) 2023-10-11T16:30:19.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a0d39a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b2bb0 0x55a5220b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:60746/0 (socket says 172.21.15.111:60746) 2023-10-11T16:30:19.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a0db9b700 1 -- 172.21.15.111:0/1388020906 learned_addr learned my addr 172.21.15.111:0/1388020906 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:19.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.113+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/1388020906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f99feffd700 1 -- 172.21.15.111:0/3768068200 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9a00006e40 con 0x7f9a08101dc0 2023-10-11T16:30:19.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f9a0e39c700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f99f40a11d0 0x7f99f40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f9a0db9b700 1 -- 172.21.15.111:0/1388020906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b2bb0 msgr2=0x55a5220b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:30:19.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/1388020906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b2bb0 0x55a5220b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f9a0db9b700 1 -- 172.21.15.111:0/1388020906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 msgr2=0x55a5220b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:30:19.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/1388020906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f9a0db9b700 1 -- 172.21.15.111:0/1388020906 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a521e855e0 con 0x55a5220b3500 2023-10-11T16:30:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/1388020906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b3500 0x55a5220b3ae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9a04005b40 tx=0x7f9a04005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b2133f455318cd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f9a0e39c700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f99f40a11d0 0x7f99f40a3680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a00099080 tx=0x7f9a00095040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f99f3fff700 1 -- 172.21.15.111:0/1388020906 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0401e070 con 0x55a5220b3500 2023-10-11T16:30:19.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f99f3fff700 1 -- 172.21.15.111:0/1388020906 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a0400bd30 con 0x55a5220b3500 2023-10-11T16:30:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.114+0000 7f99f3fff700 1 -- 172.21.15.111:0/1388020906 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04005290 con 0x55a5220b3500 2023-10-11T16:30:19.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 -- 172.21.15.111:0/1388020906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b3500 msgr2=0x55a5220b3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1388020906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b3500 0x55a5220b3ae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9a04005b40 tx=0x7f9a04005ec0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 -- 172.21.15.111:0/1388020906 shutdown_connections 2023-10-11T16:30:19.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1388020906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1388020906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b2bb0 0x55a5220b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1388020906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b3500 0x55a5220b3ae0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 -- 172.21.15.111:0/1388020906 >> 172.21.15.111:0/1388020906 conn(0x55a5220a8310 msgr2=0x55a5220ad620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 -- 172.21.15.111:0/1388020906 shutdown_connections 2023-10-11T16:30:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.115+0000 7f9a2371db80 1 -- 172.21.15.111:0/1388020906 wait complete. 2023-10-11T16:30:19.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 1 -- start start 2023-10-11T16:30:19.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 1 Processor -- start 2023-10-11T16:30:19.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 0x55a5220bc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b3500 0x55a5220bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b40c0 0x55a5220c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a521fa5890 con 0x55a5220b3500 2023-10-11T16:30:19.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a521fa5530 con 0x55a5220b2bb0 2023-10-11T16:30:19.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a521e878d0 con 0x55a5220b40c0 2023-10-11T16:30:19.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.116+0000 7f9a2371db80 10 client.0 _ll_register_callbacks cb 0x55a521fc3730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:30:19.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b3500 0x55a5220bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0d39a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 0x55a5220bc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b3500 0x55a5220bc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47768/0 (socket says 172.21.15.111:47768) 2023-10-11T16:30:19.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 -- 172.21.15.111:0/2868004594 learned_addr learned my addr 172.21.15.111:0/2868004594 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:19.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b40c0 0x55a5220c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 -- 172.21.15.111:0/2868004594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b40c0 msgr2=0x55a5220c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:30:19.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b40c0 0x55a5220c39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 -- 172.21.15.111:0/2868004594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 msgr2=0x55a5220bc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:30:19.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 0x55a5220bc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 -- 172.21.15.111:0/2868004594 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a08108fb0 con 0x55a5220b3500 2023-10-11T16:30:19.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b3500 0x55a5220bc9a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9a04005f50 tx=0x7f9a0400baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f95bcc8c11f8f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:19.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.117+0000 7f99f17fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:30:19.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.118+0000 7f99f17fa700 1 -- 172.21.15.111:0/2868004594 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0401e040 con 0x55a5220b3500 2023-10-11T16:30:19.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.118+0000 7f9a2371db80 1 -- 172.21.15.111:0/2868004594 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5220a5ad0 con 0x55a5220b3500 2023-10-11T16:30:19.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.118+0000 7f99f17fa700 1 -- 172.21.15.111:0/2868004594 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9a040052a0 con 0x55a5220b3500 2023-10-11T16:30:19.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.121+0000 7f99f17fa700 1 -- 172.21.15.111:0/2868004594 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a04005460 con 0x55a5220b3500 2023-10-11T16:30:19.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.121+0000 7f9a2371db80 10 client.14610 fetch_fsmap learned FSMap version 17 2023-10-11T16:30:19.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.121+0000 7f9a2371db80 1 -- 172.21.15.111:0/2868004594 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55a521fca140 con 0x55a5220b3500 2023-10-11T16:30:19.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f99f17fa700 1 -- 172.21.15.111:0/2868004594 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a5220a5ad0 con 0x55a5220b3500 2023-10-11T16:30:19.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f99f17fa700 1 -- 172.21.15.111:0/2868004594 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f9a0402bbe0 con 0x55a5220b3500 2023-10-11T16:30:19.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f9a2371db80 10 client.14610 fetch_fsmap finished waiting for FSMap version 17 2023-10-11T16:30:19.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f9a2371db80 10 client.14610 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-11T16:30:19.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f9a2371db80 20 client.14610 populate_metadata read hostname 'smithi111' 2023-10-11T16:30:19.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5220c7340 0x55a5220c6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f9a2371db80 4 client.14610 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:30:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f9a2371db80 1 -- 172.21.15.111:0/2868004594 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a521fc6050 con 0x55a5220c7340 2023-10-11T16:30:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.122+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5220c7340 0x55a5220c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.123+0000 7f99f17fa700 10 client.14610 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:30:19.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.123+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5220c7340 0x55a5220c6910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f03acf8889369f82 server_cookie=3bf602e338980111 in_seq=0 out_seq=0 2023-10-11T16:30:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f99f17fa700 1 -- 172.21.15.111:0/2868004594 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a521fc6050 con 0x55a5220c7340 2023-10-11T16:30:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f99f17fa700 10 client.14610 handle_command_reply: tid=0 2023-10-11T16:30:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 client.14610 shutdown 2023-10-11T16:30:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 -- 172.21.15.111:0/2868004594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b3500 msgr2=0x55a5220bc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b3500 0x55a5220bc9a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9a04005f50 tx=0x7f9a0400baa0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 -- 172.21.15.111:0/2868004594 shutdown_connections 2023-10-11T16:30:19.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5220c7340 0x55a5220c6910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b40c0 0x55a5220c39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 0x55a5220bc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f99f17fa700 0 client.14610 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:30:19.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2868004594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b3500 0x55a5220bc9a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 -- 172.21.15.111:0/2868004594 >> 172.21.15.111:0/2868004594 conn(0x55a5220a8310 msgr2=0x55a5220acd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:19.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 -- 172.21.15.111:0/2868004594 shutdown_connections 2023-10-11T16:30:19.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 1 -- 172.21.15.111:0/2868004594 wait complete. 2023-10-11T16:30:19.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.124+0000 7f9a2371db80 20 client.14610 trim_cache size 0 max 16384 2023-10-11T16:30:19.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a2371db80 1 Processor -- start 2023-10-11T16:30:19.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a2371db80 1 -- start start 2023-10-11T16:30:19.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 0x55a5220d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a2371db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a521fad730 con 0x55a5220b40c0 2023-10-11T16:30:19.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a2371db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a521f0fed0 con 0x55a5220b2bb0 2023-10-11T16:30:19.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a2371db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a521f9fe10 con 0x55a5220b3500 2023-10-11T16:30:19.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a0d39a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a0d39a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47772/0 (socket says 172.21.15.111:47772) 2023-10-11T16:30:19.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a0d39a700 1 -- 172.21.15.111:0/2135773252 learned_addr learned my addr 172.21.15.111:0/2135773252 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:19.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 0x55a5220d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.143+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f9a0d39a700 1 -- 172.21.15.111:0/2135773252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 msgr2=0x55a5220c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:30:19.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f9a0d39a700 1 -- 172.21.15.111:0/2135773252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 msgr2=0x55a5220d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:30:19.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 0x55a5220d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f9a0d39a700 1 -- 172.21.15.111:0/2135773252 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a521fc9e90 con 0x55a5220b40c0 2023-10-11T16:30:19.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b7010 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99f8013420 tx=0x7f99f8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf6df0d73843804 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:19.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f99fcbff700 1 -- 172.21.15.111:0/2135773252 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8023e10 con 0x55a5220b40c0 2023-10-11T16:30:19.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f99fcbff700 1 -- 172.21.15.111:0/2135773252 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99f8029070 con 0x55a5220b40c0 2023-10-11T16:30:19.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.144+0000 7f99fcbff700 1 -- 172.21.15.111:0/2135773252 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8015310 con 0x55a5220b40c0 2023-10-11T16:30:19.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 -- 172.21.15.111:0/2135773252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 msgr2=0x55a5220b7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b7010 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99f8013420 tx=0x7f99f8009260 comp rx=0 tx=0).stop 2023-10-11T16:30:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 -- 172.21.15.111:0/2135773252 shutdown_connections 2023-10-11T16:30:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b40c0 0x55a5220b7010 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b2bb0 0x55a5220d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 --2- 172.21.15.111:0/2135773252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 -- 172.21.15.111:0/2135773252 >> 172.21.15.111:0/2135773252 conn(0x55a5220a8310 msgr2=0x55a5220ad5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:19.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 -- 172.21.15.111:0/2135773252 shutdown_connections 2023-10-11T16:30:19.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 -- 172.21.15.111:0/2135773252 wait complete. 2023-10-11T16:30:19.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.145+0000 7f9a2371db80 1 -- start start 2023-10-11T16:30:19.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a2371db80 1 Processor -- start 2023-10-11T16:30:19.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b2bb0 0x55a5220c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a2371db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b40c0 0x55a5220c0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a2371db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52211b1e0 con 0x55a5220b2bb0 2023-10-11T16:30:19.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a2371db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52211b320 con 0x55a5220b40c0 2023-10-11T16:30:19.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a2371db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a52211b460 con 0x55a5220b3500 2023-10-11T16:30:19.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a2371db80 10 client.0 _ll_register_callbacks cb 0x7f99f8009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:30:19.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a0d39a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b2bb0 0x55a5220c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a0d39a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b2bb0 0x55a5220c39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47778/0 (socket says 172.21.15.111:47778) 2023-10-11T16:30:19.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a0d39a700 1 -- 172.21.15.111:0/1887453588 learned_addr learned my addr 172.21.15.111:0/1887453588 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:19.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.146+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b40c0 0x55a5220c0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a0d39a700 1 -- 172.21.15.111:0/1887453588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 msgr2=0x55a5220c0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:30:19.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a0d39a700 1 -- 172.21.15.111:0/1887453588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b40c0 msgr2=0x55a5220c0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:30:19.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b40c0 0x55a5220c0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a0d39a700 1 -- 172.21.15.111:0/1887453588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f80233d0 con 0x55a5220b2bb0 2023-10-11T16:30:19.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:19.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a0d39a700 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b2bb0 0x55a5220c39f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009ac0 tx=0x7f99f8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5b0ce77fed95b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:19.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f99f17fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:30:19.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f99f17fa700 1 -- 172.21.15.111:0/1887453588 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f80150d0 con 0x55a5220b2bb0 2023-10-11T16:30:19.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f99f17fa700 1 -- 172.21.15.111:0/1887453588 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f99f8015230 con 0x55a5220b2bb0 2023-10-11T16:30:19.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a2371db80 1 -- 172.21.15.111:0/1887453588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a0400bd30 con 0x55a5220b2bb0 2023-10-11T16:30:19.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f99f17fa700 1 -- 172.21.15.111:0/1887453588 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99f80153f0 con 0x55a5220b2bb0 2023-10-11T16:30:19.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a2371db80 10 client.14616 fetch_fsmap learned FSMap version 17 2023-10-11T16:30:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f9a2371db80 1 -- 172.21.15.111:0/1887453588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55a521e1ac10 con 0x55a5220b2bb0 2023-10-11T16:30:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.147+0000 7f99f17fa700 1 -- 172.21.15.111:0/1887453588 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a0400bd30 con 0x55a5220b2bb0 2023-10-11T16:30:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.148+0000 7f99f17fa700 1 -- 172.21.15.111:0/1887453588 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f99f8013890 con 0x55a5220b2bb0 2023-10-11T16:30:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.148+0000 7f9a2371db80 10 client.14616 fetch_fsmap finished waiting for FSMap version 17 2023-10-11T16:30:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.148+0000 7f9a2371db80 10 client.14616 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-11T16:30:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.148+0000 7f9a2371db80 20 client.14616 populate_metadata read hostname 'smithi111' 2023-10-11T16:30:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.148+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a522120040 0x55a52211c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.148+0000 7f9a2371db80 4 client.14616 mds_command: new command op to 24569 tid=0[{"prefix": "status"}] 2023-10-11T16:30:19.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.148+0000 7f9a2371db80 1 -- 172.21.15.111:0/1887453588 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a521fd2610 con 0x55a522120040 2023-10-11T16:30:19.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.148+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a522120040 0x55a52211c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:19.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a0db9b700 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a522120040 0x55a52211c9f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d58f98b4b6aa2edf server_cookie=cc200685e49bd69b in_seq=0 out_seq=0 2023-10-11T16:30:19.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f99f17fa700 10 client.14616 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:30:19.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f99f17fa700 1 -- 172.21.15.111:0/1887453588 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55a521fd2610 con 0x55a522120040 2023-10-11T16:30:19.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f99f17fa700 10 client.14616 handle_command_reply: tid=0 2023-10-11T16:30:19.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 client.14616 shutdown 2023-10-11T16:30:19.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 -- 172.21.15.111:0/1887453588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b2bb0 msgr2=0x55a5220c39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b2bb0 0x55a5220c39f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009ac0 tx=0x7f99f8009f60 comp rx=0 tx=0).stop 2023-10-11T16:30:19.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 -- 172.21.15.111:0/1887453588 shutdown_connections 2023-10-11T16:30:19.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a522120040 0x55a52211c9f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5220b40c0 0x55a5220c0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f99f17fa700 0 client.14616 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:30:19.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5220b2bb0 0x55a5220c39f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 --2- 172.21.15.111:0/1887453588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5220b3500 0x55a5220c0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 -- 172.21.15.111:0/1887453588 >> 172.21.15.111:0/1887453588 conn(0x55a5220a8310 msgr2=0x55a5220ad570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 -- 172.21.15.111:0/1887453588 shutdown_connections 2023-10-11T16:30:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.149+0000 7f9a2371db80 1 -- 172.21.15.111:0/1887453588 wait complete. 2023-10-11T16:30:19.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a2371db80 20 client.14616 trim_cache size 0 max 16384 2023-10-11T16:30:19.173 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:30:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:30:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-10-11T16:30:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 24569, 2023-10-11T16:30:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-10-11T16:30:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-10-11T16:30:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-10-11T16:30:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 7.5136759089999998, 2023-10-11T16:30:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 17, 2023-10-11T16:30:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 87, 2023-10-11T16:30:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 86, 2023-10-11T16:30:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 13.592639911999999 2023-10-11T16:30:19.175 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:30:19.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 -- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f99f40a11d0 msgr2=0x7f99f40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f99f40a11d0 0x7f99f40a3680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a00099080 tx=0x7f9a00095040 comp rx=0 tx=0).stop 2023-10-11T16:30:19.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 -- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08101dc0 msgr2=0x7f9a08073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08101dc0 0x7f9a08073420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b970 tx=0x7f9a0000bd30 comp rx=0 tx=0).stop 2023-10-11T16:30:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 -- 172.21.15.111:0/3768068200 shutdown_connections 2023-10-11T16:30:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f99f40a11d0 0x7f99f40a3680 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a08101dc0 0x7f9a08073420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a08100240 0x7f9a080748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 --2- 172.21.15.111:0/3768068200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a08101430 0x7f9a08072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 -- 172.21.15.111:0/3768068200 >> 172.21.15.111:0/3768068200 conn(0x7f9a080fb820 msgr2=0x7f9a080fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:19.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 -- 172.21.15.111:0/3768068200 shutdown_connections 2023-10-11T16:30:19.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:19.150+0000 7f9a0fdff700 1 -- 172.21.15.111:0/3768068200 wait complete. 2023-10-11T16:30:19.179 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:19 smithi111 ceph-mon[98527]: pgmap v276: 129 pgs: 4 creating+peering, 110 active+clean, 15 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-10-11T16:30:19.179 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:19 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3799306148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:19.179 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:19 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2242048332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:19.180 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-11T16:30:19.181 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-11T16:30:19.182 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph fs ls 2023-10-11T16:30:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:19 smithi150 ceph-mon[132217]: pgmap v276: 129 pgs: 4 creating+peering, 110 active+clean, 15 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-10-11T16:30:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:19 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3799306148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:19.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:19 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2242048332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:19.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:19 smithi130 ceph-mon[131859]: pgmap v276: 129 pgs: 4 creating+peering, 110 active+clean, 15 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-10-11T16:30:19.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:19 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3799306148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:19.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:19 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2242048332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:20.018 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:30:21.331 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:21 smithi111 ceph-mon[98527]: pgmap v277: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2023-10-11T16:30:21.338 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:21 smithi130 ceph-mon[131859]: pgmap v277: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2023-10-11T16:30:21.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:21 smithi150 ceph-mon[132217]: pgmap v277: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2023-10-11T16:30:21.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 -- 172.21.15.111:0/2668955029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 msgr2=0x7ff86c104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:21.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 --2- 172.21.15.111:0/2668955029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 0x7ff86c104eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0099c0 tx=0x7ff85c009cd0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 -- 172.21.15.111:0/2668955029 shutdown_connections 2023-10-11T16:30:21.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 --2- 172.21.15.111:0/2668955029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86c1055e0 0x7ff86c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 --2- 172.21.15.111:0/2668955029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff86c100540 0x7ff86c100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 --2- 172.21.15.111:0/2668955029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 0x7ff86c104eb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 -- 172.21.15.111:0/2668955029 >> 172.21.15.111:0/2668955029 conn(0x7ff86c0fbdc0 msgr2=0x7ff86c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:21.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 -- 172.21.15.111:0/2668955029 shutdown_connections 2023-10-11T16:30:21.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.580+0000 7ff873782700 1 -- 172.21.15.111:0/2668955029 wait complete. 2023-10-11T16:30:21.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.581+0000 7ff873782700 1 Processor -- start 2023-10-11T16:30:21.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.581+0000 7ff873782700 1 -- start start 2023-10-11T16:30:21.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff873782700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86c100540 0x7ff86c1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:21.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff873782700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 0x7ff86c196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:21.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff873782700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff86c1055e0 0x7ff86c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:21.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff873782700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c19b3b0 con 0x7ff86c100ee0 2023-10-11T16:30:21.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff873782700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c19b520 con 0x7ff86c1055e0 2023-10-11T16:30:21.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff873782700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c19b6e0 con 0x7ff86c100540 2023-10-11T16:30:21.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff870d1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 0x7ff86c196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:21.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff870d1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 0x7ff86c196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47788/0 (socket says 172.21.15.111:47788) 2023-10-11T16:30:21.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff870d1d700 1 -- 172.21.15.111:0/2314917189 learned_addr learned my addr 172.21.15.111:0/2314917189 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:21.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff871d1f700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff86c1055e0 0x7ff86c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:21.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff87151e700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86c100540 0x7ff86c1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:21.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff870d1d700 1 -- 172.21.15.111:0/2314917189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86c100540 msgr2=0x7ff86c1963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:21.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff870d1d700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86c100540 0x7ff86c1963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff870d1d700 1 -- 172.21.15.111:0/2314917189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff86c1055e0 msgr2=0x7ff86c19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:21.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff870d1d700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff86c1055e0 0x7ff86c19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.582+0000 7ff870d1d700 1 -- 172.21.15.111:0/2314917189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c0095f0 con 0x7ff86c100ee0 2023-10-11T16:30:21.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.583+0000 7ff870d1d700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 0x7ff86c196930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009fd0 tx=0x7ff85c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:21.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.583+0000 7ff8627fc700 1 -- 172.21.15.111:0/2314917189 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c01d070 con 0x7ff86c100ee0 2023-10-11T16:30:21.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.583+0000 7ff8627fc700 1 -- 172.21.15.111:0/2314917189 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff85c005050 con 0x7ff86c100ee0 2023-10-11T16:30:21.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.583+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86c110360 con 0x7ff86c100ee0 2023-10-11T16:30:21.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.583+0000 7ff8627fc700 1 -- 172.21.15.111:0/2314917189 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c004a90 con 0x7ff86c100ee0 2023-10-11T16:30:21.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.583+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c1101d0 con 0x7ff86c100ee0 2023-10-11T16:30:21.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.584+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c1104f0 con 0x7ff86c100ee0 2023-10-11T16:30:21.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.586+0000 7ff8627fc700 1 -- 172.21.15.111:0/2314917189 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ff85c005470 con 0x7ff86c100ee0 2023-10-11T16:30:21.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.588+0000 7ff8627fc700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8580a11d0 0x7ff8580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:21.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.588+0000 7ff8627fc700 1 -- 172.21.15.111:0/2314917189 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff85c0cd7e0 con 0x7ff86c100ee0 2023-10-11T16:30:21.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.589+0000 7ff87151e700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8580a11d0 0x7ff8580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:21.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.589+0000 7ff87151e700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8580a11d0 0x7ff8580a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff868009c60 tx=0x7ff868009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:21.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.592+0000 7ff8627fc700 1 -- 172.21.15.111:0/2314917189 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff85c096d70 con 0x7ff86c100ee0 2023-10-11T16:30:21.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.835+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff86c1a3470 con 0x7ff86c100ee0 2023-10-11T16:30:21.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.835+0000 7ff8627fc700 1 -- 172.21.15.111:0/2314917189 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff85c09a390 con 0x7ff86c100ee0 2023-10-11T16:30:21.837 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-11T16:30:21.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8580a11d0 msgr2=0x7ff8580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:21.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8580a11d0 0x7ff8580a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff868009c60 tx=0x7ff868009260 comp rx=0 tx=0).stop 2023-10-11T16:30:21.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 msgr2=0x7ff86c196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:21.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 0x7ff86c196930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009fd0 tx=0x7ff85c00f690 comp rx=0 tx=0).stop 2023-10-11T16:30:21.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 shutdown_connections 2023-10-11T16:30:21.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8580a11d0 0x7ff8580a3680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff86c1055e0 0x7ff86c19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86c100540 0x7ff86c1963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 --2- 172.21.15.111:0/2314917189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff86c100ee0 0x7ff86c196930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:21.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 >> 172.21.15.111:0/2314917189 conn(0x7ff86c0fbdc0 msgr2=0x7ff86c0fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:21.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 shutdown_connections 2023-10-11T16:30:21.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:21.838+0000 7ff873782700 1 -- 172.21.15.111:0/2314917189 wait complete. 2023-10-11T16:30:22.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:22 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2314917189' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-11T16:30:22.299 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-11T16:30:22.299 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-11T16:30:22.300 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-10-11T16:30:22.300 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-10-11T16:30:22.300 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-11T16:30:22.301 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-11T16:30:22.301 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-11T16:30:22.301 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-11T16:30:22.302 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-10-11T16:30:22.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:22.302 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-10-11T16:30:22.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:22 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2314917189' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-11T16:30:22.348 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-11T16:30:22.348 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-11T16:30:22.348 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-10-11T16:30:22.349 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-10-11T16:30:22.349 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-10-11T16:30:22.349 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-10-11T16:30:22.349 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-11T16:30:22.349 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-10-11T16:30:22.350 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-10-11T16:30:22.350 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-11T16:30:22.350 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-10-11T16:30:22.350 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-10-11T16:30:22.351 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-11T16:30:22.351 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-10-11T16:30:22.351 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-11T16:30:22.351 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-10-11T16:30:22.351 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-11T16:30:22.351 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41725sec preferred_lft 41725sec 2023-10-11T16:30:22.352 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::9056:a3ce:d946:5f52/64 scope link noprefixroute 2023-10-11T16:30:22.352 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-10-11T16:30:22.353 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-11T16:30:22.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:22.353 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:30:22.354 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:30:22.354 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add name ceph-brx type bridge 2023-10-11T16:30:22.355 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr flush dev ceph-brx 2023-10-11T16:30:22.355 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set ceph-brx up 2023-10-11T16:30:22.355 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-11T16:30:22.356 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:30:22.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:22 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2314917189' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-11T16:30:22.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:22.831 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-11T16:30:22.902 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-10-11T16:30:22.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:22.906 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-10-11T16:30:22.966 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-10-11T16:30:22.967 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-11T16:30:22.967 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-11T16:30:22.967 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-11T16:30:22.967 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-11T16:30:22.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:22.969 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:30:22.970 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:30:22.970 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-11T16:30:22.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-11T16:30:22.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-11T16:30:22.971 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:30:23.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:23 smithi130 ceph-mon[131859]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2023-10-11T16:30:23.337 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:23 smithi111 ceph-mon[98527]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2023-10-11T16:30:23.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:23 smithi150 ceph-mon[132217]: pgmap v278: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2023-10-11T16:30:23.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:23.707 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-10-11T16:30:23.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:23.752 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2023-10-11T16:30:23.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:23.836 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:30:23.836 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:30:23.837 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-11T16:30:23.837 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-10-11T16:30:23.837 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:30:24.021 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-10-11T16:30:24.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:24.022 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:30:24.022 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:30:24.022 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-10-11T16:30:24.023 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-11T16:30:24.026 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-10-11T16:30:24.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-10-11T16:30:24.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-10-11T16:30:24.027 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:30:24.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:24.318 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:30:24.319 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:30:24.319 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-10-11T16:30:24.320 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.0 master ceph-brx 2023-10-11T16:30:24.320 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:30:24.493 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-10-11T16:30:24.494 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-11T16:30:24.494 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-10-11T16:30:24.538 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-10-11T16:30:24.538 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-11T16:30:24.539 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-10-11T16:30:24.621 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-10-11T16:30:24.731 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-11T16:30:24.732 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-11T16:30:24.799 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-11T16:30:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-11T16:30:24.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:24.803 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-10-11T16:30:24.847 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-11T16:30:24.847 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-10-11T16:30:24.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-10-11T16:30:24.962 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-11T16:30:24.963 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-11T16:30:25.033 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-11T16:30:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-11T16:30:25.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:25.037 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-10-11T16:30:25.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:2023-10-11T16:30:25.101+0000 7fd75d86e5c0 -1 init, newargv = 0x55b785572310 newargc=15 2023-10-11T16:30:25.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[135168]: starting ceph client 2023-10-11T16:30:25.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[135168]: starting fuse 2023-10-11T16:30:25.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:25 smithi111 ceph-mon[98527]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2023-10-11T16:30:25.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:25 smithi150 ceph-mon[132217]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2023-10-11T16:30:25.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:25 smithi130 ceph-mon[131859]: pgmap v279: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2023-10-11T16:30:26.182 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2023-10-11T16:30:26.264 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-11T16:30:26.264 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-11T16:30:26.338 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-11T16:30:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-11T16:30:26.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:26.342 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2023-10-11T16:30:26.388 INFO:teuthology.orchestra.run.smithi111.stdout:121 2023-10-11T16:30:26.388 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2023-10-11T16:30:26.389 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-11T16:30:26.389 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2023-10-11T16:30:26.389 DEBUG:teuthology.orchestra.run.smithi111:> import re 2023-10-11T16:30:26.390 DEBUG:teuthology.orchestra.run.smithi111:> import os 2023-10-11T16:30:26.390 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2023-10-11T16:30:26.390 DEBUG:teuthology.orchestra.run.smithi111:> 2023-10-11T16:30:26.391 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2023-10-11T16:30:26.391 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-10-11T16:30:26.391 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2023-10-11T16:30:26.392 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-10-11T16:30:26.392 DEBUG:teuthology.orchestra.run.smithi111:> 2023-10-11T16:30:26.392 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2023-10-11T16:30:26.393 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2023-10-11T16:30:26.393 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2023-10-11T16:30:26.393 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2023-10-11T16:30:26.394 DEBUG:teuthology.orchestra.run.smithi111:> 2023-10-11T16:30:26.394 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2023-10-11T16:30:26.394 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-11T16:30:26.395 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-11T16:30:26.395 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-11T16:30:26.395 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2023-10-11T16:30:26.396 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2023-10-11T16:30:26.396 DEBUG:teuthology.orchestra.run.smithi111:> return f 2023-10-11T16:30:26.396 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-11T16:30:26.397 DEBUG:teuthology.orchestra.run.smithi111:> 2023-10-11T16:30:26.397 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.admin")) 2023-10-11T16:30:26.397 DEBUG:teuthology.orchestra.run.smithi111:> ' 2023-10-11T16:30:26.573 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.admin.135168.asok 2023-10-11T16:30:26.752 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.135168.asok 2023-10-11T16:30:26.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:26.753 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.135168.asok status 2023-10-11T16:30:26.977 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:30:26.977 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2023-10-11T16:30:26.977 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "05a43c8fb3988914c879a50970f9054a098e0612", 2023-10-11T16:30:26.978 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev)", 2023-10-11T16:30:26.978 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "admin", 2023-10-11T16:30:26.978 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2023-10-11T16:30:26.978 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-10-11T16:30:26.978 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "135168", 2023-10-11T16:30:26.979 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2023-10-11T16:30:26.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:30:26.979 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2023-10-11T16:30:26.979 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2023-10-11T16:30:26.979 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 14628, 2023-10-11T16:30:26.980 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2023-10-11T16:30:26.980 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2023-10-11T16:30:26.980 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2023-10-11T16:30:26.980 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 14628 2023-10-11T16:30:26.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:30:26.981 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-10-11T16:30:26.981 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-10-11T16:30:26.981 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-10-11T16:30:26.981 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2416502325 2023-10-11T16:30:26.981 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:30:26.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:30:26.982 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2023-10-11T16:30:26.982 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-10-11T16:30:26.982 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2023-10-11T16:30:26.983 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2416502325 2023-10-11T16:30:26.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:30:26.983 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.14628 192.168.0.1:0/2416502325", 2023-10-11T16:30:26.983 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2416502325", 2023-10-11T16:30:26.983 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2023-10-11T16:30:26.983 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 17, 2023-10-11T16:30:26.984 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 87, 2023-10-11T16:30:26.984 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2023-10-11T16:30:26.984 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2023-10-11T16:30:26.984 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2023-10-11T16:30:26.984 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:30:26.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:26.992 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2023-10-11T16:30:26.992 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-10-11T16:30:27.038 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2023-10-11T16:30:27.038 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-10-11T16:30:27.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:27.039 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-10-11T16:30:27.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:27.151 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-10-11T16:30:27.288 INFO:teuthology.orchestra.run.smithi111.stdout:# file: . 2023-10-11T16:30:27.288 INFO:teuthology.orchestra.run.smithi111.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-10-11T16:30:27.288 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:27.290 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:30:27.290 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-11T16:30:27.335 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:27.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:27 smithi111 ceph-mon[98527]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 720 B/s wr, 0 op/s 2023-10-11T16:30:27.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:27 smithi150 ceph-mon[132217]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 720 B/s wr, 0 op/s 2023-10-11T16:30:27.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 -- 172.21.15.111:0/3792699866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc100290 msgr2=0x7fdccc1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:27.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/3792699866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc100290 0x7fdccc1006a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009990 tx=0x7fdcbc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 -- 172.21.15.111:0/3792699866 shutdown_connections 2023-10-11T16:30:27.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/3792699866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdccc101e10 0x7fdccc1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/3792699866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc100290 0x7fdccc1006a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/3792699866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdccc101480 0x7fdccc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 -- 172.21.15.111:0/3792699866 >> 172.21.15.111:0/3792699866 conn(0x7fdccc0fb870 msgr2=0x7fdccc0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 -- 172.21.15.111:0/3792699866 shutdown_connections 2023-10-11T16:30:27.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.569+0000 7fdcd1dff700 1 -- 172.21.15.111:0/3792699866 wait complete. 2023-10-11T16:30:27.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.570+0000 7fdcd1dff700 1 Processor -- start 2023-10-11T16:30:27.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdcd1dff700 1 -- start start 2023-10-11T16:30:27.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdcd1dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdccc100290 0x7fdccc195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:27.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdcd1dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc101480 0x7fdccc1964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:27.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdcd1dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdccc101e10 0x7fdccc19d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:27.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdcd1dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc19db30 con 0x7fdccc101480 2023-10-11T16:30:27.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdcd1dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc19dc70 con 0x7fdccc100290 2023-10-11T16:30:27.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdcd1dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc19de00 con 0x7fdccc101e10 2023-10-11T16:30:27.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdccaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc101480 0x7fdccc1964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:27.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdccaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc101480 0x7fdccc1964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51774/0 (socket says 172.21.15.111:51774) 2023-10-11T16:30:27.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdccaffd700 1 -- 172.21.15.111:0/2037429607 learned_addr learned my addr 172.21.15.111:0/2037429607 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:27.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdccb7fe700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdccc100290 0x7fdccc195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:27.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.571+0000 7fdccbfff700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdccc101e10 0x7fdccc19d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:27.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdccaffd700 1 -- 172.21.15.111:0/2037429607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdccc101e10 msgr2=0x7fdccc19d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdccaffd700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdccc101e10 0x7fdccc19d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdccaffd700 1 -- 172.21.15.111:0/2037429607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdccc100290 msgr2=0x7fdccc195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:27.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdccaffd700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdccc100290 0x7fdccc195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdccaffd700 1 -- 172.21.15.111:0/2037429607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcbc0095f0 con 0x7fdccc101480 2023-10-11T16:30:27.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdccb7fe700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdccc100290 0x7fdccc195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:27.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdccbfff700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdccc101e10 0x7fdccc19d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:27.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdccaffd700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc101480 0x7fdccc1964f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000e9a0 tx=0x7fdcc000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdcc8ff9700 1 -- 172.21.15.111:0/2037429607 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0009790 con 0x7fdccc101480 2023-10-11T16:30:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc10bc40 con 0x7fdccc101480 2023-10-11T16:30:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdcc8ff9700 1 -- 172.21.15.111:0/2037429607 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcc000cbd0 con 0x7fdccc101480 2023-10-11T16:30:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdcc8ff9700 1 -- 172.21.15.111:0/2037429607 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0018620 con 0x7fdccc101480 2023-10-11T16:30:27.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.572+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdccc10bab0 con 0x7fdccc101480 2023-10-11T16:30:27.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.573+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdccc107880 con 0x7fdccc101480 2023-10-11T16:30:27.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.575+0000 7fdcc8ff9700 1 -- 172.21.15.111:0/2037429607 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fdcc000cd40 con 0x7fdccc101480 2023-10-11T16:30:27.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:27 smithi130 ceph-mon[131859]: pgmap v280: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 720 B/s wr, 0 op/s 2023-10-11T16:30:27.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.577+0000 7fdcc8ff9700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdcb40a1150 0x7fdcb40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:27.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.577+0000 7fdcc8ff9700 1 -- 172.21.15.111:0/2037429607 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdcc0014070 con 0x7fdccc101480 2023-10-11T16:30:27.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.577+0000 7fdccb7fe700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdcb40a1150 0x7fdcb40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:27.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.577+0000 7fdccb7fe700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdcb40a1150 0x7fdcb40a3600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00b2e0 tx=0x7fdcbc005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:27.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.581+0000 7fdcc8ff9700 1 -- 172.21.15.111:0/2037429607 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdcc0051730 con 0x7fdccc101480 2023-10-11T16:30:27.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.786+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdccc02cfe0 con 0x7fdccc101480 2023-10-11T16:30:27.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.788+0000 7fdcc8ff9700 1 -- 172.21.15.111:0/2037429607 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7fdcc0099dd0 con 0x7fdccc101480 2023-10-11T16:30:27.790 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:16.607979+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:27.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdcb40a1150 msgr2=0x7fdcb40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:27.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdcb40a1150 0x7fdcb40a3600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00b2e0 tx=0x7fdcbc005fd0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc101480 msgr2=0x7fdccc1964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc101480 0x7fdccc1964f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000e9a0 tx=0x7fdcc000ed60 comp rx=0 tx=0).stop 2023-10-11T16:30:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 shutdown_connections 2023-10-11T16:30:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdcb40a1150 0x7fdcb40a3600 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdccc101e10 0x7fdccc19d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdccc100290 0x7fdccc195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 --2- 172.21.15.111:0/2037429607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdccc101480 0x7fdccc1964f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:27.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 >> 172.21.15.111:0/2037429607 conn(0x7fdccc0fb870 msgr2=0x7fdccc1049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:27.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 shutdown_connections 2023-10-11T16:30:27.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:27.790+0000 7fdcd1dff700 1 -- 172.21.15.111:0/2037429607 wait complete. 2023-10-11T16:30:27.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:30:27.812 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:30:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 -- 172.21.15.111:0/551134235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a81014c0 msgr2=0x7f12a8101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/551134235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a81014c0 0x7f12a8101910 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1290009990 tx=0x7f1290009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 -- 172.21.15.111:0/551134235 shutdown_connections 2023-10-11T16:30:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/551134235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a8101e50 0x7f12a8108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/551134235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12a81002d0 0x7f12a81006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/551134235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a81014c0 0x7f12a8101910 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 -- 172.21.15.111:0/551134235 >> 172.21.15.111:0/551134235 conn(0x7f12a80fb8b0 msgr2=0x7f12a80fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:28.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 -- 172.21.15.111:0/551134235 shutdown_connections 2023-10-11T16:30:28.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.050+0000 7f12ad9ff700 1 -- 172.21.15.111:0/551134235 wait complete. 2023-10-11T16:30:28.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.051+0000 7f12ad9ff700 1 Processor -- start 2023-10-11T16:30:28.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12ad9ff700 1 -- start start 2023-10-11T16:30:28.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12ad9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a81002d0 0x7f12a80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12ad9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12a81014c0 0x7f12a8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12ad9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a8101e50 0x7f12a8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12ad9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8073bc0 con 0x7f12a8101e50 2023-10-11T16:30:28.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12ad9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8073d30 con 0x7f12a81014c0 2023-10-11T16:30:28.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12ad9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a81a00e0 con 0x7f12a81002d0 2023-10-11T16:30:28.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12a6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a81002d0 0x7f12a80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12a77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a8101e50 0x7f12a8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12a77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a8101e50 0x7f12a8073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51794/0 (socket says 172.21.15.111:51794) 2023-10-11T16:30:28.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f129ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12a81014c0 0x7f12a8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12a77fe700 1 -- 172.21.15.111:0/4100866519 learned_addr learned my addr 172.21.15.111:0/4100866519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:28.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.052+0000 7f12a6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a81002d0 0x7f12a80748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:60386/0 (socket says 172.21.15.111:60386) 2023-10-11T16:30:28.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12a77fe700 1 -- 172.21.15.111:0/4100866519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a81002d0 msgr2=0x7f12a80748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12a77fe700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a81002d0 0x7f12a80748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12a77fe700 1 -- 172.21.15.111:0/4100866519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12a81014c0 msgr2=0x7f12a8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12a77fe700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12a81014c0 0x7f12a8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12a77fe700 1 -- 172.21.15.111:0/4100866519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12900095f0 con 0x7f12a8101e50 2023-10-11T16:30:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12a77fe700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a8101e50 0x7f12a8073420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f12a0007ee0 tx=0x7f12a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:28.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12ac9fd700 1 -- 172.21.15.111:0/4100866519 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0017070 con 0x7f12a8101e50 2023-10-11T16:30:28.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12ac9fd700 1 -- 172.21.15.111:0/4100866519 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12a000f040 con 0x7f12a8101e50 2023-10-11T16:30:28.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12ac9fd700 1 -- 172.21.15.111:0/4100866519 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0005580 con 0x7f12a8101e50 2023-10-11T16:30:28.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a810bc80 con 0x7f12a8101e50 2023-10-11T16:30:28.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.053+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a810baf0 con 0x7f12a8101e50 2023-10-11T16:30:28.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.055+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a81a07d0 con 0x7f12a8101e50 2023-10-11T16:30:28.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.058+0000 7f12ac9fd700 1 -- 172.21.15.111:0/4100866519 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f12a000db20 con 0x7f12a8101e50 2023-10-11T16:30:28.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.060+0000 7f12ac9fd700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12940a11d0 0x7f12940a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.060+0000 7f12ac9fd700 1 -- 172.21.15.111:0/4100866519 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f12a00cc7d0 con 0x7f12a8101e50 2023-10-11T16:30:28.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.061+0000 7f12a6ffd700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12940a11d0 0x7f12940a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.061+0000 7f12a6ffd700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12940a11d0 0x7f12940a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f12a8101320 tx=0x7f1298009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:28.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.062+0000 7f12ac9fd700 1 -- 172.21.15.111:0/4100866519 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f12a0095d30 con 0x7f12a8101e50 2023-10-11T16:30:28.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.260+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12a8107200 con 0x7f12a8101e50 2023-10-11T16:30:28.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.260+0000 7f12ac9fd700 1 -- 172.21.15.111:0/4100866519 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12a0099350 con 0x7f12a8101e50 2023-10-11T16:30:28.263 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:30:28.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12940a11d0 msgr2=0x7f12940a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12940a11d0 0x7f12940a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f12a8101320 tx=0x7f1298009260 comp rx=0 tx=0).stop 2023-10-11T16:30:28.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a8101e50 msgr2=0x7f12a8073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a8101e50 0x7f12a8073420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f12a0007ee0 tx=0x7f12a000d3b0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 shutdown_connections 2023-10-11T16:30:28.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12940a11d0 0x7f12940a3680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12a8101e50 0x7f12a8073420 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a81002d0 0x7f12a80748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 --2- 172.21.15.111:0/4100866519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12a81014c0 0x7f12a8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 >> 172.21.15.111:0/4100866519 conn(0x7f12a80fb8b0 msgr2=0x7f12a80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:28.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 shutdown_connections 2023-10-11T16:30:28.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.263+0000 7f12ad9ff700 1 -- 172.21.15.111:0/4100866519 wait complete. 2023-10-11T16:30:28.282 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:30:28.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:28 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2037429607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:28.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:28 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2037429607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:28.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 -- 172.21.15.111:0/3853079345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1002c0 msgr2=0x7f44bc1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 --2- 172.21.15.111:0/3853079345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1002c0 0x7f44bc1006d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f44b8009990 tx=0x7f44b8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 -- 172.21.15.111:0/3853079345 shutdown_connections 2023-10-11T16:30:28.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 --2- 172.21.15.111:0/3853079345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44bc101e40 0x7f44bc108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 --2- 172.21.15.111:0/3853079345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1002c0 0x7f44bc1006d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 --2- 172.21.15.111:0/3853079345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc1014b0 0x7f44bc101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 -- 172.21.15.111:0/3853079345 >> 172.21.15.111:0/3853079345 conn(0x7f44bc0fb8a0 msgr2=0x7f44bc0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:28.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 -- 172.21.15.111:0/3853079345 shutdown_connections 2023-10-11T16:30:28.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.482+0000 7f44c47ff700 1 -- 172.21.15.111:0/3853079345 wait complete. 2023-10-11T16:30:28.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c47ff700 1 Processor -- start 2023-10-11T16:30:28.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c47ff700 1 -- start start 2023-10-11T16:30:28.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c47ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc1002c0 0x7f44bc198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c47ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1014b0 0x7f44bc1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c47ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44bc101e40 0x7f44bc19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c47ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc19fdf0 con 0x7f44bc1014b0 2023-10-11T16:30:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c47ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc19ff30 con 0x7f44bc101e40 2023-10-11T16:30:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c47ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc1a00c0 con 0x7f44bc1002c0 2023-10-11T16:30:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.483+0000 7f44c1d9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1014b0 0x7f44bc1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44bc101e40 0x7f44bc19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44bc101e40 0x7f44bc19f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:56936/0 (socket says 172.21.15.111:56936) 2023-10-11T16:30:28.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 -- 172.21.15.111:0/936667557 learned_addr learned my addr 172.21.15.111:0/936667557 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:28.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c259b700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc1002c0 0x7f44bc198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 -- 172.21.15.111:0/936667557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc1002c0 msgr2=0x7f44bc198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc1002c0 0x7f44bc198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 -- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1014b0 msgr2=0x7f44bc1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1014b0 0x7f44bc1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 -- 172.21.15.111:0/936667557 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b80095f0 con 0x7f44bc101e40 2023-10-11T16:30:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c1d9a700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1014b0 0x7f44bc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c2d9c700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44bc101e40 0x7f44bc19f820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44b400ef30 tx=0x7f44b400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:28.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44b37fe700 1 -- 172.21.15.111:0/936667557 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b4018070 con 0x7f44bc101e40 2023-10-11T16:30:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44b37fe700 1 -- 172.21.15.111:0/936667557 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44b400f040 con 0x7f44bc101e40 2023-10-11T16:30:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44bc10bc90 con 0x7f44bc101e40 2023-10-11T16:30:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.484+0000 7f44b37fe700 1 -- 172.21.15.111:0/936667557 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b400cd00 con 0x7f44bc101e40 2023-10-11T16:30:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.485+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44bc10bb00 con 0x7f44bc101e40 2023-10-11T16:30:28.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.485+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44bc1929f0 con 0x7f44bc101e40 2023-10-11T16:30:28.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.487+0000 7f44b37fe700 1 -- 172.21.15.111:0/936667557 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f44b4003a00 con 0x7f44bc101e40 2023-10-11T16:30:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.489+0000 7f44b37fe700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44a80a1150 0x7f44a80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.489+0000 7f44b37fe700 1 -- 172.21.15.111:0/936667557 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f44b40cc7f0 con 0x7f44bc101e40 2023-10-11T16:30:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.489+0000 7f44c259b700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44a80a1150 0x7f44a80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.489+0000 7f44c259b700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44a80a1150 0x7f44a80a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f44b800b2e0 tx=0x7f44b8005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:28.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.491+0000 7f44b37fe700 1 -- 172.21.15.111:0/936667557 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f44b4095d80 con 0x7f44bc101e40 2023-10-11T16:30:28.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2037429607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.689+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44bc061700 con 0x7f44bc101e40 2023-10-11T16:30:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.690+0000 7f44b37fe700 1 -- 172.21.15.111:0/936667557 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f44b40993a0 con 0x7f44bc101e40 2023-10-11T16:30:28.692 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:30:28.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.692+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44a80a1150 msgr2=0x7f44a80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.692+0000 7f44c47ff700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44a80a1150 0x7f44a80a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f44b800b2e0 tx=0x7f44b8005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.692+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44bc101e40 msgr2=0x7f44bc19f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.692+0000 7f44c47ff700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44bc101e40 0x7f44bc19f820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44b400ef30 tx=0x7f44b400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.693+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 shutdown_connections 2023-10-11T16:30:28.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.693+0000 7f44c47ff700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f44a80a1150 0x7f44a80a3600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.693+0000 7f44c47ff700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f44bc101e40 0x7f44bc19f820 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.693+0000 7f44c47ff700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44bc1002c0 0x7f44bc198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.693+0000 7f44c47ff700 1 --2- 172.21.15.111:0/936667557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44bc1014b0 0x7f44bc1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.693+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 >> 172.21.15.111:0/936667557 conn(0x7f44bc0fb8a0 msgr2=0x7f44bc106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.693+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 shutdown_connections 2023-10-11T16:30:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.693+0000 7f44c47ff700 1 -- 172.21.15.111:0/936667557 wait complete. 2023-10-11T16:30:28.712 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:30:28.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 -- 172.21.15.111:0/1113982849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 msgr2=0x7f850c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 --2- 172.21.15.111:0/1113982849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 0x7f850c101910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8500009990 tx=0x7f8500009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 -- 172.21.15.111:0/1113982849 shutdown_connections 2023-10-11T16:30:28.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 --2- 172.21.15.111:0/1113982849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f850c101e50 0x7f850c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 --2- 172.21.15.111:0/1113982849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f850c1002d0 0x7f850c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 --2- 172.21.15.111:0/1113982849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 0x7f850c101910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 -- 172.21.15.111:0/1113982849 >> 172.21.15.111:0/1113982849 conn(0x7f850c0fb8b0 msgr2=0x7f850c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:28.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 -- 172.21.15.111:0/1113982849 shutdown_connections 2023-10-11T16:30:28.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.912+0000 7f8511fff700 1 -- 172.21.15.111:0/1113982849 wait complete. 2023-10-11T16:30:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.913+0000 7f8511fff700 1 Processor -- start 2023-10-11T16:30:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.913+0000 7f8511fff700 1 -- start start 2023-10-11T16:30:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.913+0000 7f8511fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f850c1002d0 0x7f850c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.913+0000 7f8511fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 0x7f850c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.913+0000 7f8511fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f850c101e50 0x7f850c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.913+0000 7f8511fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c073bc0 con 0x7f850c101e50 2023-10-11T16:30:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.913+0000 7f8511fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c073d30 con 0x7f850c1014c0 2023-10-11T16:30:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.913+0000 7f8511fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c1a0100 con 0x7f850c1002d0 2023-10-11T16:30:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 0x7f850c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f850c101e50 0x7f850c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 0x7f850c072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:56954/0 (socket says 172.21.15.111:56954) 2023-10-11T16:30:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 -- 172.21.15.111:0/58358740 learned_addr learned my addr 172.21.15.111:0/58358740 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850b7fe700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f850c1002d0 0x7f850c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 -- 172.21.15.111:0/58358740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f850c1002d0 msgr2=0x7f850c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f850c1002d0 0x7f850c0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 -- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f850c101e50 msgr2=0x7f850c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f850c101e50 0x7f850c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 -- 172.21.15.111:0/58358740 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85000095f0 con 0x7f850c1014c0 2023-10-11T16:30:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.914+0000 7f850affd700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 0x7f850c072ee0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8500004760 tx=0x7f8500004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.915+0000 7f8508ff9700 1 -- 172.21.15.111:0/58358740 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850001d070 con 0x7f850c1014c0 2023-10-11T16:30:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.915+0000 7f8508ff9700 1 -- 172.21.15.111:0/58358740 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8500004830 con 0x7f850c1014c0 2023-10-11T16:30:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.915+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c10bca0 con 0x7f850c1014c0 2023-10-11T16:30:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.915+0000 7f8508ff9700 1 -- 172.21.15.111:0/58358740 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500005350 con 0x7f850c1014c0 2023-10-11T16:30:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.915+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c10bb10 con 0x7f850c1014c0 2023-10-11T16:30:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.916+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c1a0700 con 0x7f850c1014c0 2023-10-11T16:30:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.918+0000 7f8508ff9700 1 -- 172.21.15.111:0/58358740 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f85000054d0 con 0x7f850c1014c0 2023-10-11T16:30:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.919+0000 7f8508ff9700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f84f40a11d0 0x7f84f40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.919+0000 7f8508ff9700 1 -- 172.21.15.111:0/58358740 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f85000ce150 con 0x7f850c1014c0 2023-10-11T16:30:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.919+0000 7f850b7fe700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f84f40a11d0 0x7f84f40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:28.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.920+0000 7f850b7fe700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f84f40a11d0 0x7f84f40a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f850c101320 tx=0x7f84fc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:28.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:28.921+0000 7f8508ff9700 1 -- 172.21.15.111:0/58358740 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f85000976e0 con 0x7f850c1014c0 2023-10-11T16:30:29.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.120+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f850c1a0df0 con 0x7f850c1014c0 2023-10-11T16:30:29.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.121+0000 7f8508ff9700 1 -- 172.21.15.111:0/58358740 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f850009ad00 con 0x7f850c1014c0 2023-10-11T16:30:29.123 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:30:29.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f84f40a11d0 msgr2=0x7f84f40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f84f40a11d0 0x7f84f40a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f850c101320 tx=0x7f84fc009260 comp rx=0 tx=0).stop 2023-10-11T16:30:29.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 msgr2=0x7f850c072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 0x7f850c072ee0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8500004760 tx=0x7f8500004790 comp rx=0 tx=0).stop 2023-10-11T16:30:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 shutdown_connections 2023-10-11T16:30:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f84f40a11d0 0x7f84f40a3680 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f850c101e50 0x7f850c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f850c1002d0 0x7f850c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 --2- 172.21.15.111:0/58358740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f850c1014c0 0x7f850c072ee0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 >> 172.21.15.111:0/58358740 conn(0x7f850c0fb8b0 msgr2=0x7f850c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 shutdown_connections 2023-10-11T16:30:29.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.123+0000 7f8511fff700 1 -- 172.21.15.111:0/58358740 wait complete. 2023-10-11T16:30:29.144 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:30:29.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 -- 172.21.15.111:0/2902132295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41002d0 msgr2=0x7fe7f41006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/2902132295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41002d0 0x7fe7f41006e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009990 tx=0x7fe7e4009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 -- 172.21.15.111:0/2902132295 shutdown_connections 2023-10-11T16:30:29.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/2902132295 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f4101e50 0x7fe7f4108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/2902132295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41002d0 0x7fe7f41006e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/2902132295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7f41014c0 0x7fe7f4101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 -- 172.21.15.111:0/2902132295 >> 172.21.15.111:0/2902132295 conn(0x7fe7f40fb8b0 msgr2=0x7fe7f40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:29.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 -- 172.21.15.111:0/2902132295 shutdown_connections 2023-10-11T16:30:29.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.342+0000 7fe7f91ff700 1 -- 172.21.15.111:0/2902132295 wait complete. 2023-10-11T16:30:29.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.343+0000 7fe7f91ff700 1 Processor -- start 2023-10-11T16:30:29.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f91ff700 1 -- start start 2023-10-11T16:30:29.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f91ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7f41002d0 0x7fe7f41960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:29.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f91ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41014c0 0x7fe7f41965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:29.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f91ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f4101e50 0x7fe7f419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:29.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f91ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f419dc20 con 0x7fe7f41014c0 2023-10-11T16:30:29.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f91ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f419dd60 con 0x7fe7f4101e50 2023-10-11T16:30:29.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f91ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f419def0 con 0x7fe7f41002d0 2023-10-11T16:30:29.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41014c0 0x7fe7f41965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:29.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f359e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f4101e50 0x7fe7f419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:29.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41014c0 0x7fe7f41965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51848/0 (socket says 172.21.15.111:51848) 2023-10-11T16:30:29.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f259c700 1 -- 172.21.15.111:0/654715317 learned_addr learned my addr 172.21.15.111:0/654715317 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:29.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.344+0000 7fe7f2d9d700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7f41002d0 0x7fe7f41960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:29.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7f259c700 1 -- 172.21.15.111:0/654715317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7f41002d0 msgr2=0x7fe7f41960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7f259c700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7f41002d0 0x7fe7f41960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7f259c700 1 -- 172.21.15.111:0/654715317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f4101e50 msgr2=0x7fe7f419d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7f259c700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f4101e50 0x7fe7f419d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7f259c700 1 -- 172.21.15.111:0/654715317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40095f0 con 0x7fe7f41014c0 2023-10-11T16:30:29.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7f259c700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41014c0 0x7fe7f41965e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800d950 tx=0x7fe7e800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:29.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7dbfff700 1 -- 172.21.15.111:0/654715317 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800f890 con 0x7fe7f41014c0 2023-10-11T16:30:29.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7dbfff700 1 -- 172.21.15.111:0/654715317 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7e8010460 con 0x7fe7f41014c0 2023-10-11T16:30:29.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f410bca0 con 0x7fe7f41014c0 2023-10-11T16:30:29.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7dbfff700 1 -- 172.21.15.111:0/654715317 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800e610 con 0x7fe7f41014c0 2023-10-11T16:30:29.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.345+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f410bb10 con 0x7fe7f41014c0 2023-10-11T16:30:29.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.346+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f4107860 con 0x7fe7f41014c0 2023-10-11T16:30:29.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.348+0000 7fe7dbfff700 1 -- 172.21.15.111:0/654715317 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fe7e80108d0 con 0x7fe7f41014c0 2023-10-11T16:30:29.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.349+0000 7fe7dbfff700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe7dc0a1150 0x7fe7dc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:29.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.350+0000 7fe7dbfff700 1 -- 172.21.15.111:0/654715317 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe7e80cc3a0 con 0x7fe7f41014c0 2023-10-11T16:30:29.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.350+0000 7fe7f2d9d700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe7dc0a1150 0x7fe7dc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:29.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.350+0000 7fe7f2d9d700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe7dc0a1150 0x7fe7dc0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400b5c0 tx=0x7fe7e4005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:29.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.353+0000 7fe7dbfff700 1 -- 172.21.15.111:0/654715317 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe7e80958a0 con 0x7fe7f41014c0 2023-10-11T16:30:29.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:29 smithi111 ceph-mon[98527]: pgmap v281: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 984 B/s wr, 0 op/s 2023-10-11T16:30:29.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:29 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4100866519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:29.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:29 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/936667557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:30:29.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:29 smithi150 ceph-mon[132217]: pgmap v281: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 984 B/s wr, 0 op/s 2023-10-11T16:30:29.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:29 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4100866519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:29.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:29 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/936667557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:30:29.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.549+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe7f4061c00 con 0x7fe7f41014c0 2023-10-11T16:30:29.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.549+0000 7fe7dbfff700 1 -- 172.21.15.111:0/654715317 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe7e8095030 con 0x7fe7f41014c0 2023-10-11T16:30:29.552 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:30:29.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe7dc0a1150 msgr2=0x7fe7dc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe7dc0a1150 0x7fe7dc0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400b5c0 tx=0x7fe7e4005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41014c0 msgr2=0x7fe7f41965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41014c0 0x7fe7f41965e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800d950 tx=0x7fe7e800dc60 comp rx=0 tx=0).stop 2023-10-11T16:30:29.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 shutdown_connections 2023-10-11T16:30:29.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe7dc0a1150 0x7fe7dc0a3600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f4101e50 0x7fe7f419d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7f41002d0 0x7fe7f41960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 --2- 172.21.15.111:0/654715317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7f41014c0 0x7fe7f41965e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 >> 172.21.15.111:0/654715317 conn(0x7fe7f40fb8b0 msgr2=0x7fe7f41049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:29.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 shutdown_connections 2023-10-11T16:30:29.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.552+0000 7fe7f91ff700 1 -- 172.21.15.111:0/654715317 wait complete. 2023-10-11T16:30:29.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-11T16:30:29.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:29 smithi130 ceph-mon[131859]: pgmap v281: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 984 B/s wr, 0 op/s 2023-10-11T16:30:29.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:29 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4100866519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:29.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:29 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/936667557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:30:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 -- 172.21.15.111:0/3585125209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce0074b30 msgr2=0x7fbce0072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 --2- 172.21.15.111:0/3585125209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce0074b30 0x7fbce0072f80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009990 tx=0x7fbcd0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 -- 172.21.15.111:0/3585125209 shutdown_connections 2023-10-11T16:30:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 --2- 172.21.15.111:0/3585125209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbce01080f0 0x7fbce010a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 --2- 172.21.15.111:0/3585125209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce0074b30 0x7fbce0072f80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 --2- 172.21.15.111:0/3585125209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbce0073550 0x7fbce00739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 -- 172.21.15.111:0/3585125209 >> 172.21.15.111:0/3585125209 conn(0x7fbce00fb810 msgr2=0x7fbce00fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:29.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 -- 172.21.15.111:0/3585125209 shutdown_connections 2023-10-11T16:30:29.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.768+0000 7fbce57ff700 1 -- 172.21.15.111:0/3585125209 wait complete. 2023-10-11T16:30:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.769+0000 7fbce57ff700 1 Processor -- start 2023-10-11T16:30:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.769+0000 7fbce57ff700 1 -- start start 2023-10-11T16:30:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbce57ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce0073550 0x7fbce01981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbce57ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbce0074b30 0x7fbce01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbce57ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbce01080f0 0x7fbce019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbce57ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce019fd30 con 0x7fbce01080f0 2023-10-11T16:30:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbce57ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce019fe70 con 0x7fbce0074b30 2023-10-11T16:30:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbce57ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce01a0000 con 0x7fbce0073550 2023-10-11T16:30:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbce01080f0 0x7fbce019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcde7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbce0074b30 0x7fbce01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcde7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbce0074b30 0x7fbce01986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:56994/0 (socket says 172.21.15.111:56994) 2023-10-11T16:30:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbce01080f0 0x7fbce019f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51866/0 (socket says 172.21.15.111:51866) 2023-10-11T16:30:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcde7fc700 1 -- 172.21.15.111:0/243777829 learned_addr learned my addr 172.21.15.111:0/243777829 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:29.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcdeffd700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce0073550 0x7fbce01981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:29.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcdf7fe700 1 -- 172.21.15.111:0/243777829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce0073550 msgr2=0x7fbce01981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcdf7fe700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce0073550 0x7fbce01981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcdf7fe700 1 -- 172.21.15.111:0/243777829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbce0074b30 msgr2=0x7fbce01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:29.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcdf7fe700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbce0074b30 0x7fbce01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:29.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.770+0000 7fbcdf7fe700 1 -- 172.21.15.111:0/243777829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd8009c40 con 0x7fbce01080f0 2023-10-11T16:30:29.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.771+0000 7fbcdf7fe700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbce01080f0 0x7fbce019f760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbcd800efb0 tx=0x7fbcd800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:29.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.771+0000 7fbcc7fff700 1 -- 172.21.15.111:0/243777829 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8003a00 con 0x7fbce01080f0 2023-10-11T16:30:29.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.771+0000 7fbcc7fff700 1 -- 172.21.15.111:0/243777829 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcd800f040 con 0x7fbce01080f0 2023-10-11T16:30:29.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.771+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd00095f0 con 0x7fbce01080f0 2023-10-11T16:30:29.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.771+0000 7fbcc7fff700 1 -- 172.21.15.111:0/243777829 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8004550 con 0x7fbce01080f0 2023-10-11T16:30:29.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.771+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce010dda0 con 0x7fbce01080f0 2023-10-11T16:30:29.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.772+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce01067a0 con 0x7fbce01080f0 2023-10-11T16:30:29.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.774+0000 7fbcc7fff700 1 -- 172.21.15.111:0/243777829 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fbcd8003bf0 con 0x7fbce01080f0 2023-10-11T16:30:29.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.775+0000 7fbcc7fff700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc80a1150 0x7fbcc80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:29.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.775+0000 7fbcc7fff700 1 -- 172.21.15.111:0/243777829 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbcd80cc780 con 0x7fbce01080f0 2023-10-11T16:30:29.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.775+0000 7fbcdeffd700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc80a1150 0x7fbcc80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:29.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.776+0000 7fbcdeffd700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc80a1150 0x7fbcc80a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000b2e0 tx=0x7fbcd0005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:29.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.779+0000 7fbcc7fff700 1 -- 172.21.15.111:0/243777829 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbcd8095d10 con 0x7fbce01080f0 2023-10-11T16:30:29.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.995+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbce00679f0 con 0x7fbce01080f0 2023-10-11T16:30:29.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.995+0000 7fbcc7fff700 1 -- 172.21.15.111:0/243777829 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+600 (secure 0 0 0) 0x7fbcd8099330 con 0x7fbce01080f0 2023-10-11T16:30:29.997 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/4086392563 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:29.997 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/4086392563 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:29.998 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3251673175 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:29.998 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3319300422 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:29.998 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3405550474 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:29.998 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/4129094163 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:29.998 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1272467376 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:29.999 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/699479694 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:29.999 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/699479694 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:29.999 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3715184603 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:30.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.997+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc80a1150 msgr2=0x7fbcc80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.997+0000 7fbce57ff700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc80a1150 0x7fbcc80a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000b2e0 tx=0x7fbcd0005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.997+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbce01080f0 msgr2=0x7fbce019f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.997+0000 7fbce57ff700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbce01080f0 0x7fbce019f760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbcd800efb0 tx=0x7fbcd800c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.998+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 shutdown_connections 2023-10-11T16:30:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.998+0000 7fbce57ff700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbcc80a1150 0x7fbcc80a3600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.998+0000 7fbce57ff700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbce01080f0 0x7fbce019f760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.998+0000 7fbce57ff700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbce0073550 0x7fbce01981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.998+0000 7fbce57ff700 1 --2- 172.21.15.111:0/243777829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbce0074b30 0x7fbce01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.998+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 >> 172.21.15.111:0/243777829 conn(0x7fbce00fb810 msgr2=0x7fbce00fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.998+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 shutdown_connections 2023-10-11T16:30:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:29.998+0000 7fbce57ff700 1 -- 172.21.15.111:0/243777829 wait complete. 2023-10-11T16:30:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:listed 10 entries 2023-10-11T16:30:30.018 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:30:30.018 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-11T16:30:30.063 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:30.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.298+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/521775117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f041002b0 msgr2=0x7f4f041006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.298+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/521775117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f041002b0 0x7f4f041006c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f00009990 tx=0x7f4f00009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.298+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/521775117 shutdown_connections 2023-10-11T16:30:30.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.298+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/521775117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f04101e30 0x7f4f041086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.298+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/521775117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f041002b0 0x7f4f041006c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.298+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/521775117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041014a0 0x7f4f041018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.298+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/521775117 >> 172.21.15.111:0/521775117 conn(0x7f4f040fb890 msgr2=0x7f4f040fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:30.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.299+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/521775117 shutdown_connections 2023-10-11T16:30:30.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.299+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/521775117 wait complete. 2023-10-11T16:30:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.299+0000 7f4f0b5ff700 1 Processor -- start 2023-10-11T16:30:30.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0b5ff700 1 -- start start 2023-10-11T16:30:30.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0b5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041002b0 0x7f4f041981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:30.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0b5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f041014a0 0x7f4f04198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:30.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0b5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f04101e30 0x7f4f0419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:30.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0b5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0419fd60 con 0x7f4f041014a0 2023-10-11T16:30:30.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0b5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0419fea0 con 0x7f4f04101e30 2023-10-11T16:30:30.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0b5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f041a0030 con 0x7f4f041002b0 2023-10-11T16:30:30.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f08b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f041014a0 0x7f4f04198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:30.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f08b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f041014a0 0x7f4f04198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51886/0 (socket says 172.21.15.111:51886) 2023-10-11T16:30:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f08b9a700 1 -- 172.21.15.111:0/3995402403 learned_addr learned my addr 172.21.15.111:0/3995402403 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f09b9c700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f04101e30 0x7f4f0419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0939b700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041002b0 0x7f4f041981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f08b9a700 1 -- 172.21.15.111:0/3995402403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041002b0 msgr2=0x7f4f041981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f08b9a700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041002b0 0x7f4f041981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f08b9a700 1 -- 172.21.15.111:0/3995402403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f04101e30 msgr2=0x7f4f0419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f08b9a700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f04101e30 0x7f4f0419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f08b9a700 1 -- 172.21.15.111:0/3995402403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000095f0 con 0x7f4f041014a0 2023-10-11T16:30:30.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.300+0000 7f4f0939b700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041002b0 0x7f4f041981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:30.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.301+0000 7f4f08b9a700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f041014a0 0x7f4f04198720 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400e9a0 tx=0x7f4ef400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:30.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.301+0000 7f4efa7fc700 1 -- 172.21.15.111:0/3995402403 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4009790 con 0x7f4f041014a0 2023-10-11T16:30:30.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.301+0000 7f4efa7fc700 1 -- 172.21.15.111:0/3995402403 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ef400cbd0 con 0x7f4f041014a0 2023-10-11T16:30:30.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.301+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0410bc80 con 0x7f4f041014a0 2023-10-11T16:30:30.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.301+0000 7f4efa7fc700 1 -- 172.21.15.111:0/3995402403 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4018680 con 0x7f4f041014a0 2023-10-11T16:30:30.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.301+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0410baf0 con 0x7f4f041014a0 2023-10-11T16:30:30.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.302+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f04192910 con 0x7f4f041014a0 2023-10-11T16:30:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.304+0000 7f4efa7fc700 1 -- 172.21.15.111:0/3995402403 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f4ef400cd40 con 0x7f4f041014a0 2023-10-11T16:30:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.305+0000 7f4efa7fc700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4ef00a11d0 0x7f4ef00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:30.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.306+0000 7f4efa7fc700 1 -- 172.21.15.111:0/3995402403 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4ef4014070 con 0x7f4f041014a0 2023-10-11T16:30:30.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.306+0000 7f4f0939b700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4ef00a11d0 0x7f4ef00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:30.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.306+0000 7f4f0939b700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4ef00a11d0 0x7f4ef00a3680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f00000c00 tx=0x7f4f00005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:30.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.309+0000 7f4efa7fc700 1 -- 172.21.15.111:0/3995402403 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4ef4051730 con 0x7f4f041014a0 2023-10-11T16:30:30.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:30 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/58358740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:30:30.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:30 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/654715317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:30:30.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:30 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/243777829' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:30:30.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:30 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/58358740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:30:30.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:30 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/654715317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:30:30.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:30 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/243777829' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:30:30.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.511+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f0402cfe0 con 0x7f4f041014a0 2023-10-11T16:30:30.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.513+0000 7f4efa7fc700 1 -- 172.21.15.111:0/3995402403 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f4ef40962b0 con 0x7f4f041014a0 2023-10-11T16:30:30.515 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:30.515 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:16.607979+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:30.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4ef00a11d0 msgr2=0x7f4ef00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4ef00a11d0 0x7f4ef00a3680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f00000c00 tx=0x7f4f00005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f041014a0 msgr2=0x7f4f04198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f041014a0 0x7f4f04198720 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400e9a0 tx=0x7f4ef400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 shutdown_connections 2023-10-11T16:30:30.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4ef00a11d0 0x7f4ef00a3680 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f04101e30 0x7f4f0419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041002b0 0x7f4f041981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 --2- 172.21.15.111:0/3995402403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f041014a0 0x7f4f04198720 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 >> 172.21.15.111:0/3995402403 conn(0x7f4f040fb890 msgr2=0x7f4f04106db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:30.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 shutdown_connections 2023-10-11T16:30:30.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.516+0000 7f4f0b5ff700 1 -- 172.21.15.111:0/3995402403 wait complete. 2023-10-11T16:30:30.537 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:30:30.537 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:30:30.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:30 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/58358740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:30:30.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:30 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/654715317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:30:30.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:30 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/243777829' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:30:30.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 -- 172.21.15.111:0/1762358804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c0734f0 msgr2=0x7ffa3c073960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 --2- 172.21.15.111:0/1762358804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c0734f0 0x7ffa3c073960 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009990 tx=0x7ffa30009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 -- 172.21.15.111:0/1762358804 shutdown_connections 2023-10-11T16:30:30.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 --2- 172.21.15.111:0/1762358804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa3c108170 0x7ffa3c10a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 --2- 172.21.15.111:0/1762358804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa3c074ad0 0x7ffa3c072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 --2- 172.21.15.111:0/1762358804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c0734f0 0x7ffa3c073960 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 -- 172.21.15.111:0/1762358804 >> 172.21.15.111:0/1762358804 conn(0x7ffa3c0fb8e0 msgr2=0x7ffa3c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:30.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 -- 172.21.15.111:0/1762358804 shutdown_connections 2023-10-11T16:30:30.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.737+0000 7ffa417ff700 1 -- 172.21.15.111:0/1762358804 wait complete. 2023-10-11T16:30:30.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.739+0000 7ffa417ff700 1 Processor -- start 2023-10-11T16:30:30.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.739+0000 7ffa417ff700 1 -- start start 2023-10-11T16:30:30.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.739+0000 7ffa417ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa3c0734f0 0x7ffa3c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:30.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa417ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c074ad0 0x7ffa3c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:30.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa417ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa3c108170 0x7ffa3c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa417ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c19fda0 con 0x7ffa3c074ad0 2023-10-11T16:30:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa417ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c19fee0 con 0x7ffa3c108170 2023-10-11T16:30:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa417ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c1a0070 con 0x7ffa3c0734f0 2023-10-11T16:30:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa3c0734f0 0x7ffa3c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:30.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa3c0734f0 0x7ffa3c198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:60514/0 (socket says 172.21.15.111:60514) 2023-10-11T16:30:30.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c074ad0 0x7ffa3c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:30.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3affd700 1 -- 172.21.15.111:0/2887476546 learned_addr learned my addr 172.21.15.111:0/2887476546 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:30.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3b7fe700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa3c108170 0x7ffa3c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:30.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3a7fc700 1 -- 172.21.15.111:0/2887476546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa3c0734f0 msgr2=0x7ffa3c198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3a7fc700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa3c0734f0 0x7ffa3c198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3a7fc700 1 -- 172.21.15.111:0/2887476546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa3c108170 msgr2=0x7ffa3c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3a7fc700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa3c108170 0x7ffa3c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3a7fc700 1 -- 172.21.15.111:0/2887476546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa300095f0 con 0x7ffa3c074ad0 2023-10-11T16:30:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3affd700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa3c0734f0 0x7ffa3c198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:30.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.740+0000 7ffa3a7fc700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c074ad0 0x7ffa3c198760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000b680 tx=0x7ffa30009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:30.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.741+0000 7ffa23fff700 1 -- 172.21.15.111:0/2887476546 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3001d070 con 0x7ffa3c074ad0 2023-10-11T16:30:30.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.741+0000 7ffa23fff700 1 -- 172.21.15.111:0/2887476546 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa30021410 con 0x7ffa3c074ad0 2023-10-11T16:30:30.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.741+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3c10de70 con 0x7ffa3c074ad0 2023-10-11T16:30:30.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.741+0000 7ffa23fff700 1 -- 172.21.15.111:0/2887476546 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30017650 con 0x7ffa3c074ad0 2023-10-11T16:30:30.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.741+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3c10dce0 con 0x7ffa3c074ad0 2023-10-11T16:30:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.742+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3c1068b0 con 0x7ffa3c074ad0 2023-10-11T16:30:30.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.744+0000 7ffa23fff700 1 -- 172.21.15.111:0/2887476546 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ffa300217a0 con 0x7ffa3c074ad0 2023-10-11T16:30:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.745+0000 7ffa23fff700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ffa240a11d0 0x7ffa240a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.745+0000 7ffa23fff700 1 -- 172.21.15.111:0/2887476546 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffa30021e40 con 0x7ffa3c074ad0 2023-10-11T16:30:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.745+0000 7ffa3affd700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ffa240a11d0 0x7ffa240a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:30.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.746+0000 7ffa3affd700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ffa240a11d0 0x7ffa240a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c011550 tx=0x7ffa2c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:30.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.748+0000 7ffa23fff700 1 -- 172.21.15.111:0/2887476546 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ffa30097510 con 0x7ffa3c074ad0 2023-10-11T16:30:30.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.950+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffa3c061700 con 0x7ffa3c074ad0 2023-10-11T16:30:30.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.951+0000 7ffa23fff700 1 -- 172.21.15.111:0/2887476546 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffa3009ab30 con 0x7ffa3c074ad0 2023-10-11T16:30:30.953 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:30:30.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ffa240a11d0 msgr2=0x7ffa240a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ffa240a11d0 0x7ffa240a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c011550 tx=0x7ffa2c009260 comp rx=0 tx=0).stop 2023-10-11T16:30:30.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c074ad0 msgr2=0x7ffa3c198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:30.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c074ad0 0x7ffa3c198760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000b680 tx=0x7ffa30009580 comp rx=0 tx=0).stop 2023-10-11T16:30:30.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 shutdown_connections 2023-10-11T16:30:30.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ffa240a11d0 0x7ffa240a3680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffa3c108170 0x7ffa3c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa3c0734f0 0x7ffa3c198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 --2- 172.21.15.111:0/2887476546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa3c074ad0 0x7ffa3c198760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:30.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 >> 172.21.15.111:0/2887476546 conn(0x7ffa3c0fb8e0 msgr2=0x7ffa3c0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:30.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 shutdown_connections 2023-10-11T16:30:30.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:30.953+0000 7ffa417ff700 1 -- 172.21.15.111:0/2887476546 wait complete. 2023-10-11T16:30:30.973 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:30:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 -- 172.21.15.111:0/318628580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 msgr2=0x7f744c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 --2- 172.21.15.111:0/318628580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 0x7f744c1006d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f743c009990 tx=0x7f743c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 -- 172.21.15.111:0/318628580 shutdown_connections 2023-10-11T16:30:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 --2- 172.21.15.111:0/318628580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f744c101e40 0x7f744c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 --2- 172.21.15.111:0/318628580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 0x7f744c1006d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 --2- 172.21.15.111:0/318628580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f744c1014b0 0x7f744c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 -- 172.21.15.111:0/318628580 >> 172.21.15.111:0/318628580 conn(0x7f744c0fb8a0 msgr2=0x7f744c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:31.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 -- 172.21.15.111:0/318628580 shutdown_connections 2023-10-11T16:30:31.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.174+0000 7f74519ff700 1 -- 172.21.15.111:0/318628580 wait complete. 2023-10-11T16:30:31.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.175+0000 7f74519ff700 1 Processor -- start 2023-10-11T16:30:31.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.175+0000 7f74519ff700 1 -- start start 2023-10-11T16:30:31.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.175+0000 7f74519ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 0x7f744c196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.175+0000 7f74519ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f744c1014b0 0x7f744c196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.175+0000 7f74519ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f744c101e40 0x7f744c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.175+0000 7f74519ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c19dbd0 con 0x7f744c1002c0 2023-10-11T16:30:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.175+0000 7f74519ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c19dd10 con 0x7f744c101e40 2023-10-11T16:30:31.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.175+0000 7f74519ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c19dea0 con 0x7f744c1014b0 2023-10-11T16:30:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 0x7f744c196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 0x7f744c196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51936/0 (socket says 172.21.15.111:51936) 2023-10-11T16:30:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 -- 172.21.15.111:0/109517420 learned_addr learned my addr 172.21.15.111:0/109517420 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744b7fe700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f744c101e40 0x7f744c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:31.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f7443fff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f744c1014b0 0x7f744c196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:31.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 -- 172.21.15.111:0/109517420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f744c1014b0 msgr2=0x7f744c196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f744c1014b0 0x7f744c196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 -- 172.21.15.111:0/109517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f744c101e40 msgr2=0x7f744c19d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f744c101e40 0x7f744c19d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 -- 172.21.15.111:0/109517420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c0095f0 con 0x7f744c1002c0 2023-10-11T16:30:31.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f7443fff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f744c1014b0 0x7f744c196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:31.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f744affd700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 0x7f744c196050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f743c0059d0 tx=0x7f743c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:31.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.176+0000 7f74509fd700 1 -- 172.21.15.111:0/109517420 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c01d070 con 0x7f744c1002c0 2023-10-11T16:30:31.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.177+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744c10bc20 con 0x7f744c1002c0 2023-10-11T16:30:31.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.177+0000 7f74509fd700 1 -- 172.21.15.111:0/109517420 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f743c004020 con 0x7f744c1002c0 2023-10-11T16:30:31.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.177+0000 7f74509fd700 1 -- 172.21.15.111:0/109517420 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c022d00 con 0x7f744c1002c0 2023-10-11T16:30:31.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.177+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744c10ba90 con 0x7f744c1002c0 2023-10-11T16:30:31.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.178+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744c067b00 con 0x7f744c1002c0 2023-10-11T16:30:31.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.179+0000 7f74509fd700 1 -- 172.21.15.111:0/109517420 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f743c022470 con 0x7f744c1002c0 2023-10-11T16:30:31.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.181+0000 7f74509fd700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74380a1150 0x7f74380a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:31.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.182+0000 7f74509fd700 1 -- 172.21.15.111:0/109517420 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f743c0ce6b0 con 0x7f744c1002c0 2023-10-11T16:30:31.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.182+0000 7f7443fff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74380a1150 0x7f74380a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:31.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.182+0000 7f7443fff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74380a1150 0x7f74380a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7434005950 tx=0x7f74340058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:31.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.183+0000 7f74509fd700 1 -- 172.21.15.111:0/109517420 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f743c097c40 con 0x7f744c1002c0 2023-10-11T16:30:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.379+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f744c107210 con 0x7f744c1002c0 2023-10-11T16:30:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.380+0000 7f74509fd700 1 -- 172.21.15.111:0/109517420 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f743c022e60 con 0x7f744c1002c0 2023-10-11T16:30:31.382 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:30:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74380a1150 msgr2=0x7f74380a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74380a1150 0x7f74380a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7434005950 tx=0x7f74340058e0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 msgr2=0x7f744c196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 0x7f744c196050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f743c0059d0 tx=0x7f743c005740 comp rx=0 tx=0).stop 2023-10-11T16:30:31.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 shutdown_connections 2023-10-11T16:30:31.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f74380a1150 0x7f74380a3600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f744c101e40 0x7f744c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f744c1002c0 0x7f744c196050 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 --2- 172.21.15.111:0/109517420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f744c1014b0 0x7f744c196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 >> 172.21.15.111:0/109517420 conn(0x7f744c0fb8a0 msgr2=0x7f744c0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:31.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 shutdown_connections 2023-10-11T16:30:31.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.382+0000 7f74519ff700 1 -- 172.21.15.111:0/109517420 wait complete. 2023-10-11T16:30:31.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:31 smithi111 ceph-mon[98527]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:31.390 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:31 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3995402403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:31.390 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:31 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2887476546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:31.403 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:30:31.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:31 smithi150 ceph-mon[132217]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:31.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:31 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3995402403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:31.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:31 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2887476546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:31.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:31 smithi130 ceph-mon[131859]: pgmap v282: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:31.612 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:31 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3995402403' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:31.612 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:31 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2887476546' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:30:31.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 -- 172.21.15.111:0/1411270255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac101e50 msgr2=0x7fa5ac108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/1411270255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac101e50 0x7fa5ac108710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400b0a0 tx=0x7fa5a400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 -- 172.21.15.111:0/1411270255 shutdown_connections 2023-10-11T16:30:31.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/1411270255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac101e50 0x7fa5ac108710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/1411270255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5ac1002d0 0x7fa5ac1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/1411270255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5ac1014c0 0x7fa5ac101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 -- 172.21.15.111:0/1411270255 >> 172.21.15.111:0/1411270255 conn(0x7fa5ac0fb8b0 msgr2=0x7fa5ac0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:31.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 -- 172.21.15.111:0/1411270255 shutdown_connections 2023-10-11T16:30:31.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.603+0000 7fa5b35ff700 1 -- 172.21.15.111:0/1411270255 wait complete. 2023-10-11T16:30:31.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.604+0000 7fa5b35ff700 1 Processor -- start 2023-10-11T16:30:31.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.604+0000 7fa5b35ff700 1 -- start start 2023-10-11T16:30:31.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b35ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac1002d0 0x7fa5ac196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:31.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b35ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5ac1014c0 0x7fa5ac1966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:31.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b35ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5ac101e50 0x7fa5ac19d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:31.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b35ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac19dce0 con 0x7fa5ac1002d0 2023-10-11T16:30:31.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b35ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac19de20 con 0x7fa5ac1014c0 2023-10-11T16:30:31.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b35ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac19dfb0 con 0x7fa5ac101e50 2023-10-11T16:30:31.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac1002d0 0x7fa5ac196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:31.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b0b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5ac1014c0 0x7fa5ac1966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:31.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b0b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5ac1014c0 0x7fa5ac1966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:57066/0 (socket says 172.21.15.111:57066) 2023-10-11T16:30:31.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b0b9a700 1 -- 172.21.15.111:0/3149769538 learned_addr learned my addr 172.21.15.111:0/3149769538 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:31.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b1b9c700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5ac101e50 0x7fa5ac19d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:31.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b0b9a700 1 -- 172.21.15.111:0/3149769538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5ac101e50 msgr2=0x7fa5ac19d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b0b9a700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5ac101e50 0x7fa5ac19d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b0b9a700 1 -- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac1002d0 msgr2=0x7fa5ac196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b0b9a700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac1002d0 0x7fa5ac196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b0b9a700 1 -- 172.21.15.111:0/3149769538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a4009d00 con 0x7fa5ac1014c0 2023-10-11T16:30:31.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.605+0000 7fa5b139b700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac1002d0 0x7fa5ac196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:31.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.606+0000 7fa5b0b9a700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5ac1014c0 0x7fa5ac1966a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00d950 tx=0x7fa59c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:31.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.606+0000 7fa5a27fc700 1 -- 172.21.15.111:0/3149769538 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c00f890 con 0x7fa5ac1014c0 2023-10-11T16:30:31.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.606+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac10bca0 con 0x7fa5ac1014c0 2023-10-11T16:30:31.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.606+0000 7fa5a27fc700 1 -- 172.21.15.111:0/3149769538 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa59c010460 con 0x7fa5ac1014c0 2023-10-11T16:30:31.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.606+0000 7fa5a27fc700 1 -- 172.21.15.111:0/3149769538 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c00e610 con 0x7fa5ac1014c0 2023-10-11T16:30:31.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.606+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ac10bb10 con 0x7fa5ac1014c0 2023-10-11T16:30:31.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.607+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ac1077c0 con 0x7fa5ac1014c0 2023-10-11T16:30:31.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.609+0000 7fa5a27fc700 1 -- 172.21.15.111:0/3149769538 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fa59c0108d0 con 0x7fa5ac1014c0 2023-10-11T16:30:31.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.610+0000 7fa5a27fc700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5980a1150 0x7fa5980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:31.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.611+0000 7fa5a27fc700 1 -- 172.21.15.111:0/3149769538 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa59c0cc920 con 0x7fa5ac1014c0 2023-10-11T16:30:31.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.611+0000 7fa5b139b700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5980a1150 0x7fa5980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:31.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.611+0000 7fa5b139b700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5980a1150 0x7fa5980a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac101320 tx=0x7fa5a8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:31.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.613+0000 7fa5a27fc700 1 -- 172.21.15.111:0/3149769538 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa59c095e20 con 0x7fa5ac1014c0 2023-10-11T16:30:31.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.812+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa5ac061700 con 0x7fa5ac1014c0 2023-10-11T16:30:31.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.813+0000 7fa5a27fc700 1 -- 172.21.15.111:0/3149769538 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa59c020020 con 0x7fa5ac1014c0 2023-10-11T16:30:31.815 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:30:31.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5980a1150 msgr2=0x7fa5980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5980a1150 0x7fa5980a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac101320 tx=0x7fa5a8009260 comp rx=0 tx=0).stop 2023-10-11T16:30:31.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5ac1014c0 msgr2=0x7fa5ac1966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5ac1014c0 0x7fa5ac1966a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00d950 tx=0x7fa59c00dc60 comp rx=0 tx=0).stop 2023-10-11T16:30:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 shutdown_connections 2023-10-11T16:30:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa5980a1150 0x7fa5980a3600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5ac101e50 0x7fa5ac19d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5ac1002d0 0x7fa5ac196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 --2- 172.21.15.111:0/3149769538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5ac1014c0 0x7fa5ac1966a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 >> 172.21.15.111:0/3149769538 conn(0x7fa5ac0fb8b0 msgr2=0x7fa5ac104900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.815+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 shutdown_connections 2023-10-11T16:30:31.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:31.816+0000 7fa5b35ff700 1 -- 172.21.15.111:0/3149769538 wait complete. 2023-10-11T16:30:31.836 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:30:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 -- 172.21.15.111:0/3380612391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 msgr2=0x7f819c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 --2- 172.21.15.111:0/3380612391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 0x7f819c101900 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8190009990 tx=0x7f8190009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 -- 172.21.15.111:0/3380612391 shutdown_connections 2023-10-11T16:30:32.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 --2- 172.21.15.111:0/3380612391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f819c101e40 0x7f819c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 --2- 172.21.15.111:0/3380612391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f819c1002c0 0x7f819c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 --2- 172.21.15.111:0/3380612391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 0x7f819c101900 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 -- 172.21.15.111:0/3380612391 >> 172.21.15.111:0/3380612391 conn(0x7f819c0fb8a0 msgr2=0x7f819c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:32.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 -- 172.21.15.111:0/3380612391 shutdown_connections 2023-10-11T16:30:32.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.034+0000 7f81a11ff700 1 -- 172.21.15.111:0/3380612391 wait complete. 2023-10-11T16:30:32.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.035+0000 7f81a11ff700 1 Processor -- start 2023-10-11T16:30:32.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.035+0000 7f81a11ff700 1 -- start start 2023-10-11T16:30:32.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f81a11ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f819c1002c0 0x7f819c198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:32.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f81a11ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 0x7f819c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:32.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f81a11ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f819c101e40 0x7f819c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:32.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f81a11ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c19fdd0 con 0x7f819c1014b0 2023-10-11T16:30:32.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f81a11ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c19ff10 con 0x7f819c101e40 2023-10-11T16:30:32.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f81a11ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1a00a0 con 0x7f819c1002c0 2023-10-11T16:30:32.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 0x7f819c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:32.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 0x7f819c198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51960/0 (socket says 172.21.15.111:51960) 2023-10-11T16:30:32.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819a59c700 1 -- 172.21.15.111:0/1398571669 learned_addr learned my addr 172.21.15.111:0/1398571669 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:32.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819ad9d700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f819c1002c0 0x7f819c198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:32.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819b59e700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f819c101e40 0x7f819c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:32.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819a59c700 1 -- 172.21.15.111:0/1398571669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f819c1002c0 msgr2=0x7f819c198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819a59c700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f819c1002c0 0x7f819c198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819a59c700 1 -- 172.21.15.111:0/1398571669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f819c101e40 msgr2=0x7f819c19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819a59c700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f819c101e40 0x7f819c19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.036+0000 7f819a59c700 1 -- 172.21.15.111:0/1398571669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8194009c40 con 0x7f819c1014b0 2023-10-11T16:30:32.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.037+0000 7f819a59c700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 0x7f819c198790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8190004760 tx=0x7f8190004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:32.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.037+0000 7f8183fff700 1 -- 172.21.15.111:0/1398571669 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819001d070 con 0x7f819c1014b0 2023-10-11T16:30:32.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.037+0000 7f8183fff700 1 -- 172.21.15.111:0/1398571669 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8190004830 con 0x7f819c1014b0 2023-10-11T16:30:32.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.037+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81900095f0 con 0x7f819c1014b0 2023-10-11T16:30:32.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.037+0000 7f8183fff700 1 -- 172.21.15.111:0/1398571669 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190005420 con 0x7f819c1014b0 2023-10-11T16:30:32.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.037+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c10bc70 con 0x7f819c1014b0 2023-10-11T16:30:32.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.038+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c1929d0 con 0x7f819c1014b0 2023-10-11T16:30:32.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.040+0000 7f8183fff700 1 -- 172.21.15.111:0/1398571669 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f8190005580 con 0x7f819c1014b0 2023-10-11T16:30:32.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.041+0000 7f8183fff700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81840a1150 0x7f81840a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:32.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.041+0000 7f8183fff700 1 -- 172.21.15.111:0/1398571669 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f81900cdc50 con 0x7f819c1014b0 2023-10-11T16:30:32.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.042+0000 7f819ad9d700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81840a1150 0x7f81840a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:32.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.042+0000 7f819ad9d700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81840a1150 0x7f81840a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f819c101310 tx=0x7f818c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:32.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.045+0000 7f8183fff700 1 -- 172.21.15.111:0/1398571669 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f81900971e0 con 0x7f819c1014b0 2023-10-11T16:30:32.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.254+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f819c067b00 con 0x7f819c1014b0 2023-10-11T16:30:32.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.255+0000 7f8183fff700 1 -- 172.21.15.111:0/1398571669 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f819009a800 con 0x7f819c1014b0 2023-10-11T16:30:32.257 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:30:32.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81840a1150 msgr2=0x7f81840a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81840a1150 0x7f81840a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f819c101310 tx=0x7f818c009260 comp rx=0 tx=0).stop 2023-10-11T16:30:32.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 msgr2=0x7f819c198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 0x7f819c198790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8190004760 tx=0x7f8190004790 comp rx=0 tx=0).stop 2023-10-11T16:30:32.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 shutdown_connections 2023-10-11T16:30:32.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f81840a1150 0x7f81840a3600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f819c101e40 0x7f819c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f819c1002c0 0x7f819c198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 --2- 172.21.15.111:0/1398571669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f819c1014b0 0x7f819c198790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 >> 172.21.15.111:0/1398571669 conn(0x7f819c0fb8a0 msgr2=0x7f819c106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:32.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.257+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 shutdown_connections 2023-10-11T16:30:32.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.258+0000 7f81a11ff700 1 -- 172.21.15.111:0/1398571669 wait complete. 2023-10-11T16:30:32.278 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-11T16:30:32.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:32 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/109517420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:30:32.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:32 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3149769538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:30:32.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:32 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/109517420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:30:32.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:32 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3149769538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:30:32.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 -- 172.21.15.111:0/3507198462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368100240 msgr2=0x7f5368100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 --2- 172.21.15.111:0/3507198462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368100240 0x7f5368100650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5364009990 tx=0x7f5364009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 -- 172.21.15.111:0/3507198462 shutdown_connections 2023-10-11T16:30:32.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 --2- 172.21.15.111:0/3507198462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5368101dc0 0x7f5368108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 --2- 172.21.15.111:0/3507198462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368100240 0x7f5368100650 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 --2- 172.21.15.111:0/3507198462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5368101430 0x7f5368101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 -- 172.21.15.111:0/3507198462 >> 172.21.15.111:0/3507198462 conn(0x7f53680fb820 msgr2=0x7f53680fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:32.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 -- 172.21.15.111:0/3507198462 shutdown_connections 2023-10-11T16:30:32.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.475+0000 7f536f5ff700 1 -- 172.21.15.111:0/3507198462 wait complete. 2023-10-11T16:30:32.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.476+0000 7f536f5ff700 1 Processor -- start 2023-10-11T16:30:32.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.476+0000 7f536f5ff700 1 -- start start 2023-10-11T16:30:32.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536f5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5368100240 0x7f53680748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:32.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536f5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5368101430 0x7f5368072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:32.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536f5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368101dc0 0x7f5368073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:32.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536f5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368073bc0 con 0x7f5368101dc0 2023-10-11T16:30:32.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536f5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368073d30 con 0x7f5368100240 2023-10-11T16:30:32.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536f5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53681a0020 con 0x7f5368101430 2023-10-11T16:30:32.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536db9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368101dc0 0x7f5368073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:32.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536d39b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5368100240 0x7f53680748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:32.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536db9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368101dc0 0x7f5368073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51976/0 (socket says 172.21.15.111:51976) 2023-10-11T16:30:32.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536d39b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5368100240 0x7f53680748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:57106/0 (socket says 172.21.15.111:57106) 2023-10-11T16:30:32.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536db9c700 1 -- 172.21.15.111:0/2092081535 learned_addr learned my addr 172.21.15.111:0/2092081535 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:32.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536cb9a700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5368101430 0x7f5368072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:32.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536db9c700 1 -- 172.21.15.111:0/2092081535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5368101430 msgr2=0x7f5368072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536db9c700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5368101430 0x7f5368072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536db9c700 1 -- 172.21.15.111:0/2092081535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5368100240 msgr2=0x7f53680748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536db9c700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5368100240 0x7f53680748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.477+0000 7f536db9c700 1 -- 172.21.15.111:0/2092081535 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53640095f0 con 0x7f5368101dc0 2023-10-11T16:30:32.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.478+0000 7f536db9c700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368101dc0 0x7f5368073420 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f536000ef30 tx=0x7f536000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:32.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.478+0000 7f535e7fc700 1 -- 172.21.15.111:0/2092081535 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360018070 con 0x7f5368101dc0 2023-10-11T16:30:32.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.478+0000 7f535e7fc700 1 -- 172.21.15.111:0/2092081535 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f536000f040 con 0x7f5368101dc0 2023-10-11T16:30:32.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.478+0000 7f535e7fc700 1 -- 172.21.15.111:0/2092081535 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536000cd00 con 0x7f5368101dc0 2023-10-11T16:30:32.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.478+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536810bbf0 con 0x7f5368101dc0 2023-10-11T16:30:32.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.478+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536810ba60 con 0x7f5368101dc0 2023-10-11T16:30:32.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.479+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53681a0660 con 0x7f5368101dc0 2023-10-11T16:30:32.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.481+0000 7f535e7fc700 1 -- 172.21.15.111:0/2092081535 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f5360003a00 con 0x7f5368101dc0 2023-10-11T16:30:32.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.482+0000 7f535e7fc700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f53540a1150 0x7f53540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:32.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.482+0000 7f535e7fc700 1 -- 172.21.15.111:0/2092081535 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f53600cc680 con 0x7f5368101dc0 2023-10-11T16:30:32.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.482+0000 7f536d39b700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f53540a1150 0x7f53540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:32.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.483+0000 7f536d39b700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f53540a1150 0x7f53540a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f536400b620 tx=0x7f5364005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:32.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.486+0000 7f535e7fc700 1 -- 172.21.15.111:0/2092081535 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5360095c10 con 0x7f5368101dc0 2023-10-11T16:30:32.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:32 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/109517420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:30:32.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:32 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3149769538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:30:32.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.698+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5368107140 con 0x7f5368101dc0 2023-10-11T16:30:32.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.698+0000 7f535e7fc700 1 -- 172.21.15.111:0/2092081535 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+600 (secure 0 0 0) 0x7f5360099230 con 0x7f5368101dc0 2023-10-11T16:30:32.700 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/4086392563 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:32.700 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/4086392563 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:32.700 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3251673175 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:32.701 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3319300422 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:32.701 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3405550474 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:32.701 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/4129094163 2023-10-12T16:23:39.970669+0000 2023-10-11T16:30:32.701 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1272467376 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:32.701 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/699479694 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:32.702 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/699479694 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:32.702 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3715184603 2023-10-12T16:23:13.996718+0000 2023-10-11T16:30:32.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f53540a1150 msgr2=0x7f53540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f53540a1150 0x7f53540a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f536400b620 tx=0x7f5364005fd0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368101dc0 msgr2=0x7f5368073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:32.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368101dc0 0x7f5368073420 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f536000ef30 tx=0x7f536000c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 shutdown_connections 2023-10-11T16:30:32.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f53540a1150 0x7f53540a3600 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5368101dc0 0x7f5368073420 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5368100240 0x7f53680748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 --2- 172.21.15.111:0/2092081535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5368101430 0x7f5368072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:32.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.700+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 >> 172.21.15.111:0/2092081535 conn(0x7f53680fb820 msgr2=0x7f53680fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:32.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.701+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 shutdown_connections 2023-10-11T16:30:32.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:32.701+0000 7f536f5ff700 1 -- 172.21.15.111:0/2092081535 wait complete. 2023-10-11T16:30:32.706 INFO:teuthology.orchestra.run.smithi111.stderr:listed 10 entries 2023-10-11T16:30:32.720 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2023-10-11T16:30:32.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:32.721 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-10-11T16:30:32.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi111.stderr:ceph-fuse[135168]: fuse finished with error 0 and tester_r 0 2023-10-11T16:30:32.790 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T16:30:32.869 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:30:32.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:30:32.870 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-10-11T16:30:32.932 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:33.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 -- 172.21.15.111:0/480311019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 msgr2=0x7f18541018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 --2- 172.21.15.111:0/480311019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 0x7f18541018d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1848009990 tx=0x7f1848009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 -- 172.21.15.111:0/480311019 shutdown_connections 2023-10-11T16:30:33.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 --2- 172.21.15.111:0/480311019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1854101e10 0x7f18541086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 --2- 172.21.15.111:0/480311019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854100290 0x7f18541006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 --2- 172.21.15.111:0/480311019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 0x7f18541018d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 -- 172.21.15.111:0/480311019 >> 172.21.15.111:0/480311019 conn(0x7f18540fb870 msgr2=0x7f18540fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:33.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 -- 172.21.15.111:0/480311019 shutdown_connections 2023-10-11T16:30:33.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.130+0000 7f1858dff700 1 -- 172.21.15.111:0/480311019 wait complete. 2023-10-11T16:30:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.131+0000 7f1858dff700 1 Processor -- start 2023-10-11T16:30:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1858dff700 1 -- start start 2023-10-11T16:30:33.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1858dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854100290 0x7f18540748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1858dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 0x7f1854072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1858dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1854101e10 0x7f1854073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1858dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854073bc0 con 0x7f1854101480 2023-10-11T16:30:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1858dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854073d30 con 0x7f1854101e10 2023-10-11T16:30:33.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1858dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185419fff0 con 0x7f1854100290 2023-10-11T16:30:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1851d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 0x7f1854072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f185259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854100290 0x7f18540748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1851d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 0x7f1854072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52002/0 (socket says 172.21.15.111:52002) 2023-10-11T16:30:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1851d9b700 1 -- 172.21.15.111:0/681226208 learned_addr learned my addr 172.21.15.111:0/681226208 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:33.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.132+0000 7f1852d9d700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1854101e10 0x7f1854073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f185259c700 1 -- 172.21.15.111:0/681226208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1854101e10 msgr2=0x7f1854073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f185259c700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1854101e10 0x7f1854073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f185259c700 1 -- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 msgr2=0x7f1854072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f185259c700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 0x7f1854072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f185259c700 1 -- 172.21.15.111:0/681226208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18480095f0 con 0x7f1854100290 2023-10-11T16:30:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f185259c700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854100290 0x7f18540748a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f184400e9c0 tx=0x7f184400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f18437fe700 1 -- 172.21.15.111:0/681226208 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844004220 con 0x7f1854100290 2023-10-11T16:30:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f18437fe700 1 -- 172.21.15.111:0/681226208 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1844004380 con 0x7f1854100290 2023-10-11T16:30:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185410bc50 con 0x7f1854100290 2023-10-11T16:30:33.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f18437fe700 1 -- 172.21.15.111:0/681226208 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844003be0 con 0x7f1854100290 2023-10-11T16:30:33.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.133+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185410bac0 con 0x7f1854100290 2023-10-11T16:30:33.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.134+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18541a0740 con 0x7f1854100290 2023-10-11T16:30:33.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.136+0000 7f18437fe700 1 -- 172.21.15.111:0/681226208 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f1844003d40 con 0x7f1854100290 2023-10-11T16:30:33.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.137+0000 7f18437fe700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f183c0a1150 0x7f183c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:33.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.138+0000 7f18437fe700 1 -- 172.21.15.111:0/681226208 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1844014070 con 0x7f1854100290 2023-10-11T16:30:33.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.138+0000 7f1851d9b700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f183c0a1150 0x7f183c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:33.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.138+0000 7f1851d9b700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f183c0a1150 0x7f183c0a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1848006010 tx=0x7f184800b260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:33.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.142+0000 7f18437fe700 1 -- 172.21.15.111:0/681226208 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f18440962f0 con 0x7f1854100290 2023-10-11T16:30:33.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.341+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18541071d0 con 0x7f1854100290 2023-10-11T16:30:33.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.343+0000 7f18437fe700 1 -- 172.21.15.111:0/681226208 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7f1844099910 con 0x7f1854100290 2023-10-11T16:30:33.345 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:33.345 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:16.607979+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:33.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f183c0a1150 msgr2=0x7f183c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f183c0a1150 0x7f183c0a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1848006010 tx=0x7f184800b260 comp rx=0 tx=0).stop 2023-10-11T16:30:33.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854100290 msgr2=0x7f18540748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854100290 0x7f18540748a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f184400e9c0 tx=0x7f184400ecd0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 shutdown_connections 2023-10-11T16:30:33.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f183c0a1150 0x7f183c0a3600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1854101e10 0x7f1854073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1854100290 0x7f18540748a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 --2- 172.21.15.111:0/681226208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1854101480 0x7f1854072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 >> 172.21.15.111:0/681226208 conn(0x7f18540fb870 msgr2=0x7f18540fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:33.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 shutdown_connections 2023-10-11T16:30:33.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.345+0000 7f1858dff700 1 -- 172.21.15.111:0/681226208 wait complete. 2023-10-11T16:30:33.363 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:33 smithi111 ceph-mon[98527]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:33.363 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:33 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1398571669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:30:33.363 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:33 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2092081535' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:30:33.365 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:33.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:33 smithi150 ceph-mon[132217]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:33.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:33 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1398571669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:30:33.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:33 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2092081535' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:30:33.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.560+0000 7fb1837ff700 1 -- 172.21.15.111:0/675608665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c100290 msgr2=0x7fb17c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.560+0000 7fb1837ff700 1 --2- 172.21.15.111:0/675608665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c100290 0x7fb17c1006a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb178009990 tx=0x7fb178009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.561+0000 7fb1837ff700 1 -- 172.21.15.111:0/675608665 shutdown_connections 2023-10-11T16:30:33.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.561+0000 7fb1837ff700 1 --2- 172.21.15.111:0/675608665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c101e10 0x7fb17c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.561+0000 7fb1837ff700 1 --2- 172.21.15.111:0/675608665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c100290 0x7fb17c1006a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.561+0000 7fb1837ff700 1 --2- 172.21.15.111:0/675608665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb17c101480 0x7fb17c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.561+0000 7fb1837ff700 1 -- 172.21.15.111:0/675608665 >> 172.21.15.111:0/675608665 conn(0x7fb17c0fb870 msgr2=0x7fb17c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:33.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.561+0000 7fb1837ff700 1 -- 172.21.15.111:0/675608665 shutdown_connections 2023-10-11T16:30:33.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.561+0000 7fb1837ff700 1 -- 172.21.15.111:0/675608665 wait complete. 2023-10-11T16:30:33.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb1837ff700 1 Processor -- start 2023-10-11T16:30:33.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb1837ff700 1 -- start start 2023-10-11T16:30:33.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb1837ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb17c100290 0x7fb17c196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:33.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb1837ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c101480 0x7fb17c196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:33.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb1837ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c101e10 0x7fb17c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:33.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb1837ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c19db90 con 0x7fb17c101e10 2023-10-11T16:30:33.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb1837ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c19dcd0 con 0x7fb17c100290 2023-10-11T16:30:33.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb1837ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c19de60 con 0x7fb17c101480 2023-10-11T16:30:33.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb181d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c101e10 0x7fb17c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:33.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb18159b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb17c100290 0x7fb17c196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:33.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb181d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c101e10 0x7fb17c19d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52014/0 (socket says 172.21.15.111:52014) 2023-10-11T16:30:33.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.562+0000 7fb181d9c700 1 -- 172.21.15.111:0/3042139850 learned_addr learned my addr 172.21.15.111:0/3042139850 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:33.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb180d9a700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c101480 0x7fb17c196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:33.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb181d9c700 1 -- 172.21.15.111:0/3042139850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c101480 msgr2=0x7fb17c196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb181d9c700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c101480 0x7fb17c196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb181d9c700 1 -- 172.21.15.111:0/3042139850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb17c100290 msgr2=0x7fb17c196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb181d9c700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb17c100290 0x7fb17c196010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb181d9c700 1 -- 172.21.15.111:0/3042139850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1780095f0 con 0x7fb17c101e10 2023-10-11T16:30:33.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb181d9c700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c101e10 0x7fb17c19d5c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb17400ef30 tx=0x7fb17400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:33.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb1727fc700 1 -- 172.21.15.111:0/3042139850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb174018070 con 0x7fb17c101e10 2023-10-11T16:30:33.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb1727fc700 1 -- 172.21.15.111:0/3042139850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb17400f040 con 0x7fb17c101e10 2023-10-11T16:30:33.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17c10bc40 con 0x7fb17c101e10 2023-10-11T16:30:33.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb1727fc700 1 -- 172.21.15.111:0/3042139850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17400cd00 con 0x7fb17c101e10 2023-10-11T16:30:33.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.563+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17c10bab0 con 0x7fb17c101e10 2023-10-11T16:30:33.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.564+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17c107870 con 0x7fb17c101e10 2023-10-11T16:30:33.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.568+0000 7fb1727fc700 1 -- 172.21.15.111:0/3042139850 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fb174003a00 con 0x7fb17c101e10 2023-10-11T16:30:33.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.569+0000 7fb1727fc700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb1680a11d0 0x7fb1680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:33.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.569+0000 7fb1727fc700 1 -- 172.21.15.111:0/3042139850 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb1740ccf20 con 0x7fb17c101e10 2023-10-11T16:30:33.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.569+0000 7fb18159b700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb1680a11d0 0x7fb1680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:33.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.570+0000 7fb18159b700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb1680a11d0 0x7fb1680a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b2e0 tx=0x7fb178005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:33.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.572+0000 7fb1727fc700 1 -- 172.21.15.111:0/3042139850 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb1740964b0 con 0x7fb17c101e10 2023-10-11T16:30:33.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:33 smithi130 ceph-mon[131859]: pgmap v283: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:33.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:33 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1398571669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:30:33.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:33 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2092081535' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:30:33.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.774+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb17c02cfe0 con 0x7fb17c101e10 2023-10-11T16:30:33.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.776+0000 7fb1727fc700 1 -- 172.21.15.111:0/3042139850 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24030 (secure 0 0 0) 0x7fb174099ad0 con 0x7fb17c101e10 2023-10-11T16:30:33.778 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:33.778 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":87,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:16.607979+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:33.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb1680a11d0 msgr2=0x7fb1680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb1680a11d0 0x7fb1680a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb17800b2e0 tx=0x7fb178005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c101e10 msgr2=0x7fb17c19d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:33.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c101e10 0x7fb17c19d5c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb17400ef30 tx=0x7fb17400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 shutdown_connections 2023-10-11T16:30:33.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb1680a11d0 0x7fb1680a3680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb17c101e10 0x7fb17c19d5c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb17c100290 0x7fb17c196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 --2- 172.21.15.111:0/3042139850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c101480 0x7fb17c196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:33.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 >> 172.21.15.111:0/3042139850 conn(0x7fb17c0fb870 msgr2=0x7fb17c1049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:33.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 shutdown_connections 2023-10-11T16:30:33.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:33.778+0000 7fb1837ff700 1 -- 172.21.15.111:0/3042139850 wait complete. 2023-10-11T16:30:33.800 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-11T16:30:34.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 -- 172.21.15.111:0/479704653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8041002d0 msgr2=0x7fd8041006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:34.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 --2- 172.21.15.111:0/479704653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8041002d0 0x7fd8041006e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4009990 tx=0x7fd7f4009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:34.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 -- 172.21.15.111:0/479704653 shutdown_connections 2023-10-11T16:30:34.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 --2- 172.21.15.111:0/479704653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd804101e50 0x7fd804108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:34.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 --2- 172.21.15.111:0/479704653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8041002d0 0x7fd8041006e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:34.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 --2- 172.21.15.111:0/479704653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8041014c0 0x7fd804101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:34.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 -- 172.21.15.111:0/479704653 >> 172.21.15.111:0/479704653 conn(0x7fd8040fb8b0 msgr2=0x7fd8040fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:34.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 -- 172.21.15.111:0/479704653 shutdown_connections 2023-10-11T16:30:34.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.001+0000 7fd809fff700 1 -- 172.21.15.111:0/479704653 wait complete. 2023-10-11T16:30:34.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.002+0000 7fd809fff700 1 Processor -- start 2023-10-11T16:30:34.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.002+0000 7fd809fff700 1 -- start start 2023-10-11T16:30:34.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.002+0000 7fd809fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8041002d0 0x7fd804198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:34.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.002+0000 7fd809fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8041014c0 0x7fd8041987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:34.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.002+0000 7fd809fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd804101e50 0x7fd80419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:34.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.002+0000 7fd809fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80419fde0 con 0x7fd804101e50 2023-10-11T16:30:34.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.002+0000 7fd809fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80419ff20 con 0x7fd8041002d0 2023-10-11T16:30:34.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.002+0000 7fd809fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8041a00b0 con 0x7fd8041014c0 2023-10-11T16:30:34.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd804101e50 0x7fd80419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:34.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd804101e50 0x7fd80419f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52036/0 (socket says 172.21.15.111:52036) 2023-10-11T16:30:34.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 -- 172.21.15.111:0/2778482924 learned_addr learned my addr 172.21.15.111:0/2778482924 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:34.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd7fbfff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8041014c0 0x7fd8041987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:34.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd8037fe700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8041002d0 0x7fd804198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:34.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 -- 172.21.15.111:0/2778482924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8041014c0 msgr2=0x7fd8041987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:34.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8041014c0 0x7fd8041987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:34.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 -- 172.21.15.111:0/2778482924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8041002d0 msgr2=0x7fd804198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:34.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8041002d0 0x7fd804198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:34.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 -- 172.21.15.111:0/2778482924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec009520 con 0x7fd804101e50 2023-10-11T16:30:34.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd803fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd804101e50 0x7fd80419f810 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00ef30 tx=0x7fd7fc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:34.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd8017fa700 1 -- 172.21.15.111:0/2778482924 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc018070 con 0x7fd804101e50 2023-10-11T16:30:34.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.003+0000 7fd8017fa700 1 -- 172.21.15.111:0/2778482924 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7fc00f040 con 0x7fd804101e50 2023-10-11T16:30:34.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.004+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f40095f0 con 0x7fd804101e50 2023-10-11T16:30:34.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.004+0000 7fd8017fa700 1 -- 172.21.15.111:0/2778482924 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00cd00 con 0x7fd804101e50 2023-10-11T16:30:34.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.004+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80410bbd0 con 0x7fd804101e50 2023-10-11T16:30:34.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.005+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80410bd60 con 0x7fd804101e50 2023-10-11T16:30:34.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.006+0000 7fd8017fa700 1 -- 172.21.15.111:0/2778482924 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fd7fc003a00 con 0x7fd804101e50 2023-10-11T16:30:34.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.008+0000 7fd8017fa700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7f00a1150 0x7fd7f00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:34.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.008+0000 7fd8017fa700 1 -- 172.21.15.111:0/2778482924 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd7fc0cc5d0 con 0x7fd804101e50 2023-10-11T16:30:34.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.008+0000 7fd8037fe700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7f00a1150 0x7fd7f00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:34.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.009+0000 7fd8037fe700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7f00a1150 0x7fd7f00a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400b2e0 tx=0x7fd7f4005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:34.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.011+0000 7fd8017fa700 1 -- 172.21.15.111:0/2778482924 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd7fc095b60 con 0x7fd804101e50 2023-10-11T16:30:34.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:34.255+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd80402cfe0 con 0x7fd804101e50 2023-10-11T16:30:34.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:34 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/681226208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:34.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:34 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3042139850' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:34.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:34 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/681226208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:34.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:34 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3042139850' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:34.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:34 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/681226208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:34.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:34 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3042139850' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:35.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.146+0000 7fd8017fa700 1 -- 172.21.15.111:0/2778482924 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd7fc099180 con 0x7fd804101e50 2023-10-11T16:30:35.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.148+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7f00a1150 msgr2=0x7fd7f00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:35.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.148+0000 7fd809fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7f00a1150 0x7fd7f00a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7f400b2e0 tx=0x7fd7f4005960 comp rx=0 tx=0).stop 2023-10-11T16:30:35.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.148+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd804101e50 msgr2=0x7fd80419f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:35.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.148+0000 7fd809fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd804101e50 0x7fd80419f810 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00ef30 tx=0x7fd7fc00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.149+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 shutdown_connections 2023-10-11T16:30:35.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.149+0000 7fd809fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd7f00a1150 0x7fd7f00a3600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.149+0000 7fd809fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd804101e50 0x7fd80419f810 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.149+0000 7fd809fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8041002d0 0x7fd804198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.149+0000 7fd809fff700 1 --2- 172.21.15.111:0/2778482924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8041014c0 0x7fd8041987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.149+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 >> 172.21.15.111:0/2778482924 conn(0x7fd8040fb8b0 msgr2=0x7fd804106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:35.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.149+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 shutdown_connections 2023-10-11T16:30:35.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.149+0000 7fd809fff700 1 -- 172.21.15.111:0/2778482924 wait complete. 2023-10-11T16:30:35.171 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2023-10-11T16:30:35.172 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-11T16:30:35.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.372+0000 7f775fdff700 1 -- 172.21.15.111:0/1262589877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581014b0 msgr2=0x7f7758101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:35.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.372+0000 7f775fdff700 1 --2- 172.21.15.111:0/1262589877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581014b0 0x7f7758101900 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7748009990 tx=0x7f7748009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.372+0000 7f775fdff700 1 -- 172.21.15.111:0/1262589877 shutdown_connections 2023-10-11T16:30:35.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.372+0000 7f775fdff700 1 --2- 172.21.15.111:0/1262589877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7758101e40 0x7f7758108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.372+0000 7f775fdff700 1 --2- 172.21.15.111:0/1262589877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77581002c0 0x7f77581006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.372+0000 7f775fdff700 1 --2- 172.21.15.111:0/1262589877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581014b0 0x7f7758101900 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.372+0000 7f775fdff700 1 -- 172.21.15.111:0/1262589877 >> 172.21.15.111:0/1262589877 conn(0x7f77580fb8a0 msgr2=0x7f77580fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:35.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.373+0000 7f775fdff700 1 -- 172.21.15.111:0/1262589877 shutdown_connections 2023-10-11T16:30:35.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.373+0000 7f775fdff700 1 -- 172.21.15.111:0/1262589877 wait complete. 2023-10-11T16:30:35.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.374+0000 7f775fdff700 1 Processor -- start 2023-10-11T16:30:35.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.374+0000 7f775fdff700 1 -- start start 2023-10-11T16:30:35.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775fdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581002c0 0x7f77580748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:35.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775fdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77581014b0 0x7f7758072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:35.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775fdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7758101e40 0x7f7758073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:35.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775fdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7758073bc0 con 0x7f77581002c0 2023-10-11T16:30:35.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775fdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7758073d30 con 0x7f7758101e40 2023-10-11T16:30:35.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775fdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77581a0120 con 0x7f77581014b0 2023-10-11T16:30:35.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775db9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581002c0 0x7f77580748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:35.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775d39a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77581014b0 0x7f7758072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:35.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775db9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581002c0 0x7f77580748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49644/0 (socket says 172.21.15.111:49644) 2023-10-11T16:30:35.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775d39a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77581014b0 0x7f7758072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:37714/0 (socket says 172.21.15.111:37714) 2023-10-11T16:30:35.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775db9b700 1 -- 172.21.15.111:0/1023689436 learned_addr learned my addr 172.21.15.111:0/1023689436 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:35.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775e39c700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7758101e40 0x7f7758073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:35.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775d39a700 1 -- 172.21.15.111:0/1023689436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7758101e40 msgr2=0x7f7758073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:35.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775d39a700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7758101e40 0x7f7758073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775d39a700 1 -- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581002c0 msgr2=0x7f77580748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:35.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775d39a700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581002c0 0x7f77580748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:35.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775d39a700 1 -- 172.21.15.111:0/1023689436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7754009520 con 0x7f77581014b0 2023-10-11T16:30:35.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.375+0000 7f775db9b700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581002c0 0x7f77580748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:35.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.376+0000 7f775d39a700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77581014b0 0x7f7758072ee0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f774800b620 tx=0x7f7748004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:35.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.376+0000 7f774effd700 1 -- 172.21.15.111:0/1023689436 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774801d070 con 0x7f77581014b0 2023-10-11T16:30:35.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.376+0000 7f774effd700 1 -- 172.21.15.111:0/1023689436 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7748021410 con 0x7f77581014b0 2023-10-11T16:30:35.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.376+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77480095f0 con 0x7f77581014b0 2023-10-11T16:30:35.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.376+0000 7f774effd700 1 -- 172.21.15.111:0/1023689436 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748017650 con 0x7f77581014b0 2023-10-11T16:30:35.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.376+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775810bc70 con 0x7f77581014b0 2023-10-11T16:30:35.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.377+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7758107860 con 0x7f77581014b0 2023-10-11T16:30:35.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.379+0000 7f774effd700 1 -- 172.21.15.111:0/1023689436 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f77480217f0 con 0x7f77581014b0 2023-10-11T16:30:35.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.380+0000 7f774effd700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f77440a1150 0x7f77440a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:35.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.380+0000 7f774effd700 1 -- 172.21.15.111:0/1023689436 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f77480cdff0 con 0x7f77581014b0 2023-10-11T16:30:35.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.380+0000 7f775db9b700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f77440a1150 0x7f77440a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:35.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.381+0000 7f775db9b700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f77440a1150 0x7f77440a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7754009fd0 tx=0x7f7754009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:35.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.383+0000 7f774effd700 1 -- 172.21.15.111:0/1023689436 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f77480975e0 con 0x7f77581014b0 2023-10-11T16:30:35.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:35 smithi150 ceph-mon[132217]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:35.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:35 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2778482924' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-10-11T16:30:35.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:35 smithi130 ceph-mon[131859]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:35.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:35 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2778482924' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-10-11T16:30:35.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:35.623+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f7758067b00 con 0x7f77581014b0 2023-10-11T16:30:35.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:35 smithi111 ceph-mon[98527]: pgmap v284: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:35.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:35 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2778482924' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-10-11T16:30:36.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.152+0000 7f774effd700 1 -- 172.21.15.111:0/1023689436 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f774809ac00 con 0x7f77581014b0 2023-10-11T16:30:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f77440a1150 msgr2=0x7f77440a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f77440a1150 0x7f77440a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7754009fd0 tx=0x7f7754009260 comp rx=0 tx=0).stop 2023-10-11T16:30:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77581014b0 msgr2=0x7f7758072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:36.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77581014b0 0x7f7758072ee0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f774800b620 tx=0x7f7748004730 comp rx=0 tx=0).stop 2023-10-11T16:30:36.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 shutdown_connections 2023-10-11T16:30:36.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f77440a1150 0x7f77440a3600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7758101e40 0x7f7758073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77581002c0 0x7f77580748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 --2- 172.21.15.111:0/1023689436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77581014b0 0x7f7758072ee0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 >> 172.21.15.111:0/1023689436 conn(0x7f77580fb8a0 msgr2=0x7f77581049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:36.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 shutdown_connections 2023-10-11T16:30:36.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.155+0000 7f775fdff700 1 -- 172.21.15.111:0/1023689436 wait complete. 2023-10-11T16:30:36.179 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-10-11T16:30:36.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 -- 172.21.15.111:0/3385343145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 msgr2=0x7fc2341006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:36.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 --2- 172.21.15.111:0/3385343145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 0x7fc2341006e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc224009990 tx=0x7fc224009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 -- 172.21.15.111:0/3385343145 shutdown_connections 2023-10-11T16:30:36.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 --2- 172.21.15.111:0/3385343145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc234101e50 0x7fc234108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 --2- 172.21.15.111:0/3385343145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 0x7fc2341006e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 --2- 172.21.15.111:0/3385343145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2341014c0 0x7fc234101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 -- 172.21.15.111:0/3385343145 >> 172.21.15.111:0/3385343145 conn(0x7fc2340fb8b0 msgr2=0x7fc2340fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:36.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 -- 172.21.15.111:0/3385343145 shutdown_connections 2023-10-11T16:30:36.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.379+0000 7fc239dff700 1 -- 172.21.15.111:0/3385343145 wait complete. 2023-10-11T16:30:36.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc239dff700 1 Processor -- start 2023-10-11T16:30:36.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc239dff700 1 -- start start 2023-10-11T16:30:36.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc239dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 0x7fc2340748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:36.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc239dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2341014c0 0x7fc234072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:36.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc239dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc234101e50 0x7fc234073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:36.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc239dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234073bc0 con 0x7fc2341002d0 2023-10-11T16:30:36.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc239dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234073d30 con 0x7fc2341014c0 2023-10-11T16:30:36.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc239dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2341a0170 con 0x7fc234101e50 2023-10-11T16:30:36.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.380+0000 7fc2337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 0x7fc2340748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:36.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 0x7fc2340748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49656/0 (socket says 172.21.15.111:49656) 2023-10-11T16:30:36.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2337fe700 1 -- 172.21.15.111:0/2662874532 learned_addr learned my addr 172.21.15.111:0/2662874532 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:36.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2337fe700 1 -- 172.21.15.111:0/2662874532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc234101e50 msgr2=0x7fc234073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:36.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2337fe700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc234101e50 0x7fc234073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2337fe700 1 -- 172.21.15.111:0/2662874532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2341014c0 msgr2=0x7fc234072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:36.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2337fe700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2341014c0 0x7fc234072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:36.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2337fe700 1 -- 172.21.15.111:0/2662874532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2240095f0 con 0x7fc2341002d0 2023-10-11T16:30:36.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2337fe700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 0x7fc2340748a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc224004760 tx=0x7fc224009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:36.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2317fa700 1 -- 172.21.15.111:0/2662874532 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22401d070 con 0x7fc2341002d0 2023-10-11T16:30:36.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2317fa700 1 -- 172.21.15.111:0/2662874532 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc224004120 con 0x7fc2341002d0 2023-10-11T16:30:36.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23410bd10 con 0x7fc2341002d0 2023-10-11T16:30:36.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.381+0000 7fc2317fa700 1 -- 172.21.15.111:0/2662874532 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224005290 con 0x7fc2341002d0 2023-10-11T16:30:36.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.382+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23410bb80 con 0x7fc2341002d0 2023-10-11T16:30:36.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.383+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc234107870 con 0x7fc2341002d0 2023-10-11T16:30:36.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.384+0000 7fc2317fa700 1 -- 172.21.15.111:0/2662874532 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fc224004810 con 0x7fc2341002d0 2023-10-11T16:30:36.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.386+0000 7fc2317fa700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc2200a11d0 0x7fc2200a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:36.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.386+0000 7fc2317fa700 1 -- 172.21.15.111:0/2662874532 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc2240cdb20 con 0x7fc2341002d0 2023-10-11T16:30:36.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.386+0000 7fc22bfff700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc2200a11d0 0x7fc2200a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:36.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.387+0000 7fc22bfff700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc2200a11d0 0x7fc2200a3680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc21c005d90 tx=0x7fc21c005d00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:36.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.390+0000 7fc2317fa700 1 -- 172.21.15.111:0/2662874532 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc2240970b0 con 0x7fc2341002d0 2023-10-11T16:30:36.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:36 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2778482924' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-10-11T16:30:36.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:36 smithi150 ceph-mon[132217]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:36.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:36 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1023689436' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-11T16:30:36.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:36 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-11T16:30:36.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:36 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2778482924' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-10-11T16:30:36.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:36 smithi130 ceph-mon[131859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:36.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:36 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1023689436' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-11T16:30:36.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:36 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-11T16:30:36.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:36 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2778482924' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-10-11T16:30:36.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:36 smithi111 ceph-mon[98527]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:36.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:36 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1023689436' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-11T16:30:36.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:36 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-11T16:30:36.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:36.638+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fc23402cfe0 con 0x7fc2341002d0 2023-10-11T16:30:37.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.160+0000 7fc2317fa700 1 -- 172.21.15.111:0/2662874532 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fc22409a6d0 con 0x7fc2341002d0 2023-10-11T16:30:37.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc2200a11d0 msgr2=0x7fc2200a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc2200a11d0 0x7fc2200a3680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc21c005d90 tx=0x7fc21c005d00 comp rx=0 tx=0).stop 2023-10-11T16:30:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 msgr2=0x7fc2340748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 0x7fc2340748a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc224004760 tx=0x7fc224009580 comp rx=0 tx=0).stop 2023-10-11T16:30:37.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 shutdown_connections 2023-10-11T16:30:37.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc2200a11d0 0x7fc2200a3680 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc234101e50 0x7fc234073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2341002d0 0x7fc2340748a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 --2- 172.21.15.111:0/2662874532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2341014c0 0x7fc234072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 >> 172.21.15.111:0/2662874532 conn(0x7fc2340fb8b0 msgr2=0x7fc2341049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:37.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 shutdown_connections 2023-10-11T16:30:37.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.163+0000 7fc239dff700 1 -- 172.21.15.111:0/2662874532 wait complete. 2023-10-11T16:30:37.189 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-11T16:30:37.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.391+0000 7fedf0bff700 1 -- 172.21.15.111:0/3314481171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec1002d0 msgr2=0x7fedec1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:37.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.391+0000 7fedf0bff700 1 --2- 172.21.15.111:0/3314481171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec1002d0 0x7fedec1006e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feddc009990 tx=0x7feddc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.392+0000 7fedf0bff700 1 -- 172.21.15.111:0/3314481171 shutdown_connections 2023-10-11T16:30:37.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.392+0000 7fedf0bff700 1 --2- 172.21.15.111:0/3314481171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedec101e50 0x7fedec108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.392+0000 7fedf0bff700 1 --2- 172.21.15.111:0/3314481171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec1002d0 0x7fedec1006e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.392+0000 7fedf0bff700 1 --2- 172.21.15.111:0/3314481171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec1014c0 0x7fedec101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.392+0000 7fedf0bff700 1 -- 172.21.15.111:0/3314481171 >> 172.21.15.111:0/3314481171 conn(0x7fedec0fb8b0 msgr2=0x7fedec0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:37.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.392+0000 7fedf0bff700 1 -- 172.21.15.111:0/3314481171 shutdown_connections 2023-10-11T16:30:37.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.392+0000 7fedf0bff700 1 -- 172.21.15.111:0/3314481171 wait complete. 2023-10-11T16:30:37.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedf0bff700 1 Processor -- start 2023-10-11T16:30:37.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedf0bff700 1 -- start start 2023-10-11T16:30:37.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedf0bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec1002d0 0x7fedec1960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:37.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedf0bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedec1014c0 0x7fedec196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:37.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedf0bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec101e50 0x7fedec19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:37.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedf0bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec19dc40 con 0x7fedec101e50 2023-10-11T16:30:37.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedf0bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec19dd80 con 0x7fedec1014c0 2023-10-11T16:30:37.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedf0bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec19df10 con 0x7fedec1002d0 2023-10-11T16:30:37.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedead9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec101e50 0x7fedec19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:37.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedead9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec101e50 0x7fedec19d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49666/0 (socket says 172.21.15.111:49666) 2023-10-11T16:30:37.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fede9d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedec1014c0 0x7fedec196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:37.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedea59c700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec1002d0 0x7fedec1960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:37.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.393+0000 7fedead9d700 1 -- 172.21.15.111:0/1217135331 learned_addr learned my addr 172.21.15.111:0/1217135331 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:37.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7fedead9d700 1 -- 172.21.15.111:0/1217135331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec1002d0 msgr2=0x7fedec1960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:37.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7fedead9d700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec1002d0 0x7fedec1960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7fedead9d700 1 -- 172.21.15.111:0/1217135331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedec1014c0 msgr2=0x7fedec196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:37.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7fedead9d700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedec1014c0 0x7fedec196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:37.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7fedead9d700 1 -- 172.21.15.111:0/1217135331 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0095f0 con 0x7fedec101e50 2023-10-11T16:30:37.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7fedead9d700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec101e50 0x7fedec19d670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fede4007ee0 tx=0x7fede400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:37.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7feddb7fe700 1 -- 172.21.15.111:0/1217135331 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede4017070 con 0x7fedec101e50 2023-10-11T16:30:37.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7feddb7fe700 1 -- 172.21.15.111:0/1217135331 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fede400f040 con 0x7fedec101e50 2023-10-11T16:30:37.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec10bca0 con 0x7fedec101e50 2023-10-11T16:30:37.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7feddb7fe700 1 -- 172.21.15.111:0/1217135331 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede4005580 con 0x7fedec101e50 2023-10-11T16:30:37.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.394+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedec10bb10 con 0x7fedec101e50 2023-10-11T16:30:37.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.395+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedec107200 con 0x7fedec101e50 2023-10-11T16:30:37.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.397+0000 7feddb7fe700 1 -- 172.21.15.111:0/1217135331 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fede400db20 con 0x7fedec101e50 2023-10-11T16:30:37.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.399+0000 7feddb7fe700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fedd40a11d0 0x7fedd40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:37.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.400+0000 7feddb7fe700 1 -- 172.21.15.111:0/1217135331 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fede40cc830 con 0x7fedec101e50 2023-10-11T16:30:37.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.400+0000 7fedea59c700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fedd40a11d0 0x7fedd40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:37.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.400+0000 7fedea59c700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fedd40a11d0 0x7fedd40a3680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feddc00b2e0 tx=0x7feddc005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:37.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.402+0000 7feddb7fe700 1 -- 172.21.15.111:0/1217135331 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fede4095d30 con 0x7fedec101e50 2023-10-11T16:30:37.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:37 smithi150 ceph-mon[132217]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:37.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:37 smithi150 ceph-mon[132217]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-10-11T16:30:37.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:37 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-10-11T16:30:37.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:37 smithi150 ceph-mon[132217]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-10-11T16:30:37.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:37 smithi150 ceph-mon[132217]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:37.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:37 smithi150 ceph-mon[132217]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-10-11T16:30:37.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:37 smithi150 ceph-mon[132217]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-10-11T16:30:37.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:37 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2662874532' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-10-11T16:30:37.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:37 smithi130 ceph-mon[131859]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:37.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:37 smithi130 ceph-mon[131859]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-10-11T16:30:37.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:37 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-10-11T16:30:37.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:37 smithi130 ceph-mon[131859]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-10-11T16:30:37.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:37 smithi130 ceph-mon[131859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:37.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:37 smithi130 ceph-mon[131859]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-10-11T16:30:37.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:37 smithi130 ceph-mon[131859]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-10-11T16:30:37.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:37 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2662874532' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-10-11T16:30:37.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:37 smithi111 ceph-mon[98527]: pgmap v285: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2023-10-11T16:30:37.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:37 smithi111 ceph-mon[98527]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-10-11T16:30:37.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:37 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-10-11T16:30:37.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:37 smithi111 ceph-mon[98527]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-10-11T16:30:37.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:37 smithi111 ceph-mon[98527]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-11T16:30:37.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:37 smithi111 ceph-mon[98527]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-10-11T16:30:37.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:37 smithi111 ceph-mon[98527]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-10-11T16:30:37.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:37 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2662874532' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-10-11T16:30:37.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:37.645+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fedec02cfe0 con 0x7fedec101e50 2023-10-11T16:30:38.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.173+0000 7feddb7fe700 1 -- 172.21.15.111:0/1217135331 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fede4099350 con 0x7fedec101e50 2023-10-11T16:30:38.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fedd40a11d0 msgr2=0x7fedd40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fedd40a11d0 0x7fedd40a3680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feddc00b2e0 tx=0x7feddc005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec101e50 msgr2=0x7fedec19d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec101e50 0x7fedec19d670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fede4007ee0 tx=0x7fede400d3b0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 shutdown_connections 2023-10-11T16:30:38.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fedd40a11d0 0x7fedd40a3680 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedec101e50 0x7fedec19d670 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedec1002d0 0x7fedec1960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 --2- 172.21.15.111:0/1217135331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedec1014c0 0x7fedec196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 >> 172.21.15.111:0/1217135331 conn(0x7fedec0fb8b0 msgr2=0x7fedec0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:38.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 shutdown_connections 2023-10-11T16:30:38.180 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.176+0000 7fedf0bff700 1 -- 172.21.15.111:0/1217135331 wait complete. 2023-10-11T16:30:38.198 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2023-10-11T16:30:38.198 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-10-11T16:30:38.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 -- 172.21.15.111:0/4005662250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741002c0 msgr2=0x7fad741006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 --2- 172.21.15.111:0/4005662250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741002c0 0x7fad741006d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fad70009990 tx=0x7fad70009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 -- 172.21.15.111:0/4005662250 shutdown_connections 2023-10-11T16:30:38.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 --2- 172.21.15.111:0/4005662250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad74101e40 0x7fad74108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 --2- 172.21.15.111:0/4005662250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741002c0 0x7fad741006d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 --2- 172.21.15.111:0/4005662250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad741014b0 0x7fad74101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 -- 172.21.15.111:0/4005662250 >> 172.21.15.111:0/4005662250 conn(0x7fad740fb8a0 msgr2=0x7fad740fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:38.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 -- 172.21.15.111:0/4005662250 shutdown_connections 2023-10-11T16:30:38.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.399+0000 7fad7bdff700 1 -- 172.21.15.111:0/4005662250 wait complete. 2023-10-11T16:30:38.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.400+0000 7fad7bdff700 1 Processor -- start 2023-10-11T16:30:38.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.400+0000 7fad7bdff700 1 -- start start 2023-10-11T16:30:38.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7bdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad741002c0 0x7fad74198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:38.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7bdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741014b0 0x7fad74198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:38.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7bdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad74101e40 0x7fad7419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:38.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7bdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7419fea0 con 0x7fad741014b0 2023-10-11T16:30:38.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7bdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7419ffe0 con 0x7fad741002c0 2023-10-11T16:30:38.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7bdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad741a0170 con 0x7fad74101e40 2023-10-11T16:30:38.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7939a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741014b0 0x7fad74198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:38.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7939a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741014b0 0x7fad74198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49680/0 (socket says 172.21.15.111:49680) 2023-10-11T16:30:38.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7939a700 1 -- 172.21.15.111:0/3659469324 learned_addr learned my addr 172.21.15.111:0/3659469324 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:38.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad79b9b700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad741002c0 0x7fad74198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:38.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7a39c700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad74101e40 0x7fad7419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:38.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7939a700 1 -- 172.21.15.111:0/3659469324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad74101e40 msgr2=0x7fad7419f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7939a700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad74101e40 0x7fad7419f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7939a700 1 -- 172.21.15.111:0/3659469324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad741002c0 msgr2=0x7fad74198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7939a700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad741002c0 0x7fad74198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad7939a700 1 -- 172.21.15.111:0/3659469324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700095f0 con 0x7fad741014b0 2023-10-11T16:30:38.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.401+0000 7fad79b9b700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad741002c0 0x7fad74198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:38.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.402+0000 7fad7939a700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741014b0 0x7fad74198860 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fad6400d950 tx=0x7fad6400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:38.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.402+0000 7fad6affd700 1 -- 172.21.15.111:0/3659469324 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6400f890 con 0x7fad741014b0 2023-10-11T16:30:38.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.402+0000 7fad6affd700 1 -- 172.21.15.111:0/3659469324 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad64010460 con 0x7fad741014b0 2023-10-11T16:30:38.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.402+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7410bc90 con 0x7fad741014b0 2023-10-11T16:30:38.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.402+0000 7fad6affd700 1 -- 172.21.15.111:0/3659469324 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6400e610 con 0x7fad741014b0 2023-10-11T16:30:38.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.402+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7410bb00 con 0x7fad741014b0 2023-10-11T16:30:38.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.403+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad741929f0 con 0x7fad741014b0 2023-10-11T16:30:38.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.405+0000 7fad6affd700 1 -- 172.21.15.111:0/3659469324 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fad640108d0 con 0x7fad741014b0 2023-10-11T16:30:38.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.406+0000 7fad6affd700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad600a1150 0x7fad600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:38.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.406+0000 7fad6affd700 1 -- 172.21.15.111:0/3659469324 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fad640cc940 con 0x7fad741014b0 2023-10-11T16:30:38.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.407+0000 7fad79b9b700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad600a1150 0x7fad600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:38.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.407+0000 7fad79b9b700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad600a1150 0x7fad600a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fad7000b5c0 tx=0x7fad70005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:38.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.410+0000 7fad6affd700 1 -- 172.21.15.111:0/3659469324 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fad64095e40 con 0x7fad741014b0 2023-10-11T16:30:38.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] up:active 2023-10-11T16:30:38.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2662874532' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-10-11T16:30:38.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-10-11T16:30:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-10-11T16:30:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: Cluster is now healthy 2023-10-11T16:30:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-10-11T16:30:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-10-11T16:30:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-10-11T16:30:38.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:38 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1217135331' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-10-11T16:30:38.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] up:active 2023-10-11T16:30:38.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2662874532' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-10-11T16:30:38.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-10-11T16:30:38.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-10-11T16:30:38.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: Cluster is now healthy 2023-10-11T16:30:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-10-11T16:30:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-10-11T16:30:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-10-11T16:30:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:38 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1217135331' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-10-11T16:30:38.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] up:active 2023-10-11T16:30:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2662874532' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-10-11T16:30:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-10-11T16:30:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-10-11T16:30:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: Cluster is now healthy 2023-10-11T16:30:38.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-10-11T16:30:38.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-10-11T16:30:38.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-10-11T16:30:38.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1217135331' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-10-11T16:30:38.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.640+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fad74061a30 con 0x7fad600a1150 2023-10-11T16:30:38.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.682+0000 7fad6affd700 1 -- 172.21.15.111:0/3659469324 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fad74061a30 con 0x7fad600a1150 2023-10-11T16:30:38.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad600a1150 msgr2=0x7fad600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad600a1150 0x7fad600a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fad7000b5c0 tx=0x7fad70005fd0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741014b0 msgr2=0x7fad74198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741014b0 0x7fad74198860 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fad6400d950 tx=0x7fad6400dc60 comp rx=0 tx=0).stop 2023-10-11T16:30:38.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 shutdown_connections 2023-10-11T16:30:38.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fad600a1150 0x7fad600a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad74101e40 0x7fad7419f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad741002c0 0x7fad74198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 --2- 172.21.15.111:0/3659469324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad741014b0 0x7fad74198860 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 >> 172.21.15.111:0/3659469324 conn(0x7fad740fb8a0 msgr2=0x7fad74106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:38.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 shutdown_connections 2023-10-11T16:30:38.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.684+0000 7fad7bdff700 1 -- 172.21.15.111:0/3659469324 wait complete. 2023-10-11T16:30:38.704 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-10-11T16:30:38.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.903+0000 7f9df8dff700 1 -- 172.21.15.111:0/1427051343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 msgr2=0x7f9df4108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.903+0000 7f9df8dff700 1 --2- 172.21.15.111:0/1427051343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 0x7f9df4108710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00b0a0 tx=0x7f9dec00b3b0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.903+0000 7f9df8dff700 1 -- 172.21.15.111:0/1427051343 shutdown_connections 2023-10-11T16:30:38.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.903+0000 7f9df8dff700 1 --2- 172.21.15.111:0/1427051343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 0x7f9df4108710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.903+0000 7f9df8dff700 1 --2- 172.21.15.111:0/1427051343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df41002d0 0x7f9df41006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.903+0000 7f9df8dff700 1 --2- 172.21.15.111:0/1427051343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df41014c0 0x7f9df4101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.903+0000 7f9df8dff700 1 -- 172.21.15.111:0/1427051343 >> 172.21.15.111:0/1427051343 conn(0x7f9df40fb8b0 msgr2=0x7f9df40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.904+0000 7f9df8dff700 1 -- 172.21.15.111:0/1427051343 shutdown_connections 2023-10-11T16:30:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.904+0000 7f9df8dff700 1 -- 172.21.15.111:0/1427051343 wait complete. 2023-10-11T16:30:38.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df8dff700 1 Processor -- start 2023-10-11T16:30:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df8dff700 1 -- start start 2023-10-11T16:30:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df8dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df41002d0 0x7f9df4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df8dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df41014c0 0x7f9df4198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df8dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 0x7f9df419f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df8dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df419ff90 con 0x7f9df4101e50 2023-10-11T16:30:38.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df8dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41a00d0 con 0x7f9df41014c0 2023-10-11T16:30:38.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df8dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41a0260 con 0x7f9df41002d0 2023-10-11T16:30:38.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df2d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 0x7f9df419f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:38.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df2d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 0x7f9df419f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49700/0 (socket says 172.21.15.111:49700) 2023-10-11T16:30:38.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df2d9d700 1 -- 172.21.15.111:0/82341537 learned_addr learned my addr 172.21.15.111:0/82341537 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:38.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df259c700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df41002d0 0x7f9df4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:38.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.905+0000 7f9df1d9b700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df41014c0 0x7f9df4198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:38.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df2d9d700 1 -- 172.21.15.111:0/82341537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df41002d0 msgr2=0x7f9df4198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df2d9d700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df41002d0 0x7f9df4198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df2d9d700 1 -- 172.21.15.111:0/82341537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df41014c0 msgr2=0x7f9df4198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:38.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df2d9d700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df41014c0 0x7f9df4198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:38.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df2d9d700 1 -- 172.21.15.111:0/82341537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec009d00 con 0x7f9df4101e50 2023-10-11T16:30:38.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df1d9b700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df41014c0 0x7f9df4198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:38.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df2d9d700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 0x7f9df419f9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00d010 tx=0x7f9dec00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:38.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9de37fe700 1 -- 172.21.15.111:0/82341537 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00e040 con 0x7f9df4101e50 2023-10-11T16:30:38.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9de37fe700 1 -- 172.21.15.111:0/82341537 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9dec003e10 con 0x7f9df4101e50 2023-10-11T16:30:38.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df410bca0 con 0x7f9df4101e50 2023-10-11T16:30:38.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9de37fe700 1 -- 172.21.15.111:0/82341537 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec0052a0 con 0x7f9df4101e50 2023-10-11T16:30:38.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.906+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df410bb10 con 0x7f9df4101e50 2023-10-11T16:30:38.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.907+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df41a07b0 con 0x7f9df4101e50 2023-10-11T16:30:38.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.909+0000 7f9de37fe700 1 -- 172.21.15.111:0/82341537 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f9dec005400 con 0x7f9df4101e50 2023-10-11T16:30:38.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.910+0000 7f9de37fe700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ddc0a1150 0x7f9ddc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:38.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.911+0000 7f9de37fe700 1 -- 172.21.15.111:0/82341537 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9dec0cdd00 con 0x7f9df4101e50 2023-10-11T16:30:38.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.911+0000 7f9df259c700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ddc0a1150 0x7f9ddc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:38.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.911+0000 7f9df259c700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ddc0a1150 0x7f9ddc0a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9df4101320 tx=0x7f9de4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:38.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:38.914+0000 7f9de37fe700 1 -- 172.21.15.111:0/82341537 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9dec097290 con 0x7f9df4101e50 2023-10-11T16:30:39.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:30:38.647+0000 7eff26f00700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:30:39.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:30:38.647+0000 7eff26f00700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:30:39.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:30:38.647+0000 7eff26f00700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:30:39.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:30:38.647+0000 7eff26f00700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:30:39.136 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:30:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:30:38.647+0000 7eff26f00700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:30:39.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.148+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f9df41a0e20 con 0x7f9ddc0a1150 2023-10-11T16:30:39.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.153+0000 7f9de37fe700 1 -- 172.21.15.111:0/82341537 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f9df41a0e20 con 0x7f9ddc0a1150 2023-10-11T16:30:39.155 INFO:teuthology.orchestra.run.smithi111.stdout:/volumes/_nogroup/sv_0/643f2e74-74fe-4c55-8395-87bab57ea4b6 2023-10-11T16:30:39.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.155+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ddc0a1150 msgr2=0x7f9ddc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.155+0000 7f9df8dff700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ddc0a1150 0x7f9ddc0a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9df4101320 tx=0x7f9de4009260 comp rx=0 tx=0).stop 2023-10-11T16:30:39.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.155+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 msgr2=0x7f9df419f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.155+0000 7f9df8dff700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 0x7f9df419f9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00d010 tx=0x7f9dec00bc80 comp rx=0 tx=0).stop 2023-10-11T16:30:39.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.156+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 shutdown_connections 2023-10-11T16:30:39.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.156+0000 7f9df8dff700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ddc0a1150 0x7f9ddc0a3600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.156+0000 7f9df8dff700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9df4101e50 0x7f9df419f9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.156+0000 7f9df8dff700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9df41002d0 0x7f9df4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.156+0000 7f9df8dff700 1 --2- 172.21.15.111:0/82341537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9df41014c0 0x7f9df4198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.156+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 >> 172.21.15.111:0/82341537 conn(0x7f9df40fb8b0 msgr2=0x7f9df4106d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.156+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 shutdown_connections 2023-10-11T16:30:39.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.156+0000 7f9df8dff700 1 -- 172.21.15.111:0/82341537 wait complete. 2023-10-11T16:30:39.177 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-10-11T16:30:39.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 -- 172.21.15.111:0/1937710014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f64107f00 msgr2=0x7f9f6410a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/1937710014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f64107f00 0x7f9f6410a320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00b0a0 tx=0x7f9f5c00b3b0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 -- 172.21.15.111:0/1937710014 shutdown_connections 2023-10-11T16:30:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/1937710014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f64107f00 0x7f9f6410a320 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/1937710014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f64074a90 0x7f9f64072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/1937710014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f640734c0 0x7f9f64073930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 -- 172.21.15.111:0/1937710014 >> 172.21.15.111:0/1937710014 conn(0x7f9f640fb6a0 msgr2=0x7f9f640fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:39.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 -- 172.21.15.111:0/1937710014 shutdown_connections 2023-10-11T16:30:39.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.377+0000 7f9f6b19d700 1 -- 172.21.15.111:0/1937710014 wait complete. 2023-10-11T16:30:39.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.378+0000 7f9f6b19d700 1 Processor -- start 2023-10-11T16:30:39.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.378+0000 7f9f6b19d700 1 -- start start 2023-10-11T16:30:39.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6b19d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f640734c0 0x7f9f641980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6b19d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f64074a90 0x7f9f64198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6b19d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f64107f00 0x7f9f6419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6b19d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6419fc70 con 0x7f9f640734c0 2023-10-11T16:30:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6b19d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6419fdb0 con 0x7f9f64107f00 2023-10-11T16:30:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6b19d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6419ff40 con 0x7f9f64074a90 2023-10-11T16:30:39.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a19b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f640734c0 0x7f9f641980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:39.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a19b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f640734c0 0x7f9f641980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49714/0 (socket says 172.21.15.111:49714) 2023-10-11T16:30:39.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a19b700 1 -- 172.21.15.111:0/2442448696 learned_addr learned my addr 172.21.15.111:0/2442448696 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:39.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6999a700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f64074a90 0x7f9f64198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:39.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a99c700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f64107f00 0x7f9f6419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:39.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a19b700 1 -- 172.21.15.111:0/2442448696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f64074a90 msgr2=0x7f9f64198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a19b700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f64074a90 0x7f9f64198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a19b700 1 -- 172.21.15.111:0/2442448696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f64107f00 msgr2=0x7f9f6419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a19b700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f64107f00 0x7f9f6419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.379+0000 7f9f6a19b700 1 -- 172.21.15.111:0/2442448696 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f60009520 con 0x7f9f640734c0 2023-10-11T16:30:39.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.380+0000 7f9f6a19b700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f640734c0 0x7f9f641980f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000ea50 tx=0x7f9f6000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:39.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.380+0000 7f9f5b7fe700 1 -- 172.21.15.111:0/2442448696 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000cbd0 con 0x7f9f640734c0 2023-10-11T16:30:39.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.380+0000 7f9f5b7fe700 1 -- 172.21.15.111:0/2442448696 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f6000cd30 con 0x7f9f640734c0 2023-10-11T16:30:39.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.380+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c009d00 con 0x7f9f640734c0 2023-10-11T16:30:39.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.380+0000 7f9f5b7fe700 1 -- 172.21.15.111:0/2442448696 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60018680 con 0x7f9f640734c0 2023-10-11T16:30:39.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.380+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6410dc50 con 0x7f9f640734c0 2023-10-11T16:30:39.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.381+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f64106690 con 0x7f9f640734c0 2023-10-11T16:30:39.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.383+0000 7f9f5b7fe700 1 -- 172.21.15.111:0/2442448696 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f9f60010430 con 0x7f9f640734c0 2023-10-11T16:30:39.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.384+0000 7f9f5b7fe700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f500a11d0 0x7f9f500a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:39.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.384+0000 7f9f5b7fe700 1 -- 172.21.15.111:0/2442448696 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9f60014070 con 0x7f9f640734c0 2023-10-11T16:30:39.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.384+0000 7f9f6999a700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f500a11d0 0x7f9f500a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:39.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.385+0000 7f9f6999a700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f500a11d0 0x7f9f500a3680 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009c60 tx=0x7f9f54009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:39.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.388+0000 7f9f5b7fe700 1 -- 172.21.15.111:0/2442448696 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9f600964b0 con 0x7f9f640734c0 2023-10-11T16:30:39.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:39 smithi150 ceph-mon[132217]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 2 op/s 2023-10-11T16:30:39.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:39 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] up:active 2023-10-11T16:30:39.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:39 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1217135331' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-10-11T16:30:39.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:39 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2023-10-11T16:30:39.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:39 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:39.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:39 smithi150 ceph-mon[132217]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:39.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:39 smithi130 ceph-mon[131859]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 2 op/s 2023-10-11T16:30:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:39 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] up:active 2023-10-11T16:30:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:39 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1217135331' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-10-11T16:30:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:39 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2023-10-11T16:30:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:39 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:39.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:39 smithi130 ceph-mon[131859]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:39.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.616+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f64060f30 con 0x7f9f500a11d0 2023-10-11T16:30:39.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:39 smithi111 ceph-mon[98527]: pgmap v286: 129 pgs: 129 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 2 op/s 2023-10-11T16:30:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:39 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] up:active 2023-10-11T16:30:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:39 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1217135331' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-10-11T16:30:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:39 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2023-10-11T16:30:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:39 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:39 smithi111 ceph-mon[98527]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:39.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.641+0000 7f9f5b7fe700 1 -- 172.21.15.111:0/2442448696 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9f64060f30 con 0x7f9f500a11d0 2023-10-11T16:30:39.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f500a11d0 msgr2=0x7f9f500a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f500a11d0 0x7f9f500a3680 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009c60 tx=0x7f9f54009260 comp rx=0 tx=0).stop 2023-10-11T16:30:39.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f640734c0 msgr2=0x7f9f641980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f640734c0 0x7f9f641980f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000ea50 tx=0x7f9f6000ed60 comp rx=0 tx=0).stop 2023-10-11T16:30:39.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 shutdown_connections 2023-10-11T16:30:39.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f500a11d0 0x7f9f500a3680 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f64107f00 0x7f9f6419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f640734c0 0x7f9f641980f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 --2- 172.21.15.111:0/2442448696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f64074a90 0x7f9f64198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.643+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 >> 172.21.15.111:0/2442448696 conn(0x7f9f640fb6a0 msgr2=0x7f9f6410a110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:39.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.644+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 shutdown_connections 2023-10-11T16:30:39.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.644+0000 7f9f6b19d700 1 -- 172.21.15.111:0/2442448696 wait complete. 2023-10-11T16:30:39.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-10-11T16:30:39.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 -- 172.21.15.111:0/2391745119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc1002d0 msgr2=0x7fbbcc1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/2391745119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc1002d0 0x7fbbcc1006e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009990 tx=0x7fbbbc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 -- 172.21.15.111:0/2391745119 shutdown_connections 2023-10-11T16:30:39.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/2391745119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbcc101e50 0x7fbbcc108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/2391745119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc1002d0 0x7fbbcc1006e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/2391745119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc1014c0 0x7fbbcc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 -- 172.21.15.111:0/2391745119 >> 172.21.15.111:0/2391745119 conn(0x7fbbcc0fb8b0 msgr2=0x7fbbcc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:39.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 -- 172.21.15.111:0/2391745119 shutdown_connections 2023-10-11T16:30:39.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.863+0000 7fbbd11ff700 1 -- 172.21.15.111:0/2391745119 wait complete. 2023-10-11T16:30:39.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.864+0000 7fbbd11ff700 1 Processor -- start 2023-10-11T16:30:39.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.864+0000 7fbbd11ff700 1 -- start start 2023-10-11T16:30:39.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbd11ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc1002d0 0x7fbbcc0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:39.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbd11ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbcc1014c0 0x7fbbcc072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbd11ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc101e50 0x7fbbcc073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbd11ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc073bc0 con 0x7fbbcc1002d0 2023-10-11T16:30:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbd11ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc073d30 con 0x7fbbcc1014c0 2023-10-11T16:30:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbd11ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc1a00a0 con 0x7fbbcc101e50 2023-10-11T16:30:39.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc1002d0 0x7fbbcc0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc1002d0 0x7fbbcc0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49722/0 (socket says 172.21.15.111:49722) 2023-10-11T16:30:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcad9d700 1 -- 172.21.15.111:0/3323447472 learned_addr learned my addr 172.21.15.111:0/3323447472 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbc3fff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbcc1014c0 0x7fbbcc072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:39.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcb59e700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc101e50 0x7fbbcc073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:39.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcad9d700 1 -- 172.21.15.111:0/3323447472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc101e50 msgr2=0x7fbbcc073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcad9d700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc101e50 0x7fbbcc073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcad9d700 1 -- 172.21.15.111:0/3323447472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbcc1014c0 msgr2=0x7fbbcc072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:39.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcad9d700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbcc1014c0 0x7fbbcc072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:39.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcad9d700 1 -- 172.21.15.111:0/3323447472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4009520 con 0x7fbbcc1002d0 2023-10-11T16:30:39.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.865+0000 7fbbcb59e700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc101e50 0x7fbbcc073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:39.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.866+0000 7fbbcad9d700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc1002d0 0x7fbbcc0748a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc004710 tx=0x7fbbbc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:39.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.866+0000 7fbbc8d99700 1 -- 172.21.15.111:0/3323447472 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc01d070 con 0x7fbbcc1002d0 2023-10-11T16:30:39.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.866+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc0095f0 con 0x7fbbcc1002d0 2023-10-11T16:30:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.866+0000 7fbbc8d99700 1 -- 172.21.15.111:0/3323447472 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbbc021410 con 0x7fbbcc1002d0 2023-10-11T16:30:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.866+0000 7fbbc8d99700 1 -- 172.21.15.111:0/3323447472 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc017650 con 0x7fbbcc1002d0 2023-10-11T16:30:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.866+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbcc10bc20 con 0x7fbbcc1002d0 2023-10-11T16:30:39.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.867+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbcc10bdb0 con 0x7fbbcc1002d0 2023-10-11T16:30:39.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.869+0000 7fbbc8d99700 1 -- 172.21.15.111:0/3323447472 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fbbbc021580 con 0x7fbbcc1002d0 2023-10-11T16:30:39.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.870+0000 7fbbc8d99700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbbb80a1150 0x7fbbb80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:39.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.870+0000 7fbbc8d99700 1 -- 172.21.15.111:0/3323447472 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbbbc0cdf90 con 0x7fbbcc1002d0 2023-10-11T16:30:39.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.870+0000 7fbbc3fff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbbb80a1150 0x7fbbb80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:39.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.871+0000 7fbbc3fff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbbb80a1150 0x7fbbb80a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009df0 tx=0x7fbbb4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:39.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:39.874+0000 7fbbc8d99700 1 -- 172.21.15.111:0/3323447472 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbbbc097520 con 0x7fbbcc1002d0 2023-10-11T16:30:40.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.111+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbcc060f30 con 0x7fbbb80a1150 2023-10-11T16:30:40.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.115+0000 7fbbc8d99700 1 -- 172.21.15.111:0/3323447472 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fbbcc060f30 con 0x7fbbb80a1150 2023-10-11T16:30:40.117 INFO:teuthology.orchestra.run.smithi111.stdout:/volumes/_nogroup/sv_1/291f0a93-6079-41ae-a1a4-65679e80ef94 2023-10-11T16:30:40.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.117+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbbb80a1150 msgr2=0x7fbbb80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.117+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbbb80a1150 0x7fbbb80a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009df0 tx=0x7fbbb4009260 comp rx=0 tx=0).stop 2023-10-11T16:30:40.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.117+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc1002d0 msgr2=0x7fbbcc0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.117+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc1002d0 0x7fbbcc0748a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc004710 tx=0x7fbbbc009580 comp rx=0 tx=0).stop 2023-10-11T16:30:40.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.118+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 shutdown_connections 2023-10-11T16:30:40.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.118+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbbb80a1150 0x7fbbb80a3600 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.118+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbcc101e50 0x7fbbcc073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.118+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbcc1002d0 0x7fbbcc0748a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.118+0000 7fbbd11ff700 1 --2- 172.21.15.111:0/3323447472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbbcc1014c0 0x7fbbcc072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.118+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 >> 172.21.15.111:0/3323447472 conn(0x7fbbcc0fb8b0 msgr2=0x7fbbcc0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:40.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.118+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 shutdown_connections 2023-10-11T16:30:40.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.118+0000 7fbbd11ff700 1 -- 172.21.15.111:0/3323447472 wait complete. 2023-10-11T16:30:40.140 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:30:40.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 -- 172.21.15.111:0/463844101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41014a0 msgr2=0x7f15a41018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/463844101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41014a0 0x7f15a41018f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15940098f0 tx=0x7f1594009c00 comp rx=0 tx=0).stop 2023-10-11T16:30:40.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 -- 172.21.15.111:0/463844101 shutdown_connections 2023-10-11T16:30:40.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/463844101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a4101e30 0x7f15a41086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/463844101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15a41002b0 0x7f15a41006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/463844101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41014a0 0x7f15a41018f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 -- 172.21.15.111:0/463844101 >> 172.21.15.111:0/463844101 conn(0x7f15a40fb890 msgr2=0x7f15a40fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:40.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 -- 172.21.15.111:0/463844101 shutdown_connections 2023-10-11T16:30:40.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.343+0000 7f15ab3ff700 1 -- 172.21.15.111:0/463844101 wait complete. 2023-10-11T16:30:40.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.344+0000 7f15ab3ff700 1 Processor -- start 2023-10-11T16:30:40.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.344+0000 7f15ab3ff700 1 -- start start 2023-10-11T16:30:40.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.344+0000 7f15ab3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41002b0 0x7f15a40748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:40.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.344+0000 7f15ab3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a41014a0 0x7f15a4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:40.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.344+0000 7f15ab3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15a4101e30 0x7f15a4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:40.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.344+0000 7f15ab3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4073bc0 con 0x7f15a41014a0 2023-10-11T16:30:40.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.344+0000 7f15ab3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4073d30 con 0x7f15a41002b0 2023-10-11T16:30:40.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.344+0000 7f15ab3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a41a0080 con 0x7f15a4101e30 2023-10-11T16:30:40.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a41014a0 0x7f15a4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:40.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a919b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41002b0 0x7f15a40748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:40.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a41014a0 0x7f15a4072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49734/0 (socket says 172.21.15.111:49734) 2023-10-11T16:30:40.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 -- 172.21.15.111:0/1875096587 learned_addr learned my addr 172.21.15.111:0/1875096587 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:40.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a999c700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15a4101e30 0x7f15a4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:40.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 -- 172.21.15.111:0/1875096587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15a4101e30 msgr2=0x7f15a4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15a4101e30 0x7f15a4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 -- 172.21.15.111:0/1875096587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41002b0 msgr2=0x7f15a40748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41002b0 0x7f15a40748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 -- 172.21.15.111:0/1875096587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c009d00 con 0x7f15a41014a0 2023-10-11T16:30:40.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a919b700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41002b0 0x7f15a40748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:40.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.345+0000 7f15a899a700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a41014a0 0x7f15a4072ee0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1594005e90 tx=0x7f159400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:40.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.346+0000 7f159a7fc700 1 -- 172.21.15.111:0/1875096587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159401c070 con 0x7f15a41014a0 2023-10-11T16:30:40.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.346+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1594009520 con 0x7f15a41014a0 2023-10-11T16:30:40.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.346+0000 7f159a7fc700 1 -- 172.21.15.111:0/1875096587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f159400f7f0 con 0x7f15a41014a0 2023-10-11T16:30:40.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.346+0000 7f159a7fc700 1 -- 172.21.15.111:0/1875096587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15940049c0 con 0x7f15a41014a0 2023-10-11T16:30:40.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.346+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a410bc60 con 0x7f15a41014a0 2023-10-11T16:30:40.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.347+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a410bdf0 con 0x7f15a41014a0 2023-10-11T16:30:40.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.349+0000 7f159a7fc700 1 -- 172.21.15.111:0/1875096587 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f1594020070 con 0x7f15a41014a0 2023-10-11T16:30:40.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.350+0000 7f159a7fc700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15900a1150 0x7f15900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:40.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.350+0000 7f159a7fc700 1 -- 172.21.15.111:0/1875096587 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15940ce080 con 0x7f15a41014a0 2023-10-11T16:30:40.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.350+0000 7f15a919b700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15900a1150 0x7f15900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:40.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.351+0000 7f15a919b700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15900a1150 0x7f15900a3600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f15a4101300 tx=0x7f15a0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:40.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.352+0000 7f159a7fc700 1 -- 172.21.15.111:0/1875096587 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f15940974f0 con 0x7f15a41014a0 2023-10-11T16:30:40.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:40.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:40.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-10-11T16:30:40.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2023-10-11T16:30:40.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2023-10-11T16:30:40.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-10-11T16:30:40.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-10-11T16:30:40.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2023-10-11T16:30:40.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2023-10-11T16:30:40.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-10-11T16:30:40.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-10-11T16:30:40.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:40 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-10-11T16:30:40.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:40.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:40.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-10-11T16:30:40.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2023-10-11T16:30:40.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2023-10-11T16:30:40.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-10-11T16:30:40.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-10-11T16:30:40.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2023-10-11T16:30:40.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2023-10-11T16:30:40.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-10-11T16:30:40.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-10-11T16:30:40.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:40 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-10-11T16:30:40.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.593+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15a4107230 con 0x7f15a41014a0 2023-10-11T16:30:40.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.594+0000 7f159a7fc700 1 -- 172.21.15.111:0/1875096587 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10820 (secure 0 0 0) 0x7f159409ab10 con 0x7f15a41014a0 2023-10-11T16:30:40.596 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:40.596 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:38.174800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:30:40.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15900a1150 msgr2=0x7f15900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15900a1150 0x7f15900a3600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f15a4101300 tx=0x7f15a0009260 comp rx=0 tx=0).stop 2023-10-11T16:30:40.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a41014a0 msgr2=0x7f15a4072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a41014a0 0x7f15a4072ee0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1594005e90 tx=0x7f159400b730 comp rx=0 tx=0).stop 2023-10-11T16:30:40.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 shutdown_connections 2023-10-11T16:30:40.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15900a1150 0x7f15900a3600 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15a4101e30 0x7f15a4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15a41002b0 0x7f15a40748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 --2- 172.21.15.111:0/1875096587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a41014a0 0x7f15a4072ee0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 >> 172.21.15.111:0/1875096587 conn(0x7f15a40fb890 msgr2=0x7f15a40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:40.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 shutdown_connections 2023-10-11T16:30:40.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.596+0000 7f15ab3ff700 1 -- 172.21.15.111:0/1875096587 wait complete. 2023-10-11T16:30:40.601 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 24 2023-10-11T16:30:40.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:40.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:40.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-10-11T16:30:40.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2023-10-11T16:30:40.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2023-10-11T16:30:40.615 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-10-11T16:30:40.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-10-11T16:30:40.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2023-10-11T16:30:40.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2023-10-11T16:30:40.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-10-11T16:30:40.616 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-10-11T16:30:40.617 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:40 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-10-11T16:30:40.618 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:30:40.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 -- 172.21.15.111:0/775507069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c101480 msgr2=0x7f4d4c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 --2- 172.21.15.111:0/775507069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c101480 0x7f4d4c1018d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009990 tx=0x7f4d38009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 -- 172.21.15.111:0/775507069 shutdown_connections 2023-10-11T16:30:40.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 --2- 172.21.15.111:0/775507069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d4c101e10 0x7f4d4c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 --2- 172.21.15.111:0/775507069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d4c100290 0x7f4d4c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 --2- 172.21.15.111:0/775507069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c101480 0x7f4d4c1018d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 -- 172.21.15.111:0/775507069 >> 172.21.15.111:0/775507069 conn(0x7f4d4c0fb870 msgr2=0x7f4d4c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:40.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 -- 172.21.15.111:0/775507069 shutdown_connections 2023-10-11T16:30:40.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.815+0000 7f4d52bff700 1 -- 172.21.15.111:0/775507069 wait complete. 2023-10-11T16:30:40.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.816+0000 7f4d52bff700 1 Processor -- start 2023-10-11T16:30:40.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.816+0000 7f4d52bff700 1 -- start start 2023-10-11T16:30:40.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d52bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c100290 0x7f4d4c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:40.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d52bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d4c101480 0x7f4d4c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:40.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d52bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d4c101e10 0x7f4d4c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:40.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d52bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c073bc0 con 0x7f4d4c100290 2023-10-11T16:30:40.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d52bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c073d30 con 0x7f4d4c101e10 2023-10-11T16:30:40.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d52bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c1a0000 con 0x7f4d4c101480 2023-10-11T16:30:40.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5099b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c100290 0x7f4d4c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:40.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5099b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c100290 0x7f4d4c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49742/0 (socket says 172.21.15.111:49742) 2023-10-11T16:30:40.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5099b700 1 -- 172.21.15.111:0/3809853712 learned_addr learned my addr 172.21.15.111:0/3809853712 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:40.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d4bfff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d4c101480 0x7f4d4c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:40.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5119c700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d4c101e10 0x7f4d4c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:40.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5099b700 1 -- 172.21.15.111:0/3809853712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d4c101480 msgr2=0x7f4d4c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5099b700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d4c101480 0x7f4d4c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5099b700 1 -- 172.21.15.111:0/3809853712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d4c101e10 msgr2=0x7f4d4c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:40.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5099b700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d4c101e10 0x7f4d4c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:40.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.817+0000 7f4d5099b700 1 -- 172.21.15.111:0/3809853712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d380095f0 con 0x7f4d4c100290 2023-10-11T16:30:40.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.818+0000 7f4d5099b700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c100290 0x7f4d4c0748a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000d980 tx=0x7f4d4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:40.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.818+0000 7f4d49ffb700 1 -- 172.21.15.111:0/3809853712 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4000b950 con 0x7f4d4c100290 2023-10-11T16:30:40.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.818+0000 7f4d49ffb700 1 -- 172.21.15.111:0/3809853712 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d40007620 con 0x7f4d4c100290 2023-10-11T16:30:40.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.818+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c10bc50 con 0x7f4d4c100290 2023-10-11T16:30:40.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.818+0000 7f4d49ffb700 1 -- 172.21.15.111:0/3809853712 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4000f3c0 con 0x7f4d4c100290 2023-10-11T16:30:40.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.818+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c10bac0 con 0x7f4d4c100290 2023-10-11T16:30:40.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.819+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c1071c0 con 0x7f4d4c100290 2023-10-11T16:30:40.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.821+0000 7f4d49ffb700 1 -- 172.21.15.111:0/3809853712 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f4d40010640 con 0x7f4d4c100290 2023-10-11T16:30:40.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.822+0000 7f4d49ffb700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4d3c0a11d0 0x7f4d3c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:40.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.822+0000 7f4d49ffb700 1 -- 172.21.15.111:0/3809853712 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4d400cca50 con 0x7f4d4c100290 2023-10-11T16:30:40.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.822+0000 7f4d4bfff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4d3c0a11d0 0x7f4d3c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:40.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.823+0000 7f4d4bfff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4d3c0a11d0 0x7f4d3c0a3680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800b2e0 tx=0x7f4d38005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:40.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:40.826+0000 7f4d49ffb700 1 -- 172.21.15.111:0/3809853712 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4d40095e90 con 0x7f4d4c100290 2023-10-11T16:30:41.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.027+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d4c02cfe0 con 0x7f4d4c100290 2023-10-11T16:30:41.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.029+0000 7f4d49ffb700 1 -- 172.21.15.111:0/3809853712 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24555 (secure 0 0 0) 0x7f4d40016090 con 0x7f4d4c100290 2023-10-11T16:30:41.031 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:41.031 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":88,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:40.174382+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:41.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4d3c0a11d0 msgr2=0x7f4d3c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4d3c0a11d0 0x7f4d3c0a3680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800b2e0 tx=0x7f4d38005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c100290 msgr2=0x7f4d4c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c100290 0x7f4d4c0748a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000d980 tx=0x7f4d4000dc90 comp rx=0 tx=0).stop 2023-10-11T16:30:41.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 shutdown_connections 2023-10-11T16:30:41.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f4d3c0a11d0 0x7f4d3c0a3680 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d4c101e10 0x7f4d4c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d4c100290 0x7f4d4c0748a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 --2- 172.21.15.111:0/3809853712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d4c101480 0x7f4d4c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 >> 172.21.15.111:0/3809853712 conn(0x7f4d4c0fb870 msgr2=0x7f4d4c0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:41.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 shutdown_connections 2023-10-11T16:30:41.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.031+0000 7f4d52bff700 1 -- 172.21.15.111:0/3809853712 wait complete. 2023-10-11T16:30:41.053 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-11T16:30:41.053 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T16:30:41.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 -- 172.21.15.111:0/3689644148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c074b30 msgr2=0x7f671c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 --2- 172.21.15.111:0/3689644148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c074b30 0x7f671c072f80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6718009990 tx=0x7f6718009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 -- 172.21.15.111:0/3689644148 shutdown_connections 2023-10-11T16:30:41.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 --2- 172.21.15.111:0/3689644148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f671c10c860 0x7f671c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 --2- 172.21.15.111:0/3689644148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c074b30 0x7f671c072f80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 --2- 172.21.15.111:0/3689644148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f671c073550 0x7f671c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 -- 172.21.15.111:0/3689644148 >> 172.21.15.111:0/3689644148 conn(0x7f671c0fb870 msgr2=0x7f671c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:41.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 -- 172.21.15.111:0/3689644148 shutdown_connections 2023-10-11T16:30:41.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.261+0000 7f6723bff700 1 -- 172.21.15.111:0/3689644148 wait complete. 2023-10-11T16:30:41.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.262+0000 7f6723bff700 1 Processor -- start 2023-10-11T16:30:41.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.262+0000 7f6723bff700 1 -- start start 2023-10-11T16:30:41.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.262+0000 7f6723bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f671c073550 0x7f671c19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:41.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.262+0000 7f6723bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f671c074b30 0x7f671c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:41.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f6723bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c10c860 0x7f671c1a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:41.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f6723bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1a4180 con 0x7f671c10c860 2023-10-11T16:30:41.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f6723bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1a42c0 con 0x7f671c073550 2023-10-11T16:30:41.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f6723bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1a4450 con 0x7f671c074b30 2023-10-11T16:30:41.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f671c073550 0x7f671c19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:41.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f671c073550 0x7f671c19c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:34298/0 (socket says 172.21.15.111:34298) 2023-10-11T16:30:41.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 -- 172.21.15.111:0/1981913342 learned_addr learned my addr 172.21.15.111:0/1981913342 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:41.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672119a700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f671c074b30 0x7f671c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:41.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672219c700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c10c860 0x7f671c1a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:41.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 -- 172.21.15.111:0/1981913342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f671c074b30 msgr2=0x7f671c19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f671c074b30 0x7f671c19cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 -- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c10c860 msgr2=0x7f671c1a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c10c860 0x7f671c1a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 -- 172.21.15.111:0/1981913342 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67180095f0 con 0x7f671c073550 2023-10-11T16:30:41.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.263+0000 7f672199b700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f671c073550 0x7f671c19c600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f67180098e0 tx=0x7f6718004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:41.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.264+0000 7f6712ffd700 1 -- 172.21.15.111:0/1981913342 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671801d070 con 0x7f671c073550 2023-10-11T16:30:41.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.264+0000 7f6712ffd700 1 -- 172.21.15.111:0/1981913342 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67180054f0 con 0x7f671c073550 2023-10-11T16:30:41.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.264+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c112250 con 0x7f671c073550 2023-10-11T16:30:41.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.264+0000 7f6712ffd700 1 -- 172.21.15.111:0/1981913342 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718004be0 con 0x7f671c073550 2023-10-11T16:30:41.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.264+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c1120c0 con 0x7f671c073550 2023-10-11T16:30:41.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.265+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c1023b0 con 0x7f671c073550 2023-10-11T16:30:41.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.267+0000 7f6712ffd700 1 -- 172.21.15.111:0/1981913342 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f671800fbc0 con 0x7f671c073550 2023-10-11T16:30:41.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.268+0000 7f6712ffd700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67080a1260 0x7f67080a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:41.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.269+0000 7f6712ffd700 1 -- 172.21.15.111:0/1981913342 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f67180cdfb0 con 0x7f671c073550 2023-10-11T16:30:41.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.269+0000 7f672119a700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67080a1260 0x7f67080a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:41.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.269+0000 7f672119a700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67080a1260 0x7f67080a3710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f671c074850 tx=0x7f670c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:41.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.271+0000 7f6712ffd700 1 -- 172.21.15.111:0/1981913342 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f67180972c0 con 0x7f671c073550 2023-10-11T16:30:41.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: pgmap v287: 129 pgs: 129 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.9 KiB/s wr, 5 op/s 2023-10-11T16:30:41.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:41.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-10-11T16:30:41.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2023-10-11T16:30:41.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2023-10-11T16:30:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-10-11T16:30:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-10-11T16:30:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2023-10-11T16:30:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2023-10-11T16:30:41.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-10-11T16:30:41.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-10-11T16:30:41.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-10-11T16:30:41.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: osdmap e88: 12 total, 12 up, 12 in 2023-10-11T16:30:41.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1875096587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:41.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3809853712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:41.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-10-11T16:30:41.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:41 smithi111 ceph-mon[98527]: osdmap e89: 12 total, 12 up, 12 in 2023-10-11T16:30:41.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: pgmap v287: 129 pgs: 129 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.9 KiB/s wr, 5 op/s 2023-10-11T16:30:41.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-10-11T16:30:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2023-10-11T16:30:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2023-10-11T16:30:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-10-11T16:30:41.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-10-11T16:30:41.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2023-10-11T16:30:41.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2023-10-11T16:30:41.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-10-11T16:30:41.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-10-11T16:30:41.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-10-11T16:30:41.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: osdmap e88: 12 total, 12 up, 12 in 2023-10-11T16:30:41.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1875096587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:41.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3809853712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:41.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-10-11T16:30:41.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:41 smithi150 ceph-mon[132217]: osdmap e89: 12 total, 12 up, 12 in 2023-10-11T16:30:41.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.468+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f671c061cb0 con 0x7f67080a1260 2023-10-11T16:30:41.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.483+0000 7f6712ffd700 1 -- 172.21.15.111:0/1981913342 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423860 (secure 0 0 0) 0x7f671c061cb0 con 0x7f67080a1260 2023-10-11T16:30:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:41.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67080a1260 msgr2=0x7f67080a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67080a1260 0x7f67080a3710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f671c074850 tx=0x7f670c009260 comp rx=0 tx=0).stop 2023-10-11T16:30:41.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f671c073550 msgr2=0x7f671c19c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f671c073550 0x7f671c19c600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f67180098e0 tx=0x7f6718004730 comp rx=0 tx=0).stop 2023-10-11T16:30:41.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 shutdown_connections 2023-10-11T16:30:41.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67080a1260 0x7f67080a3710 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f671c10c860 0x7f671c1a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f671c073550 0x7f671c19c600 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 --2- 172.21.15.111:0/1981913342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f671c074b30 0x7f671c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 >> 172.21.15.111:0/1981913342 conn(0x7f671c0fb870 msgr2=0x7f671c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:41.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 shutdown_connections 2023-10-11T16:30:41.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.489+0000 7f6723bff700 1 -- 172.21.15.111:0/1981913342 wait complete. 2023-10-11T16:30:41.498 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:30:41.509 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2023-10-11T16:30:41.180362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597340,"num_objects":56,"num_object_clones":0,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":156,"num_read_kb":134,"num_write":294,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79552,"kb_used_data":11360,"kb_used_omap":0,"kb_used_meta":68096,"kb_avail":1124862272,"statfs":{"total":1151940427776,"available":1151858966528,"internally_reserved":0,"allocated":11632640,"data_stored":7401916,"data_compressed":249989,"data_compressed_allocated":335872,"data_compressed_original":2170880,"omap_allocated":0,"internal_metadata":69730304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2521,"num_objects":32,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":65,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.124599"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284183+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:30:17.284183+0000","last_peered":"2023-10-11T16:30:17.284183+0000","last_clean":"2023-10-11T16:30:17.284183+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:30:17.284183+0000","last_undegraded":"2023-10-11T16:30:17.284183+0000","last_fullsized":"2023-10-11T16:30:17.284183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:26.542965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612056+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:30:16.612056+0000","last_peered":"2023-10-11T16:30:16.612056+0000","last_clean":"2023-10-11T16:30:16.612056+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:30:16.612056+0000","last_undegraded":"2023-10-11T16:30:16.612056+0000","last_fullsized":"2023-10-11T16:30:16.612056+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:08:41.790985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612043+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:30:16.612043+0000","last_peered":"2023-10-11T16:30:16.612043+0000","last_clean":"2023-10-11T16:30:16.612043+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:30:16.612043+0000","last_undegraded":"2023-10-11T16:30:16.612043+0000","last_fullsized":"2023-10-11T16:30:16.612043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:43.188283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668622+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:30:17.668622+0000","last_peered":"2023-10-11T16:30:17.668622+0000","last_clean":"2023-10-11T16:30:17.668622+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:30:17.668622+0000","last_undegraded":"2023-10-11T16:30:17.668622+0000","last_fullsized":"2023-10-11T16:30:17.668622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:33:52.126432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614398+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:30:16.614398+0000","last_peered":"2023-10-11T16:30:16.614398+0000","last_clean":"2023-10-11T16:30:16.614398+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:30:16.614398+0000","last_undegraded":"2023-10-11T16:30:16.614398+0000","last_fullsized":"2023-10-11T16:30:16.614398+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:59.778582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669108+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:30:17.669108+0000","last_peered":"2023-10-11T16:30:17.669108+0000","last_clean":"2023-10-11T16:30:17.669108+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:30:17.669108+0000","last_undegraded":"2023-10-11T16:30:17.669108+0000","last_fullsized":"2023-10-11T16:30:17.669108+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:37:10.884146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612956+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:30:16.612956+0000","last_peered":"2023-10-11T16:30:16.612956+0000","last_clean":"2023-10-11T16:30:16.612956+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:30:16.612956+0000","last_undegraded":"2023-10-11T16:30:16.612956+0000","last_fullsized":"2023-10-11T16:30:16.612956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:14:14.116406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668148+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:30:17.668148+0000","last_peered":"2023-10-11T16:30:17.668148+0000","last_clean":"2023-10-11T16:30:17.668148+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:30:17.668148+0000","last_undegraded":"2023-10-11T16:30:17.668148+0000","last_fullsized":"2023-10-11T16:30:17.668148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:24:57.989531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668680+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:30:17.668680+0000","last_peered":"2023-10-11T16:30:17.668680+0000","last_clean":"2023-10-11T16:30:17.668680+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:30:17.668680+0000","last_undegraded":"2023-10-11T16:30:17.668680+0000","last_fullsized":"2023-10-11T16:30:17.668680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:41:45.811568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668330+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:30:17.668330+0000","last_peered":"2023-10-11T16:30:17.668330+0000","last_clean":"2023-10-11T16:30:17.668330+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:30:17.668330+0000","last_undegraded":"2023-10-11T16:30:17.668330+0000","last_fullsized":"2023-10-11T16:30:17.668330+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:28.841961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613837+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:30:16.613837+0000","last_peered":"2023-10-11T16:30:16.613837+0000","last_clean":"2023-10-11T16:30:16.613837+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:30:16.613837+0000","last_undegraded":"2023-10-11T16:30:16.613837+0000","last_fullsized":"2023-10-11T16:30:16.613837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:13:18.450772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668462+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:30:17.668462+0000","last_peered":"2023-10-11T16:30:17.668462+0000","last_clean":"2023-10-11T16:30:17.668462+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:30:17.668462+0000","last_undegraded":"2023-10-11T16:30:17.668462+0000","last_fullsized":"2023-10-11T16:30:17.668462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:17:32.551928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669875+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:30:17.669875+0000","last_peered":"2023-10-11T16:30:17.669875+0000","last_clean":"2023-10-11T16:30:17.669875+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:30:17.669875+0000","last_undegraded":"2023-10-11T16:30:17.669875+0000","last_fullsized":"2023-10-11T16:30:17.669875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:10:27.344516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611789+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:30:16.611789+0000","last_peered":"2023-10-11T16:30:16.611789+0000","last_clean":"2023-10-11T16:30:16.611789+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:30:16.611789+0000","last_undegraded":"2023-10-11T16:30:16.611789+0000","last_fullsized":"2023-10-11T16:30:16.611789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:34:22.094507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611778+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:30:16.611778+0000","last_peered":"2023-10-11T16:30:16.611778+0000","last_clean":"2023-10-11T16:30:16.611778+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:30:16.611778+0000","last_undegraded":"2023-10-11T16:30:16.611778+0000","last_fullsized":"2023-10-11T16:30:16.611778+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:15.576706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667849+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:30:17.667849+0000","last_peered":"2023-10-11T16:30:17.667849+0000","last_clean":"2023-10-11T16:30:17.667849+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:30:17.667849+0000","last_undegraded":"2023-10-11T16:30:17.667849+0000","last_fullsized":"2023-10-11T16:30:17.667849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:36:36.528407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284123+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:30:17.284123+0000","last_peered":"2023-10-11T16:30:17.284123+0000","last_clean":"2023-10-11T16:30:17.284123+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:30:17.284123+0000","last_undegraded":"2023-10-11T16:30:17.284123+0000","last_fullsized":"2023-10-11T16:30:17.284123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:47:55.247727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669370+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:30:17.669370+0000","last_peered":"2023-10-11T16:30:17.669370+0000","last_clean":"2023-10-11T16:30:17.669370+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:30:17.669370+0000","last_undegraded":"2023-10-11T16:30:17.669370+0000","last_fullsized":"2023-10-11T16:30:17.669370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:02:10.477569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667926+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:30:17.667926+0000","last_peered":"2023-10-11T16:30:17.667926+0000","last_clean":"2023-10-11T16:30:17.667926+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:30:17.667926+0000","last_undegraded":"2023-10-11T16:30:17.667926+0000","last_fullsized":"2023-10-11T16:30:17.667926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:12:46.918768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667256+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:17.667256+0000","last_peered":"2023-10-11T16:30:17.667256+0000","last_clean":"2023-10-11T16:30:17.667256+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:30:17.667256+0000","last_undegraded":"2023-10-11T16:30:17.667256+0000","last_fullsized":"2023-10-11T16:30:17.667256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:02:31.283177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669122+0000","last_change":"2023-10-11T16:30:11.121744+0000","last_active":"2023-10-11T16:30:17.669122+0000","last_peered":"2023-10-11T16:30:17.669122+0000","last_clean":"2023-10-11T16:30:17.669122+0000","last_became_active":"2023-10-11T16:30:11.121370+0000","last_became_peered":"2023-10-11T16:30:11.121370+0000","last_unstale":"2023-10-11T16:30:17.669122+0000","last_undegraded":"2023-10-11T16:30:17.669122+0000","last_fullsized":"2023-10-11T16:30:17.669122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:00:50.716941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668383+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:30:17.668383+0000","last_peered":"2023-10-11T16:30:17.668383+0000","last_clean":"2023-10-11T16:30:17.668383+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:30:17.668383+0000","last_undegraded":"2023-10-11T16:30:17.668383+0000","last_fullsized":"2023-10-11T16:30:17.668383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:10:24.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667627+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:30:17.667627+0000","last_peered":"2023-10-11T16:30:17.667627+0000","last_clean":"2023-10-11T16:30:17.667627+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:30:17.667627+0000","last_undegraded":"2023-10-11T16:30:17.667627+0000","last_fullsized":"2023-10-11T16:30:17.667627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:59:03.310068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668885+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:30:17.668885+0000","last_peered":"2023-10-11T16:30:17.668885+0000","last_clean":"2023-10-11T16:30:17.668885+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:30:17.668885+0000","last_undegraded":"2023-10-11T16:30:17.668885+0000","last_fullsized":"2023-10-11T16:30:17.668885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:29:49.809378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612215+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:30:16.612215+0000","last_peered":"2023-10-11T16:30:16.612215+0000","last_clean":"2023-10-11T16:30:16.612215+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:30:16.612215+0000","last_undegraded":"2023-10-11T16:30:16.612215+0000","last_fullsized":"2023-10-11T16:30:16.612215+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:45:37.863112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668530+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:30:17.668530+0000","last_peered":"2023-10-11T16:30:17.668530+0000","last_clean":"2023-10-11T16:30:17.668530+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:30:17.668530+0000","last_undegraded":"2023-10-11T16:30:17.668530+0000","last_fullsized":"2023-10-11T16:30:17.668530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:47:05.876657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.173667+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:30:37.173667+0000","last_peered":"2023-10-11T16:30:37.173667+0000","last_clean":"2023-10-11T16:30:37.173667+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:30:37.173667+0000","last_undegraded":"2023-10-11T16:30:37.173667+0000","last_fullsized":"2023-10-11T16:30:37.173667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:55:47.819813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668277+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:30:17.668277+0000","last_peered":"2023-10-11T16:30:17.668277+0000","last_clean":"2023-10-11T16:30:17.668277+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:30:17.668277+0000","last_undegraded":"2023-10-11T16:30:17.668277+0000","last_fullsized":"2023-10-11T16:30:17.668277+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:27:30.832381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667964+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:30:17.667964+0000","last_peered":"2023-10-11T16:30:17.667964+0000","last_clean":"2023-10-11T16:30:17.667964+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:30:17.667964+0000","last_undegraded":"2023-10-11T16:30:17.667964+0000","last_fullsized":"2023-10-11T16:30:17.667964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:49:38.616403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614611+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:30:16.614611+0000","last_peered":"2023-10-11T16:30:16.614611+0000","last_clean":"2023-10-11T16:30:16.614611+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:30:16.614611+0000","last_undegraded":"2023-10-11T16:30:16.614611+0000","last_fullsized":"2023-10-11T16:30:16.614611+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:36:36.308880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":87,"state":"peering","last_fresh":"2023-10-11T16:30:17.668751+0000","last_change":"2023-10-11T16:30:40.187333+0000","last_active":"2023-10-11T16:30:40.187333+0000","last_peered":"2023-10-11T16:30:17.668751+0000","last_clean":"2023-10-11T16:30:17.668751+0000","last_became_active":"2023-10-11T16:30:11.658658+0000","last_became_peered":"2023-10-11T16:30:11.658658+0000","last_unstale":"2023-10-11T16:30:17.668751+0000","last_undegraded":"2023-10-11T16:30:17.668751+0000","last_fullsized":"2023-10-11T16:30:17.668751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,8],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667669+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:30:17.667669+0000","last_peered":"2023-10-11T16:30:17.667669+0000","last_clean":"2023-10-11T16:30:17.667669+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:30:17.667669+0000","last_undegraded":"2023-10-11T16:30:17.667669+0000","last_fullsized":"2023-10-11T16:30:17.667669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:18:27.476930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187858+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:30:40.187858+0000","last_peered":"2023-10-11T16:30:40.187858+0000","last_clean":"2023-10-11T16:30:40.187858+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:30:40.187858+0000","last_undegraded":"2023-10-11T16:30:40.187858+0000","last_fullsized":"2023-10-11T16:30:40.187858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:51.299737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614368+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:30:16.614368+0000","last_peered":"2023-10-11T16:30:16.614368+0000","last_clean":"2023-10-11T16:30:16.614368+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:30:16.614368+0000","last_undegraded":"2023-10-11T16:30:16.614368+0000","last_fullsized":"2023-10-11T16:30:16.614368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:59:43.658557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667731+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:30:17.667731+0000","last_peered":"2023-10-11T16:30:17.667731+0000","last_clean":"2023-10-11T16:30:17.667731+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:30:17.667731+0000","last_undegraded":"2023-10-11T16:30:17.667731+0000","last_fullsized":"2023-10-11T16:30:17.667731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:41:13.591057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176047+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:30:37.176047+0000","last_peered":"2023-10-11T16:30:37.176047+0000","last_clean":"2023-10-11T16:30:37.176047+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:30:37.176047+0000","last_undegraded":"2023-10-11T16:30:37.176047+0000","last_fullsized":"2023-10-11T16:30:37.176047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:38:45.180274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.188012+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:30:40.188012+0000","last_peered":"2023-10-11T16:30:40.188012+0000","last_clean":"2023-10-11T16:30:40.188012+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:30:40.188012+0000","last_undegraded":"2023-10-11T16:30:40.188012+0000","last_fullsized":"2023-10-11T16:30:40.188012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:36:53.041380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284122+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:30:17.284122+0000","last_peered":"2023-10-11T16:30:17.284122+0000","last_clean":"2023-10-11T16:30:17.284122+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:30:17.284122+0000","last_undegraded":"2023-10-11T16:30:17.284122+0000","last_fullsized":"2023-10-11T16:30:17.284122+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:35:11.414446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.177276+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:30:37.177276+0000","last_peered":"2023-10-11T16:30:37.177276+0000","last_clean":"2023-10-11T16:30:37.177276+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:30:37.177276+0000","last_undegraded":"2023-10-11T16:30:37.177276+0000","last_fullsized":"2023-10-11T16:30:37.177276+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:44:31.081697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668117+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:30:17.668117+0000","last_peered":"2023-10-11T16:30:17.668117+0000","last_clean":"2023-10-11T16:30:17.668117+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:30:17.668117+0000","last_undegraded":"2023-10-11T16:30:17.668117+0000","last_fullsized":"2023-10-11T16:30:17.668117+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:20:03.123703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612081+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:30:16.612081+0000","last_peered":"2023-10-11T16:30:16.612081+0000","last_clean":"2023-10-11T16:30:16.612081+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:30:16.612081+0000","last_undegraded":"2023-10-11T16:30:16.612081+0000","last_fullsized":"2023-10-11T16:30:16.612081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:44:54.955030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186294+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:30:40.186294+0000","last_peered":"2023-10-11T16:30:40.186294+0000","last_clean":"2023-10-11T16:30:40.186294+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:30:40.186294+0000","last_undegraded":"2023-10-11T16:30:40.186294+0000","last_fullsized":"2023-10-11T16:30:40.186294+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:19:55.442658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668217+0000","last_change":"2023-10-11T16:30:11.662850+0000","last_active":"2023-10-11T16:30:17.668217+0000","last_peered":"2023-10-11T16:30:17.668217+0000","last_clean":"2023-10-11T16:30:17.668217+0000","last_became_active":"2023-10-11T16:30:11.661052+0000","last_became_peered":"2023-10-11T16:30:11.661052+0000","last_unstale":"2023-10-11T16:30:17.668217+0000","last_undegraded":"2023-10-11T16:30:17.668217+0000","last_fullsized":"2023-10-11T16:30:17.668217+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:33:50.024445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667905+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:30:17.667905+0000","last_peered":"2023-10-11T16:30:17.667905+0000","last_clean":"2023-10-11T16:30:17.667905+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:30:17.667905+0000","last_undegraded":"2023-10-11T16:30:17.667905+0000","last_fullsized":"2023-10-11T16:30:17.667905+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:07.189663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166341+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:30:36.166341+0000","last_peered":"2023-10-11T16:30:36.166341+0000","last_clean":"2023-10-11T16:30:36.166341+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:30:36.166341+0000","last_undegraded":"2023-10-11T16:30:36.166341+0000","last_fullsized":"2023-10-11T16:30:36.166341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:36:57.466108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669129+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:30:17.669129+0000","last_peered":"2023-10-11T16:30:17.669129+0000","last_clean":"2023-10-11T16:30:17.669129+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:30:17.669129+0000","last_undegraded":"2023-10-11T16:30:17.669129+0000","last_fullsized":"2023-10-11T16:30:17.669129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:44:17.713245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667932+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:30:17.667932+0000","last_peered":"2023-10-11T16:30:17.667932+0000","last_clean":"2023-10-11T16:30:17.667932+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:30:17.667932+0000","last_undegraded":"2023-10-11T16:30:17.667932+0000","last_fullsized":"2023-10-11T16:30:17.667932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:11:42.295504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165661+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:30:36.165661+0000","last_peered":"2023-10-11T16:30:36.165661+0000","last_clean":"2023-10-11T16:30:36.165661+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:30:36.165661+0000","last_undegraded":"2023-10-11T16:30:36.165661+0000","last_fullsized":"2023-10-11T16:30:36.165661+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:24:23.101409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612235+0000","last_change":"2023-10-11T16:30:11.666124+0000","last_active":"2023-10-11T16:30:16.612235+0000","last_peered":"2023-10-11T16:30:16.612235+0000","last_clean":"2023-10-11T16:30:16.612235+0000","last_became_active":"2023-10-11T16:30:11.665786+0000","last_became_peered":"2023-10-11T16:30:11.665786+0000","last_unstale":"2023-10-11T16:30:16.612235+0000","last_undegraded":"2023-10-11T16:30:16.612235+0000","last_fullsized":"2023-10-11T16:30:16.612235+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:45:32.355550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669108+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:30:17.669108+0000","last_peered":"2023-10-11T16:30:17.669108+0000","last_clean":"2023-10-11T16:30:17.669108+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:30:17.669108+0000","last_undegraded":"2023-10-11T16:30:17.669108+0000","last_fullsized":"2023-10-11T16:30:17.669108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:20:20.621738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187532+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:30:40.187532+0000","last_peered":"2023-10-11T16:30:40.187532+0000","last_clean":"2023-10-11T16:30:40.187532+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:30:40.187532+0000","last_undegraded":"2023-10-11T16:30:40.187532+0000","last_fullsized":"2023-10-11T16:30:40.187532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:45:20.227087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668685+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:30:17.668685+0000","last_peered":"2023-10-11T16:30:17.668685+0000","last_clean":"2023-10-11T16:30:17.668685+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:30:17.668685+0000","last_undegraded":"2023-10-11T16:30:17.668685+0000","last_fullsized":"2023-10-11T16:30:17.668685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:02:09.993229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667958+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:30:17.667958+0000","last_peered":"2023-10-11T16:30:17.667958+0000","last_clean":"2023-10-11T16:30:17.667958+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:30:17.667958+0000","last_undegraded":"2023-10-11T16:30:17.667958+0000","last_fullsized":"2023-10-11T16:30:17.667958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:33.934527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'12","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165516+0000","last_change":"2023-10-11T16:30:11.112780+0000","last_active":"2023-10-11T16:30:36.165516+0000","last_peered":"2023-10-11T16:30:36.165516+0000","last_clean":"2023-10-11T16:30:36.165516+0000","last_became_active":"2023-10-11T16:30:11.112481+0000","last_became_peered":"2023-10-11T16:30:11.112481+0000","last_unstale":"2023-10-11T16:30:36.165516+0000","last_undegraded":"2023-10-11T16:30:36.165516+0000","last_fullsized":"2023-10-11T16:30:36.165516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:44.261584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614775+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:30:16.614775+0000","last_peered":"2023-10-11T16:30:16.614775+0000","last_clean":"2023-10-11T16:30:16.614775+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:30:16.614775+0000","last_undegraded":"2023-10-11T16:30:16.614775+0000","last_fullsized":"2023-10-11T16:30:16.614775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:14:04.119677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.189579+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:30:40.189579+0000","last_peered":"2023-10-11T16:30:40.189579+0000","last_clean":"2023-10-11T16:30:40.189579+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:30:40.189579+0000","last_undegraded":"2023-10-11T16:30:40.189579+0000","last_fullsized":"2023-10-11T16:30:40.189579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:26.634024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165984+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:30:36.165984+0000","last_peered":"2023-10-11T16:30:36.165984+0000","last_clean":"2023-10-11T16:30:36.165984+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:30:36.165984+0000","last_undegraded":"2023-10-11T16:30:36.165984+0000","last_fullsized":"2023-10-11T16:30:36.165984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:45:39.491656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668870+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:30:17.668870+0000","last_peered":"2023-10-11T16:30:17.668870+0000","last_clean":"2023-10-11T16:30:17.668870+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:30:17.668870+0000","last_undegraded":"2023-10-11T16:30:17.668870+0000","last_fullsized":"2023-10-11T16:30:17.668870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:15:57.948212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283849+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:30:17.283849+0000","last_peered":"2023-10-11T16:30:17.283849+0000","last_clean":"2023-10-11T16:30:17.283849+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:30:17.283849+0000","last_undegraded":"2023-10-11T16:30:17.283849+0000","last_fullsized":"2023-10-11T16:30:17.283849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:27:49.373457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187910+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:30:40.187910+0000","last_peered":"2023-10-11T16:30:40.187910+0000","last_clean":"2023-10-11T16:30:40.187910+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:30:40.187910+0000","last_undegraded":"2023-10-11T16:30:40.187910+0000","last_fullsized":"2023-10-11T16:30:40.187910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:56:20.300209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613601+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:30:16.613601+0000","last_peered":"2023-10-11T16:30:16.613601+0000","last_clean":"2023-10-11T16:30:16.613601+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:30:16.613601+0000","last_undegraded":"2023-10-11T16:30:16.613601+0000","last_fullsized":"2023-10-11T16:30:16.613601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:34.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668783+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:17.668783+0000","last_peered":"2023-10-11T16:30:17.668783+0000","last_clean":"2023-10-11T16:30:17.668783+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:30:17.668783+0000","last_undegraded":"2023-10-11T16:30:17.668783+0000","last_fullsized":"2023-10-11T16:30:17.668783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:33:16.429523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:38.189202+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:30:38.189202+0000","last_peered":"2023-10-11T16:30:38.189202+0000","last_clean":"2023-10-11T16:30:38.189202+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:30:38.189202+0000","last_undegraded":"2023-10-11T16:30:38.189202+0000","last_fullsized":"2023-10-11T16:30:38.189202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:25:21.420979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613021+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:30:16.613021+0000","last_peered":"2023-10-11T16:30:16.613021+0000","last_clean":"2023-10-11T16:30:16.613021+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:30:16.613021+0000","last_undegraded":"2023-10-11T16:30:16.613021+0000","last_fullsized":"2023-10-11T16:30:16.613021+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:10:00.701910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669925+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:30:17.669925+0000","last_peered":"2023-10-11T16:30:17.669925+0000","last_clean":"2023-10-11T16:30:17.669925+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:30:17.669925+0000","last_undegraded":"2023-10-11T16:30:17.669925+0000","last_fullsized":"2023-10-11T16:30:17.669925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:46:40.422035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176084+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:30:37.176084+0000","last_peered":"2023-10-11T16:30:37.176084+0000","last_clean":"2023-10-11T16:30:37.176084+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:30:37.176084+0000","last_undegraded":"2023-10-11T16:30:37.176084+0000","last_fullsized":"2023-10-11T16:30:37.176084+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:15.680165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187485+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:30:40.187485+0000","last_peered":"2023-10-11T16:30:40.187485+0000","last_clean":"2023-10-11T16:30:40.187485+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:30:40.187485+0000","last_undegraded":"2023-10-11T16:30:40.187485+0000","last_fullsized":"2023-10-11T16:30:40.187485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:58:29.097161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187329+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:30:40.187329+0000","last_peered":"2023-10-11T16:30:40.187329+0000","last_clean":"2023-10-11T16:30:40.187329+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:30:40.187329+0000","last_undegraded":"2023-10-11T16:30:40.187329+0000","last_fullsized":"2023-10-11T16:30:40.187329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:21:39.470036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.615240+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:30:16.615240+0000","last_peered":"2023-10-11T16:30:16.615240+0000","last_clean":"2023-10-11T16:30:16.615240+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:30:16.615240+0000","last_undegraded":"2023-10-11T16:30:16.615240+0000","last_fullsized":"2023-10-11T16:30:16.615240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:02:48.445205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284055+0000","last_change":"2023-10-11T16:30:11.670547+0000","last_active":"2023-10-11T16:30:17.284055+0000","last_peered":"2023-10-11T16:30:17.284055+0000","last_clean":"2023-10-11T16:30:17.284055+0000","last_became_active":"2023-10-11T16:30:11.669842+0000","last_became_peered":"2023-10-11T16:30:11.669842+0000","last_unstale":"2023-10-11T16:30:17.284055+0000","last_undegraded":"2023-10-11T16:30:17.284055+0000","last_fullsized":"2023-10-11T16:30:17.284055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:20:40.999317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614361+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:30:16.614361+0000","last_peered":"2023-10-11T16:30:16.614361+0000","last_clean":"2023-10-11T16:30:16.614361+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:30:16.614361+0000","last_undegraded":"2023-10-11T16:30:16.614361+0000","last_fullsized":"2023-10-11T16:30:16.614361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:13.782416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669256+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:30:17.669256+0000","last_peered":"2023-10-11T16:30:17.669256+0000","last_clean":"2023-10-11T16:30:17.669256+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:30:17.669256+0000","last_undegraded":"2023-10-11T16:30:17.669256+0000","last_fullsized":"2023-10-11T16:30:17.669256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:10:47.487424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668683+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:30:17.668683+0000","last_peered":"2023-10-11T16:30:17.668683+0000","last_clean":"2023-10-11T16:30:17.668683+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:30:17.668683+0000","last_undegraded":"2023-10-11T16:30:17.668683+0000","last_fullsized":"2023-10-11T16:30:17.668683+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:10:24.647448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669638+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:30:17.669638+0000","last_peered":"2023-10-11T16:30:17.669638+0000","last_clean":"2023-10-11T16:30:17.669638+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:30:17.669638+0000","last_undegraded":"2023-10-11T16:30:17.669638+0000","last_fullsized":"2023-10-11T16:30:17.669638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:12:58.596577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613284+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:30:16.613284+0000","last_peered":"2023-10-11T16:30:16.613284+0000","last_clean":"2023-10-11T16:30:16.613284+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:30:16.613284+0000","last_undegraded":"2023-10-11T16:30:16.613284+0000","last_fullsized":"2023-10-11T16:30:16.613284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:33:51.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668320+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:30:17.668320+0000","last_peered":"2023-10-11T16:30:17.668320+0000","last_clean":"2023-10-11T16:30:17.668320+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:30:17.668320+0000","last_undegraded":"2023-10-11T16:30:17.668320+0000","last_fullsized":"2023-10-11T16:30:17.668320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:32:12.254443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669773+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:30:17.669773+0000","last_peered":"2023-10-11T16:30:17.669773+0000","last_clean":"2023-10-11T16:30:17.669773+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:30:17.669773+0000","last_undegraded":"2023-10-11T16:30:17.669773+0000","last_fullsized":"2023-10-11T16:30:17.669773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:09:37.408810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186478+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:30:40.186478+0000","last_peered":"2023-10-11T16:30:40.186478+0000","last_clean":"2023-10-11T16:30:40.186478+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:30:40.186478+0000","last_undegraded":"2023-10-11T16:30:40.186478+0000","last_fullsized":"2023-10-11T16:30:40.186478+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:23.165604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176027+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:30:37.176027+0000","last_peered":"2023-10-11T16:30:37.176027+0000","last_clean":"2023-10-11T16:30:37.176027+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:30:37.176027+0000","last_undegraded":"2023-10-11T16:30:37.176027+0000","last_fullsized":"2023-10-11T16:30:37.176027+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:26:18.864499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:19.956667+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:30:19.956667+0000","last_peered":"2023-10-11T16:30:19.956667+0000","last_clean":"2023-10-11T16:30:19.956667+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:30:19.956667+0000","last_undegraded":"2023-10-11T16:30:19.956667+0000","last_fullsized":"2023-10-11T16:30:19.956667+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:53:51.765779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667662+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:30:17.667662+0000","last_peered":"2023-10-11T16:30:17.667662+0000","last_clean":"2023-10-11T16:30:17.667662+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:30:17.667662+0000","last_undegraded":"2023-10-11T16:30:17.667662+0000","last_fullsized":"2023-10-11T16:30:17.667662+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:23.176208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:38.183142+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:30:38.183142+0000","last_peered":"2023-10-11T16:30:38.183142+0000","last_clean":"2023-10-11T16:30:38.183142+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:30:38.183142+0000","last_undegraded":"2023-10-11T16:30:38.183142+0000","last_fullsized":"2023-10-11T16:30:38.183142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:02:22.009759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612829+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:30:16.612829+0000","last_peered":"2023-10-11T16:30:16.612829+0000","last_clean":"2023-10-11T16:30:16.612829+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:30:16.612829+0000","last_undegraded":"2023-10-11T16:30:16.612829+0000","last_fullsized":"2023-10-11T16:30:16.612829+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:27:41.571741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187873+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:30:40.187873+0000","last_peered":"2023-10-11T16:30:40.187873+0000","last_clean":"2023-10-11T16:30:40.187873+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:30:40.187873+0000","last_undegraded":"2023-10-11T16:30:40.187873+0000","last_fullsized":"2023-10-11T16:30:40.187873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:05:27.804658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667984+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:30:17.667984+0000","last_peered":"2023-10-11T16:30:17.667984+0000","last_clean":"2023-10-11T16:30:17.667984+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:30:17.667984+0000","last_undegraded":"2023-10-11T16:30:17.667984+0000","last_fullsized":"2023-10-11T16:30:17.667984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:32:59.074750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.173678+0000","last_change":"2023-10-11T16:30:11.117164+0000","last_active":"2023-10-11T16:30:37.173678+0000","last_peered":"2023-10-11T16:30:37.173678+0000","last_clean":"2023-10-11T16:30:37.173678+0000","last_became_active":"2023-10-11T16:30:11.116887+0000","last_became_peered":"2023-10-11T16:30:11.116887+0000","last_unstale":"2023-10-11T16:30:37.173678+0000","last_undegraded":"2023-10-11T16:30:37.173678+0000","last_fullsized":"2023-10-11T16:30:37.173678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:39:08.785806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283813+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:30:17.283813+0000","last_peered":"2023-10-11T16:30:17.283813+0000","last_clean":"2023-10-11T16:30:17.283813+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:30:17.283813+0000","last_undegraded":"2023-10-11T16:30:17.283813+0000","last_fullsized":"2023-10-11T16:30:17.283813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:43.398632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166054+0000","last_change":"2023-10-11T16:30:11.129313+0000","last_active":"2023-10-11T16:30:36.166054+0000","last_peered":"2023-10-11T16:30:36.166054+0000","last_clean":"2023-10-11T16:30:36.166054+0000","last_became_active":"2023-10-11T16:30:11.129051+0000","last_became_peered":"2023-10-11T16:30:11.129051+0000","last_unstale":"2023-10-11T16:30:36.166054+0000","last_undegraded":"2023-10-11T16:30:36.166054+0000","last_fullsized":"2023-10-11T16:30:36.166054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:02:06.341952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669886+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:30:17.669886+0000","last_peered":"2023-10-11T16:30:17.669886+0000","last_clean":"2023-10-11T16:30:17.669886+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:30:17.669886+0000","last_undegraded":"2023-10-11T16:30:17.669886+0000","last_fullsized":"2023-10-11T16:30:17.669886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:47:23.558003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667532+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:30:17.667532+0000","last_peered":"2023-10-11T16:30:17.667532+0000","last_clean":"2023-10-11T16:30:17.667532+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:30:17.667532+0000","last_undegraded":"2023-10-11T16:30:17.667532+0000","last_fullsized":"2023-10-11T16:30:17.667532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:55:02.983828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669364+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:30:17.669364+0000","last_peered":"2023-10-11T16:30:17.669364+0000","last_clean":"2023-10-11T16:30:17.669364+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:30:17.669364+0000","last_undegraded":"2023-10-11T16:30:17.669364+0000","last_fullsized":"2023-10-11T16:30:17.669364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:02:51.786440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668839+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:30:17.668839+0000","last_peered":"2023-10-11T16:30:17.668839+0000","last_clean":"2023-10-11T16:30:17.668839+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:30:17.668839+0000","last_undegraded":"2023-10-11T16:30:17.668839+0000","last_fullsized":"2023-10-11T16:30:17.668839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:55.881039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667028+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:30:17.667028+0000","last_peered":"2023-10-11T16:30:17.667028+0000","last_clean":"2023-10-11T16:30:17.667028+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:30:17.667028+0000","last_undegraded":"2023-10-11T16:30:17.667028+0000","last_fullsized":"2023-10-11T16:30:17.667028+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:18:05.052999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.164814+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:30:36.164814+0000","last_peered":"2023-10-11T16:30:36.164814+0000","last_clean":"2023-10-11T16:30:36.164814+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:30:36.164814+0000","last_undegraded":"2023-10-11T16:30:36.164814+0000","last_fullsized":"2023-10-11T16:30:36.164814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:09:01.841380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612174+0000","last_change":"2023-10-11T16:30:11.661388+0000","last_active":"2023-10-11T16:30:16.612174+0000","last_peered":"2023-10-11T16:30:16.612174+0000","last_clean":"2023-10-11T16:30:16.612174+0000","last_became_active":"2023-10-11T16:30:11.661073+0000","last_became_peered":"2023-10-11T16:30:11.661073+0000","last_unstale":"2023-10-11T16:30:16.612174+0000","last_undegraded":"2023-10-11T16:30:16.612174+0000","last_fullsized":"2023-10-11T16:30:16.612174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:10:29.755199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668643+0000","last_change":"2023-10-11T16:30:16.171535+0000","last_active":"2023-10-11T16:30:17.668643+0000","last_peered":"2023-10-11T16:30:17.668643+0000","last_clean":"2023-10-11T16:30:17.668643+0000","last_became_active":"2023-10-11T16:30:16.171051+0000","last_became_peered":"2023-10-11T16:30:16.171051+0000","last_unstale":"2023-10-11T16:30:17.668643+0000","last_undegraded":"2023-10-11T16:30:17.668643+0000","last_fullsized":"2023-10-11T16:30:17.668643+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:36:53.674481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.164950+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:30:36.164950+0000","last_peered":"2023-10-11T16:30:36.164950+0000","last_clean":"2023-10-11T16:30:36.164950+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:30:36.164950+0000","last_undegraded":"2023-10-11T16:30:36.164950+0000","last_fullsized":"2023-10-11T16:30:36.164950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:06:21.073342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614260+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:30:16.614260+0000","last_peered":"2023-10-11T16:30:16.614260+0000","last_clean":"2023-10-11T16:30:16.614260+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:30:16.614260+0000","last_undegraded":"2023-10-11T16:30:16.614260+0000","last_fullsized":"2023-10-11T16:30:16.614260+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:28:45.994452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668995+0000","last_change":"2023-10-11T16:30:16.131117+0000","last_active":"2023-10-11T16:30:17.668995+0000","last_peered":"2023-10-11T16:30:17.668995+0000","last_clean":"2023-10-11T16:30:17.668995+0000","last_became_active":"2023-10-11T16:30:16.130776+0000","last_became_peered":"2023-10-11T16:30:16.130776+0000","last_unstale":"2023-10-11T16:30:17.668995+0000","last_undegraded":"2023-10-11T16:30:17.668995+0000","last_fullsized":"2023-10-11T16:30:17.668995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:24:15.931186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166640+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:30:36.166640+0000","last_peered":"2023-10-11T16:30:36.166640+0000","last_clean":"2023-10-11T16:30:36.166640+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:30:36.166640+0000","last_undegraded":"2023-10-11T16:30:36.166640+0000","last_fullsized":"2023-10-11T16:30:36.166640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:12:26.496915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283586+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:30:17.283586+0000","last_peered":"2023-10-11T16:30:17.283586+0000","last_clean":"2023-10-11T16:30:17.283586+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:30:17.283586+0000","last_undegraded":"2023-10-11T16:30:17.283586+0000","last_fullsized":"2023-10-11T16:30:17.283586+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:10:18.200065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:19.940473+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:30:19.940473+0000","last_peered":"2023-10-11T16:30:19.940473+0000","last_clean":"2023-10-11T16:30:19.940473+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:30:19.940473+0000","last_undegraded":"2023-10-11T16:30:19.940473+0000","last_fullsized":"2023-10-11T16:30:19.940473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:39.565360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667561+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:30:17.667561+0000","last_peered":"2023-10-11T16:30:17.667561+0000","last_clean":"2023-10-11T16:30:17.667561+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:30:17.667561+0000","last_undegraded":"2023-10-11T16:30:17.667561+0000","last_fullsized":"2023-10-11T16:30:17.667561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:26:26.416627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667992+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:30:17.667992+0000","last_peered":"2023-10-11T16:30:17.667992+0000","last_clean":"2023-10-11T16:30:17.667992+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:30:17.667992+0000","last_undegraded":"2023-10-11T16:30:17.667992+0000","last_fullsized":"2023-10-11T16:30:17.667992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:18:04.314307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669109+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:30:17.669109+0000","last_peered":"2023-10-11T16:30:17.669109+0000","last_clean":"2023-10-11T16:30:17.669109+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:30:17.669109+0000","last_undegraded":"2023-10-11T16:30:17.669109+0000","last_fullsized":"2023-10-11T16:30:17.669109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:39:11.522213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.167157+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:30:36.167157+0000","last_peered":"2023-10-11T16:30:36.167157+0000","last_clean":"2023-10-11T16:30:36.167157+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:30:36.167157+0000","last_undegraded":"2023-10-11T16:30:36.167157+0000","last_fullsized":"2023-10-11T16:30:36.167157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:06.688871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613531+0000","last_change":"2023-10-11T16:30:11.648776+0000","last_active":"2023-10-11T16:30:16.613531+0000","last_peered":"2023-10-11T16:30:16.613531+0000","last_clean":"2023-10-11T16:30:16.613531+0000","last_became_active":"2023-10-11T16:30:11.648465+0000","last_became_peered":"2023-10-11T16:30:11.648465+0000","last_unstale":"2023-10-11T16:30:16.613531+0000","last_undegraded":"2023-10-11T16:30:16.613531+0000","last_fullsized":"2023-10-11T16:30:16.613531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:24:06.619577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667818+0000","last_change":"2023-10-11T16:30:16.129703+0000","last_active":"2023-10-11T16:30:17.667818+0000","last_peered":"2023-10-11T16:30:17.667818+0000","last_clean":"2023-10-11T16:30:17.667818+0000","last_became_active":"2023-10-11T16:30:16.129231+0000","last_became_peered":"2023-10-11T16:30:16.129231+0000","last_unstale":"2023-10-11T16:30:17.667818+0000","last_undegraded":"2023-10-11T16:30:17.667818+0000","last_fullsized":"2023-10-11T16:30:17.667818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:47:29.769705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667910+0000","last_change":"2023-10-11T16:30:11.127305+0000","last_active":"2023-10-11T16:30:17.667910+0000","last_peered":"2023-10-11T16:30:17.667910+0000","last_clean":"2023-10-11T16:30:17.667910+0000","last_became_active":"2023-10-11T16:30:11.126986+0000","last_became_peered":"2023-10-11T16:30:11.126986+0000","last_unstale":"2023-10-11T16:30:17.667910+0000","last_undegraded":"2023-10-11T16:30:17.667910+0000","last_fullsized":"2023-10-11T16:30:17.667910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:33:23.033612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613872+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:30:16.613872+0000","last_peered":"2023-10-11T16:30:16.613872+0000","last_clean":"2023-10-11T16:30:16.613872+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:30:16.613872+0000","last_undegraded":"2023-10-11T16:30:16.613872+0000","last_fullsized":"2023-10-11T16:30:16.613872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:49.187595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612035+0000","last_change":"2023-10-11T16:30:16.135709+0000","last_active":"2023-10-11T16:30:16.612035+0000","last_peered":"2023-10-11T16:30:16.612035+0000","last_clean":"2023-10-11T16:30:16.612035+0000","last_became_active":"2023-10-11T16:30:16.135294+0000","last_became_peered":"2023-10-11T16:30:16.135294+0000","last_unstale":"2023-10-11T16:30:16.612035+0000","last_undegraded":"2023-10-11T16:30:16.612035+0000","last_fullsized":"2023-10-11T16:30:16.612035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:50:08.906768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.189235+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:30:40.189235+0000","last_peered":"2023-10-11T16:30:40.189235+0000","last_clean":"2023-10-11T16:30:40.189235+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:30:40.189235+0000","last_undegraded":"2023-10-11T16:30:40.189235+0000","last_fullsized":"2023-10-11T16:30:40.189235+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:38:09.758963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613738+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:30:16.613738+0000","last_peered":"2023-10-11T16:30:16.613738+0000","last_clean":"2023-10-11T16:30:16.613738+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:30:16.613738+0000","last_undegraded":"2023-10-11T16:30:16.613738+0000","last_fullsized":"2023-10-11T16:30:16.613738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:04:56.528036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186085+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:30:40.186085+0000","last_peered":"2023-10-11T16:30:40.186085+0000","last_clean":"2023-10-11T16:30:40.186085+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:30:40.186085+0000","last_undegraded":"2023-10-11T16:30:40.186085+0000","last_fullsized":"2023-10-11T16:30:40.186085+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:23:35.338891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669638+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:30:17.669638+0000","last_peered":"2023-10-11T16:30:17.669638+0000","last_clean":"2023-10-11T16:30:17.669638+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:30:17.669638+0000","last_undegraded":"2023-10-11T16:30:17.669638+0000","last_fullsized":"2023-10-11T16:30:17.669638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:25:06.130705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613643+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:30:16.613643+0000","last_peered":"2023-10-11T16:30:16.613643+0000","last_clean":"2023-10-11T16:30:16.613643+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:30:16.613643+0000","last_undegraded":"2023-10-11T16:30:16.613643+0000","last_fullsized":"2023-10-11T16:30:16.613643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:11:20.783196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283653+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:30:17.283653+0000","last_peered":"2023-10-11T16:30:17.283653+0000","last_clean":"2023-10-11T16:30:17.283653+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:30:17.283653+0000","last_undegraded":"2023-10-11T16:30:17.283653+0000","last_fullsized":"2023-10-11T16:30:17.283653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:05:33.114445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667198+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:30:17.667198+0000","last_peered":"2023-10-11T16:30:17.667198+0000","last_clean":"2023-10-11T16:30:17.667198+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:30:17.667198+0000","last_undegraded":"2023-10-11T16:30:17.667198+0000","last_fullsized":"2023-10-11T16:30:17.667198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:03:56.127687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667360+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:30:17.667360+0000","last_peered":"2023-10-11T16:30:17.667360+0000","last_clean":"2023-10-11T16:30:17.667360+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:30:17.667360+0000","last_undegraded":"2023-10-11T16:30:17.667360+0000","last_fullsized":"2023-10-11T16:30:17.667360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:33:42.559787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611986+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:30:16.611986+0000","last_peered":"2023-10-11T16:30:16.611986+0000","last_clean":"2023-10-11T16:30:16.611986+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:30:16.611986+0000","last_undegraded":"2023-10-11T16:30:16.611986+0000","last_fullsized":"2023-10-11T16:30:16.611986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:59:39.821639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669892+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:30:17.669892+0000","last_peered":"2023-10-11T16:30:17.669892+0000","last_clean":"2023-10-11T16:30:17.669892+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:30:17.669892+0000","last_undegraded":"2023-10-11T16:30:17.669892+0000","last_fullsized":"2023-10-11T16:30:17.669892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:44.023609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284031+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:30:17.284031+0000","last_peered":"2023-10-11T16:30:17.284031+0000","last_clean":"2023-10-11T16:30:17.284031+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:30:17.284031+0000","last_undegraded":"2023-10-11T16:30:17.284031+0000","last_fullsized":"2023-10-11T16:30:17.284031+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:58.558512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667794+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:30:17.667794+0000","last_peered":"2023-10-11T16:30:17.667794+0000","last_clean":"2023-10-11T16:30:17.667794+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:30:17.667794+0000","last_undegraded":"2023-10-11T16:30:17.667794+0000","last_fullsized":"2023-10-11T16:30:17.667794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:11:57.944077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669256+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:30:17.669256+0000","last_peered":"2023-10-11T16:30:17.669256+0000","last_clean":"2023-10-11T16:30:17.669256+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:30:17.669256+0000","last_undegraded":"2023-10-11T16:30:17.669256+0000","last_fullsized":"2023-10-11T16:30:17.669256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:16:39.680829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186430+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:30:40.186430+0000","last_peered":"2023-10-11T16:30:40.186430+0000","last_clean":"2023-10-11T16:30:40.186430+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:30:40.186430+0000","last_undegraded":"2023-10-11T16:30:40.186430+0000","last_fullsized":"2023-10-11T16:30:40.186430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:27:45.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668100+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:30:17.668100+0000","last_peered":"2023-10-11T16:30:17.668100+0000","last_clean":"2023-10-11T16:30:17.668100+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:30:17.668100+0000","last_undegraded":"2023-10-11T16:30:17.668100+0000","last_fullsized":"2023-10-11T16:30:17.668100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:06:16.966258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669079+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:30:17.669079+0000","last_peered":"2023-10-11T16:30:17.669079+0000","last_clean":"2023-10-11T16:30:17.669079+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:30:17.669079+0000","last_undegraded":"2023-10-11T16:30:17.669079+0000","last_fullsized":"2023-10-11T16:30:17.669079+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:32.524195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669665+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:30:17.669665+0000","last_peered":"2023-10-11T16:30:17.669665+0000","last_clean":"2023-10-11T16:30:17.669665+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:30:17.669665+0000","last_undegraded":"2023-10-11T16:30:17.669665+0000","last_fullsized":"2023-10-11T16:30:17.669665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:17.866867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283801+0000","last_change":"2023-10-11T16:30:11.668522+0000","last_active":"2023-10-11T16:30:17.283801+0000","last_peered":"2023-10-11T16:30:17.283801+0000","last_clean":"2023-10-11T16:30:17.283801+0000","last_became_active":"2023-10-11T16:30:11.654192+0000","last_became_peered":"2023-10-11T16:30:11.654192+0000","last_unstale":"2023-10-11T16:30:17.283801+0000","last_undegraded":"2023-10-11T16:30:17.283801+0000","last_fullsized":"2023-10-11T16:30:17.283801+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:51:10.675716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6726,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":111,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":38577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449105,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":946176,"data_stored":468750,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.373},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.622}]},{"osd":5,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.499},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60399999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.56899999999999995}]},{"osd":7,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.86099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.77300000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.70699999999999996}]}]},{"osd":10,"up_from":71,"seq":304942678037,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6320,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738832,"statfs":{"total":95995035648,"available":95988563968,"internally_reserved":0,"allocated":1024000,"data_stored":1059190,"data_compressed":47399,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56100000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.61599999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81000000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.78600000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":0.89600000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.878},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.86499999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.82699999999999996}]}]},{"osd":9,"up_from":66,"seq":283467841560,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6176,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738976,"statfs":{"total":95995035648,"available":95988711424,"internally_reserved":0,"allocated":942080,"data_stored":464654,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.499}]},{"osd":2,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.52700000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.439}]},{"osd":4,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.66500000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.59999999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.57799999999999996}]},{"osd":8,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.629},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.61599999999999999}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.69999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":8,"up_from":58,"seq":249108103199,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6380,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738772,"statfs":{"total":95995035648,"available":95988502528,"internally_reserved":0,"allocated":954368,"data_stored":464479,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57799999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.625},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.443,"15min":0.44},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.55900000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.55800000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.42299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.505}]},{"osd":6,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65000000000000002,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.66800000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.77200000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73699999999999999}]},{"osd":10,"last update":"Wed Oct 11 16:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":1011712,"data_stored":1064328,"data_compressed":47399,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.33800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.629,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81499999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78000000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.748,"5min":1.103,"15min":1.103},"last":0.74299999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77100000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.628},"min":{"1min":0.45400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79500000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.72499999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.70699999999999996,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.82399999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.71799999999999997,"15min":0.72599999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.85599999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.89800000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.873}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":937984,"data_stored":464072,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.54100000000000004,"15min":0.52300000000000002},"min":{"1min":0.44400000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91500000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.46899999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84599999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.44800000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.65000000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.48599999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94099999999999995,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.52200000000000002}]},{"osd":4,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.52000000000000002,"5min":0.498,"15min":0.498},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.66000000000000003,"15min":0.64900000000000002},"min":{"1min":0.56499999999999995,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.67200000000000004,"15min":0.65700000000000003},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.871,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.76100000000000001,"15min":0.745},"min":{"1min":0.56999999999999995,"5min":0.434,"15min":0.434},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.78100000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:29:55 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.77300000000000002,"15min":0.75700000000000001},"min":{"1min":0.66300000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73199999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.753}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74099999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.84799999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.85699999999999998,"15min":0.86799999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0569999999999999,"5min":1.254,"15min":1.254},"last":0.85899999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.89000000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.877,"15min":0.877},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.872}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7352,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737800,"statfs":{"total":95995035648,"available":95987507200,"internally_reserved":0,"allocated":966656,"data_stored":484916,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57399999999999995,"15min":0.621},"min":{"1min":0.371,"5min":0.251,"15min":0.251},"max":{"1min":0.754,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.53500000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76600000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.505,"15min":0.501},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.50900000000000001,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.307,"15min":0.307},"max":{"1min":0.68600000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.629}]},{"osd":3,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.438,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:29:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41999999999999998,"15min":0.40899999999999997},"min":{"1min":0.33300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.439},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.55100000000000005,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.72199999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71399999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.504,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.84499999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.878},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.82399999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.91600000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.79700000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":465550,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.40899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.497},{"interface":"front","average":{"1min":0.63,"5min":0.55500000000000005,"15min":0.52600000000000002},"min":{"1min":0.441,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60099999999999998,"15min":0.57799999999999996},"min":{"1min":0.38500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.622,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.44500000000000001,"15min":0.433},"min":{"1min":0.35799999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44600000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:29:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57599999999999996,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72699999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64700000000000002,"15min":0.63400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.621}]},{"osd":6,"last update":"Wed Oct 11 16:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.52200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96799999999999997,"5min":0.999,"15min":0.999},"last":0.52000000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.75},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.78400000000000003}]},{"osd":8,"last update":"Wed Oct 11 16:30:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.73199999999999998,"15min":0.73899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75900000000000001},"min":{"1min":0.56599999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91800000000000004,"5min":1.133,"15min":1.133},"last":0.73499999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.76400000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.80500000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:30:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.75800000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7276,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737876,"statfs":{"total":95995035648,"available":95987585024,"internally_reserved":0,"allocated":954368,"data_stored":467900,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79800000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.496}]},{"osd":1,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78300000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.83599999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.755}]},{"osd":3,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56699999999999995,"15min":0.54200000000000004},"min":{"1min":0.49399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59499999999999997,"15min":0.56299999999999994},"min":{"1min":0.47299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74399999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75900000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.81499999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66600000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.49099999999999999,"15min":0.47799999999999998},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Wed Oct 11 16:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.63100000000000001,"15min":0.61899999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.50800000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.80000000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77900000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.874},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.80700000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.86599999999999999}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.627,"15min":0.61599999999999999},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92200000000000004,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72099999999999997}]},{"osd":9,"last update":"Wed Oct 11 16:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.84299999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.88800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6628,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738524,"statfs":{"total":95995035648,"available":95988248576,"internally_reserved":0,"allocated":946176,"data_stored":463935,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.38200000000000001},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69999999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.373},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.40699999999999997},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.40400000000000003}]},{"osd":1,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60599999999999998,"15min":0.56999999999999995},"min":{"1min":0.433,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63500000000000001,"15min":0.58899999999999997},"min":{"1min":0.48199999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64500000000000002,"15min":0.63400000000000001},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90900000000000003,"5min":1.101,"15min":1.101},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69399999999999995,"15min":0.68000000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0509999999999999,"5min":1.194,"15min":1.194},"last":0.73499999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85399999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69199999999999995,"15min":0.68899999999999995},"min":{"1min":0.51300000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91600000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.66800000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:30:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91500000000000004,"5min":1.268,"15min":1.268},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.60999999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84599999999999997,"5min":1.119,"15min":1.119},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.071,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.629}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70699999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.755}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":950272,"data_stored":464668,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.379,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.36899999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.55200000000000005,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79600000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51800000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71999999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.2560000000000002,"5min":4.5,"15min":4.5},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68899999999999995,"15min":0.72999999999999998},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.75900000000000001}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.58299999999999996}]},{"osd":7,"last update":"Wed Oct 11 16:30:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48999999999999999,"15min":0.48199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56000000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.60999999999999999,"15min":0.59499999999999997},"min":{"1min":0.47799999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60599999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.84699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6712,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738440,"statfs":{"total":95995035648,"available":95988162560,"internally_reserved":0,"allocated":1032192,"data_stored":1069474,"data_compressed":50350,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.745,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78700000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.873,"5min":0.874,"15min":0.874},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.70599999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.438,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63200000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.71399999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90600000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.76500000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58899999999999997,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85299999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.99299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78100000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.83899999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70799999999999996,"15min":0.71499999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69599999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.89200000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64600000000000002,"15min":0.64800000000000002},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84299999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79000000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.93600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.97499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:30:41.520 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T16:30:41.581 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b[157084]: 2023-10-11T16:30:41.385+0000 7fe8647ed700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-11T16:30:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: pgmap v287: 129 pgs: 129 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.9 KiB/s wr, 5 op/s 2023-10-11T16:30:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:30:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-10-11T16:30:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2023-10-11T16:30:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2023-10-11T16:30:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-10-11T16:30:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-10-11T16:30:41.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2023-10-11T16:30:41.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2023-10-11T16:30:41.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-10-11T16:30:41.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-10-11T16:30:41.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-10-11T16:30:41.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: osdmap e88: 12 total, 12 up, 12 in 2023-10-11T16:30:41.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1875096587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:30:41.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3809853712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:41.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-10-11T16:30:41.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:41 smithi130 ceph-mon[131859]: osdmap e89: 12 total, 12 up, 12 in 2023-10-11T16:30:41.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 -- 172.21.15.111:0/212266485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae181014a0 msgr2=0x7fae181018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 --2- 172.21.15.111:0/212266485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae181014a0 0x7fae181018f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009990 tx=0x7fae0c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 -- 172.21.15.111:0/212266485 shutdown_connections 2023-10-11T16:30:41.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 --2- 172.21.15.111:0/212266485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae18101e30 0x7fae181086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 --2- 172.21.15.111:0/212266485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae181002b0 0x7fae181006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 --2- 172.21.15.111:0/212266485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae181014a0 0x7fae181018f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 -- 172.21.15.111:0/212266485 >> 172.21.15.111:0/212266485 conn(0x7fae180fb890 msgr2=0x7fae180fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:41.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 -- 172.21.15.111:0/212266485 shutdown_connections 2023-10-11T16:30:41.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.746+0000 7fae1ddff700 1 -- 172.21.15.111:0/212266485 wait complete. 2023-10-11T16:30:41.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.747+0000 7fae1ddff700 1 Processor -- start 2023-10-11T16:30:41.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.747+0000 7fae1ddff700 1 -- start start 2023-10-11T16:30:41.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae1ddff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae181002b0 0x7fae180748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:41.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae1ddff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae181014a0 0x7fae18072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:41.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae1ddff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae18101e30 0x7fae18073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:41.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae1ddff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae18073bc0 con 0x7fae18101e30 2023-10-11T16:30:41.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae1ddff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae18073d30 con 0x7fae181014a0 2023-10-11T16:30:41.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae1ddff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae181a0080 con 0x7fae181002b0 2023-10-11T16:30:41.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae16ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae181014a0 0x7fae18072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:41.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae16ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae181014a0 0x7fae18072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:34324/0 (socket says 172.21.15.111:34324) 2023-10-11T16:30:41.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae16ffd700 1 -- 172.21.15.111:0/3180357327 learned_addr learned my addr 172.21.15.111:0/3180357327 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:41.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae177fe700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae181002b0 0x7fae180748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:41.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae17fff700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae18101e30 0x7fae18073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:41.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae16ffd700 1 -- 172.21.15.111:0/3180357327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae181002b0 msgr2=0x7fae180748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae16ffd700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae181002b0 0x7fae180748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae16ffd700 1 -- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae18101e30 msgr2=0x7fae18073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae16ffd700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae18101e30 0x7fae18073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.748+0000 7fae16ffd700 1 -- 172.21.15.111:0/3180357327 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c0095f0 con 0x7fae181014a0 2023-10-11T16:30:41.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.749+0000 7fae16ffd700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae181014a0 0x7fae18072ee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00c010 tx=0x7fae0c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:41.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.749+0000 7fae14ff9700 1 -- 172.21.15.111:0/3180357327 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c01d070 con 0x7fae181014a0 2023-10-11T16:30:41.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.749+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1810bcf0 con 0x7fae181014a0 2023-10-11T16:30:41.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.749+0000 7fae14ff9700 1 -- 172.21.15.111:0/3180357327 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae0c005030 con 0x7fae181014a0 2023-10-11T16:30:41.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.749+0000 7fae14ff9700 1 -- 172.21.15.111:0/3180357327 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c004b90 con 0x7fae181014a0 2023-10-11T16:30:41.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.749+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1810bb60 con 0x7fae181014a0 2023-10-11T16:30:41.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.750+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae18107850 con 0x7fae181014a0 2023-10-11T16:30:41.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.752+0000 7fae14ff9700 1 -- 172.21.15.111:0/3180357327 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fae0c0051a0 con 0x7fae181014a0 2023-10-11T16:30:41.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.753+0000 7fae14ff9700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fae000a11e0 0x7fae000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:41.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.754+0000 7fae14ff9700 1 -- 172.21.15.111:0/3180357327 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fae0c0ce4a0 con 0x7fae181014a0 2023-10-11T16:30:41.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.754+0000 7fae177fe700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fae000a11e0 0x7fae000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:41.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.754+0000 7fae177fe700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fae000a11e0 0x7fae000a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fae18101300 tx=0x7fae08009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:41.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.756+0000 7fae14ff9700 1 -- 172.21.15.111:0/3180357327 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fae0c0977b0 con 0x7fae181014a0 2023-10-11T16:30:41.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.949+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae180619c0 con 0x7fae000a11e0 2023-10-11T16:30:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.962+0000 7fae14ff9700 1 -- 172.21.15.111:0/3180357327 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423860 (secure 0 0 0) 0x7fae180619c0 con 0x7fae000a11e0 2023-10-11T16:30:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:41.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fae000a11e0 msgr2=0x7fae000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fae000a11e0 0x7fae000a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fae18101300 tx=0x7fae08009260 comp rx=0 tx=0).stop 2023-10-11T16:30:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae181014a0 msgr2=0x7fae18072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae181014a0 0x7fae18072ee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00c010 tx=0x7fae0c004730 comp rx=0 tx=0).stop 2023-10-11T16:30:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 shutdown_connections 2023-10-11T16:30:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fae000a11e0 0x7fae000a3690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae18101e30 0x7fae18073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae181002b0 0x7fae180748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 --2- 172.21.15.111:0/3180357327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae181014a0 0x7fae18072ee0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 >> 172.21.15.111:0/3180357327 conn(0x7fae180fb890 msgr2=0x7fae18104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 shutdown_connections 2023-10-11T16:30:41.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:41.967+0000 7fae1ddff700 1 -- 172.21.15.111:0/3180357327 wait complete. 2023-10-11T16:30:41.975 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:30:41.988 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2023-10-11T16:30:41.180362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597340,"num_objects":56,"num_object_clones":0,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":156,"num_read_kb":134,"num_write":294,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79552,"kb_used_data":11360,"kb_used_omap":0,"kb_used_meta":68096,"kb_avail":1124862272,"statfs":{"total":1151940427776,"available":1151858966528,"internally_reserved":0,"allocated":11632640,"data_stored":7401916,"data_compressed":249989,"data_compressed_allocated":335872,"data_compressed_original":2170880,"omap_allocated":0,"internal_metadata":69730304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2521,"num_objects":32,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":65,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.124599"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284183+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:30:17.284183+0000","last_peered":"2023-10-11T16:30:17.284183+0000","last_clean":"2023-10-11T16:30:17.284183+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:30:17.284183+0000","last_undegraded":"2023-10-11T16:30:17.284183+0000","last_fullsized":"2023-10-11T16:30:17.284183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:26.542965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612056+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:30:16.612056+0000","last_peered":"2023-10-11T16:30:16.612056+0000","last_clean":"2023-10-11T16:30:16.612056+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:30:16.612056+0000","last_undegraded":"2023-10-11T16:30:16.612056+0000","last_fullsized":"2023-10-11T16:30:16.612056+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:08:41.790985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612043+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:30:16.612043+0000","last_peered":"2023-10-11T16:30:16.612043+0000","last_clean":"2023-10-11T16:30:16.612043+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:30:16.612043+0000","last_undegraded":"2023-10-11T16:30:16.612043+0000","last_fullsized":"2023-10-11T16:30:16.612043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:43.188283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668622+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:30:17.668622+0000","last_peered":"2023-10-11T16:30:17.668622+0000","last_clean":"2023-10-11T16:30:17.668622+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:30:17.668622+0000","last_undegraded":"2023-10-11T16:30:17.668622+0000","last_fullsized":"2023-10-11T16:30:17.668622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:33:52.126432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614398+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:30:16.614398+0000","last_peered":"2023-10-11T16:30:16.614398+0000","last_clean":"2023-10-11T16:30:16.614398+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:30:16.614398+0000","last_undegraded":"2023-10-11T16:30:16.614398+0000","last_fullsized":"2023-10-11T16:30:16.614398+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:59.778582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669108+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:30:17.669108+0000","last_peered":"2023-10-11T16:30:17.669108+0000","last_clean":"2023-10-11T16:30:17.669108+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:30:17.669108+0000","last_undegraded":"2023-10-11T16:30:17.669108+0000","last_fullsized":"2023-10-11T16:30:17.669108+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:37:10.884146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612956+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:30:16.612956+0000","last_peered":"2023-10-11T16:30:16.612956+0000","last_clean":"2023-10-11T16:30:16.612956+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:30:16.612956+0000","last_undegraded":"2023-10-11T16:30:16.612956+0000","last_fullsized":"2023-10-11T16:30:16.612956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:14:14.116406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668148+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:30:17.668148+0000","last_peered":"2023-10-11T16:30:17.668148+0000","last_clean":"2023-10-11T16:30:17.668148+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:30:17.668148+0000","last_undegraded":"2023-10-11T16:30:17.668148+0000","last_fullsized":"2023-10-11T16:30:17.668148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:24:57.989531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668680+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:30:17.668680+0000","last_peered":"2023-10-11T16:30:17.668680+0000","last_clean":"2023-10-11T16:30:17.668680+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:30:17.668680+0000","last_undegraded":"2023-10-11T16:30:17.668680+0000","last_fullsized":"2023-10-11T16:30:17.668680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:41:45.811568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668330+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:30:17.668330+0000","last_peered":"2023-10-11T16:30:17.668330+0000","last_clean":"2023-10-11T16:30:17.668330+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:30:17.668330+0000","last_undegraded":"2023-10-11T16:30:17.668330+0000","last_fullsized":"2023-10-11T16:30:17.668330+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:28.841961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613837+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:30:16.613837+0000","last_peered":"2023-10-11T16:30:16.613837+0000","last_clean":"2023-10-11T16:30:16.613837+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:30:16.613837+0000","last_undegraded":"2023-10-11T16:30:16.613837+0000","last_fullsized":"2023-10-11T16:30:16.613837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:13:18.450772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668462+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:30:17.668462+0000","last_peered":"2023-10-11T16:30:17.668462+0000","last_clean":"2023-10-11T16:30:17.668462+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:30:17.668462+0000","last_undegraded":"2023-10-11T16:30:17.668462+0000","last_fullsized":"2023-10-11T16:30:17.668462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:17:32.551928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669875+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:30:17.669875+0000","last_peered":"2023-10-11T16:30:17.669875+0000","last_clean":"2023-10-11T16:30:17.669875+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:30:17.669875+0000","last_undegraded":"2023-10-11T16:30:17.669875+0000","last_fullsized":"2023-10-11T16:30:17.669875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:10:27.344516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611789+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:30:16.611789+0000","last_peered":"2023-10-11T16:30:16.611789+0000","last_clean":"2023-10-11T16:30:16.611789+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:30:16.611789+0000","last_undegraded":"2023-10-11T16:30:16.611789+0000","last_fullsized":"2023-10-11T16:30:16.611789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:34:22.094507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611778+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:30:16.611778+0000","last_peered":"2023-10-11T16:30:16.611778+0000","last_clean":"2023-10-11T16:30:16.611778+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:30:16.611778+0000","last_undegraded":"2023-10-11T16:30:16.611778+0000","last_fullsized":"2023-10-11T16:30:16.611778+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:15.576706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667849+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:30:17.667849+0000","last_peered":"2023-10-11T16:30:17.667849+0000","last_clean":"2023-10-11T16:30:17.667849+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:30:17.667849+0000","last_undegraded":"2023-10-11T16:30:17.667849+0000","last_fullsized":"2023-10-11T16:30:17.667849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:36:36.528407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284123+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:30:17.284123+0000","last_peered":"2023-10-11T16:30:17.284123+0000","last_clean":"2023-10-11T16:30:17.284123+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:30:17.284123+0000","last_undegraded":"2023-10-11T16:30:17.284123+0000","last_fullsized":"2023-10-11T16:30:17.284123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:47:55.247727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669370+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:30:17.669370+0000","last_peered":"2023-10-11T16:30:17.669370+0000","last_clean":"2023-10-11T16:30:17.669370+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:30:17.669370+0000","last_undegraded":"2023-10-11T16:30:17.669370+0000","last_fullsized":"2023-10-11T16:30:17.669370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:02:10.477569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667926+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:30:17.667926+0000","last_peered":"2023-10-11T16:30:17.667926+0000","last_clean":"2023-10-11T16:30:17.667926+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:30:17.667926+0000","last_undegraded":"2023-10-11T16:30:17.667926+0000","last_fullsized":"2023-10-11T16:30:17.667926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:12:46.918768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667256+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:17.667256+0000","last_peered":"2023-10-11T16:30:17.667256+0000","last_clean":"2023-10-11T16:30:17.667256+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:30:17.667256+0000","last_undegraded":"2023-10-11T16:30:17.667256+0000","last_fullsized":"2023-10-11T16:30:17.667256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:02:31.283177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669122+0000","last_change":"2023-10-11T16:30:11.121744+0000","last_active":"2023-10-11T16:30:17.669122+0000","last_peered":"2023-10-11T16:30:17.669122+0000","last_clean":"2023-10-11T16:30:17.669122+0000","last_became_active":"2023-10-11T16:30:11.121370+0000","last_became_peered":"2023-10-11T16:30:11.121370+0000","last_unstale":"2023-10-11T16:30:17.669122+0000","last_undegraded":"2023-10-11T16:30:17.669122+0000","last_fullsized":"2023-10-11T16:30:17.669122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:00:50.716941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668383+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:30:17.668383+0000","last_peered":"2023-10-11T16:30:17.668383+0000","last_clean":"2023-10-11T16:30:17.668383+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:30:17.668383+0000","last_undegraded":"2023-10-11T16:30:17.668383+0000","last_fullsized":"2023-10-11T16:30:17.668383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:10:24.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667627+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:30:17.667627+0000","last_peered":"2023-10-11T16:30:17.667627+0000","last_clean":"2023-10-11T16:30:17.667627+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:30:17.667627+0000","last_undegraded":"2023-10-11T16:30:17.667627+0000","last_fullsized":"2023-10-11T16:30:17.667627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:59:03.310068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668885+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:30:17.668885+0000","last_peered":"2023-10-11T16:30:17.668885+0000","last_clean":"2023-10-11T16:30:17.668885+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:30:17.668885+0000","last_undegraded":"2023-10-11T16:30:17.668885+0000","last_fullsized":"2023-10-11T16:30:17.668885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:29:49.809378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612215+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:30:16.612215+0000","last_peered":"2023-10-11T16:30:16.612215+0000","last_clean":"2023-10-11T16:30:16.612215+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:30:16.612215+0000","last_undegraded":"2023-10-11T16:30:16.612215+0000","last_fullsized":"2023-10-11T16:30:16.612215+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:45:37.863112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668530+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:30:17.668530+0000","last_peered":"2023-10-11T16:30:17.668530+0000","last_clean":"2023-10-11T16:30:17.668530+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:30:17.668530+0000","last_undegraded":"2023-10-11T16:30:17.668530+0000","last_fullsized":"2023-10-11T16:30:17.668530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:47:05.876657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.173667+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:30:37.173667+0000","last_peered":"2023-10-11T16:30:37.173667+0000","last_clean":"2023-10-11T16:30:37.173667+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:30:37.173667+0000","last_undegraded":"2023-10-11T16:30:37.173667+0000","last_fullsized":"2023-10-11T16:30:37.173667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:55:47.819813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668277+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:30:17.668277+0000","last_peered":"2023-10-11T16:30:17.668277+0000","last_clean":"2023-10-11T16:30:17.668277+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:30:17.668277+0000","last_undegraded":"2023-10-11T16:30:17.668277+0000","last_fullsized":"2023-10-11T16:30:17.668277+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:27:30.832381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667964+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:30:17.667964+0000","last_peered":"2023-10-11T16:30:17.667964+0000","last_clean":"2023-10-11T16:30:17.667964+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:30:17.667964+0000","last_undegraded":"2023-10-11T16:30:17.667964+0000","last_fullsized":"2023-10-11T16:30:17.667964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:49:38.616403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614611+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:30:16.614611+0000","last_peered":"2023-10-11T16:30:16.614611+0000","last_clean":"2023-10-11T16:30:16.614611+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:30:16.614611+0000","last_undegraded":"2023-10-11T16:30:16.614611+0000","last_fullsized":"2023-10-11T16:30:16.614611+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:36:36.308880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":87,"state":"peering","last_fresh":"2023-10-11T16:30:17.668751+0000","last_change":"2023-10-11T16:30:40.187333+0000","last_active":"2023-10-11T16:30:40.187333+0000","last_peered":"2023-10-11T16:30:17.668751+0000","last_clean":"2023-10-11T16:30:17.668751+0000","last_became_active":"2023-10-11T16:30:11.658658+0000","last_became_peered":"2023-10-11T16:30:11.658658+0000","last_unstale":"2023-10-11T16:30:17.668751+0000","last_undegraded":"2023-10-11T16:30:17.668751+0000","last_fullsized":"2023-10-11T16:30:17.668751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,8],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667669+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:30:17.667669+0000","last_peered":"2023-10-11T16:30:17.667669+0000","last_clean":"2023-10-11T16:30:17.667669+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:30:17.667669+0000","last_undegraded":"2023-10-11T16:30:17.667669+0000","last_fullsized":"2023-10-11T16:30:17.667669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:18:27.476930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187858+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:30:40.187858+0000","last_peered":"2023-10-11T16:30:40.187858+0000","last_clean":"2023-10-11T16:30:40.187858+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:30:40.187858+0000","last_undegraded":"2023-10-11T16:30:40.187858+0000","last_fullsized":"2023-10-11T16:30:40.187858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:51.299737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614368+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:30:16.614368+0000","last_peered":"2023-10-11T16:30:16.614368+0000","last_clean":"2023-10-11T16:30:16.614368+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:30:16.614368+0000","last_undegraded":"2023-10-11T16:30:16.614368+0000","last_fullsized":"2023-10-11T16:30:16.614368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:59:43.658557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667731+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:30:17.667731+0000","last_peered":"2023-10-11T16:30:17.667731+0000","last_clean":"2023-10-11T16:30:17.667731+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:30:17.667731+0000","last_undegraded":"2023-10-11T16:30:17.667731+0000","last_fullsized":"2023-10-11T16:30:17.667731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:41:13.591057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176047+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:30:37.176047+0000","last_peered":"2023-10-11T16:30:37.176047+0000","last_clean":"2023-10-11T16:30:37.176047+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:30:37.176047+0000","last_undegraded":"2023-10-11T16:30:37.176047+0000","last_fullsized":"2023-10-11T16:30:37.176047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:38:45.180274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.188012+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:30:40.188012+0000","last_peered":"2023-10-11T16:30:40.188012+0000","last_clean":"2023-10-11T16:30:40.188012+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:30:40.188012+0000","last_undegraded":"2023-10-11T16:30:40.188012+0000","last_fullsized":"2023-10-11T16:30:40.188012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:36:53.041380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284122+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:30:17.284122+0000","last_peered":"2023-10-11T16:30:17.284122+0000","last_clean":"2023-10-11T16:30:17.284122+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:30:17.284122+0000","last_undegraded":"2023-10-11T16:30:17.284122+0000","last_fullsized":"2023-10-11T16:30:17.284122+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:35:11.414446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.177276+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:30:37.177276+0000","last_peered":"2023-10-11T16:30:37.177276+0000","last_clean":"2023-10-11T16:30:37.177276+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:30:37.177276+0000","last_undegraded":"2023-10-11T16:30:37.177276+0000","last_fullsized":"2023-10-11T16:30:37.177276+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:44:31.081697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668117+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:30:17.668117+0000","last_peered":"2023-10-11T16:30:17.668117+0000","last_clean":"2023-10-11T16:30:17.668117+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:30:17.668117+0000","last_undegraded":"2023-10-11T16:30:17.668117+0000","last_fullsized":"2023-10-11T16:30:17.668117+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:20:03.123703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612081+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:30:16.612081+0000","last_peered":"2023-10-11T16:30:16.612081+0000","last_clean":"2023-10-11T16:30:16.612081+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:30:16.612081+0000","last_undegraded":"2023-10-11T16:30:16.612081+0000","last_fullsized":"2023-10-11T16:30:16.612081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:44:54.955030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186294+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:30:40.186294+0000","last_peered":"2023-10-11T16:30:40.186294+0000","last_clean":"2023-10-11T16:30:40.186294+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:30:40.186294+0000","last_undegraded":"2023-10-11T16:30:40.186294+0000","last_fullsized":"2023-10-11T16:30:40.186294+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:19:55.442658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668217+0000","last_change":"2023-10-11T16:30:11.662850+0000","last_active":"2023-10-11T16:30:17.668217+0000","last_peered":"2023-10-11T16:30:17.668217+0000","last_clean":"2023-10-11T16:30:17.668217+0000","last_became_active":"2023-10-11T16:30:11.661052+0000","last_became_peered":"2023-10-11T16:30:11.661052+0000","last_unstale":"2023-10-11T16:30:17.668217+0000","last_undegraded":"2023-10-11T16:30:17.668217+0000","last_fullsized":"2023-10-11T16:30:17.668217+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:33:50.024445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667905+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:30:17.667905+0000","last_peered":"2023-10-11T16:30:17.667905+0000","last_clean":"2023-10-11T16:30:17.667905+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:30:17.667905+0000","last_undegraded":"2023-10-11T16:30:17.667905+0000","last_fullsized":"2023-10-11T16:30:17.667905+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:07.189663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166341+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:30:36.166341+0000","last_peered":"2023-10-11T16:30:36.166341+0000","last_clean":"2023-10-11T16:30:36.166341+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:30:36.166341+0000","last_undegraded":"2023-10-11T16:30:36.166341+0000","last_fullsized":"2023-10-11T16:30:36.166341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:36:57.466108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669129+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:30:17.669129+0000","last_peered":"2023-10-11T16:30:17.669129+0000","last_clean":"2023-10-11T16:30:17.669129+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:30:17.669129+0000","last_undegraded":"2023-10-11T16:30:17.669129+0000","last_fullsized":"2023-10-11T16:30:17.669129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:44:17.713245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667932+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:30:17.667932+0000","last_peered":"2023-10-11T16:30:17.667932+0000","last_clean":"2023-10-11T16:30:17.667932+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:30:17.667932+0000","last_undegraded":"2023-10-11T16:30:17.667932+0000","last_fullsized":"2023-10-11T16:30:17.667932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:11:42.295504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165661+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:30:36.165661+0000","last_peered":"2023-10-11T16:30:36.165661+0000","last_clean":"2023-10-11T16:30:36.165661+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:30:36.165661+0000","last_undegraded":"2023-10-11T16:30:36.165661+0000","last_fullsized":"2023-10-11T16:30:36.165661+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:24:23.101409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612235+0000","last_change":"2023-10-11T16:30:11.666124+0000","last_active":"2023-10-11T16:30:16.612235+0000","last_peered":"2023-10-11T16:30:16.612235+0000","last_clean":"2023-10-11T16:30:16.612235+0000","last_became_active":"2023-10-11T16:30:11.665786+0000","last_became_peered":"2023-10-11T16:30:11.665786+0000","last_unstale":"2023-10-11T16:30:16.612235+0000","last_undegraded":"2023-10-11T16:30:16.612235+0000","last_fullsized":"2023-10-11T16:30:16.612235+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:45:32.355550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669108+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:30:17.669108+0000","last_peered":"2023-10-11T16:30:17.669108+0000","last_clean":"2023-10-11T16:30:17.669108+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:30:17.669108+0000","last_undegraded":"2023-10-11T16:30:17.669108+0000","last_fullsized":"2023-10-11T16:30:17.669108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:20:20.621738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187532+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:30:40.187532+0000","last_peered":"2023-10-11T16:30:40.187532+0000","last_clean":"2023-10-11T16:30:40.187532+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:30:40.187532+0000","last_undegraded":"2023-10-11T16:30:40.187532+0000","last_fullsized":"2023-10-11T16:30:40.187532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:45:20.227087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668685+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:30:17.668685+0000","last_peered":"2023-10-11T16:30:17.668685+0000","last_clean":"2023-10-11T16:30:17.668685+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:30:17.668685+0000","last_undegraded":"2023-10-11T16:30:17.668685+0000","last_fullsized":"2023-10-11T16:30:17.668685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:02:09.993229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667958+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:30:17.667958+0000","last_peered":"2023-10-11T16:30:17.667958+0000","last_clean":"2023-10-11T16:30:17.667958+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:30:17.667958+0000","last_undegraded":"2023-10-11T16:30:17.667958+0000","last_fullsized":"2023-10-11T16:30:17.667958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:33.934527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'12","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165516+0000","last_change":"2023-10-11T16:30:11.112780+0000","last_active":"2023-10-11T16:30:36.165516+0000","last_peered":"2023-10-11T16:30:36.165516+0000","last_clean":"2023-10-11T16:30:36.165516+0000","last_became_active":"2023-10-11T16:30:11.112481+0000","last_became_peered":"2023-10-11T16:30:11.112481+0000","last_unstale":"2023-10-11T16:30:36.165516+0000","last_undegraded":"2023-10-11T16:30:36.165516+0000","last_fullsized":"2023-10-11T16:30:36.165516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:44.261584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614775+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:30:16.614775+0000","last_peered":"2023-10-11T16:30:16.614775+0000","last_clean":"2023-10-11T16:30:16.614775+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:30:16.614775+0000","last_undegraded":"2023-10-11T16:30:16.614775+0000","last_fullsized":"2023-10-11T16:30:16.614775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:14:04.119677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.189579+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:30:40.189579+0000","last_peered":"2023-10-11T16:30:40.189579+0000","last_clean":"2023-10-11T16:30:40.189579+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:30:40.189579+0000","last_undegraded":"2023-10-11T16:30:40.189579+0000","last_fullsized":"2023-10-11T16:30:40.189579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:26.634024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165984+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:30:36.165984+0000","last_peered":"2023-10-11T16:30:36.165984+0000","last_clean":"2023-10-11T16:30:36.165984+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:30:36.165984+0000","last_undegraded":"2023-10-11T16:30:36.165984+0000","last_fullsized":"2023-10-11T16:30:36.165984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:45:39.491656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668870+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:30:17.668870+0000","last_peered":"2023-10-11T16:30:17.668870+0000","last_clean":"2023-10-11T16:30:17.668870+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:30:17.668870+0000","last_undegraded":"2023-10-11T16:30:17.668870+0000","last_fullsized":"2023-10-11T16:30:17.668870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:15:57.948212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283849+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:30:17.283849+0000","last_peered":"2023-10-11T16:30:17.283849+0000","last_clean":"2023-10-11T16:30:17.283849+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:30:17.283849+0000","last_undegraded":"2023-10-11T16:30:17.283849+0000","last_fullsized":"2023-10-11T16:30:17.283849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:27:49.373457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187910+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:30:40.187910+0000","last_peered":"2023-10-11T16:30:40.187910+0000","last_clean":"2023-10-11T16:30:40.187910+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:30:40.187910+0000","last_undegraded":"2023-10-11T16:30:40.187910+0000","last_fullsized":"2023-10-11T16:30:40.187910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:56:20.300209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613601+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:30:16.613601+0000","last_peered":"2023-10-11T16:30:16.613601+0000","last_clean":"2023-10-11T16:30:16.613601+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:30:16.613601+0000","last_undegraded":"2023-10-11T16:30:16.613601+0000","last_fullsized":"2023-10-11T16:30:16.613601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:34.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668783+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:17.668783+0000","last_peered":"2023-10-11T16:30:17.668783+0000","last_clean":"2023-10-11T16:30:17.668783+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:30:17.668783+0000","last_undegraded":"2023-10-11T16:30:17.668783+0000","last_fullsized":"2023-10-11T16:30:17.668783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:33:16.429523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:38.189202+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:30:38.189202+0000","last_peered":"2023-10-11T16:30:38.189202+0000","last_clean":"2023-10-11T16:30:38.189202+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:30:38.189202+0000","last_undegraded":"2023-10-11T16:30:38.189202+0000","last_fullsized":"2023-10-11T16:30:38.189202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:25:21.420979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613021+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:30:16.613021+0000","last_peered":"2023-10-11T16:30:16.613021+0000","last_clean":"2023-10-11T16:30:16.613021+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:30:16.613021+0000","last_undegraded":"2023-10-11T16:30:16.613021+0000","last_fullsized":"2023-10-11T16:30:16.613021+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:10:00.701910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669925+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:30:17.669925+0000","last_peered":"2023-10-11T16:30:17.669925+0000","last_clean":"2023-10-11T16:30:17.669925+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:30:17.669925+0000","last_undegraded":"2023-10-11T16:30:17.669925+0000","last_fullsized":"2023-10-11T16:30:17.669925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:46:40.422035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176084+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:30:37.176084+0000","last_peered":"2023-10-11T16:30:37.176084+0000","last_clean":"2023-10-11T16:30:37.176084+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:30:37.176084+0000","last_undegraded":"2023-10-11T16:30:37.176084+0000","last_fullsized":"2023-10-11T16:30:37.176084+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:15.680165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187485+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:30:40.187485+0000","last_peered":"2023-10-11T16:30:40.187485+0000","last_clean":"2023-10-11T16:30:40.187485+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:30:40.187485+0000","last_undegraded":"2023-10-11T16:30:40.187485+0000","last_fullsized":"2023-10-11T16:30:40.187485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:58:29.097161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187329+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:30:40.187329+0000","last_peered":"2023-10-11T16:30:40.187329+0000","last_clean":"2023-10-11T16:30:40.187329+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:30:40.187329+0000","last_undegraded":"2023-10-11T16:30:40.187329+0000","last_fullsized":"2023-10-11T16:30:40.187329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:21:39.470036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.615240+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:30:16.615240+0000","last_peered":"2023-10-11T16:30:16.615240+0000","last_clean":"2023-10-11T16:30:16.615240+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:30:16.615240+0000","last_undegraded":"2023-10-11T16:30:16.615240+0000","last_fullsized":"2023-10-11T16:30:16.615240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:02:48.445205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284055+0000","last_change":"2023-10-11T16:30:11.670547+0000","last_active":"2023-10-11T16:30:17.284055+0000","last_peered":"2023-10-11T16:30:17.284055+0000","last_clean":"2023-10-11T16:30:17.284055+0000","last_became_active":"2023-10-11T16:30:11.669842+0000","last_became_peered":"2023-10-11T16:30:11.669842+0000","last_unstale":"2023-10-11T16:30:17.284055+0000","last_undegraded":"2023-10-11T16:30:17.284055+0000","last_fullsized":"2023-10-11T16:30:17.284055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:20:40.999317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614361+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:30:16.614361+0000","last_peered":"2023-10-11T16:30:16.614361+0000","last_clean":"2023-10-11T16:30:16.614361+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:30:16.614361+0000","last_undegraded":"2023-10-11T16:30:16.614361+0000","last_fullsized":"2023-10-11T16:30:16.614361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:13.782416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669256+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:30:17.669256+0000","last_peered":"2023-10-11T16:30:17.669256+0000","last_clean":"2023-10-11T16:30:17.669256+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:30:17.669256+0000","last_undegraded":"2023-10-11T16:30:17.669256+0000","last_fullsized":"2023-10-11T16:30:17.669256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:10:47.487424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668683+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:30:17.668683+0000","last_peered":"2023-10-11T16:30:17.668683+0000","last_clean":"2023-10-11T16:30:17.668683+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:30:17.668683+0000","last_undegraded":"2023-10-11T16:30:17.668683+0000","last_fullsized":"2023-10-11T16:30:17.668683+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:10:24.647448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669638+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:30:17.669638+0000","last_peered":"2023-10-11T16:30:17.669638+0000","last_clean":"2023-10-11T16:30:17.669638+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:30:17.669638+0000","last_undegraded":"2023-10-11T16:30:17.669638+0000","last_fullsized":"2023-10-11T16:30:17.669638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:12:58.596577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613284+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:30:16.613284+0000","last_peered":"2023-10-11T16:30:16.613284+0000","last_clean":"2023-10-11T16:30:16.613284+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:30:16.613284+0000","last_undegraded":"2023-10-11T16:30:16.613284+0000","last_fullsized":"2023-10-11T16:30:16.613284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:33:51.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668320+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:30:17.668320+0000","last_peered":"2023-10-11T16:30:17.668320+0000","last_clean":"2023-10-11T16:30:17.668320+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:30:17.668320+0000","last_undegraded":"2023-10-11T16:30:17.668320+0000","last_fullsized":"2023-10-11T16:30:17.668320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:32:12.254443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669773+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:30:17.669773+0000","last_peered":"2023-10-11T16:30:17.669773+0000","last_clean":"2023-10-11T16:30:17.669773+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:30:17.669773+0000","last_undegraded":"2023-10-11T16:30:17.669773+0000","last_fullsized":"2023-10-11T16:30:17.669773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:09:37.408810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186478+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:30:40.186478+0000","last_peered":"2023-10-11T16:30:40.186478+0000","last_clean":"2023-10-11T16:30:40.186478+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:30:40.186478+0000","last_undegraded":"2023-10-11T16:30:40.186478+0000","last_fullsized":"2023-10-11T16:30:40.186478+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:23.165604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176027+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:30:37.176027+0000","last_peered":"2023-10-11T16:30:37.176027+0000","last_clean":"2023-10-11T16:30:37.176027+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:30:37.176027+0000","last_undegraded":"2023-10-11T16:30:37.176027+0000","last_fullsized":"2023-10-11T16:30:37.176027+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:26:18.864499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:19.956667+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:30:19.956667+0000","last_peered":"2023-10-11T16:30:19.956667+0000","last_clean":"2023-10-11T16:30:19.956667+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:30:19.956667+0000","last_undegraded":"2023-10-11T16:30:19.956667+0000","last_fullsized":"2023-10-11T16:30:19.956667+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:53:51.765779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667662+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:30:17.667662+0000","last_peered":"2023-10-11T16:30:17.667662+0000","last_clean":"2023-10-11T16:30:17.667662+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:30:17.667662+0000","last_undegraded":"2023-10-11T16:30:17.667662+0000","last_fullsized":"2023-10-11T16:30:17.667662+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:23.176208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:38.183142+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:30:38.183142+0000","last_peered":"2023-10-11T16:30:38.183142+0000","last_clean":"2023-10-11T16:30:38.183142+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:30:38.183142+0000","last_undegraded":"2023-10-11T16:30:38.183142+0000","last_fullsized":"2023-10-11T16:30:38.183142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:02:22.009759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612829+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:30:16.612829+0000","last_peered":"2023-10-11T16:30:16.612829+0000","last_clean":"2023-10-11T16:30:16.612829+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:30:16.612829+0000","last_undegraded":"2023-10-11T16:30:16.612829+0000","last_fullsized":"2023-10-11T16:30:16.612829+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:27:41.571741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187873+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:30:40.187873+0000","last_peered":"2023-10-11T16:30:40.187873+0000","last_clean":"2023-10-11T16:30:40.187873+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:30:40.187873+0000","last_undegraded":"2023-10-11T16:30:40.187873+0000","last_fullsized":"2023-10-11T16:30:40.187873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:05:27.804658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667984+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:30:17.667984+0000","last_peered":"2023-10-11T16:30:17.667984+0000","last_clean":"2023-10-11T16:30:17.667984+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:30:17.667984+0000","last_undegraded":"2023-10-11T16:30:17.667984+0000","last_fullsized":"2023-10-11T16:30:17.667984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:32:59.074750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.173678+0000","last_change":"2023-10-11T16:30:11.117164+0000","last_active":"2023-10-11T16:30:37.173678+0000","last_peered":"2023-10-11T16:30:37.173678+0000","last_clean":"2023-10-11T16:30:37.173678+0000","last_became_active":"2023-10-11T16:30:11.116887+0000","last_became_peered":"2023-10-11T16:30:11.116887+0000","last_unstale":"2023-10-11T16:30:37.173678+0000","last_undegraded":"2023-10-11T16:30:37.173678+0000","last_fullsized":"2023-10-11T16:30:37.173678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:39:08.785806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283813+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:30:17.283813+0000","last_peered":"2023-10-11T16:30:17.283813+0000","last_clean":"2023-10-11T16:30:17.283813+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:30:17.283813+0000","last_undegraded":"2023-10-11T16:30:17.283813+0000","last_fullsized":"2023-10-11T16:30:17.283813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:43.398632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166054+0000","last_change":"2023-10-11T16:30:11.129313+0000","last_active":"2023-10-11T16:30:36.166054+0000","last_peered":"2023-10-11T16:30:36.166054+0000","last_clean":"2023-10-11T16:30:36.166054+0000","last_became_active":"2023-10-11T16:30:11.129051+0000","last_became_peered":"2023-10-11T16:30:11.129051+0000","last_unstale":"2023-10-11T16:30:36.166054+0000","last_undegraded":"2023-10-11T16:30:36.166054+0000","last_fullsized":"2023-10-11T16:30:36.166054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:02:06.341952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669886+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:30:17.669886+0000","last_peered":"2023-10-11T16:30:17.669886+0000","last_clean":"2023-10-11T16:30:17.669886+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:30:17.669886+0000","last_undegraded":"2023-10-11T16:30:17.669886+0000","last_fullsized":"2023-10-11T16:30:17.669886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:47:23.558003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667532+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:30:17.667532+0000","last_peered":"2023-10-11T16:30:17.667532+0000","last_clean":"2023-10-11T16:30:17.667532+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:30:17.667532+0000","last_undegraded":"2023-10-11T16:30:17.667532+0000","last_fullsized":"2023-10-11T16:30:17.667532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:55:02.983828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669364+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:30:17.669364+0000","last_peered":"2023-10-11T16:30:17.669364+0000","last_clean":"2023-10-11T16:30:17.669364+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:30:17.669364+0000","last_undegraded":"2023-10-11T16:30:17.669364+0000","last_fullsized":"2023-10-11T16:30:17.669364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:02:51.786440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668839+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:30:17.668839+0000","last_peered":"2023-10-11T16:30:17.668839+0000","last_clean":"2023-10-11T16:30:17.668839+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:30:17.668839+0000","last_undegraded":"2023-10-11T16:30:17.668839+0000","last_fullsized":"2023-10-11T16:30:17.668839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:55.881039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667028+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:30:17.667028+0000","last_peered":"2023-10-11T16:30:17.667028+0000","last_clean":"2023-10-11T16:30:17.667028+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:30:17.667028+0000","last_undegraded":"2023-10-11T16:30:17.667028+0000","last_fullsized":"2023-10-11T16:30:17.667028+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:18:05.052999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.164814+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:30:36.164814+0000","last_peered":"2023-10-11T16:30:36.164814+0000","last_clean":"2023-10-11T16:30:36.164814+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:30:36.164814+0000","last_undegraded":"2023-10-11T16:30:36.164814+0000","last_fullsized":"2023-10-11T16:30:36.164814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:09:01.841380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612174+0000","last_change":"2023-10-11T16:30:11.661388+0000","last_active":"2023-10-11T16:30:16.612174+0000","last_peered":"2023-10-11T16:30:16.612174+0000","last_clean":"2023-10-11T16:30:16.612174+0000","last_became_active":"2023-10-11T16:30:11.661073+0000","last_became_peered":"2023-10-11T16:30:11.661073+0000","last_unstale":"2023-10-11T16:30:16.612174+0000","last_undegraded":"2023-10-11T16:30:16.612174+0000","last_fullsized":"2023-10-11T16:30:16.612174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:10:29.755199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668643+0000","last_change":"2023-10-11T16:30:16.171535+0000","last_active":"2023-10-11T16:30:17.668643+0000","last_peered":"2023-10-11T16:30:17.668643+0000","last_clean":"2023-10-11T16:30:17.668643+0000","last_became_active":"2023-10-11T16:30:16.171051+0000","last_became_peered":"2023-10-11T16:30:16.171051+0000","last_unstale":"2023-10-11T16:30:17.668643+0000","last_undegraded":"2023-10-11T16:30:17.668643+0000","last_fullsized":"2023-10-11T16:30:17.668643+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:36:53.674481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.164950+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:30:36.164950+0000","last_peered":"2023-10-11T16:30:36.164950+0000","last_clean":"2023-10-11T16:30:36.164950+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:30:36.164950+0000","last_undegraded":"2023-10-11T16:30:36.164950+0000","last_fullsized":"2023-10-11T16:30:36.164950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:06:21.073342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614260+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:30:16.614260+0000","last_peered":"2023-10-11T16:30:16.614260+0000","last_clean":"2023-10-11T16:30:16.614260+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:30:16.614260+0000","last_undegraded":"2023-10-11T16:30:16.614260+0000","last_fullsized":"2023-10-11T16:30:16.614260+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:28:45.994452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668995+0000","last_change":"2023-10-11T16:30:16.131117+0000","last_active":"2023-10-11T16:30:17.668995+0000","last_peered":"2023-10-11T16:30:17.668995+0000","last_clean":"2023-10-11T16:30:17.668995+0000","last_became_active":"2023-10-11T16:30:16.130776+0000","last_became_peered":"2023-10-11T16:30:16.130776+0000","last_unstale":"2023-10-11T16:30:17.668995+0000","last_undegraded":"2023-10-11T16:30:17.668995+0000","last_fullsized":"2023-10-11T16:30:17.668995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:24:15.931186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166640+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:30:36.166640+0000","last_peered":"2023-10-11T16:30:36.166640+0000","last_clean":"2023-10-11T16:30:36.166640+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:30:36.166640+0000","last_undegraded":"2023-10-11T16:30:36.166640+0000","last_fullsized":"2023-10-11T16:30:36.166640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:12:26.496915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283586+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:30:17.283586+0000","last_peered":"2023-10-11T16:30:17.283586+0000","last_clean":"2023-10-11T16:30:17.283586+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:30:17.283586+0000","last_undegraded":"2023-10-11T16:30:17.283586+0000","last_fullsized":"2023-10-11T16:30:17.283586+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:10:18.200065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:19.940473+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:30:19.940473+0000","last_peered":"2023-10-11T16:30:19.940473+0000","last_clean":"2023-10-11T16:30:19.940473+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:30:19.940473+0000","last_undegraded":"2023-10-11T16:30:19.940473+0000","last_fullsized":"2023-10-11T16:30:19.940473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:39.565360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667561+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:30:17.667561+0000","last_peered":"2023-10-11T16:30:17.667561+0000","last_clean":"2023-10-11T16:30:17.667561+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:30:17.667561+0000","last_undegraded":"2023-10-11T16:30:17.667561+0000","last_fullsized":"2023-10-11T16:30:17.667561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:26:26.416627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667992+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:30:17.667992+0000","last_peered":"2023-10-11T16:30:17.667992+0000","last_clean":"2023-10-11T16:30:17.667992+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:30:17.667992+0000","last_undegraded":"2023-10-11T16:30:17.667992+0000","last_fullsized":"2023-10-11T16:30:17.667992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:18:04.314307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669109+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:30:17.669109+0000","last_peered":"2023-10-11T16:30:17.669109+0000","last_clean":"2023-10-11T16:30:17.669109+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:30:17.669109+0000","last_undegraded":"2023-10-11T16:30:17.669109+0000","last_fullsized":"2023-10-11T16:30:17.669109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:39:11.522213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.167157+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:30:36.167157+0000","last_peered":"2023-10-11T16:30:36.167157+0000","last_clean":"2023-10-11T16:30:36.167157+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:30:36.167157+0000","last_undegraded":"2023-10-11T16:30:36.167157+0000","last_fullsized":"2023-10-11T16:30:36.167157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:06.688871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613531+0000","last_change":"2023-10-11T16:30:11.648776+0000","last_active":"2023-10-11T16:30:16.613531+0000","last_peered":"2023-10-11T16:30:16.613531+0000","last_clean":"2023-10-11T16:30:16.613531+0000","last_became_active":"2023-10-11T16:30:11.648465+0000","last_became_peered":"2023-10-11T16:30:11.648465+0000","last_unstale":"2023-10-11T16:30:16.613531+0000","last_undegraded":"2023-10-11T16:30:16.613531+0000","last_fullsized":"2023-10-11T16:30:16.613531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:24:06.619577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667818+0000","last_change":"2023-10-11T16:30:16.129703+0000","last_active":"2023-10-11T16:30:17.667818+0000","last_peered":"2023-10-11T16:30:17.667818+0000","last_clean":"2023-10-11T16:30:17.667818+0000","last_became_active":"2023-10-11T16:30:16.129231+0000","last_became_peered":"2023-10-11T16:30:16.129231+0000","last_unstale":"2023-10-11T16:30:17.667818+0000","last_undegraded":"2023-10-11T16:30:17.667818+0000","last_fullsized":"2023-10-11T16:30:17.667818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:47:29.769705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667910+0000","last_change":"2023-10-11T16:30:11.127305+0000","last_active":"2023-10-11T16:30:17.667910+0000","last_peered":"2023-10-11T16:30:17.667910+0000","last_clean":"2023-10-11T16:30:17.667910+0000","last_became_active":"2023-10-11T16:30:11.126986+0000","last_became_peered":"2023-10-11T16:30:11.126986+0000","last_unstale":"2023-10-11T16:30:17.667910+0000","last_undegraded":"2023-10-11T16:30:17.667910+0000","last_fullsized":"2023-10-11T16:30:17.667910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:33:23.033612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613872+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:30:16.613872+0000","last_peered":"2023-10-11T16:30:16.613872+0000","last_clean":"2023-10-11T16:30:16.613872+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:30:16.613872+0000","last_undegraded":"2023-10-11T16:30:16.613872+0000","last_fullsized":"2023-10-11T16:30:16.613872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:49.187595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612035+0000","last_change":"2023-10-11T16:30:16.135709+0000","last_active":"2023-10-11T16:30:16.612035+0000","last_peered":"2023-10-11T16:30:16.612035+0000","last_clean":"2023-10-11T16:30:16.612035+0000","last_became_active":"2023-10-11T16:30:16.135294+0000","last_became_peered":"2023-10-11T16:30:16.135294+0000","last_unstale":"2023-10-11T16:30:16.612035+0000","last_undegraded":"2023-10-11T16:30:16.612035+0000","last_fullsized":"2023-10-11T16:30:16.612035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:50:08.906768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.189235+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:30:40.189235+0000","last_peered":"2023-10-11T16:30:40.189235+0000","last_clean":"2023-10-11T16:30:40.189235+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:30:40.189235+0000","last_undegraded":"2023-10-11T16:30:40.189235+0000","last_fullsized":"2023-10-11T16:30:40.189235+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:38:09.758963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613738+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:30:16.613738+0000","last_peered":"2023-10-11T16:30:16.613738+0000","last_clean":"2023-10-11T16:30:16.613738+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:30:16.613738+0000","last_undegraded":"2023-10-11T16:30:16.613738+0000","last_fullsized":"2023-10-11T16:30:16.613738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:04:56.528036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186085+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:30:40.186085+0000","last_peered":"2023-10-11T16:30:40.186085+0000","last_clean":"2023-10-11T16:30:40.186085+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:30:40.186085+0000","last_undegraded":"2023-10-11T16:30:40.186085+0000","last_fullsized":"2023-10-11T16:30:40.186085+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:23:35.338891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669638+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:30:17.669638+0000","last_peered":"2023-10-11T16:30:17.669638+0000","last_clean":"2023-10-11T16:30:17.669638+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:30:17.669638+0000","last_undegraded":"2023-10-11T16:30:17.669638+0000","last_fullsized":"2023-10-11T16:30:17.669638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:25:06.130705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613643+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:30:16.613643+0000","last_peered":"2023-10-11T16:30:16.613643+0000","last_clean":"2023-10-11T16:30:16.613643+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:30:16.613643+0000","last_undegraded":"2023-10-11T16:30:16.613643+0000","last_fullsized":"2023-10-11T16:30:16.613643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:11:20.783196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283653+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:30:17.283653+0000","last_peered":"2023-10-11T16:30:17.283653+0000","last_clean":"2023-10-11T16:30:17.283653+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:30:17.283653+0000","last_undegraded":"2023-10-11T16:30:17.283653+0000","last_fullsized":"2023-10-11T16:30:17.283653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:05:33.114445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667198+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:30:17.667198+0000","last_peered":"2023-10-11T16:30:17.667198+0000","last_clean":"2023-10-11T16:30:17.667198+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:30:17.667198+0000","last_undegraded":"2023-10-11T16:30:17.667198+0000","last_fullsized":"2023-10-11T16:30:17.667198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:03:56.127687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667360+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:30:17.667360+0000","last_peered":"2023-10-11T16:30:17.667360+0000","last_clean":"2023-10-11T16:30:17.667360+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:30:17.667360+0000","last_undegraded":"2023-10-11T16:30:17.667360+0000","last_fullsized":"2023-10-11T16:30:17.667360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:33:42.559787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611986+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:30:16.611986+0000","last_peered":"2023-10-11T16:30:16.611986+0000","last_clean":"2023-10-11T16:30:16.611986+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:30:16.611986+0000","last_undegraded":"2023-10-11T16:30:16.611986+0000","last_fullsized":"2023-10-11T16:30:16.611986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:59:39.821639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669892+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:30:17.669892+0000","last_peered":"2023-10-11T16:30:17.669892+0000","last_clean":"2023-10-11T16:30:17.669892+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:30:17.669892+0000","last_undegraded":"2023-10-11T16:30:17.669892+0000","last_fullsized":"2023-10-11T16:30:17.669892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:44.023609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284031+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:30:17.284031+0000","last_peered":"2023-10-11T16:30:17.284031+0000","last_clean":"2023-10-11T16:30:17.284031+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:30:17.284031+0000","last_undegraded":"2023-10-11T16:30:17.284031+0000","last_fullsized":"2023-10-11T16:30:17.284031+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:58.558512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667794+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:30:17.667794+0000","last_peered":"2023-10-11T16:30:17.667794+0000","last_clean":"2023-10-11T16:30:17.667794+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:30:17.667794+0000","last_undegraded":"2023-10-11T16:30:17.667794+0000","last_fullsized":"2023-10-11T16:30:17.667794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:11:57.944077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669256+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:30:17.669256+0000","last_peered":"2023-10-11T16:30:17.669256+0000","last_clean":"2023-10-11T16:30:17.669256+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:30:17.669256+0000","last_undegraded":"2023-10-11T16:30:17.669256+0000","last_fullsized":"2023-10-11T16:30:17.669256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:16:39.680829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186430+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:30:40.186430+0000","last_peered":"2023-10-11T16:30:40.186430+0000","last_clean":"2023-10-11T16:30:40.186430+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:30:40.186430+0000","last_undegraded":"2023-10-11T16:30:40.186430+0000","last_fullsized":"2023-10-11T16:30:40.186430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:27:45.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668100+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:30:17.668100+0000","last_peered":"2023-10-11T16:30:17.668100+0000","last_clean":"2023-10-11T16:30:17.668100+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:30:17.668100+0000","last_undegraded":"2023-10-11T16:30:17.668100+0000","last_fullsized":"2023-10-11T16:30:17.668100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:06:16.966258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669079+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:30:17.669079+0000","last_peered":"2023-10-11T16:30:17.669079+0000","last_clean":"2023-10-11T16:30:17.669079+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:30:17.669079+0000","last_undegraded":"2023-10-11T16:30:17.669079+0000","last_fullsized":"2023-10-11T16:30:17.669079+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:32.524195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669665+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:30:17.669665+0000","last_peered":"2023-10-11T16:30:17.669665+0000","last_clean":"2023-10-11T16:30:17.669665+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:30:17.669665+0000","last_undegraded":"2023-10-11T16:30:17.669665+0000","last_fullsized":"2023-10-11T16:30:17.669665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:17.866867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283801+0000","last_change":"2023-10-11T16:30:11.668522+0000","last_active":"2023-10-11T16:30:17.283801+0000","last_peered":"2023-10-11T16:30:17.283801+0000","last_clean":"2023-10-11T16:30:17.283801+0000","last_became_active":"2023-10-11T16:30:11.654192+0000","last_became_peered":"2023-10-11T16:30:11.654192+0000","last_unstale":"2023-10-11T16:30:17.283801+0000","last_undegraded":"2023-10-11T16:30:17.283801+0000","last_fullsized":"2023-10-11T16:30:17.283801+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:51:10.675716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6726,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":111,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":38577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449105,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":946176,"data_stored":468750,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.373},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.622}]},{"osd":5,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.499},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60399999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.56899999999999995}]},{"osd":7,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.86099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.77300000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.70699999999999996}]}]},{"osd":10,"up_from":71,"seq":304942678037,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6320,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738832,"statfs":{"total":95995035648,"available":95988563968,"internally_reserved":0,"allocated":1024000,"data_stored":1059190,"data_compressed":47399,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56100000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.61599999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81000000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.78600000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":0.89600000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.878},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.86499999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.82699999999999996}]}]},{"osd":9,"up_from":66,"seq":283467841560,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6176,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738976,"statfs":{"total":95995035648,"available":95988711424,"internally_reserved":0,"allocated":942080,"data_stored":464654,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.499}]},{"osd":2,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.52700000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.439}]},{"osd":4,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.66500000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.59999999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.57799999999999996}]},{"osd":8,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.629},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.61599999999999999}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.69999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":8,"up_from":58,"seq":249108103199,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6380,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738772,"statfs":{"total":95995035648,"available":95988502528,"internally_reserved":0,"allocated":954368,"data_stored":464479,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57799999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.625},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.443,"15min":0.44},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.55900000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.55800000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.42299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.505}]},{"osd":6,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65000000000000002,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.66800000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.77200000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73699999999999999}]},{"osd":10,"last update":"Wed Oct 11 16:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":1011712,"data_stored":1064328,"data_compressed":47399,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.33800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.629,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81499999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78000000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.748,"5min":1.103,"15min":1.103},"last":0.74299999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77100000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.628},"min":{"1min":0.45400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79500000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.72499999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.70699999999999996,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.82399999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.71799999999999997,"15min":0.72599999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.85599999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.89800000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.873}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":937984,"data_stored":464072,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.54100000000000004,"15min":0.52300000000000002},"min":{"1min":0.44400000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91500000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.46899999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84599999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.44800000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.65000000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.48599999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94099999999999995,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.52200000000000002}]},{"osd":4,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.52000000000000002,"5min":0.498,"15min":0.498},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.66000000000000003,"15min":0.64900000000000002},"min":{"1min":0.56499999999999995,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.67200000000000004,"15min":0.65700000000000003},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.871,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.76100000000000001,"15min":0.745},"min":{"1min":0.56999999999999995,"5min":0.434,"15min":0.434},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.78100000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:29:55 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.77300000000000002,"15min":0.75700000000000001},"min":{"1min":0.66300000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73199999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.753}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74099999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.84799999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.85699999999999998,"15min":0.86799999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0569999999999999,"5min":1.254,"15min":1.254},"last":0.85899999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.89000000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.877,"15min":0.877},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.872}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7352,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737800,"statfs":{"total":95995035648,"available":95987507200,"internally_reserved":0,"allocated":966656,"data_stored":484916,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57399999999999995,"15min":0.621},"min":{"1min":0.371,"5min":0.251,"15min":0.251},"max":{"1min":0.754,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.53500000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76600000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.505,"15min":0.501},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.50900000000000001,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.307,"15min":0.307},"max":{"1min":0.68600000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.629}]},{"osd":3,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.438,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:29:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41999999999999998,"15min":0.40899999999999997},"min":{"1min":0.33300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.439},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.55100000000000005,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.72199999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71399999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.504,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.84499999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.878},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.82399999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.91600000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.79700000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":465550,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.40899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.497},{"interface":"front","average":{"1min":0.63,"5min":0.55500000000000005,"15min":0.52600000000000002},"min":{"1min":0.441,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60099999999999998,"15min":0.57799999999999996},"min":{"1min":0.38500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.622,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.44500000000000001,"15min":0.433},"min":{"1min":0.35799999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44600000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:29:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57599999999999996,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72699999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64700000000000002,"15min":0.63400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.621}]},{"osd":6,"last update":"Wed Oct 11 16:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.52200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96799999999999997,"5min":0.999,"15min":0.999},"last":0.52000000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.75},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.78400000000000003}]},{"osd":8,"last update":"Wed Oct 11 16:30:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.73199999999999998,"15min":0.73899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75900000000000001},"min":{"1min":0.56599999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91800000000000004,"5min":1.133,"15min":1.133},"last":0.73499999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.76400000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.80500000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:30:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.75800000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7276,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737876,"statfs":{"total":95995035648,"available":95987585024,"internally_reserved":0,"allocated":954368,"data_stored":467900,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79800000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.496}]},{"osd":1,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78300000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.83599999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.755}]},{"osd":3,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56699999999999995,"15min":0.54200000000000004},"min":{"1min":0.49399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59499999999999997,"15min":0.56299999999999994},"min":{"1min":0.47299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74399999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75900000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.81499999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66600000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.49099999999999999,"15min":0.47799999999999998},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Wed Oct 11 16:29:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.63100000000000001,"15min":0.61899999999999999},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.50800000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.80000000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77900000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.874},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.80700000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.86599999999999999}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.627,"15min":0.61599999999999999},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92200000000000004,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72099999999999997}]},{"osd":9,"last update":"Wed Oct 11 16:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.84299999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.88800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6628,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738524,"statfs":{"total":95995035648,"available":95988248576,"internally_reserved":0,"allocated":946176,"data_stored":463935,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.38200000000000001},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69999999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.373},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.40699999999999997},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.40400000000000003}]},{"osd":1,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60599999999999998,"15min":0.56999999999999995},"min":{"1min":0.433,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63500000000000001,"15min":0.58899999999999997},"min":{"1min":0.48199999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64500000000000002,"15min":0.63400000000000001},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90900000000000003,"5min":1.101,"15min":1.101},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69399999999999995,"15min":0.68000000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0509999999999999,"5min":1.194,"15min":1.194},"last":0.73499999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85399999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69199999999999995,"15min":0.68899999999999995},"min":{"1min":0.51300000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91600000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.66800000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:30:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91500000000000004,"5min":1.268,"15min":1.268},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.60999999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84599999999999997,"5min":1.119,"15min":1.119},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.071,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.629}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70699999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.755}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":950272,"data_stored":464668,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.379,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.36899999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.55200000000000005,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79600000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51800000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71999999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.2560000000000002,"5min":4.5,"15min":4.5},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68899999999999995,"15min":0.72999999999999998},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.75900000000000001}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.58299999999999996}]},{"osd":7,"last update":"Wed Oct 11 16:30:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48999999999999999,"15min":0.48199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56000000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.60999999999999999,"15min":0.59499999999999997},"min":{"1min":0.47799999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60599999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.84699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6712,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738440,"statfs":{"total":95995035648,"available":95988162560,"internally_reserved":0,"allocated":1032192,"data_stored":1069474,"data_compressed":50350,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.745,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78700000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.873,"5min":0.874,"15min":0.874},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.70599999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.438,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63200000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.71399999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90600000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.76500000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58899999999999997,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85299999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.99299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78100000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.83899999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70799999999999996,"15min":0.71499999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69599999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.89200000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64600000000000002,"15min":0.64800000000000002},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84299999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79000000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.93600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.97499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:30:41.999 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T16:30:42.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.227+0000 7efc20bff700 1 -- 172.21.15.111:0/482983782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1002b0 msgr2=0x7efc1c1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.227+0000 7efc20bff700 1 --2- 172.21.15.111:0/482983782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1002b0 0x7efc1c1006c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009990 tx=0x7efc0c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.228+0000 7efc20bff700 1 -- 172.21.15.111:0/482983782 shutdown_connections 2023-10-11T16:30:42.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.228+0000 7efc20bff700 1 --2- 172.21.15.111:0/482983782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc1c101e30 0x7efc1c1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.228+0000 7efc20bff700 1 --2- 172.21.15.111:0/482983782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1002b0 0x7efc1c1006c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.228+0000 7efc20bff700 1 --2- 172.21.15.111:0/482983782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1c1014a0 0x7efc1c1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.228+0000 7efc20bff700 1 -- 172.21.15.111:0/482983782 >> 172.21.15.111:0/482983782 conn(0x7efc1c0fb890 msgr2=0x7efc1c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:42.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.228+0000 7efc20bff700 1 -- 172.21.15.111:0/482983782 shutdown_connections 2023-10-11T16:30:42.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.228+0000 7efc20bff700 1 -- 172.21.15.111:0/482983782 wait complete. 2023-10-11T16:30:42.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc20bff700 1 Processor -- start 2023-10-11T16:30:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc20bff700 1 -- start start 2023-10-11T16:30:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc20bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc1c1002b0 0x7efc1c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc20bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1014a0 0x7efc1c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:42.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc20bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1c101e30 0x7efc1c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc20bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c19dbc0 con 0x7efc1c1014a0 2023-10-11T16:30:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc20bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c19dd00 con 0x7efc1c1002b0 2023-10-11T16:30:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc20bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c19de90 con 0x7efc1c101e30 2023-10-11T16:30:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc13fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1014a0 0x7efc1c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:42.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc13fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1014a0 0x7efc1c196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49810/0 (socket says 172.21.15.111:49810) 2023-10-11T16:30:42.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc13fff700 1 -- 172.21.15.111:0/3129915258 learned_addr learned my addr 172.21.15.111:0/3129915258 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:42.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.229+0000 7efc1a59c700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc1c1002b0 0x7efc1c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:42.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc1ad9d700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1c101e30 0x7efc1c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:42.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc13fff700 1 -- 172.21.15.111:0/3129915258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1c101e30 msgr2=0x7efc1c19d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc13fff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1c101e30 0x7efc1c19d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc13fff700 1 -- 172.21.15.111:0/3129915258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc1c1002b0 msgr2=0x7efc1c196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc13fff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc1c1002b0 0x7efc1c196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc13fff700 1 -- 172.21.15.111:0/3129915258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c0095f0 con 0x7efc1c1014a0 2023-10-11T16:30:42.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc13fff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1014a0 0x7efc1c196580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efc0400d950 tx=0x7efc0400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:42.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc137fe700 1 -- 172.21.15.111:0/3129915258 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc04009940 con 0x7efc1c1014a0 2023-10-11T16:30:42.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc137fe700 1 -- 172.21.15.111:0/3129915258 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc04010460 con 0x7efc1c1014a0 2023-10-11T16:30:42.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c10bc80 con 0x7efc1c1014a0 2023-10-11T16:30:42.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc137fe700 1 -- 172.21.15.111:0/3129915258 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0400f620 con 0x7efc1c1014a0 2023-10-11T16:30:42.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.230+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c10baf0 con 0x7efc1c1014a0 2023-10-11T16:30:42.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.231+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c1078c0 con 0x7efc1c1014a0 2023-10-11T16:30:42.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.233+0000 7efc137fe700 1 -- 172.21.15.111:0/3129915258 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7efc040108d0 con 0x7efc1c1014a0 2023-10-11T16:30:42.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.235+0000 7efc137fe700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7efc080a11e0 0x7efc080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:42.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.235+0000 7efc137fe700 1 -- 172.21.15.111:0/3129915258 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efc040ccc20 con 0x7efc1c1014a0 2023-10-11T16:30:42.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.235+0000 7efc1a59c700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7efc080a11e0 0x7efc080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:42.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.235+0000 7efc1a59c700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7efc080a11e0 0x7efc080a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00b5c0 tx=0x7efc0c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:42.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.239+0000 7efc137fe700 1 -- 172.21.15.111:0/3129915258 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efc04095f00 con 0x7efc1c1014a0 2023-10-11T16:30:42.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:42 smithi111 ceph-mon[98527]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:42.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.432+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc1c061a10 con 0x7efc080a11e0 2023-10-11T16:30:42.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.444+0000 7efc137fe700 1 -- 172.21.15.111:0/3129915258 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423887 (secure 0 0 0) 0x7efc1c061a10 con 0x7efc080a11e0 2023-10-11T16:30:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:42.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.449+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7efc080a11e0 msgr2=0x7efc080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7efc080a11e0 0x7efc080a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00b5c0 tx=0x7efc0c005fd0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1014a0 msgr2=0x7efc1c196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1014a0 0x7efc1c196580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efc0400d950 tx=0x7efc0400dc60 comp rx=0 tx=0).stop 2023-10-11T16:30:42.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 shutdown_connections 2023-10-11T16:30:42.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7efc080a11e0 0x7efc080a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1c101e30 0x7efc1c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efc1c1002b0 0x7efc1c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 --2- 172.21.15.111:0/3129915258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc1c1014a0 0x7efc1c196580 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 >> 172.21.15.111:0/3129915258 conn(0x7efc1c0fb890 msgr2=0x7efc1c104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 shutdown_connections 2023-10-11T16:30:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.450+0000 7efc20bff700 1 -- 172.21.15.111:0/3129915258 wait complete. 2023-10-11T16:30:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:30:42.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:42 smithi150 ceph-mon[132217]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:42.470 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2023-10-11T16:30:42.058445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597340,"num_objects":60,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":157,"num_read_kb":135,"num_write":308,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":248,"ondisk_log_size":248,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":416,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79848,"kb_used_data":11464,"kb_used_omap":0,"kb_used_meta":68288,"kb_avail":1124861976,"statfs":{"total":1151940427776,"available":1151858663424,"internally_reserved":0,"allocated":11739136,"data_stored":7465353,"data_compressed":255891,"data_compressed_allocated":344064,"data_compressed_original":2187264,"omap_allocated":0,"internal_metadata":69926912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2521,"num_objects":36,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":79,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002222"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284183+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:30:17.284183+0000","last_peered":"2023-10-11T16:30:17.284183+0000","last_clean":"2023-10-11T16:30:17.284183+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:30:17.284183+0000","last_undegraded":"2023-10-11T16:30:17.284183+0000","last_fullsized":"2023-10-11T16:30:17.284183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:26.542965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612056+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:30:16.612056+0000","last_peered":"2023-10-11T16:30:16.612056+0000","last_clean":"2023-10-11T16:30:16.612056+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:30:16.612056+0000","last_undegraded":"2023-10-11T16:30:16.612056+0000","last_fullsized":"2023-10-11T16:30:16.612056+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:08:41.790985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612043+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:30:16.612043+0000","last_peered":"2023-10-11T16:30:16.612043+0000","last_clean":"2023-10-11T16:30:16.612043+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:30:16.612043+0000","last_undegraded":"2023-10-11T16:30:16.612043+0000","last_fullsized":"2023-10-11T16:30:16.612043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:43.188283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668622+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:30:17.668622+0000","last_peered":"2023-10-11T16:30:17.668622+0000","last_clean":"2023-10-11T16:30:17.668622+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:30:17.668622+0000","last_undegraded":"2023-10-11T16:30:17.668622+0000","last_fullsized":"2023-10-11T16:30:17.668622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:33:52.126432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614398+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:30:16.614398+0000","last_peered":"2023-10-11T16:30:16.614398+0000","last_clean":"2023-10-11T16:30:16.614398+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:30:16.614398+0000","last_undegraded":"2023-10-11T16:30:16.614398+0000","last_fullsized":"2023-10-11T16:30:16.614398+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:59.778582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669108+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:30:17.669108+0000","last_peered":"2023-10-11T16:30:17.669108+0000","last_clean":"2023-10-11T16:30:17.669108+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:30:17.669108+0000","last_undegraded":"2023-10-11T16:30:17.669108+0000","last_fullsized":"2023-10-11T16:30:17.669108+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:37:10.884146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612956+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:30:16.612956+0000","last_peered":"2023-10-11T16:30:16.612956+0000","last_clean":"2023-10-11T16:30:16.612956+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:30:16.612956+0000","last_undegraded":"2023-10-11T16:30:16.612956+0000","last_fullsized":"2023-10-11T16:30:16.612956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:14:14.116406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668148+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:30:17.668148+0000","last_peered":"2023-10-11T16:30:17.668148+0000","last_clean":"2023-10-11T16:30:17.668148+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:30:17.668148+0000","last_undegraded":"2023-10-11T16:30:17.668148+0000","last_fullsized":"2023-10-11T16:30:17.668148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:24:57.989531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668680+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:30:17.668680+0000","last_peered":"2023-10-11T16:30:17.668680+0000","last_clean":"2023-10-11T16:30:17.668680+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:30:17.668680+0000","last_undegraded":"2023-10-11T16:30:17.668680+0000","last_fullsized":"2023-10-11T16:30:17.668680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:41:45.811568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668330+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:30:17.668330+0000","last_peered":"2023-10-11T16:30:17.668330+0000","last_clean":"2023-10-11T16:30:17.668330+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:30:17.668330+0000","last_undegraded":"2023-10-11T16:30:17.668330+0000","last_fullsized":"2023-10-11T16:30:17.668330+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:28.841961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613837+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:30:16.613837+0000","last_peered":"2023-10-11T16:30:16.613837+0000","last_clean":"2023-10-11T16:30:16.613837+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:30:16.613837+0000","last_undegraded":"2023-10-11T16:30:16.613837+0000","last_fullsized":"2023-10-11T16:30:16.613837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:13:18.450772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668462+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:30:17.668462+0000","last_peered":"2023-10-11T16:30:17.668462+0000","last_clean":"2023-10-11T16:30:17.668462+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:30:17.668462+0000","last_undegraded":"2023-10-11T16:30:17.668462+0000","last_fullsized":"2023-10-11T16:30:17.668462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:17:32.551928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669875+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:30:17.669875+0000","last_peered":"2023-10-11T16:30:17.669875+0000","last_clean":"2023-10-11T16:30:17.669875+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:30:17.669875+0000","last_undegraded":"2023-10-11T16:30:17.669875+0000","last_fullsized":"2023-10-11T16:30:17.669875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:10:27.344516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611789+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:30:16.611789+0000","last_peered":"2023-10-11T16:30:16.611789+0000","last_clean":"2023-10-11T16:30:16.611789+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:30:16.611789+0000","last_undegraded":"2023-10-11T16:30:16.611789+0000","last_fullsized":"2023-10-11T16:30:16.611789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:34:22.094507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611778+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:30:16.611778+0000","last_peered":"2023-10-11T16:30:16.611778+0000","last_clean":"2023-10-11T16:30:16.611778+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:30:16.611778+0000","last_undegraded":"2023-10-11T16:30:16.611778+0000","last_fullsized":"2023-10-11T16:30:16.611778+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:15.576706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667849+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:30:17.667849+0000","last_peered":"2023-10-11T16:30:17.667849+0000","last_clean":"2023-10-11T16:30:17.667849+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:30:17.667849+0000","last_undegraded":"2023-10-11T16:30:17.667849+0000","last_fullsized":"2023-10-11T16:30:17.667849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:36:36.528407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284123+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:30:17.284123+0000","last_peered":"2023-10-11T16:30:17.284123+0000","last_clean":"2023-10-11T16:30:17.284123+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:30:17.284123+0000","last_undegraded":"2023-10-11T16:30:17.284123+0000","last_fullsized":"2023-10-11T16:30:17.284123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:47:55.247727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182794+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:30:41.182794+0000","last_peered":"2023-10-11T16:30:41.182794+0000","last_clean":"2023-10-11T16:30:41.182794+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:30:41.182794+0000","last_undegraded":"2023-10-11T16:30:41.182794+0000","last_fullsized":"2023-10-11T16:30:41.182794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:09:27.215051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185311+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:30:41.185311+0000","last_peered":"2023-10-11T16:30:41.185311+0000","last_clean":"2023-10-11T16:30:41.185311+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:30:41.185311+0000","last_undegraded":"2023-10-11T16:30:41.185311+0000","last_fullsized":"2023-10-11T16:30:41.185311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:46:35.027109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186180+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.186180+0000","last_peered":"2023-10-11T16:30:41.186180+0000","last_clean":"2023-10-11T16:30:41.186180+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:30:41.186180+0000","last_undegraded":"2023-10-11T16:30:41.186180+0000","last_fullsized":"2023-10-11T16:30:41.186180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:05.403679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198854+0000","last_change":"2023-10-11T16:30:41.198854+0000","last_active":"2023-10-11T16:30:41.198854+0000","last_peered":"2023-10-11T16:30:41.198854+0000","last_clean":"2023-10-11T16:30:41.198854+0000","last_became_active":"2023-10-11T16:30:41.189677+0000","last_became_peered":"2023-10-11T16:30:41.189677+0000","last_unstale":"2023-10-11T16:30:41.198854+0000","last_undegraded":"2023-10-11T16:30:41.198854+0000","last_fullsized":"2023-10-11T16:30:41.198854+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:58:40.952333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668383+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:30:17.668383+0000","last_peered":"2023-10-11T16:30:17.668383+0000","last_clean":"2023-10-11T16:30:17.668383+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:30:17.668383+0000","last_undegraded":"2023-10-11T16:30:17.668383+0000","last_fullsized":"2023-10-11T16:30:17.668383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:10:24.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667627+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:30:17.667627+0000","last_peered":"2023-10-11T16:30:17.667627+0000","last_clean":"2023-10-11T16:30:17.667627+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:30:17.667627+0000","last_undegraded":"2023-10-11T16:30:17.667627+0000","last_fullsized":"2023-10-11T16:30:17.667627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:59:03.310068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668885+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:30:17.668885+0000","last_peered":"2023-10-11T16:30:17.668885+0000","last_clean":"2023-10-11T16:30:17.668885+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:30:17.668885+0000","last_undegraded":"2023-10-11T16:30:17.668885+0000","last_fullsized":"2023-10-11T16:30:17.668885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:29:49.809378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612215+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:30:16.612215+0000","last_peered":"2023-10-11T16:30:16.612215+0000","last_clean":"2023-10-11T16:30:16.612215+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:30:16.612215+0000","last_undegraded":"2023-10-11T16:30:16.612215+0000","last_fullsized":"2023-10-11T16:30:16.612215+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:45:37.863112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668530+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:30:17.668530+0000","last_peered":"2023-10-11T16:30:17.668530+0000","last_clean":"2023-10-11T16:30:17.668530+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:30:17.668530+0000","last_undegraded":"2023-10-11T16:30:17.668530+0000","last_fullsized":"2023-10-11T16:30:17.668530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:47:05.876657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.173667+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:30:37.173667+0000","last_peered":"2023-10-11T16:30:37.173667+0000","last_clean":"2023-10-11T16:30:37.173667+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:30:37.173667+0000","last_undegraded":"2023-10-11T16:30:37.173667+0000","last_fullsized":"2023-10-11T16:30:37.173667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:55:47.819813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668277+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:30:17.668277+0000","last_peered":"2023-10-11T16:30:17.668277+0000","last_clean":"2023-10-11T16:30:17.668277+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:30:17.668277+0000","last_undegraded":"2023-10-11T16:30:17.668277+0000","last_fullsized":"2023-10-11T16:30:17.668277+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:27:30.832381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667964+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:30:17.667964+0000","last_peered":"2023-10-11T16:30:17.667964+0000","last_clean":"2023-10-11T16:30:17.667964+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:30:17.667964+0000","last_undegraded":"2023-10-11T16:30:17.667964+0000","last_fullsized":"2023-10-11T16:30:17.667964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:49:38.616403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614611+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:30:16.614611+0000","last_peered":"2023-10-11T16:30:16.614611+0000","last_clean":"2023-10-11T16:30:16.614611+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:30:16.614611+0000","last_undegraded":"2023-10-11T16:30:16.614611+0000","last_fullsized":"2023-10-11T16:30:16.614611+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:36:36.308880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":87,"state":"peering","last_fresh":"2023-10-11T16:30:17.668751+0000","last_change":"2023-10-11T16:30:40.187333+0000","last_active":"2023-10-11T16:30:40.187333+0000","last_peered":"2023-10-11T16:30:17.668751+0000","last_clean":"2023-10-11T16:30:17.668751+0000","last_became_active":"2023-10-11T16:30:11.658658+0000","last_became_peered":"2023-10-11T16:30:11.658658+0000","last_unstale":"2023-10-11T16:30:17.668751+0000","last_undegraded":"2023-10-11T16:30:17.668751+0000","last_fullsized":"2023-10-11T16:30:17.668751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,8],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185067+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:30:41.185067+0000","last_peered":"2023-10-11T16:30:41.185067+0000","last_clean":"2023-10-11T16:30:41.185067+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:30:41.185067+0000","last_undegraded":"2023-10-11T16:30:41.185067+0000","last_fullsized":"2023-10-11T16:30:41.185067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:48.316789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187858+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:30:40.187858+0000","last_peered":"2023-10-11T16:30:40.187858+0000","last_clean":"2023-10-11T16:30:40.187858+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:30:40.187858+0000","last_undegraded":"2023-10-11T16:30:40.187858+0000","last_fullsized":"2023-10-11T16:30:40.187858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:51.299737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614368+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:30:16.614368+0000","last_peered":"2023-10-11T16:30:16.614368+0000","last_clean":"2023-10-11T16:30:16.614368+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:30:16.614368+0000","last_undegraded":"2023-10-11T16:30:16.614368+0000","last_fullsized":"2023-10-11T16:30:16.614368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:59:43.658557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186478+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:30:41.186478+0000","last_peered":"2023-10-11T16:30:41.186478+0000","last_clean":"2023-10-11T16:30:41.186478+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:30:41.186478+0000","last_undegraded":"2023-10-11T16:30:41.186478+0000","last_fullsized":"2023-10-11T16:30:41.186478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:20:03.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176047+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:30:37.176047+0000","last_peered":"2023-10-11T16:30:37.176047+0000","last_clean":"2023-10-11T16:30:37.176047+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:30:37.176047+0000","last_undegraded":"2023-10-11T16:30:37.176047+0000","last_fullsized":"2023-10-11T16:30:37.176047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:38:45.180274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.188012+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:30:40.188012+0000","last_peered":"2023-10-11T16:30:40.188012+0000","last_clean":"2023-10-11T16:30:40.188012+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:30:40.188012+0000","last_undegraded":"2023-10-11T16:30:40.188012+0000","last_fullsized":"2023-10-11T16:30:40.188012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:36:53.041380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284122+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:30:17.284122+0000","last_peered":"2023-10-11T16:30:17.284122+0000","last_clean":"2023-10-11T16:30:17.284122+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:30:17.284122+0000","last_undegraded":"2023-10-11T16:30:17.284122+0000","last_fullsized":"2023-10-11T16:30:17.284122+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:35:11.414446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.177276+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:30:37.177276+0000","last_peered":"2023-10-11T16:30:37.177276+0000","last_clean":"2023-10-11T16:30:37.177276+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:30:37.177276+0000","last_undegraded":"2023-10-11T16:30:37.177276+0000","last_fullsized":"2023-10-11T16:30:37.177276+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:44:31.081697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668117+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:30:17.668117+0000","last_peered":"2023-10-11T16:30:17.668117+0000","last_clean":"2023-10-11T16:30:17.668117+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:30:17.668117+0000","last_undegraded":"2023-10-11T16:30:17.668117+0000","last_fullsized":"2023-10-11T16:30:17.668117+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:20:03.123703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612081+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:30:16.612081+0000","last_peered":"2023-10-11T16:30:16.612081+0000","last_clean":"2023-10-11T16:30:16.612081+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:30:16.612081+0000","last_undegraded":"2023-10-11T16:30:16.612081+0000","last_fullsized":"2023-10-11T16:30:16.612081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:44:54.955030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186294+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:30:40.186294+0000","last_peered":"2023-10-11T16:30:40.186294+0000","last_clean":"2023-10-11T16:30:40.186294+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:30:40.186294+0000","last_undegraded":"2023-10-11T16:30:40.186294+0000","last_fullsized":"2023-10-11T16:30:40.186294+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:19:55.442658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181702+0000","last_change":"2023-10-11T16:30:11.662850+0000","last_active":"2023-10-11T16:30:41.181702+0000","last_peered":"2023-10-11T16:30:41.181702+0000","last_clean":"2023-10-11T16:30:41.181702+0000","last_became_active":"2023-10-11T16:30:11.661052+0000","last_became_peered":"2023-10-11T16:30:11.661052+0000","last_unstale":"2023-10-11T16:30:41.181702+0000","last_undegraded":"2023-10-11T16:30:41.181702+0000","last_fullsized":"2023-10-11T16:30:41.181702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:23.324421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667905+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:30:17.667905+0000","last_peered":"2023-10-11T16:30:17.667905+0000","last_clean":"2023-10-11T16:30:17.667905+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:30:17.667905+0000","last_undegraded":"2023-10-11T16:30:17.667905+0000","last_fullsized":"2023-10-11T16:30:17.667905+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:07.189663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166341+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:30:36.166341+0000","last_peered":"2023-10-11T16:30:36.166341+0000","last_clean":"2023-10-11T16:30:36.166341+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:30:36.166341+0000","last_undegraded":"2023-10-11T16:30:36.166341+0000","last_fullsized":"2023-10-11T16:30:36.166341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:36:57.466108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669129+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:30:17.669129+0000","last_peered":"2023-10-11T16:30:17.669129+0000","last_clean":"2023-10-11T16:30:17.669129+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:30:17.669129+0000","last_undegraded":"2023-10-11T16:30:17.669129+0000","last_fullsized":"2023-10-11T16:30:17.669129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:44:17.713245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181694+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:30:41.181694+0000","last_peered":"2023-10-11T16:30:41.181694+0000","last_clean":"2023-10-11T16:30:41.181694+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:30:41.181694+0000","last_undegraded":"2023-10-11T16:30:41.181694+0000","last_fullsized":"2023-10-11T16:30:41.181694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:32.776111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165661+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:30:36.165661+0000","last_peered":"2023-10-11T16:30:36.165661+0000","last_clean":"2023-10-11T16:30:36.165661+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:30:36.165661+0000","last_undegraded":"2023-10-11T16:30:36.165661+0000","last_fullsized":"2023-10-11T16:30:36.165661+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:24:23.101409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612235+0000","last_change":"2023-10-11T16:30:11.666124+0000","last_active":"2023-10-11T16:30:16.612235+0000","last_peered":"2023-10-11T16:30:16.612235+0000","last_clean":"2023-10-11T16:30:16.612235+0000","last_became_active":"2023-10-11T16:30:11.665786+0000","last_became_peered":"2023-10-11T16:30:11.665786+0000","last_unstale":"2023-10-11T16:30:16.612235+0000","last_undegraded":"2023-10-11T16:30:16.612235+0000","last_fullsized":"2023-10-11T16:30:16.612235+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:45:32.355550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669108+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:30:17.669108+0000","last_peered":"2023-10-11T16:30:17.669108+0000","last_clean":"2023-10-11T16:30:17.669108+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:30:17.669108+0000","last_undegraded":"2023-10-11T16:30:17.669108+0000","last_fullsized":"2023-10-11T16:30:17.669108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:20:20.621738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187532+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:30:40.187532+0000","last_peered":"2023-10-11T16:30:40.187532+0000","last_clean":"2023-10-11T16:30:40.187532+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:30:40.187532+0000","last_undegraded":"2023-10-11T16:30:40.187532+0000","last_fullsized":"2023-10-11T16:30:40.187532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:45:20.227087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668685+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:30:17.668685+0000","last_peered":"2023-10-11T16:30:17.668685+0000","last_clean":"2023-10-11T16:30:17.668685+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:30:17.668685+0000","last_undegraded":"2023-10-11T16:30:17.668685+0000","last_fullsized":"2023-10-11T16:30:17.668685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:02:09.993229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181930+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:30:41.181930+0000","last_peered":"2023-10-11T16:30:41.181930+0000","last_clean":"2023-10-11T16:30:41.181930+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:30:41.181930+0000","last_undegraded":"2023-10-11T16:30:41.181930+0000","last_fullsized":"2023-10-11T16:30:41.181930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:27:45.123842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'12","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165516+0000","last_change":"2023-10-11T16:30:11.112780+0000","last_active":"2023-10-11T16:30:36.165516+0000","last_peered":"2023-10-11T16:30:36.165516+0000","last_clean":"2023-10-11T16:30:36.165516+0000","last_became_active":"2023-10-11T16:30:11.112481+0000","last_became_peered":"2023-10-11T16:30:11.112481+0000","last_unstale":"2023-10-11T16:30:36.165516+0000","last_undegraded":"2023-10-11T16:30:36.165516+0000","last_fullsized":"2023-10-11T16:30:36.165516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:44.261584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614775+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:30:16.614775+0000","last_peered":"2023-10-11T16:30:16.614775+0000","last_clean":"2023-10-11T16:30:16.614775+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:30:16.614775+0000","last_undegraded":"2023-10-11T16:30:16.614775+0000","last_fullsized":"2023-10-11T16:30:16.614775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:14:04.119677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.189579+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:30:40.189579+0000","last_peered":"2023-10-11T16:30:40.189579+0000","last_clean":"2023-10-11T16:30:40.189579+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:30:40.189579+0000","last_undegraded":"2023-10-11T16:30:40.189579+0000","last_fullsized":"2023-10-11T16:30:40.189579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:26.634024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185142+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:30:41.185142+0000","last_peered":"2023-10-11T16:30:41.185142+0000","last_clean":"2023-10-11T16:30:41.185142+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:30:41.185142+0000","last_undegraded":"2023-10-11T16:30:41.185142+0000","last_fullsized":"2023-10-11T16:30:41.185142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:25:53.623192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668870+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:30:17.668870+0000","last_peered":"2023-10-11T16:30:17.668870+0000","last_clean":"2023-10-11T16:30:17.668870+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:30:17.668870+0000","last_undegraded":"2023-10-11T16:30:17.668870+0000","last_fullsized":"2023-10-11T16:30:17.668870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:15:57.948212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283849+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:30:17.283849+0000","last_peered":"2023-10-11T16:30:17.283849+0000","last_clean":"2023-10-11T16:30:17.283849+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:30:17.283849+0000","last_undegraded":"2023-10-11T16:30:17.283849+0000","last_fullsized":"2023-10-11T16:30:17.283849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:27:49.373457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187910+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:30:40.187910+0000","last_peered":"2023-10-11T16:30:40.187910+0000","last_clean":"2023-10-11T16:30:40.187910+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:30:40.187910+0000","last_undegraded":"2023-10-11T16:30:40.187910+0000","last_fullsized":"2023-10-11T16:30:40.187910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:56:20.300209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613601+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:30:16.613601+0000","last_peered":"2023-10-11T16:30:16.613601+0000","last_clean":"2023-10-11T16:30:16.613601+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:30:16.613601+0000","last_undegraded":"2023-10-11T16:30:16.613601+0000","last_fullsized":"2023-10-11T16:30:16.613601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:34.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182281+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.182281+0000","last_peered":"2023-10-11T16:30:41.182281+0000","last_clean":"2023-10-11T16:30:41.182281+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:30:41.182281+0000","last_undegraded":"2023-10-11T16:30:41.182281+0000","last_fullsized":"2023-10-11T16:30:41.182281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:35:47.449805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:38.189202+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:30:38.189202+0000","last_peered":"2023-10-11T16:30:38.189202+0000","last_clean":"2023-10-11T16:30:38.189202+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:30:38.189202+0000","last_undegraded":"2023-10-11T16:30:38.189202+0000","last_fullsized":"2023-10-11T16:30:38.189202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:25:21.420979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613021+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:30:16.613021+0000","last_peered":"2023-10-11T16:30:16.613021+0000","last_clean":"2023-10-11T16:30:16.613021+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:30:16.613021+0000","last_undegraded":"2023-10-11T16:30:16.613021+0000","last_fullsized":"2023-10-11T16:30:16.613021+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:10:00.701910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669925+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:30:17.669925+0000","last_peered":"2023-10-11T16:30:17.669925+0000","last_clean":"2023-10-11T16:30:17.669925+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:30:17.669925+0000","last_undegraded":"2023-10-11T16:30:17.669925+0000","last_fullsized":"2023-10-11T16:30:17.669925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:46:40.422035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176084+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:30:37.176084+0000","last_peered":"2023-10-11T16:30:37.176084+0000","last_clean":"2023-10-11T16:30:37.176084+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:30:37.176084+0000","last_undegraded":"2023-10-11T16:30:37.176084+0000","last_fullsized":"2023-10-11T16:30:37.176084+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:15.680165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187485+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:30:40.187485+0000","last_peered":"2023-10-11T16:30:40.187485+0000","last_clean":"2023-10-11T16:30:40.187485+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:30:40.187485+0000","last_undegraded":"2023-10-11T16:30:40.187485+0000","last_fullsized":"2023-10-11T16:30:40.187485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:58:29.097161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187329+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:30:40.187329+0000","last_peered":"2023-10-11T16:30:40.187329+0000","last_clean":"2023-10-11T16:30:40.187329+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:30:40.187329+0000","last_undegraded":"2023-10-11T16:30:40.187329+0000","last_fullsized":"2023-10-11T16:30:40.187329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:21:39.470036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.615240+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:30:16.615240+0000","last_peered":"2023-10-11T16:30:16.615240+0000","last_clean":"2023-10-11T16:30:16.615240+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:30:16.615240+0000","last_undegraded":"2023-10-11T16:30:16.615240+0000","last_fullsized":"2023-10-11T16:30:16.615240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:02:48.445205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284055+0000","last_change":"2023-10-11T16:30:11.670547+0000","last_active":"2023-10-11T16:30:17.284055+0000","last_peered":"2023-10-11T16:30:17.284055+0000","last_clean":"2023-10-11T16:30:17.284055+0000","last_became_active":"2023-10-11T16:30:11.669842+0000","last_became_peered":"2023-10-11T16:30:11.669842+0000","last_unstale":"2023-10-11T16:30:17.284055+0000","last_undegraded":"2023-10-11T16:30:17.284055+0000","last_fullsized":"2023-10-11T16:30:17.284055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:20:40.999317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614361+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:30:16.614361+0000","last_peered":"2023-10-11T16:30:16.614361+0000","last_clean":"2023-10-11T16:30:16.614361+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:30:16.614361+0000","last_undegraded":"2023-10-11T16:30:16.614361+0000","last_fullsized":"2023-10-11T16:30:16.614361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:13.782416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182101+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:30:41.182101+0000","last_peered":"2023-10-11T16:30:41.182101+0000","last_clean":"2023-10-11T16:30:41.182101+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:30:41.182101+0000","last_undegraded":"2023-10-11T16:30:41.182101+0000","last_fullsized":"2023-10-11T16:30:41.182101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:36.264713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668683+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:30:17.668683+0000","last_peered":"2023-10-11T16:30:17.668683+0000","last_clean":"2023-10-11T16:30:17.668683+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:30:17.668683+0000","last_undegraded":"2023-10-11T16:30:17.668683+0000","last_fullsized":"2023-10-11T16:30:17.668683+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:10:24.647448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669638+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:30:17.669638+0000","last_peered":"2023-10-11T16:30:17.669638+0000","last_clean":"2023-10-11T16:30:17.669638+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:30:17.669638+0000","last_undegraded":"2023-10-11T16:30:17.669638+0000","last_fullsized":"2023-10-11T16:30:17.669638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:12:58.596577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613284+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:30:16.613284+0000","last_peered":"2023-10-11T16:30:16.613284+0000","last_clean":"2023-10-11T16:30:16.613284+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:30:16.613284+0000","last_undegraded":"2023-10-11T16:30:16.613284+0000","last_fullsized":"2023-10-11T16:30:16.613284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:33:51.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668320+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:30:17.668320+0000","last_peered":"2023-10-11T16:30:17.668320+0000","last_clean":"2023-10-11T16:30:17.668320+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:30:17.668320+0000","last_undegraded":"2023-10-11T16:30:17.668320+0000","last_fullsized":"2023-10-11T16:30:17.668320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:32:12.254443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669773+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:30:17.669773+0000","last_peered":"2023-10-11T16:30:17.669773+0000","last_clean":"2023-10-11T16:30:17.669773+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:30:17.669773+0000","last_undegraded":"2023-10-11T16:30:17.669773+0000","last_fullsized":"2023-10-11T16:30:17.669773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:09:37.408810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186478+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:30:40.186478+0000","last_peered":"2023-10-11T16:30:40.186478+0000","last_clean":"2023-10-11T16:30:40.186478+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:30:40.186478+0000","last_undegraded":"2023-10-11T16:30:40.186478+0000","last_fullsized":"2023-10-11T16:30:40.186478+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:23.165604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176027+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:30:37.176027+0000","last_peered":"2023-10-11T16:30:37.176027+0000","last_clean":"2023-10-11T16:30:37.176027+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:30:37.176027+0000","last_undegraded":"2023-10-11T16:30:37.176027+0000","last_fullsized":"2023-10-11T16:30:37.176027+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:26:18.864499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:19.956667+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:30:19.956667+0000","last_peered":"2023-10-11T16:30:19.956667+0000","last_clean":"2023-10-11T16:30:19.956667+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:30:19.956667+0000","last_undegraded":"2023-10-11T16:30:19.956667+0000","last_fullsized":"2023-10-11T16:30:19.956667+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:53:51.765779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667662+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:30:17.667662+0000","last_peered":"2023-10-11T16:30:17.667662+0000","last_clean":"2023-10-11T16:30:17.667662+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:30:17.667662+0000","last_undegraded":"2023-10-11T16:30:17.667662+0000","last_fullsized":"2023-10-11T16:30:17.667662+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:23.176208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:38.183142+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:30:38.183142+0000","last_peered":"2023-10-11T16:30:38.183142+0000","last_clean":"2023-10-11T16:30:38.183142+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:30:38.183142+0000","last_undegraded":"2023-10-11T16:30:38.183142+0000","last_fullsized":"2023-10-11T16:30:38.183142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:02:22.009759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612829+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:30:16.612829+0000","last_peered":"2023-10-11T16:30:16.612829+0000","last_clean":"2023-10-11T16:30:16.612829+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:30:16.612829+0000","last_undegraded":"2023-10-11T16:30:16.612829+0000","last_fullsized":"2023-10-11T16:30:16.612829+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:27:41.571741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187873+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:30:40.187873+0000","last_peered":"2023-10-11T16:30:40.187873+0000","last_clean":"2023-10-11T16:30:40.187873+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:30:40.187873+0000","last_undegraded":"2023-10-11T16:30:40.187873+0000","last_fullsized":"2023-10-11T16:30:40.187873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:05:27.804658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667984+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:30:17.667984+0000","last_peered":"2023-10-11T16:30:17.667984+0000","last_clean":"2023-10-11T16:30:17.667984+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:30:17.667984+0000","last_undegraded":"2023-10-11T16:30:17.667984+0000","last_fullsized":"2023-10-11T16:30:17.667984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:32:59.074750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.173678+0000","last_change":"2023-10-11T16:30:11.117164+0000","last_active":"2023-10-11T16:30:37.173678+0000","last_peered":"2023-10-11T16:30:37.173678+0000","last_clean":"2023-10-11T16:30:37.173678+0000","last_became_active":"2023-10-11T16:30:11.116887+0000","last_became_peered":"2023-10-11T16:30:11.116887+0000","last_unstale":"2023-10-11T16:30:37.173678+0000","last_undegraded":"2023-10-11T16:30:37.173678+0000","last_fullsized":"2023-10-11T16:30:37.173678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:39:08.785806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283813+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:30:17.283813+0000","last_peered":"2023-10-11T16:30:17.283813+0000","last_clean":"2023-10-11T16:30:17.283813+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:30:17.283813+0000","last_undegraded":"2023-10-11T16:30:17.283813+0000","last_fullsized":"2023-10-11T16:30:17.283813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:43.398632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166054+0000","last_change":"2023-10-11T16:30:11.129313+0000","last_active":"2023-10-11T16:30:36.166054+0000","last_peered":"2023-10-11T16:30:36.166054+0000","last_clean":"2023-10-11T16:30:36.166054+0000","last_became_active":"2023-10-11T16:30:11.129051+0000","last_became_peered":"2023-10-11T16:30:11.129051+0000","last_unstale":"2023-10-11T16:30:36.166054+0000","last_undegraded":"2023-10-11T16:30:36.166054+0000","last_fullsized":"2023-10-11T16:30:36.166054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:02:06.341952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669886+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:30:17.669886+0000","last_peered":"2023-10-11T16:30:17.669886+0000","last_clean":"2023-10-11T16:30:17.669886+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:30:17.669886+0000","last_undegraded":"2023-10-11T16:30:17.669886+0000","last_fullsized":"2023-10-11T16:30:17.669886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:47:23.558003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667532+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:30:17.667532+0000","last_peered":"2023-10-11T16:30:17.667532+0000","last_clean":"2023-10-11T16:30:17.667532+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:30:17.667532+0000","last_undegraded":"2023-10-11T16:30:17.667532+0000","last_fullsized":"2023-10-11T16:30:17.667532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:55:02.983828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182666+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:30:41.182666+0000","last_peered":"2023-10-11T16:30:41.182666+0000","last_clean":"2023-10-11T16:30:41.182666+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:30:41.182666+0000","last_undegraded":"2023-10-11T16:30:41.182666+0000","last_fullsized":"2023-10-11T16:30:41.182666+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:35.861737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668839+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:30:17.668839+0000","last_peered":"2023-10-11T16:30:17.668839+0000","last_clean":"2023-10-11T16:30:17.668839+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:30:17.668839+0000","last_undegraded":"2023-10-11T16:30:17.668839+0000","last_fullsized":"2023-10-11T16:30:17.668839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:55.881039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185784+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:30:41.185784+0000","last_peered":"2023-10-11T16:30:41.185784+0000","last_clean":"2023-10-11T16:30:41.185784+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:30:41.185784+0000","last_undegraded":"2023-10-11T16:30:41.185784+0000","last_fullsized":"2023-10-11T16:30:41.185784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:57:36.382181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.164814+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:30:36.164814+0000","last_peered":"2023-10-11T16:30:36.164814+0000","last_clean":"2023-10-11T16:30:36.164814+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:30:36.164814+0000","last_undegraded":"2023-10-11T16:30:36.164814+0000","last_fullsized":"2023-10-11T16:30:36.164814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:09:01.841380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612174+0000","last_change":"2023-10-11T16:30:11.661388+0000","last_active":"2023-10-11T16:30:16.612174+0000","last_peered":"2023-10-11T16:30:16.612174+0000","last_clean":"2023-10-11T16:30:16.612174+0000","last_became_active":"2023-10-11T16:30:11.661073+0000","last_became_peered":"2023-10-11T16:30:11.661073+0000","last_unstale":"2023-10-11T16:30:16.612174+0000","last_undegraded":"2023-10-11T16:30:16.612174+0000","last_fullsized":"2023-10-11T16:30:16.612174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:10:29.755199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668643+0000","last_change":"2023-10-11T16:30:16.171535+0000","last_active":"2023-10-11T16:30:17.668643+0000","last_peered":"2023-10-11T16:30:17.668643+0000","last_clean":"2023-10-11T16:30:17.668643+0000","last_became_active":"2023-10-11T16:30:16.171051+0000","last_became_peered":"2023-10-11T16:30:16.171051+0000","last_unstale":"2023-10-11T16:30:17.668643+0000","last_undegraded":"2023-10-11T16:30:17.668643+0000","last_fullsized":"2023-10-11T16:30:17.668643+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:36:53.674481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.164950+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:30:36.164950+0000","last_peered":"2023-10-11T16:30:36.164950+0000","last_clean":"2023-10-11T16:30:36.164950+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:30:36.164950+0000","last_undegraded":"2023-10-11T16:30:36.164950+0000","last_fullsized":"2023-10-11T16:30:36.164950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:06:21.073342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614260+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:30:16.614260+0000","last_peered":"2023-10-11T16:30:16.614260+0000","last_clean":"2023-10-11T16:30:16.614260+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:30:16.614260+0000","last_undegraded":"2023-10-11T16:30:16.614260+0000","last_fullsized":"2023-10-11T16:30:16.614260+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:28:45.994452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668995+0000","last_change":"2023-10-11T16:30:16.131117+0000","last_active":"2023-10-11T16:30:17.668995+0000","last_peered":"2023-10-11T16:30:17.668995+0000","last_clean":"2023-10-11T16:30:17.668995+0000","last_became_active":"2023-10-11T16:30:16.130776+0000","last_became_peered":"2023-10-11T16:30:16.130776+0000","last_unstale":"2023-10-11T16:30:17.668995+0000","last_undegraded":"2023-10-11T16:30:17.668995+0000","last_fullsized":"2023-10-11T16:30:17.668995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:24:15.931186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166640+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:30:36.166640+0000","last_peered":"2023-10-11T16:30:36.166640+0000","last_clean":"2023-10-11T16:30:36.166640+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:30:36.166640+0000","last_undegraded":"2023-10-11T16:30:36.166640+0000","last_fullsized":"2023-10-11T16:30:36.166640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:12:26.496915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283586+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:30:17.283586+0000","last_peered":"2023-10-11T16:30:17.283586+0000","last_clean":"2023-10-11T16:30:17.283586+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:30:17.283586+0000","last_undegraded":"2023-10-11T16:30:17.283586+0000","last_fullsized":"2023-10-11T16:30:17.283586+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:10:18.200065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:19.940473+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:30:19.940473+0000","last_peered":"2023-10-11T16:30:19.940473+0000","last_clean":"2023-10-11T16:30:19.940473+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:30:19.940473+0000","last_undegraded":"2023-10-11T16:30:19.940473+0000","last_fullsized":"2023-10-11T16:30:19.940473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:39.565360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185977+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:30:41.185977+0000","last_peered":"2023-10-11T16:30:41.185977+0000","last_clean":"2023-10-11T16:30:41.185977+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:30:41.185977+0000","last_undegraded":"2023-10-11T16:30:41.185977+0000","last_fullsized":"2023-10-11T16:30:41.185977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:32:39.882844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667992+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:30:17.667992+0000","last_peered":"2023-10-11T16:30:17.667992+0000","last_clean":"2023-10-11T16:30:17.667992+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:30:17.667992+0000","last_undegraded":"2023-10-11T16:30:17.667992+0000","last_fullsized":"2023-10-11T16:30:17.667992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:18:04.314307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669109+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:30:17.669109+0000","last_peered":"2023-10-11T16:30:17.669109+0000","last_clean":"2023-10-11T16:30:17.669109+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:30:17.669109+0000","last_undegraded":"2023-10-11T16:30:17.669109+0000","last_fullsized":"2023-10-11T16:30:17.669109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:39:11.522213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.167157+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:30:36.167157+0000","last_peered":"2023-10-11T16:30:36.167157+0000","last_clean":"2023-10-11T16:30:36.167157+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:30:36.167157+0000","last_undegraded":"2023-10-11T16:30:36.167157+0000","last_fullsized":"2023-10-11T16:30:36.167157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:06.688871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613531+0000","last_change":"2023-10-11T16:30:11.648776+0000","last_active":"2023-10-11T16:30:16.613531+0000","last_peered":"2023-10-11T16:30:16.613531+0000","last_clean":"2023-10-11T16:30:16.613531+0000","last_became_active":"2023-10-11T16:30:11.648465+0000","last_became_peered":"2023-10-11T16:30:11.648465+0000","last_unstale":"2023-10-11T16:30:16.613531+0000","last_undegraded":"2023-10-11T16:30:16.613531+0000","last_fullsized":"2023-10-11T16:30:16.613531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:24:06.619577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667818+0000","last_change":"2023-10-11T16:30:16.129703+0000","last_active":"2023-10-11T16:30:17.667818+0000","last_peered":"2023-10-11T16:30:17.667818+0000","last_clean":"2023-10-11T16:30:17.667818+0000","last_became_active":"2023-10-11T16:30:16.129231+0000","last_became_peered":"2023-10-11T16:30:16.129231+0000","last_unstale":"2023-10-11T16:30:17.667818+0000","last_undegraded":"2023-10-11T16:30:17.667818+0000","last_fullsized":"2023-10-11T16:30:17.667818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:47:29.769705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667910+0000","last_change":"2023-10-11T16:30:11.127305+0000","last_active":"2023-10-11T16:30:17.667910+0000","last_peered":"2023-10-11T16:30:17.667910+0000","last_clean":"2023-10-11T16:30:17.667910+0000","last_became_active":"2023-10-11T16:30:11.126986+0000","last_became_peered":"2023-10-11T16:30:11.126986+0000","last_unstale":"2023-10-11T16:30:17.667910+0000","last_undegraded":"2023-10-11T16:30:17.667910+0000","last_fullsized":"2023-10-11T16:30:17.667910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:33:23.033612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613872+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:30:16.613872+0000","last_peered":"2023-10-11T16:30:16.613872+0000","last_clean":"2023-10-11T16:30:16.613872+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:30:16.613872+0000","last_undegraded":"2023-10-11T16:30:16.613872+0000","last_fullsized":"2023-10-11T16:30:16.613872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:49.187595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612035+0000","last_change":"2023-10-11T16:30:16.135709+0000","last_active":"2023-10-11T16:30:16.612035+0000","last_peered":"2023-10-11T16:30:16.612035+0000","last_clean":"2023-10-11T16:30:16.612035+0000","last_became_active":"2023-10-11T16:30:16.135294+0000","last_became_peered":"2023-10-11T16:30:16.135294+0000","last_unstale":"2023-10-11T16:30:16.612035+0000","last_undegraded":"2023-10-11T16:30:16.612035+0000","last_fullsized":"2023-10-11T16:30:16.612035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:50:08.906768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.189235+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:30:40.189235+0000","last_peered":"2023-10-11T16:30:40.189235+0000","last_clean":"2023-10-11T16:30:40.189235+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:30:40.189235+0000","last_undegraded":"2023-10-11T16:30:40.189235+0000","last_fullsized":"2023-10-11T16:30:40.189235+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:38:09.758963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613738+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:30:16.613738+0000","last_peered":"2023-10-11T16:30:16.613738+0000","last_clean":"2023-10-11T16:30:16.613738+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:30:16.613738+0000","last_undegraded":"2023-10-11T16:30:16.613738+0000","last_fullsized":"2023-10-11T16:30:16.613738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:04:56.528036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186085+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:30:40.186085+0000","last_peered":"2023-10-11T16:30:40.186085+0000","last_clean":"2023-10-11T16:30:40.186085+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:30:40.186085+0000","last_undegraded":"2023-10-11T16:30:40.186085+0000","last_fullsized":"2023-10-11T16:30:40.186085+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:23:35.338891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669638+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:30:17.669638+0000","last_peered":"2023-10-11T16:30:17.669638+0000","last_clean":"2023-10-11T16:30:17.669638+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:30:17.669638+0000","last_undegraded":"2023-10-11T16:30:17.669638+0000","last_fullsized":"2023-10-11T16:30:17.669638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:25:06.130705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613643+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:30:16.613643+0000","last_peered":"2023-10-11T16:30:16.613643+0000","last_clean":"2023-10-11T16:30:16.613643+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:30:16.613643+0000","last_undegraded":"2023-10-11T16:30:16.613643+0000","last_fullsized":"2023-10-11T16:30:16.613643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:11:20.783196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283653+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:30:17.283653+0000","last_peered":"2023-10-11T16:30:17.283653+0000","last_clean":"2023-10-11T16:30:17.283653+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:30:17.283653+0000","last_undegraded":"2023-10-11T16:30:17.283653+0000","last_fullsized":"2023-10-11T16:30:17.283653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:05:33.114445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186048+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:30:41.186048+0000","last_peered":"2023-10-11T16:30:41.186048+0000","last_clean":"2023-10-11T16:30:41.186048+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:30:41.186048+0000","last_undegraded":"2023-10-11T16:30:41.186048+0000","last_fullsized":"2023-10-11T16:30:41.186048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:50.699880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185837+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:30:41.185837+0000","last_peered":"2023-10-11T16:30:41.185837+0000","last_clean":"2023-10-11T16:30:41.185837+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:30:41.185837+0000","last_undegraded":"2023-10-11T16:30:41.185837+0000","last_fullsized":"2023-10-11T16:30:41.185837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:03:23.234106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611986+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:30:16.611986+0000","last_peered":"2023-10-11T16:30:16.611986+0000","last_clean":"2023-10-11T16:30:16.611986+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:30:16.611986+0000","last_undegraded":"2023-10-11T16:30:16.611986+0000","last_fullsized":"2023-10-11T16:30:16.611986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:59:39.821639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669892+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:30:17.669892+0000","last_peered":"2023-10-11T16:30:17.669892+0000","last_clean":"2023-10-11T16:30:17.669892+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:30:17.669892+0000","last_undegraded":"2023-10-11T16:30:17.669892+0000","last_fullsized":"2023-10-11T16:30:17.669892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:44.023609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284031+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:30:17.284031+0000","last_peered":"2023-10-11T16:30:17.284031+0000","last_clean":"2023-10-11T16:30:17.284031+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:30:17.284031+0000","last_undegraded":"2023-10-11T16:30:17.284031+0000","last_fullsized":"2023-10-11T16:30:17.284031+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:58.558512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185560+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:30:41.185560+0000","last_peered":"2023-10-11T16:30:41.185560+0000","last_clean":"2023-10-11T16:30:41.185560+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:30:41.185560+0000","last_undegraded":"2023-10-11T16:30:41.185560+0000","last_fullsized":"2023-10-11T16:30:41.185560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:19:06.908283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669256+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:30:17.669256+0000","last_peered":"2023-10-11T16:30:17.669256+0000","last_clean":"2023-10-11T16:30:17.669256+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:30:17.669256+0000","last_undegraded":"2023-10-11T16:30:17.669256+0000","last_fullsized":"2023-10-11T16:30:17.669256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:16:39.680829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186430+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:30:40.186430+0000","last_peered":"2023-10-11T16:30:40.186430+0000","last_clean":"2023-10-11T16:30:40.186430+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:30:40.186430+0000","last_undegraded":"2023-10-11T16:30:40.186430+0000","last_fullsized":"2023-10-11T16:30:40.186430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:27:45.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668100+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:30:17.668100+0000","last_peered":"2023-10-11T16:30:17.668100+0000","last_clean":"2023-10-11T16:30:17.668100+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:30:17.668100+0000","last_undegraded":"2023-10-11T16:30:17.668100+0000","last_fullsized":"2023-10-11T16:30:17.668100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:06:16.966258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669079+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:30:17.669079+0000","last_peered":"2023-10-11T16:30:17.669079+0000","last_clean":"2023-10-11T16:30:17.669079+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:30:17.669079+0000","last_undegraded":"2023-10-11T16:30:17.669079+0000","last_fullsized":"2023-10-11T16:30:17.669079+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:32.524195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669665+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:30:17.669665+0000","last_peered":"2023-10-11T16:30:17.669665+0000","last_clean":"2023-10-11T16:30:17.669665+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:30:17.669665+0000","last_undegraded":"2023-10-11T16:30:17.669665+0000","last_fullsized":"2023-10-11T16:30:17.669665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:17.866867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283801+0000","last_change":"2023-10-11T16:30:11.668522+0000","last_active":"2023-10-11T16:30:17.283801+0000","last_peered":"2023-10-11T16:30:17.283801+0000","last_clean":"2023-10-11T16:30:17.283801+0000","last_became_active":"2023-10-11T16:30:11.654192+0000","last_became_peered":"2023-10-11T16:30:11.654192+0000","last_unstale":"2023-10-11T16:30:17.283801+0000","last_undegraded":"2023-10-11T16:30:17.283801+0000","last_fullsized":"2023-10-11T16:30:17.283801+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:51:10.675716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6726,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":125,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":73444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449105,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":946176,"data_stored":468750,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.373},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.622}]},{"osd":5,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.499},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60399999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.56899999999999995}]},{"osd":7,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.86099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.77300000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.70699999999999996}]}]},{"osd":10,"up_from":71,"seq":304942678037,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6320,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738832,"statfs":{"total":95995035648,"available":95988563968,"internally_reserved":0,"allocated":1024000,"data_stored":1059190,"data_compressed":47399,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56100000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.61599999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81000000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.78600000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":0.89600000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.878},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.86499999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.82699999999999996}]}]},{"osd":9,"up_from":66,"seq":283467841561,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6276,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":978944,"data_stored":482592,"data_compressed":14600,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.622},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.42499999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.77300000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.495}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.59199999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.85099999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.754},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.70199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":8,"up_from":58,"seq":249108103199,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6380,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738772,"statfs":{"total":95995035648,"available":95988502528,"internally_reserved":0,"allocated":954368,"data_stored":464479,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57799999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.625},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.443,"15min":0.44},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.55900000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.55800000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.42299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.505}]},{"osd":6,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65000000000000002,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.66800000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.77200000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73699999999999999}]},{"osd":10,"last update":"Wed Oct 11 16:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":1011712,"data_stored":1064328,"data_compressed":47399,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.33800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.629,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81499999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78000000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.748,"5min":1.103,"15min":1.103},"last":0.74299999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77100000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.628},"min":{"1min":0.45400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79500000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.72499999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.70699999999999996,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.82399999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.71799999999999997,"15min":0.72599999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.85599999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.89800000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.873}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":937984,"data_stored":464072,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.54100000000000004,"15min":0.52300000000000002},"min":{"1min":0.44400000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91500000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.46899999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84599999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.44800000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.65000000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.48599999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94099999999999995,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.52200000000000002}]},{"osd":4,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.52000000000000002,"5min":0.498,"15min":0.498},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.66000000000000003,"15min":0.64900000000000002},"min":{"1min":0.56499999999999995,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.67200000000000004,"15min":0.65700000000000003},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.871,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.76100000000000001,"15min":0.745},"min":{"1min":0.56999999999999995,"5min":0.434,"15min":0.434},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.78100000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:29:55 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.77300000000000002,"15min":0.75700000000000001},"min":{"1min":0.66300000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73199999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.753}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74099999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.84799999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.85699999999999998,"15min":0.86799999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0569999999999999,"5min":1.254,"15min":1.254},"last":0.85899999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.89000000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.877,"15min":0.877},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.872}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7352,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737800,"statfs":{"total":95995035648,"available":95987507200,"internally_reserved":0,"allocated":966656,"data_stored":484916,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57399999999999995,"15min":0.621},"min":{"1min":0.371,"5min":0.251,"15min":0.251},"max":{"1min":0.754,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.53500000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76600000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.505,"15min":0.501},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.50900000000000001,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.307,"15min":0.307},"max":{"1min":0.68600000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.629}]},{"osd":3,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.438,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:29:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41999999999999998,"15min":0.40899999999999997},"min":{"1min":0.33300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.439},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.55100000000000005,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.72199999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71399999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.504,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.84499999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.878},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.82399999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.91600000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.79700000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":465550,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.40899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.497},{"interface":"front","average":{"1min":0.63,"5min":0.55500000000000005,"15min":0.52600000000000002},"min":{"1min":0.441,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60099999999999998,"15min":0.57799999999999996},"min":{"1min":0.38500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.622,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.44500000000000001,"15min":0.433},"min":{"1min":0.35799999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44600000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:29:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57599999999999996,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72699999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64700000000000002,"15min":0.63400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.621}]},{"osd":6,"last update":"Wed Oct 11 16:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.52200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96799999999999997,"5min":0.999,"15min":0.999},"last":0.52000000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.75},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.78400000000000003}]},{"osd":8,"last update":"Wed Oct 11 16:30:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.73199999999999998,"15min":0.73899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75900000000000001},"min":{"1min":0.56599999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91800000000000004,"5min":1.133,"15min":1.133},"last":0.73499999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.76400000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.80500000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:30:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.75800000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7472,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737680,"statfs":{"total":95995035648,"available":95987384320,"internally_reserved":0,"allocated":1024000,"data_stored":513399,"data_compressed":14600,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79800000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78300000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.83599999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.84199999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56699999999999995,"15min":0.54200000000000004},"min":{"1min":0.49399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59499999999999997,"15min":0.56299999999999994},"min":{"1min":0.47299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68600000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74399999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75900000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.875}]},{"osd":5,"last update":"Wed Oct 11 16:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66600000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.49099999999999999,"15min":0.47799999999999998},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.61499999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.64200000000000002,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86599999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.65600000000000003,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76100000000000001}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77900000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.80700000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":1.0149999999999999}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.627,"15min":0.61599999999999999},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92200000000000004,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.91500000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.52600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":1},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.73299999999999998,"15min":0.71999999999999997},"min":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.92800000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":1.036}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6628,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738524,"statfs":{"total":95995035648,"available":95988248576,"internally_reserved":0,"allocated":946176,"data_stored":463935,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.38200000000000001},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69999999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.373},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.40699999999999997},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.40400000000000003}]},{"osd":1,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60599999999999998,"15min":0.56999999999999995},"min":{"1min":0.433,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63500000000000001,"15min":0.58899999999999997},"min":{"1min":0.48199999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64500000000000002,"15min":0.63400000000000001},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90900000000000003,"5min":1.101,"15min":1.101},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69399999999999995,"15min":0.68000000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0509999999999999,"5min":1.194,"15min":1.194},"last":0.73499999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85399999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69199999999999995,"15min":0.68899999999999995},"min":{"1min":0.51300000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91600000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.66800000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:30:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91500000000000004,"5min":1.268,"15min":1.268},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.60999999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84599999999999997,"5min":1.119,"15min":1.119},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.071,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.629}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70699999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.755}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":950272,"data_stored":464668,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.379,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.36899999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.55200000000000005,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79600000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51800000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71999999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.2560000000000002,"5min":4.5,"15min":4.5},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68899999999999995,"15min":0.72999999999999998},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.75900000000000001}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.58299999999999996}]},{"osd":7,"last update":"Wed Oct 11 16:30:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48999999999999999,"15min":0.48199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56000000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.60999999999999999,"15min":0.59499999999999997},"min":{"1min":0.47799999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60599999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.84699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6712,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738440,"statfs":{"total":95995035648,"available":95988162560,"internally_reserved":0,"allocated":1032192,"data_stored":1069474,"data_compressed":50350,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.745,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78700000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.873,"5min":0.874,"15min":0.874},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.70599999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.438,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63200000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.71399999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90600000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.76500000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58899999999999997,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85299999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.99299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78100000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.83899999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70799999999999996,"15min":0.71499999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69599999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.89200000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64600000000000002,"15min":0.64800000000000002},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84299999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79000000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.93600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.97499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":39235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:30:42.481 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1d is not active+clean 2023-10-11T16:30:42.481 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1d', 'version': "0'0", 'reported_seq': 21, 'reported_epoch': 87, 'state': 'peering', 'last_fresh': '2023-10-11T16:30:17.668751+0000', 'last_change': '2023-10-11T16:30:40.187333+0000', 'last_active': '2023-10-11T16:30:40.187333+0000', 'last_peered': '2023-10-11T16:30:17.668751+0000', 'last_clean': '2023-10-11T16:30:17.668751+0000', 'last_became_active': '2023-10-11T16:30:11.658658+0000', 'last_became_peered': '2023-10-11T16:30:11.658658+0000', 'last_unstale': '2023-10-11T16:30:17.668751+0000', 'last_undegraded': '2023-10-11T16:30:17.668751+0000', 'last_fullsized': '2023-10-11T16:30:17.668751+0000', 'mapping_epoch': 88, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 82, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-11T16:30:11.090383+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-11T16:30:11.090383+0000', 'last_clean_scrub_stamp': '2023-10-11T16:30:11.090383+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 8, 1], 'acting': [5, 8, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 4, 8], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-11T16:30:42.482 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-11T16:30:42.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:42 smithi130 ceph-mon[131859]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:42.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3480419253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd0073550 msgr2=0x7f2dd00739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3480419253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd0073550 0x7f2dd00739c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009990 tx=0x7f2dbc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3480419253 shutdown_connections 2023-10-11T16:30:42.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3480419253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dd010c860 0x7f2dd010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3480419253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd0074b30 0x7f2dd0072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3480419253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd0073550 0x7f2dd00739c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3480419253 >> 172.21.15.111:0/3480419253 conn(0x7f2dd00fb870 msgr2=0x7f2dd00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:42.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3480419253 shutdown_connections 2023-10-11T16:30:42.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.706+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3480419253 wait complete. 2023-10-11T16:30:42.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.707+0000 7f2dd6dff700 1 Processor -- start 2023-10-11T16:30:42.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.707+0000 7f2dd6dff700 1 -- start start 2023-10-11T16:30:42.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.707+0000 7f2dd6dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd0073550 0x7f2dd019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.707+0000 7f2dd6dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dd0074b30 0x7f2dd019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.707+0000 7f2dd6dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd010c860 0x7f2dd01a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.707+0000 7f2dd6dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd01a4170 con 0x7f2dd010c860 2023-10-11T16:30:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.707+0000 7f2dd6dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd01a42b0 con 0x7f2dd0074b30 2023-10-11T16:30:42.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.707+0000 7f2dd6dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd01a4440 con 0x7f2dd0073550 2023-10-11T16:30:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd010c860 0x7f2dd01a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd010c860 0x7f2dd01a3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49836/0 (socket says 172.21.15.111:49836) 2023-10-11T16:30:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 -- 172.21.15.111:0/3180012360 learned_addr learned my addr 172.21.15.111:0/3180012360 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:42.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd4b9b700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd0073550 0x7f2dd019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:42.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dcffff700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dd0074b30 0x7f2dd019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:42.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 -- 172.21.15.111:0/3180012360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd0073550 msgr2=0x7f2dd019c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd0073550 0x7f2dd019c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 -- 172.21.15.111:0/3180012360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dd0074b30 msgr2=0x7f2dd019cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dd0074b30 0x7f2dd019cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 -- 172.21.15.111:0/3180012360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4009520 con 0x7f2dd010c860 2023-10-11T16:30:42.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd539c700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd010c860 0x7f2dd01a3ba0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800ef30 tx=0x7f2dc800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:42.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dcdffb700 1 -- 172.21.15.111:0/3180012360 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8018070 con 0x7f2dd010c860 2023-10-11T16:30:42.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dcdffb700 1 -- 172.21.15.111:0/3180012360 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dc800f040 con 0x7f2dd010c860 2023-10-11T16:30:42.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.708+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dbc0095f0 con 0x7f2dd010c860 2023-10-11T16:30:42.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.709+0000 7f2dcdffb700 1 -- 172.21.15.111:0/3180012360 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc800cd00 con 0x7f2dd010c860 2023-10-11T16:30:42.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.709+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd0112270 con 0x7f2dd010c860 2023-10-11T16:30:42.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.710+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd01023b0 con 0x7f2dd010c860 2023-10-11T16:30:42.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.711+0000 7f2dcdffb700 1 -- 172.21.15.111:0/3180012360 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f2dc8003a00 con 0x7f2dd010c860 2023-10-11T16:30:42.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.713+0000 7f2dcdffb700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2dc00a11e0 0x7f2dc00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:42.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.713+0000 7f2dcdffb700 1 -- 172.21.15.111:0/3180012360 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2dc80cc8d0 con 0x7f2dd010c860 2023-10-11T16:30:42.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.713+0000 7f2dd4b9b700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2dc00a11e0 0x7f2dc00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:42.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.714+0000 7f2dd4b9b700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2dc00a11e0 0x7f2dc00a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0074f00 tx=0x7f2dc4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:42.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.716+0000 7f2dcdffb700 1 -- 172.21.15.111:0/3180012360 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2dc8095c40 con 0x7f2dd010c860 2023-10-11T16:30:42.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.992+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2dd002cfe0 con 0x7f2dd010c860 2023-10-11T16:30:42.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.993+0000 7f2dcdffb700 1 -- 172.21.15.111:0/3180012360 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3059 (secure 0 0 0) 0x7f2dc8099260 con 0x7f2dd010c860 2023-10-11T16:30:42.995 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:42.995 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":367,"monmap":{"epoch":3,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":89,"num_osds":12,"num_up_osds":12,"osd_up_since":1697041763,"num_in_osds":12,"osd_in_since":1697041746,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":128},{"state_name":"peering","count":1}],"num_pgs":129,"num_pools":4,"num_objects":60,"data_bytes":597340,"bytes_used":81764352,"bytes_avail":1151858663424,"bytes_total":1151940427776,"inactive_pgs_ratio":0.0077519379556179047,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":72,"recovering_keys_per_sec":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"read_bytes_sec":2047,"write_bytes_sec":3455,"read_op_per_sec":2,"write_op_per_sec":9},"fsmap":{"epoch":24,"id":1,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":1,"rank":2,"name":"e","status":"up:standby-replay","gid":24488},{"filesystem_id":1,"rank":1,"name":"f","status":"up:standby-replay","gid":24520},{"filesystem_id":1,"rank":0,"name":"h","status":"up:standby-replay","gid":24521},{"filesystem_id":1,"rank":2,"name":"c","status":"up:active","gid":24532},{"filesystem_id":1,"rank":1,"name":"i","status":"up:active","gid":24550},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24569}],"up:standby":6},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.111:8443/"}},"servicemap":{"epoch":20,"modified":"2023-10-11T16:30:42.058592+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"e":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"f":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"g":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"h":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"i":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"j":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-10-11T16:30:42.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2dc00a11e0 msgr2=0x7f2dc00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2dc00a11e0 0x7f2dc00a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0074f00 tx=0x7f2dc4009260 comp rx=0 tx=0).stop 2023-10-11T16:30:42.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd010c860 msgr2=0x7f2dd01a3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:42.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd010c860 0x7f2dd01a3ba0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800ef30 tx=0x7f2dc800c3a0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 shutdown_connections 2023-10-11T16:30:42.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2dc00a11e0 0x7f2dc00a3690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dd010c860 0x7f2dd01a3ba0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dd0073550 0x7f2dd019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 --2- 172.21.15.111:0/3180012360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dd0074b30 0x7f2dd019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:42.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 >> 172.21.15.111:0/3180012360 conn(0x7f2dd00fb870 msgr2=0x7f2dd00fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:42.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 shutdown_connections 2023-10-11T16:30:42.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:42.995+0000 7f2dd6dff700 1 -- 172.21.15.111:0/3180012360 wait complete. 2023-10-11T16:30:43.015 INFO:tasks.cephfs.filesystem.ceph_manager:making progress, resetting timeout 2023-10-11T16:30:43.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T16:30:43.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 -- 172.21.15.111:0/689311051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 msgr2=0x7fc050103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:43.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 --2- 172.21.15.111:0/689311051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 0x7fc050103b80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc044009990 tx=0x7fc044009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 -- 172.21.15.111:0/689311051 shutdown_connections 2023-10-11T16:30:43.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 --2- 172.21.15.111:0/689311051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0501040c0 0x7fc05010a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 --2- 172.21.15.111:0/689311051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc050102540 0x7fc050102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 --2- 172.21.15.111:0/689311051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 0x7fc050103b80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 -- 172.21.15.111:0/689311051 >> 172.21.15.111:0/689311051 conn(0x7fc0500fdb20 msgr2=0x7fc0500fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:43.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 -- 172.21.15.111:0/689311051 shutdown_connections 2023-10-11T16:30:43.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.215+0000 7fc0555ff700 1 -- 172.21.15.111:0/689311051 wait complete. 2023-10-11T16:30:43.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc0555ff700 1 Processor -- start 2023-10-11T16:30:43.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc0555ff700 1 -- start start 2023-10-11T16:30:43.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc0555ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc050102540 0x7fc050198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:43.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc0555ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 0x7fc050198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:43.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc0555ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0501040c0 0x7fc05019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:43.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc0555ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05019fdd0 con 0x7fc050103730 2023-10-11T16:30:43.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc0555ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05019ff10 con 0x7fc050102540 2023-10-11T16:30:43.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc0555ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0501a00a0 con 0x7fc0501040c0 2023-10-11T16:30:43.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc04e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 0x7fc050198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:43.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc04effd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc050102540 0x7fc050198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:43.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc04e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 0x7fc050198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49862/0 (socket says 172.21.15.111:49862) 2023-10-11T16:30:43.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.216+0000 7fc04e7fc700 1 -- 172.21.15.111:0/2207283901 learned_addr learned my addr 172.21.15.111:0/2207283901 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:43.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc04f7fe700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0501040c0 0x7fc05019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:43.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc04e7fc700 1 -- 172.21.15.111:0/2207283901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0501040c0 msgr2=0x7fc05019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:43.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc04e7fc700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0501040c0 0x7fc05019f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc04e7fc700 1 -- 172.21.15.111:0/2207283901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc050102540 msgr2=0x7fc050198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:43.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc04e7fc700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc050102540 0x7fc050198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc04e7fc700 1 -- 172.21.15.111:0/2207283901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440095f0 con 0x7fc050103730 2023-10-11T16:30:43.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc04e7fc700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 0x7fc050198790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc044004760 tx=0x7fc044004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:43.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc037fff700 1 -- 172.21.15.111:0/2207283901 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04401d070 con 0x7fc050103730 2023-10-11T16:30:43.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc037fff700 1 -- 172.21.15.111:0/2207283901 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc044004830 con 0x7fc050103730 2023-10-11T16:30:43.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05010de90 con 0x7fc050103730 2023-10-11T16:30:43.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.217+0000 7fc037fff700 1 -- 172.21.15.111:0/2207283901 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044005350 con 0x7fc050103730 2023-10-11T16:30:43.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.218+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05010dd00 con 0x7fc050103730 2023-10-11T16:30:43.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.219+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc050109b50 con 0x7fc050103730 2023-10-11T16:30:43.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.220+0000 7fc037fff700 1 -- 172.21.15.111:0/2207283901 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fc044005510 con 0x7fc050103730 2023-10-11T16:30:43.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.222+0000 7fc037fff700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0380a11e0 0x7fc0380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:43.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.222+0000 7fc037fff700 1 -- 172.21.15.111:0/2207283901 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc0440cdfb0 con 0x7fc050103730 2023-10-11T16:30:43.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.222+0000 7fc04effd700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0380a11e0 0x7fc0380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:43.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.223+0000 7fc04effd700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0380a11e0 0x7fc0380a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc050103590 tx=0x7fc040009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:43.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.225+0000 7fc037fff700 1 -- 172.21.15.111:0/2207283901 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc04405a730 con 0x7fc050103730 2023-10-11T16:30:43.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:43 smithi111 ceph-mon[98527]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:43 smithi111 ceph-mon[98527]: pgmap v290: 129 pgs: 1 peering, 128 active+clean; 583 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3.4 KiB/s wr, 12 op/s; 72 B/s, 0 objects/s recovering 2023-10-11T16:30:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:43 smithi111 ceph-mon[98527]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:43 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3180012360' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-10-11T16:30:43.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.421+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc050061b50 con 0x7fc0380a11e0 2023-10-11T16:30:43.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.433+0000 7fc037fff700 1 -- 172.21.15.111:0/2207283901 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423887 (secure 0 0 0) 0x7fc050061b50 con 0x7fc0380a11e0 2023-10-11T16:30:43.440 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:43.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0380a11e0 msgr2=0x7fc0380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:43.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0380a11e0 0x7fc0380a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc050103590 tx=0x7fc040009260 comp rx=0 tx=0).stop 2023-10-11T16:30:43.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 msgr2=0x7fc050198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:43.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 0x7fc050198790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc044004760 tx=0x7fc044004790 comp rx=0 tx=0).stop 2023-10-11T16:30:43.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 shutdown_connections 2023-10-11T16:30:43.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc0380a11e0 0x7fc0380a3690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0501040c0 0x7fc05019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc050102540 0x7fc050198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 --2- 172.21.15.111:0/2207283901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc050103730 0x7fc050198790 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:43.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 >> 172.21.15.111:0/2207283901 conn(0x7fc0500fdb20 msgr2=0x7fc050106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:43.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 shutdown_connections 2023-10-11T16:30:43.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:43.438+0000 7fc0555ff700 1 -- 172.21.15.111:0/2207283901 wait complete. 2023-10-11T16:30:43.446 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:30:43.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:43 smithi150 ceph-mon[132217]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:43.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:43 smithi150 ceph-mon[132217]: pgmap v290: 129 pgs: 1 peering, 128 active+clean; 583 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3.4 KiB/s wr, 12 op/s; 72 B/s, 0 objects/s recovering 2023-10-11T16:30:43.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:43 smithi150 ceph-mon[132217]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:43.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:43 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3180012360' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-10-11T16:30:43.459 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2023-10-11T16:30:42.058445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597340,"num_objects":60,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":157,"num_read_kb":135,"num_write":308,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":248,"ondisk_log_size":248,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":416,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79848,"kb_used_data":11464,"kb_used_omap":0,"kb_used_meta":68288,"kb_avail":1124861976,"statfs":{"total":1151940427776,"available":1151858663424,"internally_reserved":0,"allocated":11739136,"data_stored":7465353,"data_compressed":255891,"data_compressed_allocated":344064,"data_compressed_original":2187264,"omap_allocated":0,"internal_metadata":69926912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2521,"num_objects":36,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":79,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002222"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284183+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:30:17.284183+0000","last_peered":"2023-10-11T16:30:17.284183+0000","last_clean":"2023-10-11T16:30:17.284183+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:30:17.284183+0000","last_undegraded":"2023-10-11T16:30:17.284183+0000","last_fullsized":"2023-10-11T16:30:17.284183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:26.542965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612056+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:30:16.612056+0000","last_peered":"2023-10-11T16:30:16.612056+0000","last_clean":"2023-10-11T16:30:16.612056+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:30:16.612056+0000","last_undegraded":"2023-10-11T16:30:16.612056+0000","last_fullsized":"2023-10-11T16:30:16.612056+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:08:41.790985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612043+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:30:16.612043+0000","last_peered":"2023-10-11T16:30:16.612043+0000","last_clean":"2023-10-11T16:30:16.612043+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:30:16.612043+0000","last_undegraded":"2023-10-11T16:30:16.612043+0000","last_fullsized":"2023-10-11T16:30:16.612043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:43.188283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668622+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:30:17.668622+0000","last_peered":"2023-10-11T16:30:17.668622+0000","last_clean":"2023-10-11T16:30:17.668622+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:30:17.668622+0000","last_undegraded":"2023-10-11T16:30:17.668622+0000","last_fullsized":"2023-10-11T16:30:17.668622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:33:52.126432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614398+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:30:16.614398+0000","last_peered":"2023-10-11T16:30:16.614398+0000","last_clean":"2023-10-11T16:30:16.614398+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:30:16.614398+0000","last_undegraded":"2023-10-11T16:30:16.614398+0000","last_fullsized":"2023-10-11T16:30:16.614398+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:59.778582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669108+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:30:17.669108+0000","last_peered":"2023-10-11T16:30:17.669108+0000","last_clean":"2023-10-11T16:30:17.669108+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:30:17.669108+0000","last_undegraded":"2023-10-11T16:30:17.669108+0000","last_fullsized":"2023-10-11T16:30:17.669108+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:37:10.884146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612956+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:30:16.612956+0000","last_peered":"2023-10-11T16:30:16.612956+0000","last_clean":"2023-10-11T16:30:16.612956+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:30:16.612956+0000","last_undegraded":"2023-10-11T16:30:16.612956+0000","last_fullsized":"2023-10-11T16:30:16.612956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:14:14.116406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668148+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:30:17.668148+0000","last_peered":"2023-10-11T16:30:17.668148+0000","last_clean":"2023-10-11T16:30:17.668148+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:30:17.668148+0000","last_undegraded":"2023-10-11T16:30:17.668148+0000","last_fullsized":"2023-10-11T16:30:17.668148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:24:57.989531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668680+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:30:17.668680+0000","last_peered":"2023-10-11T16:30:17.668680+0000","last_clean":"2023-10-11T16:30:17.668680+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:30:17.668680+0000","last_undegraded":"2023-10-11T16:30:17.668680+0000","last_fullsized":"2023-10-11T16:30:17.668680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:41:45.811568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668330+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:30:17.668330+0000","last_peered":"2023-10-11T16:30:17.668330+0000","last_clean":"2023-10-11T16:30:17.668330+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:30:17.668330+0000","last_undegraded":"2023-10-11T16:30:17.668330+0000","last_fullsized":"2023-10-11T16:30:17.668330+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:28.841961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613837+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:30:16.613837+0000","last_peered":"2023-10-11T16:30:16.613837+0000","last_clean":"2023-10-11T16:30:16.613837+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:30:16.613837+0000","last_undegraded":"2023-10-11T16:30:16.613837+0000","last_fullsized":"2023-10-11T16:30:16.613837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:13:18.450772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668462+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:30:17.668462+0000","last_peered":"2023-10-11T16:30:17.668462+0000","last_clean":"2023-10-11T16:30:17.668462+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:30:17.668462+0000","last_undegraded":"2023-10-11T16:30:17.668462+0000","last_fullsized":"2023-10-11T16:30:17.668462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:17:32.551928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669875+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:30:17.669875+0000","last_peered":"2023-10-11T16:30:17.669875+0000","last_clean":"2023-10-11T16:30:17.669875+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:30:17.669875+0000","last_undegraded":"2023-10-11T16:30:17.669875+0000","last_fullsized":"2023-10-11T16:30:17.669875+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:10:27.344516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611789+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:30:16.611789+0000","last_peered":"2023-10-11T16:30:16.611789+0000","last_clean":"2023-10-11T16:30:16.611789+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:30:16.611789+0000","last_undegraded":"2023-10-11T16:30:16.611789+0000","last_fullsized":"2023-10-11T16:30:16.611789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:34:22.094507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611778+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:30:16.611778+0000","last_peered":"2023-10-11T16:30:16.611778+0000","last_clean":"2023-10-11T16:30:16.611778+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:30:16.611778+0000","last_undegraded":"2023-10-11T16:30:16.611778+0000","last_fullsized":"2023-10-11T16:30:16.611778+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:15.576706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667849+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:30:17.667849+0000","last_peered":"2023-10-11T16:30:17.667849+0000","last_clean":"2023-10-11T16:30:17.667849+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:30:17.667849+0000","last_undegraded":"2023-10-11T16:30:17.667849+0000","last_fullsized":"2023-10-11T16:30:17.667849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:36:36.528407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284123+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:30:17.284123+0000","last_peered":"2023-10-11T16:30:17.284123+0000","last_clean":"2023-10-11T16:30:17.284123+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:30:17.284123+0000","last_undegraded":"2023-10-11T16:30:17.284123+0000","last_fullsized":"2023-10-11T16:30:17.284123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:47:55.247727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182794+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:30:41.182794+0000","last_peered":"2023-10-11T16:30:41.182794+0000","last_clean":"2023-10-11T16:30:41.182794+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:30:41.182794+0000","last_undegraded":"2023-10-11T16:30:41.182794+0000","last_fullsized":"2023-10-11T16:30:41.182794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:09:27.215051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185311+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:30:41.185311+0000","last_peered":"2023-10-11T16:30:41.185311+0000","last_clean":"2023-10-11T16:30:41.185311+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:30:41.185311+0000","last_undegraded":"2023-10-11T16:30:41.185311+0000","last_fullsized":"2023-10-11T16:30:41.185311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:46:35.027109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186180+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.186180+0000","last_peered":"2023-10-11T16:30:41.186180+0000","last_clean":"2023-10-11T16:30:41.186180+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:30:41.186180+0000","last_undegraded":"2023-10-11T16:30:41.186180+0000","last_fullsized":"2023-10-11T16:30:41.186180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:05.403679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198854+0000","last_change":"2023-10-11T16:30:41.198854+0000","last_active":"2023-10-11T16:30:41.198854+0000","last_peered":"2023-10-11T16:30:41.198854+0000","last_clean":"2023-10-11T16:30:41.198854+0000","last_became_active":"2023-10-11T16:30:41.189677+0000","last_became_peered":"2023-10-11T16:30:41.189677+0000","last_unstale":"2023-10-11T16:30:41.198854+0000","last_undegraded":"2023-10-11T16:30:41.198854+0000","last_fullsized":"2023-10-11T16:30:41.198854+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:58:40.952333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668383+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:30:17.668383+0000","last_peered":"2023-10-11T16:30:17.668383+0000","last_clean":"2023-10-11T16:30:17.668383+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:30:17.668383+0000","last_undegraded":"2023-10-11T16:30:17.668383+0000","last_fullsized":"2023-10-11T16:30:17.668383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:10:24.304620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667627+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:30:17.667627+0000","last_peered":"2023-10-11T16:30:17.667627+0000","last_clean":"2023-10-11T16:30:17.667627+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:30:17.667627+0000","last_undegraded":"2023-10-11T16:30:17.667627+0000","last_fullsized":"2023-10-11T16:30:17.667627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:59:03.310068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668885+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:30:17.668885+0000","last_peered":"2023-10-11T16:30:17.668885+0000","last_clean":"2023-10-11T16:30:17.668885+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:30:17.668885+0000","last_undegraded":"2023-10-11T16:30:17.668885+0000","last_fullsized":"2023-10-11T16:30:17.668885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:29:49.809378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612215+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:30:16.612215+0000","last_peered":"2023-10-11T16:30:16.612215+0000","last_clean":"2023-10-11T16:30:16.612215+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:30:16.612215+0000","last_undegraded":"2023-10-11T16:30:16.612215+0000","last_fullsized":"2023-10-11T16:30:16.612215+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:45:37.863112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668530+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:30:17.668530+0000","last_peered":"2023-10-11T16:30:17.668530+0000","last_clean":"2023-10-11T16:30:17.668530+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:30:17.668530+0000","last_undegraded":"2023-10-11T16:30:17.668530+0000","last_fullsized":"2023-10-11T16:30:17.668530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:47:05.876657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.173667+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:30:37.173667+0000","last_peered":"2023-10-11T16:30:37.173667+0000","last_clean":"2023-10-11T16:30:37.173667+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:30:37.173667+0000","last_undegraded":"2023-10-11T16:30:37.173667+0000","last_fullsized":"2023-10-11T16:30:37.173667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:55:47.819813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668277+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:30:17.668277+0000","last_peered":"2023-10-11T16:30:17.668277+0000","last_clean":"2023-10-11T16:30:17.668277+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:30:17.668277+0000","last_undegraded":"2023-10-11T16:30:17.668277+0000","last_fullsized":"2023-10-11T16:30:17.668277+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:27:30.832381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667964+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:30:17.667964+0000","last_peered":"2023-10-11T16:30:17.667964+0000","last_clean":"2023-10-11T16:30:17.667964+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:30:17.667964+0000","last_undegraded":"2023-10-11T16:30:17.667964+0000","last_fullsized":"2023-10-11T16:30:17.667964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:49:38.616403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614611+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:30:16.614611+0000","last_peered":"2023-10-11T16:30:16.614611+0000","last_clean":"2023-10-11T16:30:16.614611+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:30:16.614611+0000","last_undegraded":"2023-10-11T16:30:16.614611+0000","last_fullsized":"2023-10-11T16:30:16.614611+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:36:36.308880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":87,"state":"peering","last_fresh":"2023-10-11T16:30:17.668751+0000","last_change":"2023-10-11T16:30:40.187333+0000","last_active":"2023-10-11T16:30:40.187333+0000","last_peered":"2023-10-11T16:30:17.668751+0000","last_clean":"2023-10-11T16:30:17.668751+0000","last_became_active":"2023-10-11T16:30:11.658658+0000","last_became_peered":"2023-10-11T16:30:11.658658+0000","last_unstale":"2023-10-11T16:30:17.668751+0000","last_undegraded":"2023-10-11T16:30:17.668751+0000","last_fullsized":"2023-10-11T16:30:17.668751+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,8],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185067+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:30:41.185067+0000","last_peered":"2023-10-11T16:30:41.185067+0000","last_clean":"2023-10-11T16:30:41.185067+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:30:41.185067+0000","last_undegraded":"2023-10-11T16:30:41.185067+0000","last_fullsized":"2023-10-11T16:30:41.185067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:48.316789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187858+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:30:40.187858+0000","last_peered":"2023-10-11T16:30:40.187858+0000","last_clean":"2023-10-11T16:30:40.187858+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:30:40.187858+0000","last_undegraded":"2023-10-11T16:30:40.187858+0000","last_fullsized":"2023-10-11T16:30:40.187858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:51.299737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614368+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:30:16.614368+0000","last_peered":"2023-10-11T16:30:16.614368+0000","last_clean":"2023-10-11T16:30:16.614368+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:30:16.614368+0000","last_undegraded":"2023-10-11T16:30:16.614368+0000","last_fullsized":"2023-10-11T16:30:16.614368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:59:43.658557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186478+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:30:41.186478+0000","last_peered":"2023-10-11T16:30:41.186478+0000","last_clean":"2023-10-11T16:30:41.186478+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:30:41.186478+0000","last_undegraded":"2023-10-11T16:30:41.186478+0000","last_fullsized":"2023-10-11T16:30:41.186478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:20:03.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176047+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:30:37.176047+0000","last_peered":"2023-10-11T16:30:37.176047+0000","last_clean":"2023-10-11T16:30:37.176047+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:30:37.176047+0000","last_undegraded":"2023-10-11T16:30:37.176047+0000","last_fullsized":"2023-10-11T16:30:37.176047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:38:45.180274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.188012+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:30:40.188012+0000","last_peered":"2023-10-11T16:30:40.188012+0000","last_clean":"2023-10-11T16:30:40.188012+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:30:40.188012+0000","last_undegraded":"2023-10-11T16:30:40.188012+0000","last_fullsized":"2023-10-11T16:30:40.188012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:36:53.041380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284122+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:30:17.284122+0000","last_peered":"2023-10-11T16:30:17.284122+0000","last_clean":"2023-10-11T16:30:17.284122+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:30:17.284122+0000","last_undegraded":"2023-10-11T16:30:17.284122+0000","last_fullsized":"2023-10-11T16:30:17.284122+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:35:11.414446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.177276+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:30:37.177276+0000","last_peered":"2023-10-11T16:30:37.177276+0000","last_clean":"2023-10-11T16:30:37.177276+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:30:37.177276+0000","last_undegraded":"2023-10-11T16:30:37.177276+0000","last_fullsized":"2023-10-11T16:30:37.177276+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:44:31.081697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668117+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:30:17.668117+0000","last_peered":"2023-10-11T16:30:17.668117+0000","last_clean":"2023-10-11T16:30:17.668117+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:30:17.668117+0000","last_undegraded":"2023-10-11T16:30:17.668117+0000","last_fullsized":"2023-10-11T16:30:17.668117+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:20:03.123703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612081+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:30:16.612081+0000","last_peered":"2023-10-11T16:30:16.612081+0000","last_clean":"2023-10-11T16:30:16.612081+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:30:16.612081+0000","last_undegraded":"2023-10-11T16:30:16.612081+0000","last_fullsized":"2023-10-11T16:30:16.612081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:44:54.955030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186294+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:30:40.186294+0000","last_peered":"2023-10-11T16:30:40.186294+0000","last_clean":"2023-10-11T16:30:40.186294+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:30:40.186294+0000","last_undegraded":"2023-10-11T16:30:40.186294+0000","last_fullsized":"2023-10-11T16:30:40.186294+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:19:55.442658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181702+0000","last_change":"2023-10-11T16:30:11.662850+0000","last_active":"2023-10-11T16:30:41.181702+0000","last_peered":"2023-10-11T16:30:41.181702+0000","last_clean":"2023-10-11T16:30:41.181702+0000","last_became_active":"2023-10-11T16:30:11.661052+0000","last_became_peered":"2023-10-11T16:30:11.661052+0000","last_unstale":"2023-10-11T16:30:41.181702+0000","last_undegraded":"2023-10-11T16:30:41.181702+0000","last_fullsized":"2023-10-11T16:30:41.181702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:23.324421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667905+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:30:17.667905+0000","last_peered":"2023-10-11T16:30:17.667905+0000","last_clean":"2023-10-11T16:30:17.667905+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:30:17.667905+0000","last_undegraded":"2023-10-11T16:30:17.667905+0000","last_fullsized":"2023-10-11T16:30:17.667905+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:07.189663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166341+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:30:36.166341+0000","last_peered":"2023-10-11T16:30:36.166341+0000","last_clean":"2023-10-11T16:30:36.166341+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:30:36.166341+0000","last_undegraded":"2023-10-11T16:30:36.166341+0000","last_fullsized":"2023-10-11T16:30:36.166341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:36:57.466108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669129+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:30:17.669129+0000","last_peered":"2023-10-11T16:30:17.669129+0000","last_clean":"2023-10-11T16:30:17.669129+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:30:17.669129+0000","last_undegraded":"2023-10-11T16:30:17.669129+0000","last_fullsized":"2023-10-11T16:30:17.669129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:44:17.713245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181694+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:30:41.181694+0000","last_peered":"2023-10-11T16:30:41.181694+0000","last_clean":"2023-10-11T16:30:41.181694+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:30:41.181694+0000","last_undegraded":"2023-10-11T16:30:41.181694+0000","last_fullsized":"2023-10-11T16:30:41.181694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:32.776111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165661+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:30:36.165661+0000","last_peered":"2023-10-11T16:30:36.165661+0000","last_clean":"2023-10-11T16:30:36.165661+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:30:36.165661+0000","last_undegraded":"2023-10-11T16:30:36.165661+0000","last_fullsized":"2023-10-11T16:30:36.165661+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:24:23.101409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612235+0000","last_change":"2023-10-11T16:30:11.666124+0000","last_active":"2023-10-11T16:30:16.612235+0000","last_peered":"2023-10-11T16:30:16.612235+0000","last_clean":"2023-10-11T16:30:16.612235+0000","last_became_active":"2023-10-11T16:30:11.665786+0000","last_became_peered":"2023-10-11T16:30:11.665786+0000","last_unstale":"2023-10-11T16:30:16.612235+0000","last_undegraded":"2023-10-11T16:30:16.612235+0000","last_fullsized":"2023-10-11T16:30:16.612235+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:45:32.355550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669108+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:30:17.669108+0000","last_peered":"2023-10-11T16:30:17.669108+0000","last_clean":"2023-10-11T16:30:17.669108+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:30:17.669108+0000","last_undegraded":"2023-10-11T16:30:17.669108+0000","last_fullsized":"2023-10-11T16:30:17.669108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:20:20.621738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187532+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:30:40.187532+0000","last_peered":"2023-10-11T16:30:40.187532+0000","last_clean":"2023-10-11T16:30:40.187532+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:30:40.187532+0000","last_undegraded":"2023-10-11T16:30:40.187532+0000","last_fullsized":"2023-10-11T16:30:40.187532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:45:20.227087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668685+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:30:17.668685+0000","last_peered":"2023-10-11T16:30:17.668685+0000","last_clean":"2023-10-11T16:30:17.668685+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:30:17.668685+0000","last_undegraded":"2023-10-11T16:30:17.668685+0000","last_fullsized":"2023-10-11T16:30:17.668685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:02:09.993229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181930+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:30:41.181930+0000","last_peered":"2023-10-11T16:30:41.181930+0000","last_clean":"2023-10-11T16:30:41.181930+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:30:41.181930+0000","last_undegraded":"2023-10-11T16:30:41.181930+0000","last_fullsized":"2023-10-11T16:30:41.181930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:27:45.123842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'12","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.165516+0000","last_change":"2023-10-11T16:30:11.112780+0000","last_active":"2023-10-11T16:30:36.165516+0000","last_peered":"2023-10-11T16:30:36.165516+0000","last_clean":"2023-10-11T16:30:36.165516+0000","last_became_active":"2023-10-11T16:30:11.112481+0000","last_became_peered":"2023-10-11T16:30:11.112481+0000","last_unstale":"2023-10-11T16:30:36.165516+0000","last_undegraded":"2023-10-11T16:30:36.165516+0000","last_fullsized":"2023-10-11T16:30:36.165516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:44.261584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614775+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:30:16.614775+0000","last_peered":"2023-10-11T16:30:16.614775+0000","last_clean":"2023-10-11T16:30:16.614775+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:30:16.614775+0000","last_undegraded":"2023-10-11T16:30:16.614775+0000","last_fullsized":"2023-10-11T16:30:16.614775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:14:04.119677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.189579+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:30:40.189579+0000","last_peered":"2023-10-11T16:30:40.189579+0000","last_clean":"2023-10-11T16:30:40.189579+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:30:40.189579+0000","last_undegraded":"2023-10-11T16:30:40.189579+0000","last_fullsized":"2023-10-11T16:30:40.189579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:26.634024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185142+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:30:41.185142+0000","last_peered":"2023-10-11T16:30:41.185142+0000","last_clean":"2023-10-11T16:30:41.185142+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:30:41.185142+0000","last_undegraded":"2023-10-11T16:30:41.185142+0000","last_fullsized":"2023-10-11T16:30:41.185142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:25:53.623192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668870+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:30:17.668870+0000","last_peered":"2023-10-11T16:30:17.668870+0000","last_clean":"2023-10-11T16:30:17.668870+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:30:17.668870+0000","last_undegraded":"2023-10-11T16:30:17.668870+0000","last_fullsized":"2023-10-11T16:30:17.668870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:15:57.948212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283849+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:30:17.283849+0000","last_peered":"2023-10-11T16:30:17.283849+0000","last_clean":"2023-10-11T16:30:17.283849+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:30:17.283849+0000","last_undegraded":"2023-10-11T16:30:17.283849+0000","last_fullsized":"2023-10-11T16:30:17.283849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:27:49.373457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187910+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:30:40.187910+0000","last_peered":"2023-10-11T16:30:40.187910+0000","last_clean":"2023-10-11T16:30:40.187910+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:30:40.187910+0000","last_undegraded":"2023-10-11T16:30:40.187910+0000","last_fullsized":"2023-10-11T16:30:40.187910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:56:20.300209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613601+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:30:16.613601+0000","last_peered":"2023-10-11T16:30:16.613601+0000","last_clean":"2023-10-11T16:30:16.613601+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:30:16.613601+0000","last_undegraded":"2023-10-11T16:30:16.613601+0000","last_fullsized":"2023-10-11T16:30:16.613601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:34.553676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182281+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.182281+0000","last_peered":"2023-10-11T16:30:41.182281+0000","last_clean":"2023-10-11T16:30:41.182281+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:30:41.182281+0000","last_undegraded":"2023-10-11T16:30:41.182281+0000","last_fullsized":"2023-10-11T16:30:41.182281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:35:47.449805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:38.189202+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:30:38.189202+0000","last_peered":"2023-10-11T16:30:38.189202+0000","last_clean":"2023-10-11T16:30:38.189202+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:30:38.189202+0000","last_undegraded":"2023-10-11T16:30:38.189202+0000","last_fullsized":"2023-10-11T16:30:38.189202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:25:21.420979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613021+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:30:16.613021+0000","last_peered":"2023-10-11T16:30:16.613021+0000","last_clean":"2023-10-11T16:30:16.613021+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:30:16.613021+0000","last_undegraded":"2023-10-11T16:30:16.613021+0000","last_fullsized":"2023-10-11T16:30:16.613021+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:10:00.701910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669925+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:30:17.669925+0000","last_peered":"2023-10-11T16:30:17.669925+0000","last_clean":"2023-10-11T16:30:17.669925+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:30:17.669925+0000","last_undegraded":"2023-10-11T16:30:17.669925+0000","last_fullsized":"2023-10-11T16:30:17.669925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:46:40.422035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176084+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:30:37.176084+0000","last_peered":"2023-10-11T16:30:37.176084+0000","last_clean":"2023-10-11T16:30:37.176084+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:30:37.176084+0000","last_undegraded":"2023-10-11T16:30:37.176084+0000","last_fullsized":"2023-10-11T16:30:37.176084+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:15.680165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187485+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:30:40.187485+0000","last_peered":"2023-10-11T16:30:40.187485+0000","last_clean":"2023-10-11T16:30:40.187485+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:30:40.187485+0000","last_undegraded":"2023-10-11T16:30:40.187485+0000","last_fullsized":"2023-10-11T16:30:40.187485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:58:29.097161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187329+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:30:40.187329+0000","last_peered":"2023-10-11T16:30:40.187329+0000","last_clean":"2023-10-11T16:30:40.187329+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:30:40.187329+0000","last_undegraded":"2023-10-11T16:30:40.187329+0000","last_fullsized":"2023-10-11T16:30:40.187329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:21:39.470036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.615240+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:30:16.615240+0000","last_peered":"2023-10-11T16:30:16.615240+0000","last_clean":"2023-10-11T16:30:16.615240+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:30:16.615240+0000","last_undegraded":"2023-10-11T16:30:16.615240+0000","last_fullsized":"2023-10-11T16:30:16.615240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:02:48.445205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284055+0000","last_change":"2023-10-11T16:30:11.670547+0000","last_active":"2023-10-11T16:30:17.284055+0000","last_peered":"2023-10-11T16:30:17.284055+0000","last_clean":"2023-10-11T16:30:17.284055+0000","last_became_active":"2023-10-11T16:30:11.669842+0000","last_became_peered":"2023-10-11T16:30:11.669842+0000","last_unstale":"2023-10-11T16:30:17.284055+0000","last_undegraded":"2023-10-11T16:30:17.284055+0000","last_fullsized":"2023-10-11T16:30:17.284055+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:20:40.999317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614361+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:30:16.614361+0000","last_peered":"2023-10-11T16:30:16.614361+0000","last_clean":"2023-10-11T16:30:16.614361+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:30:16.614361+0000","last_undegraded":"2023-10-11T16:30:16.614361+0000","last_fullsized":"2023-10-11T16:30:16.614361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:13.782416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182101+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:30:41.182101+0000","last_peered":"2023-10-11T16:30:41.182101+0000","last_clean":"2023-10-11T16:30:41.182101+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:30:41.182101+0000","last_undegraded":"2023-10-11T16:30:41.182101+0000","last_fullsized":"2023-10-11T16:30:41.182101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:36.264713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668683+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:30:17.668683+0000","last_peered":"2023-10-11T16:30:17.668683+0000","last_clean":"2023-10-11T16:30:17.668683+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:30:17.668683+0000","last_undegraded":"2023-10-11T16:30:17.668683+0000","last_fullsized":"2023-10-11T16:30:17.668683+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:10:24.647448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669638+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:30:17.669638+0000","last_peered":"2023-10-11T16:30:17.669638+0000","last_clean":"2023-10-11T16:30:17.669638+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:30:17.669638+0000","last_undegraded":"2023-10-11T16:30:17.669638+0000","last_fullsized":"2023-10-11T16:30:17.669638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:12:58.596577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613284+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:30:16.613284+0000","last_peered":"2023-10-11T16:30:16.613284+0000","last_clean":"2023-10-11T16:30:16.613284+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:30:16.613284+0000","last_undegraded":"2023-10-11T16:30:16.613284+0000","last_fullsized":"2023-10-11T16:30:16.613284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:33:51.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668320+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:30:17.668320+0000","last_peered":"2023-10-11T16:30:17.668320+0000","last_clean":"2023-10-11T16:30:17.668320+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:30:17.668320+0000","last_undegraded":"2023-10-11T16:30:17.668320+0000","last_fullsized":"2023-10-11T16:30:17.668320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:32:12.254443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669773+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:30:17.669773+0000","last_peered":"2023-10-11T16:30:17.669773+0000","last_clean":"2023-10-11T16:30:17.669773+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:30:17.669773+0000","last_undegraded":"2023-10-11T16:30:17.669773+0000","last_fullsized":"2023-10-11T16:30:17.669773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:09:37.408810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186478+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:30:40.186478+0000","last_peered":"2023-10-11T16:30:40.186478+0000","last_clean":"2023-10-11T16:30:40.186478+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:30:40.186478+0000","last_undegraded":"2023-10-11T16:30:40.186478+0000","last_fullsized":"2023-10-11T16:30:40.186478+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:38:23.165604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.176027+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:30:37.176027+0000","last_peered":"2023-10-11T16:30:37.176027+0000","last_clean":"2023-10-11T16:30:37.176027+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:30:37.176027+0000","last_undegraded":"2023-10-11T16:30:37.176027+0000","last_fullsized":"2023-10-11T16:30:37.176027+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:26:18.864499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:19.956667+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:30:19.956667+0000","last_peered":"2023-10-11T16:30:19.956667+0000","last_clean":"2023-10-11T16:30:19.956667+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:30:19.956667+0000","last_undegraded":"2023-10-11T16:30:19.956667+0000","last_fullsized":"2023-10-11T16:30:19.956667+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:53:51.765779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667662+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:30:17.667662+0000","last_peered":"2023-10-11T16:30:17.667662+0000","last_clean":"2023-10-11T16:30:17.667662+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:30:17.667662+0000","last_undegraded":"2023-10-11T16:30:17.667662+0000","last_fullsized":"2023-10-11T16:30:17.667662+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:23.176208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:38.183142+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:30:38.183142+0000","last_peered":"2023-10-11T16:30:38.183142+0000","last_clean":"2023-10-11T16:30:38.183142+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:30:38.183142+0000","last_undegraded":"2023-10-11T16:30:38.183142+0000","last_fullsized":"2023-10-11T16:30:38.183142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:02:22.009759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612829+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:30:16.612829+0000","last_peered":"2023-10-11T16:30:16.612829+0000","last_clean":"2023-10-11T16:30:16.612829+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:30:16.612829+0000","last_undegraded":"2023-10-11T16:30:16.612829+0000","last_fullsized":"2023-10-11T16:30:16.612829+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:27:41.571741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.187873+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:30:40.187873+0000","last_peered":"2023-10-11T16:30:40.187873+0000","last_clean":"2023-10-11T16:30:40.187873+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:30:40.187873+0000","last_undegraded":"2023-10-11T16:30:40.187873+0000","last_fullsized":"2023-10-11T16:30:40.187873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:05:27.804658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667984+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:30:17.667984+0000","last_peered":"2023-10-11T16:30:17.667984+0000","last_clean":"2023-10-11T16:30:17.667984+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:30:17.667984+0000","last_undegraded":"2023-10-11T16:30:17.667984+0000","last_fullsized":"2023-10-11T16:30:17.667984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:32:59.074750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:37.173678+0000","last_change":"2023-10-11T16:30:11.117164+0000","last_active":"2023-10-11T16:30:37.173678+0000","last_peered":"2023-10-11T16:30:37.173678+0000","last_clean":"2023-10-11T16:30:37.173678+0000","last_became_active":"2023-10-11T16:30:11.116887+0000","last_became_peered":"2023-10-11T16:30:11.116887+0000","last_unstale":"2023-10-11T16:30:37.173678+0000","last_undegraded":"2023-10-11T16:30:37.173678+0000","last_fullsized":"2023-10-11T16:30:37.173678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:39:08.785806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283813+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:30:17.283813+0000","last_peered":"2023-10-11T16:30:17.283813+0000","last_clean":"2023-10-11T16:30:17.283813+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:30:17.283813+0000","last_undegraded":"2023-10-11T16:30:17.283813+0000","last_fullsized":"2023-10-11T16:30:17.283813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:43.398632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166054+0000","last_change":"2023-10-11T16:30:11.129313+0000","last_active":"2023-10-11T16:30:36.166054+0000","last_peered":"2023-10-11T16:30:36.166054+0000","last_clean":"2023-10-11T16:30:36.166054+0000","last_became_active":"2023-10-11T16:30:11.129051+0000","last_became_peered":"2023-10-11T16:30:11.129051+0000","last_unstale":"2023-10-11T16:30:36.166054+0000","last_undegraded":"2023-10-11T16:30:36.166054+0000","last_fullsized":"2023-10-11T16:30:36.166054+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:02:06.341952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669886+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:30:17.669886+0000","last_peered":"2023-10-11T16:30:17.669886+0000","last_clean":"2023-10-11T16:30:17.669886+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:30:17.669886+0000","last_undegraded":"2023-10-11T16:30:17.669886+0000","last_fullsized":"2023-10-11T16:30:17.669886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:47:23.558003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667532+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:30:17.667532+0000","last_peered":"2023-10-11T16:30:17.667532+0000","last_clean":"2023-10-11T16:30:17.667532+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:30:17.667532+0000","last_undegraded":"2023-10-11T16:30:17.667532+0000","last_fullsized":"2023-10-11T16:30:17.667532+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:55:02.983828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182666+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:30:41.182666+0000","last_peered":"2023-10-11T16:30:41.182666+0000","last_clean":"2023-10-11T16:30:41.182666+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:30:41.182666+0000","last_undegraded":"2023-10-11T16:30:41.182666+0000","last_fullsized":"2023-10-11T16:30:41.182666+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:35.861737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668839+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:30:17.668839+0000","last_peered":"2023-10-11T16:30:17.668839+0000","last_clean":"2023-10-11T16:30:17.668839+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:30:17.668839+0000","last_undegraded":"2023-10-11T16:30:17.668839+0000","last_fullsized":"2023-10-11T16:30:17.668839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:55.881039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185784+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:30:41.185784+0000","last_peered":"2023-10-11T16:30:41.185784+0000","last_clean":"2023-10-11T16:30:41.185784+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:30:41.185784+0000","last_undegraded":"2023-10-11T16:30:41.185784+0000","last_fullsized":"2023-10-11T16:30:41.185784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:57:36.382181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.164814+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:30:36.164814+0000","last_peered":"2023-10-11T16:30:36.164814+0000","last_clean":"2023-10-11T16:30:36.164814+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:30:36.164814+0000","last_undegraded":"2023-10-11T16:30:36.164814+0000","last_fullsized":"2023-10-11T16:30:36.164814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:09:01.841380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612174+0000","last_change":"2023-10-11T16:30:11.661388+0000","last_active":"2023-10-11T16:30:16.612174+0000","last_peered":"2023-10-11T16:30:16.612174+0000","last_clean":"2023-10-11T16:30:16.612174+0000","last_became_active":"2023-10-11T16:30:11.661073+0000","last_became_peered":"2023-10-11T16:30:11.661073+0000","last_unstale":"2023-10-11T16:30:16.612174+0000","last_undegraded":"2023-10-11T16:30:16.612174+0000","last_fullsized":"2023-10-11T16:30:16.612174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:10:29.755199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668643+0000","last_change":"2023-10-11T16:30:16.171535+0000","last_active":"2023-10-11T16:30:17.668643+0000","last_peered":"2023-10-11T16:30:17.668643+0000","last_clean":"2023-10-11T16:30:17.668643+0000","last_became_active":"2023-10-11T16:30:16.171051+0000","last_became_peered":"2023-10-11T16:30:16.171051+0000","last_unstale":"2023-10-11T16:30:17.668643+0000","last_undegraded":"2023-10-11T16:30:17.668643+0000","last_fullsized":"2023-10-11T16:30:17.668643+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:36:53.674481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.164950+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:30:36.164950+0000","last_peered":"2023-10-11T16:30:36.164950+0000","last_clean":"2023-10-11T16:30:36.164950+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:30:36.164950+0000","last_undegraded":"2023-10-11T16:30:36.164950+0000","last_fullsized":"2023-10-11T16:30:36.164950+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:06:21.073342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.614260+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:30:16.614260+0000","last_peered":"2023-10-11T16:30:16.614260+0000","last_clean":"2023-10-11T16:30:16.614260+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:30:16.614260+0000","last_undegraded":"2023-10-11T16:30:16.614260+0000","last_fullsized":"2023-10-11T16:30:16.614260+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:28:45.994452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668995+0000","last_change":"2023-10-11T16:30:16.131117+0000","last_active":"2023-10-11T16:30:17.668995+0000","last_peered":"2023-10-11T16:30:17.668995+0000","last_clean":"2023-10-11T16:30:17.668995+0000","last_became_active":"2023-10-11T16:30:16.130776+0000","last_became_peered":"2023-10-11T16:30:16.130776+0000","last_unstale":"2023-10-11T16:30:17.668995+0000","last_undegraded":"2023-10-11T16:30:17.668995+0000","last_fullsized":"2023-10-11T16:30:17.668995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:24:15.931186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.166640+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:30:36.166640+0000","last_peered":"2023-10-11T16:30:36.166640+0000","last_clean":"2023-10-11T16:30:36.166640+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:30:36.166640+0000","last_undegraded":"2023-10-11T16:30:36.166640+0000","last_fullsized":"2023-10-11T16:30:36.166640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:12:26.496915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283586+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:30:17.283586+0000","last_peered":"2023-10-11T16:30:17.283586+0000","last_clean":"2023-10-11T16:30:17.283586+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:30:17.283586+0000","last_undegraded":"2023-10-11T16:30:17.283586+0000","last_fullsized":"2023-10-11T16:30:17.283586+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:10:18.200065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:19.940473+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:30:19.940473+0000","last_peered":"2023-10-11T16:30:19.940473+0000","last_clean":"2023-10-11T16:30:19.940473+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:30:19.940473+0000","last_undegraded":"2023-10-11T16:30:19.940473+0000","last_fullsized":"2023-10-11T16:30:19.940473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:39.565360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185977+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:30:41.185977+0000","last_peered":"2023-10-11T16:30:41.185977+0000","last_clean":"2023-10-11T16:30:41.185977+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:30:41.185977+0000","last_undegraded":"2023-10-11T16:30:41.185977+0000","last_fullsized":"2023-10-11T16:30:41.185977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:32:39.882844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667992+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:30:17.667992+0000","last_peered":"2023-10-11T16:30:17.667992+0000","last_clean":"2023-10-11T16:30:17.667992+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:30:17.667992+0000","last_undegraded":"2023-10-11T16:30:17.667992+0000","last_fullsized":"2023-10-11T16:30:17.667992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:18:04.314307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669109+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:30:17.669109+0000","last_peered":"2023-10-11T16:30:17.669109+0000","last_clean":"2023-10-11T16:30:17.669109+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:30:17.669109+0000","last_undegraded":"2023-10-11T16:30:17.669109+0000","last_fullsized":"2023-10-11T16:30:17.669109+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:39:11.522213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:36.167157+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:30:36.167157+0000","last_peered":"2023-10-11T16:30:36.167157+0000","last_clean":"2023-10-11T16:30:36.167157+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:30:36.167157+0000","last_undegraded":"2023-10-11T16:30:36.167157+0000","last_fullsized":"2023-10-11T16:30:36.167157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:06.688871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613531+0000","last_change":"2023-10-11T16:30:11.648776+0000","last_active":"2023-10-11T16:30:16.613531+0000","last_peered":"2023-10-11T16:30:16.613531+0000","last_clean":"2023-10-11T16:30:16.613531+0000","last_became_active":"2023-10-11T16:30:11.648465+0000","last_became_peered":"2023-10-11T16:30:11.648465+0000","last_unstale":"2023-10-11T16:30:16.613531+0000","last_undegraded":"2023-10-11T16:30:16.613531+0000","last_fullsized":"2023-10-11T16:30:16.613531+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:24:06.619577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667818+0000","last_change":"2023-10-11T16:30:16.129703+0000","last_active":"2023-10-11T16:30:17.667818+0000","last_peered":"2023-10-11T16:30:17.667818+0000","last_clean":"2023-10-11T16:30:17.667818+0000","last_became_active":"2023-10-11T16:30:16.129231+0000","last_became_peered":"2023-10-11T16:30:16.129231+0000","last_unstale":"2023-10-11T16:30:17.667818+0000","last_undegraded":"2023-10-11T16:30:17.667818+0000","last_fullsized":"2023-10-11T16:30:17.667818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:47:29.769705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.667910+0000","last_change":"2023-10-11T16:30:11.127305+0000","last_active":"2023-10-11T16:30:17.667910+0000","last_peered":"2023-10-11T16:30:17.667910+0000","last_clean":"2023-10-11T16:30:17.667910+0000","last_became_active":"2023-10-11T16:30:11.126986+0000","last_became_peered":"2023-10-11T16:30:11.126986+0000","last_unstale":"2023-10-11T16:30:17.667910+0000","last_undegraded":"2023-10-11T16:30:17.667910+0000","last_fullsized":"2023-10-11T16:30:17.667910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:33:23.033612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613872+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:30:16.613872+0000","last_peered":"2023-10-11T16:30:16.613872+0000","last_clean":"2023-10-11T16:30:16.613872+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:30:16.613872+0000","last_undegraded":"2023-10-11T16:30:16.613872+0000","last_fullsized":"2023-10-11T16:30:16.613872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:49.187595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.612035+0000","last_change":"2023-10-11T16:30:16.135709+0000","last_active":"2023-10-11T16:30:16.612035+0000","last_peered":"2023-10-11T16:30:16.612035+0000","last_clean":"2023-10-11T16:30:16.612035+0000","last_became_active":"2023-10-11T16:30:16.135294+0000","last_became_peered":"2023-10-11T16:30:16.135294+0000","last_unstale":"2023-10-11T16:30:16.612035+0000","last_undegraded":"2023-10-11T16:30:16.612035+0000","last_fullsized":"2023-10-11T16:30:16.612035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:50:08.906768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.189235+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:30:40.189235+0000","last_peered":"2023-10-11T16:30:40.189235+0000","last_clean":"2023-10-11T16:30:40.189235+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:30:40.189235+0000","last_undegraded":"2023-10-11T16:30:40.189235+0000","last_fullsized":"2023-10-11T16:30:40.189235+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:38:09.758963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613738+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:30:16.613738+0000","last_peered":"2023-10-11T16:30:16.613738+0000","last_clean":"2023-10-11T16:30:16.613738+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:30:16.613738+0000","last_undegraded":"2023-10-11T16:30:16.613738+0000","last_fullsized":"2023-10-11T16:30:16.613738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:04:56.528036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186085+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:30:40.186085+0000","last_peered":"2023-10-11T16:30:40.186085+0000","last_clean":"2023-10-11T16:30:40.186085+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:30:40.186085+0000","last_undegraded":"2023-10-11T16:30:40.186085+0000","last_fullsized":"2023-10-11T16:30:40.186085+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:23:35.338891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669638+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:30:17.669638+0000","last_peered":"2023-10-11T16:30:17.669638+0000","last_clean":"2023-10-11T16:30:17.669638+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:30:17.669638+0000","last_undegraded":"2023-10-11T16:30:17.669638+0000","last_fullsized":"2023-10-11T16:30:17.669638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:25:06.130705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.613643+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:30:16.613643+0000","last_peered":"2023-10-11T16:30:16.613643+0000","last_clean":"2023-10-11T16:30:16.613643+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:30:16.613643+0000","last_undegraded":"2023-10-11T16:30:16.613643+0000","last_fullsized":"2023-10-11T16:30:16.613643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:11:20.783196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283653+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:30:17.283653+0000","last_peered":"2023-10-11T16:30:17.283653+0000","last_clean":"2023-10-11T16:30:17.283653+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:30:17.283653+0000","last_undegraded":"2023-10-11T16:30:17.283653+0000","last_fullsized":"2023-10-11T16:30:17.283653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:05:33.114445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186048+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:30:41.186048+0000","last_peered":"2023-10-11T16:30:41.186048+0000","last_clean":"2023-10-11T16:30:41.186048+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:30:41.186048+0000","last_undegraded":"2023-10-11T16:30:41.186048+0000","last_fullsized":"2023-10-11T16:30:41.186048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:50.699880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185837+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:30:41.185837+0000","last_peered":"2023-10-11T16:30:41.185837+0000","last_clean":"2023-10-11T16:30:41.185837+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:30:41.185837+0000","last_undegraded":"2023-10-11T16:30:41.185837+0000","last_fullsized":"2023-10-11T16:30:41.185837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:03:23.234106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:16.611986+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:30:16.611986+0000","last_peered":"2023-10-11T16:30:16.611986+0000","last_clean":"2023-10-11T16:30:16.611986+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:30:16.611986+0000","last_undegraded":"2023-10-11T16:30:16.611986+0000","last_fullsized":"2023-10-11T16:30:16.611986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:59:39.821639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669892+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:30:17.669892+0000","last_peered":"2023-10-11T16:30:17.669892+0000","last_clean":"2023-10-11T16:30:17.669892+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:30:17.669892+0000","last_undegraded":"2023-10-11T16:30:17.669892+0000","last_fullsized":"2023-10-11T16:30:17.669892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:44.023609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.284031+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:30:17.284031+0000","last_peered":"2023-10-11T16:30:17.284031+0000","last_clean":"2023-10-11T16:30:17.284031+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:30:17.284031+0000","last_undegraded":"2023-10-11T16:30:17.284031+0000","last_fullsized":"2023-10-11T16:30:17.284031+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:58.558512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185560+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:30:41.185560+0000","last_peered":"2023-10-11T16:30:41.185560+0000","last_clean":"2023-10-11T16:30:41.185560+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:30:41.185560+0000","last_undegraded":"2023-10-11T16:30:41.185560+0000","last_fullsized":"2023-10-11T16:30:41.185560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:19:06.908283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669256+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:30:17.669256+0000","last_peered":"2023-10-11T16:30:17.669256+0000","last_clean":"2023-10-11T16:30:17.669256+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:30:17.669256+0000","last_undegraded":"2023-10-11T16:30:17.669256+0000","last_fullsized":"2023-10-11T16:30:17.669256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:16:39.680829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-11T16:30:40.186430+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:30:40.186430+0000","last_peered":"2023-10-11T16:30:40.186430+0000","last_clean":"2023-10-11T16:30:40.186430+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:30:40.186430+0000","last_undegraded":"2023-10-11T16:30:40.186430+0000","last_fullsized":"2023-10-11T16:30:40.186430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:27:45.879779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.668100+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:30:17.668100+0000","last_peered":"2023-10-11T16:30:17.668100+0000","last_clean":"2023-10-11T16:30:17.668100+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:30:17.668100+0000","last_undegraded":"2023-10-11T16:30:17.668100+0000","last_fullsized":"2023-10-11T16:30:17.668100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:06:16.966258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669079+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:30:17.669079+0000","last_peered":"2023-10-11T16:30:17.669079+0000","last_clean":"2023-10-11T16:30:17.669079+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:30:17.669079+0000","last_undegraded":"2023-10-11T16:30:17.669079+0000","last_fullsized":"2023-10-11T16:30:17.669079+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:32.524195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.669665+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:30:17.669665+0000","last_peered":"2023-10-11T16:30:17.669665+0000","last_clean":"2023-10-11T16:30:17.669665+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:30:17.669665+0000","last_undegraded":"2023-10-11T16:30:17.669665+0000","last_fullsized":"2023-10-11T16:30:17.669665+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:17.866867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-11T16:30:17.283801+0000","last_change":"2023-10-11T16:30:11.668522+0000","last_active":"2023-10-11T16:30:17.283801+0000","last_peered":"2023-10-11T16:30:17.283801+0000","last_clean":"2023-10-11T16:30:17.283801+0000","last_became_active":"2023-10-11T16:30:11.654192+0000","last_became_peered":"2023-10-11T16:30:11.654192+0000","last_unstale":"2023-10-11T16:30:17.283801+0000","last_undegraded":"2023-10-11T16:30:17.283801+0000","last_fullsized":"2023-10-11T16:30:17.283801+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:51:10.675716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6726,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":125,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":73444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449105,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":946176,"data_stored":468750,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.373},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.622}]},{"osd":5,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.499},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60399999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.56899999999999995}]},{"osd":7,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.86099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.77300000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.70699999999999996}]}]},{"osd":10,"up_from":71,"seq":304942678037,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6320,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738832,"statfs":{"total":95995035648,"available":95988563968,"internally_reserved":0,"allocated":1024000,"data_stored":1059190,"data_compressed":47399,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56100000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.61599999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81000000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.78600000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":0.89600000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.878},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.86499999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.82699999999999996}]}]},{"osd":9,"up_from":66,"seq":283467841561,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6276,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":978944,"data_stored":482592,"data_compressed":14600,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.622},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.42499999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.77300000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.495}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.59199999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.85099999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.754},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.70199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":8,"up_from":58,"seq":249108103199,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6380,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738772,"statfs":{"total":95995035648,"available":95988502528,"internally_reserved":0,"allocated":954368,"data_stored":464479,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57799999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.625},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.443,"15min":0.44},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.55900000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.55800000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.42299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.505}]},{"osd":6,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65000000000000002,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.66800000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.77200000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73699999999999999}]},{"osd":10,"last update":"Wed Oct 11 16:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":1011712,"data_stored":1064328,"data_compressed":47399,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.33800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.629,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81499999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78000000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.748,"5min":1.103,"15min":1.103},"last":0.74299999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77100000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.628},"min":{"1min":0.45400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79500000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.72499999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.70699999999999996,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.82399999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.71799999999999997,"15min":0.72599999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.85599999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.89800000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.873}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":937984,"data_stored":464072,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.54100000000000004,"15min":0.52300000000000002},"min":{"1min":0.44400000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91500000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.46899999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84599999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.44800000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.65000000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.48599999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.64300000000000002,"15min":0.63400000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94099999999999995,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.56499999999999995,"15min":0.55500000000000005},"min":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55900000000000005,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.52200000000000002}]},{"osd":4,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70899999999999996,"15min":0.69699999999999995},"min":{"1min":0.52000000000000002,"5min":0.498,"15min":0.498},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.66000000000000003,"15min":0.64900000000000002},"min":{"1min":0.56499999999999995,"5min":0.502,"15min":0.502},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67900000000000005}]},{"osd":5,"last update":"Wed Oct 11 16:29:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.67200000000000004,"15min":0.65700000000000003},"min":{"1min":0.55800000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.871,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.76100000000000001,"15min":0.745},"min":{"1min":0.56999999999999995,"5min":0.434,"15min":0.434},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.78100000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:29:55 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.77300000000000002,"15min":0.75700000000000001},"min":{"1min":0.66300000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73199999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.753}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74099999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.84799999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.85699999999999998,"15min":0.86799999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0569999999999999,"5min":1.254,"15min":1.254},"last":0.85899999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.89000000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.877,"5min":0.877,"15min":0.877},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.872}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7352,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737800,"statfs":{"total":95995035648,"available":95987507200,"internally_reserved":0,"allocated":966656,"data_stored":484916,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57399999999999995,"15min":0.621},"min":{"1min":0.371,"5min":0.251,"15min":0.251},"max":{"1min":0.754,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.53500000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76600000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.505,"15min":0.501},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.50900000000000001,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.307,"15min":0.307},"max":{"1min":0.68600000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.629}]},{"osd":3,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.438,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:29:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41999999999999998,"15min":0.40899999999999997},"min":{"1min":0.33300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.439},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.55100000000000005,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.72199999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71399999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.504,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.84499999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.878},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.82399999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.91600000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.79700000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7288,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737864,"statfs":{"total":95995035648,"available":95987572736,"internally_reserved":0,"allocated":966656,"data_stored":465550,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.40899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.497},{"interface":"front","average":{"1min":0.63,"5min":0.55500000000000005,"15min":0.52600000000000002},"min":{"1min":0.441,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60099999999999998,"15min":0.57799999999999996},"min":{"1min":0.38500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.622,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.44500000000000001,"15min":0.433},"min":{"1min":0.35799999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44600000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:29:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57599999999999996,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72699999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64700000000000002,"15min":0.63400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.621}]},{"osd":6,"last update":"Wed Oct 11 16:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.64200000000000002,"15min":0.622},"min":{"1min":0.52200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96799999999999997,"5min":0.999,"15min":0.999},"last":0.52000000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.75},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.78400000000000003}]},{"osd":8,"last update":"Wed Oct 11 16:30:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.73199999999999998,"15min":0.73899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75900000000000001},"min":{"1min":0.56599999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91800000000000004,"5min":1.133,"15min":1.133},"last":0.73499999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.76400000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.80500000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:30:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.75800000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7472,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737680,"statfs":{"total":95995035648,"available":95987384320,"internally_reserved":0,"allocated":1024000,"data_stored":513399,"data_compressed":14600,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79800000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78300000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.83599999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.84199999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56699999999999995,"15min":0.54200000000000004},"min":{"1min":0.49399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59499999999999997,"15min":0.56299999999999994},"min":{"1min":0.47299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68600000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74399999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75900000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.875}]},{"osd":5,"last update":"Wed Oct 11 16:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66600000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.49099999999999999,"15min":0.47799999999999998},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.61499999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.64200000000000002,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86599999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.65600000000000003,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76100000000000001}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77900000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.80700000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":1.0149999999999999}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.627,"15min":0.61599999999999999},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92200000000000004,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.91500000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.52600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":1},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.73299999999999998,"15min":0.71999999999999997},"min":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.92800000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":1.036}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6628,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738524,"statfs":{"total":95995035648,"available":95988248576,"internally_reserved":0,"allocated":946176,"data_stored":463935,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.38200000000000001},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69999999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.373},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.40699999999999997},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.40400000000000003}]},{"osd":1,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60599999999999998,"15min":0.56999999999999995},"min":{"1min":0.433,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63500000000000001,"15min":0.58899999999999997},"min":{"1min":0.48199999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64500000000000002,"15min":0.63400000000000001},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90900000000000003,"5min":1.101,"15min":1.101},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69399999999999995,"15min":0.68000000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0509999999999999,"5min":1.194,"15min":1.194},"last":0.73499999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85399999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69199999999999995,"15min":0.68899999999999995},"min":{"1min":0.51300000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91600000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.66800000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:30:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91500000000000004,"5min":1.268,"15min":1.268},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.60999999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84599999999999997,"5min":1.119,"15min":1.119},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.071,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.629}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70699999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.755}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":950272,"data_stored":464668,"data_compressed":11649,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.379,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.36899999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.55200000000000005,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79600000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51800000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71999999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.2560000000000002,"5min":4.5,"15min":4.5},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68899999999999995,"15min":0.72999999999999998},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.75900000000000001}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.58299999999999996}]},{"osd":7,"last update":"Wed Oct 11 16:30:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48999999999999999,"15min":0.48199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56000000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.60999999999999999,"15min":0.59499999999999997},"min":{"1min":0.47799999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.72199999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60599999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.84699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6712,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738440,"statfs":{"total":95995035648,"available":95988162560,"internally_reserved":0,"allocated":1032192,"data_stored":1069474,"data_compressed":50350,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.745,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78700000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.873,"5min":0.874,"15min":0.874},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.70599999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.438,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63200000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.71399999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90600000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.76500000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58899999999999997,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85299999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.99299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78100000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.83899999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70799999999999996,"15min":0.71499999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69599999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.89200000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64600000000000002,"15min":0.64800000000000002},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84299999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79000000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.93600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.97499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":39235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:30:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:43 smithi130 ceph-mon[131859]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:43 smithi130 ceph-mon[131859]: pgmap v290: 129 pgs: 1 peering, 128 active+clean; 583 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3.4 KiB/s wr, 12 op/s; 72 B/s, 0 objects/s recovering 2023-10-11T16:30:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:43 smithi130 ceph-mon[131859]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:43 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3180012360' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-10-11T16:30:44.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:44 smithi130 ceph-mon[131859]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:44.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:44 smithi111 ceph-mon[98527]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:44.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:44 smithi150 ceph-mon[132217]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:45.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:45 smithi130 ceph-mon[131859]: pgmap v291: 129 pgs: 1 peering, 128 active+clean; 622 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.6 KiB/s wr, 16 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:45 smithi111 ceph-mon[98527]: pgmap v291: 129 pgs: 1 peering, 128 active+clean; 622 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.6 KiB/s wr, 16 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:45 smithi150 ceph-mon[132217]: pgmap v291: 129 pgs: 1 peering, 128 active+clean; 622 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.6 KiB/s wr, 16 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:46.470 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T16:30:46.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 -- 172.21.15.111:0/1531447008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c101480 msgr2=0x7f636c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:46.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 --2- 172.21.15.111:0/1531447008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c101480 0x7f636c1018d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6354009990 tx=0x7f6354009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 -- 172.21.15.111:0/1531447008 shutdown_connections 2023-10-11T16:30:46.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 --2- 172.21.15.111:0/1531447008 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f636c101e10 0x7f636c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 --2- 172.21.15.111:0/1531447008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636c100290 0x7f636c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 --2- 172.21.15.111:0/1531447008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c101480 0x7f636c1018d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 -- 172.21.15.111:0/1531447008 >> 172.21.15.111:0/1531447008 conn(0x7f636c0fb870 msgr2=0x7f636c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:46.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 -- 172.21.15.111:0/1531447008 shutdown_connections 2023-10-11T16:30:46.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.677+0000 7f63723ff700 1 -- 172.21.15.111:0/1531447008 wait complete. 2023-10-11T16:30:46.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.678+0000 7f63723ff700 1 Processor -- start 2023-10-11T16:30:46.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.678+0000 7f63723ff700 1 -- start start 2023-10-11T16:30:46.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.678+0000 7f63723ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c100290 0x7f636c195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:46.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.678+0000 7f63723ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f636c101480 0x7f636c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:46.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.678+0000 7f63723ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636c101e10 0x7f636c19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:46.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.678+0000 7f63723ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c19db10 con 0x7f636c100290 2023-10-11T16:30:46.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.678+0000 7f63723ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c19dc50 con 0x7f636c101480 2023-10-11T16:30:46.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.678+0000 7f63723ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c19dde0 con 0x7f636c101e10 2023-10-11T16:30:46.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c100290 0x7f636c195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:46.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c100290 0x7f636c195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42982/0 (socket says 172.21.15.111:42982) 2023-10-11T16:30:46.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 -- 172.21.15.111:0/1352394464 learned_addr learned my addr 172.21.15.111:0/1352394464 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:46.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f6363fff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f636c101480 0x7f636c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:46.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f637099c700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636c101e10 0x7f636c19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:46.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 -- 172.21.15.111:0/1352394464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636c101e10 msgr2=0x7f636c19d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:46.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636c101e10 0x7f636c19d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 -- 172.21.15.111:0/1352394464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f636c101480 msgr2=0x7f636c1964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:46.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f636c101480 0x7f636c1964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 -- 172.21.15.111:0/1352394464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63540095f0 con 0x7f636c100290 2023-10-11T16:30:46.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.679+0000 7f636bfff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c100290 0x7f636c195f90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f635c00e970 tx=0x7f635c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:46.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.680+0000 7f6369ffb700 1 -- 172.21.15.111:0/1352394464 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c009790 con 0x7f636c100290 2023-10-11T16:30:46.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.680+0000 7f6369ffb700 1 -- 172.21.15.111:0/1352394464 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f635c00c7e0 con 0x7f636c100290 2023-10-11T16:30:46.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.680+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c10bc50 con 0x7f636c100290 2023-10-11T16:30:46.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.680+0000 7f6369ffb700 1 -- 172.21.15.111:0/1352394464 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c003c70 con 0x7f636c100290 2023-10-11T16:30:46.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.680+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c10bac0 con 0x7f636c100290 2023-10-11T16:30:46.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.681+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636c067a60 con 0x7f636c100290 2023-10-11T16:30:46.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.683+0000 7f6369ffb700 1 -- 172.21.15.111:0/1352394464 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f635c00cc50 con 0x7f636c100290 2023-10-11T16:30:46.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.684+0000 7f6369ffb700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f63580a11e0 0x7f63580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:46.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.684+0000 7f6369ffb700 1 -- 172.21.15.111:0/1352394464 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f635c014070 con 0x7f636c100290 2023-10-11T16:30:46.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.684+0000 7f6363fff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f63580a11e0 0x7f63580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:46.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.685+0000 7f6363fff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f63580a11e0 0x7f63580a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f635400b2e0 tx=0x7f6354005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:46.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.686+0000 7f6369ffb700 1 -- 172.21.15.111:0/1352394464 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f635c096380 con 0x7f636c100290 2023-10-11T16:30:46.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.882+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f636c19e950 con 0x7f63580a11e0 2023-10-11T16:30:46.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.895+0000 7f6369ffb700 1 -- 172.21.15.111:0/1352394464 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423876 (secure 0 0 0) 0x7f636c19e950 con 0x7f63580a11e0 2023-10-11T16:30:46.901 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:46.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f63580a11e0 msgr2=0x7f63580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:46.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f63580a11e0 0x7f63580a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f635400b2e0 tx=0x7f6354005fb0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c100290 msgr2=0x7f636c195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:46.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c100290 0x7f636c195f90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f635c00e970 tx=0x7f635c00ec80 comp rx=0 tx=0).stop 2023-10-11T16:30:46.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 shutdown_connections 2023-10-11T16:30:46.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f63580a11e0 0x7f63580a3690 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636c101e10 0x7f636c19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f636c100290 0x7f636c195f90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 --2- 172.21.15.111:0/1352394464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f636c101480 0x7f636c1964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:46.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 >> 172.21.15.111:0/1352394464 conn(0x7f636c0fb870 msgr2=0x7f636c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:46.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 shutdown_connections 2023-10-11T16:30:46.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:46.900+0000 7f63723ff700 1 -- 172.21.15.111:0/1352394464 wait complete. 2023-10-11T16:30:46.908 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:30:46.920 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2023-10-11T16:30:46.059576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637256,"num_objects":63,"num_object_clones":0,"num_object_copies":191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":188,"num_read_kb":192,"num_write":324,"num_write_kb":3026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1220437,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":263,"ondisk_log_size":263,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":422,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80764,"kb_used_data":11804,"kb_used_omap":0,"kb_used_meta":68864,"kb_avail":1124861060,"statfs":{"total":1151940427776,"available":1151857725440,"internally_reserved":0,"allocated":12087296,"data_stored":7760706,"data_compressed":318660,"data_compressed_allocated":430080,"data_compressed_original":2379776,"omap_allocated":0,"internal_metadata":70516736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42026,"num_objects":29,"num_object_clones":0,"num_object_copies":89,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":52,"num_read_kb":73,"num_write":70,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":39701,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002212"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185713+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:30:41.185713+0000","last_peered":"2023-10-11T16:30:41.185713+0000","last_clean":"2023-10-11T16:30:41.185713+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:30:41.185713+0000","last_undegraded":"2023-10-11T16:30:41.185713+0000","last_fullsized":"2023-10-11T16:30:41.185713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:47:02.731639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185549+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:30:41.185549+0000","last_peered":"2023-10-11T16:30:41.185549+0000","last_clean":"2023-10-11T16:30:41.185549+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:30:41.185549+0000","last_undegraded":"2023-10-11T16:30:41.185549+0000","last_fullsized":"2023-10-11T16:30:41.185549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:04:44.769304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185196+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:30:41.185196+0000","last_peered":"2023-10-11T16:30:41.185196+0000","last_clean":"2023-10-11T16:30:41.185196+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:30:41.185196+0000","last_undegraded":"2023-10-11T16:30:41.185196+0000","last_fullsized":"2023-10-11T16:30:41.185196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:22:01.830634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188320+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:30:41.188320+0000","last_peered":"2023-10-11T16:30:41.188320+0000","last_clean":"2023-10-11T16:30:41.188320+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:30:41.188320+0000","last_undegraded":"2023-10-11T16:30:41.188320+0000","last_fullsized":"2023-10-11T16:30:41.188320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:21:55.362917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185730+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:30:41.185730+0000","last_peered":"2023-10-11T16:30:41.185730+0000","last_clean":"2023-10-11T16:30:41.185730+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:30:41.185730+0000","last_undegraded":"2023-10-11T16:30:41.185730+0000","last_fullsized":"2023-10-11T16:30:41.185730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:47.175695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189802+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:30:41.189802+0000","last_peered":"2023-10-11T16:30:41.189802+0000","last_clean":"2023-10-11T16:30:41.189802+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:30:41.189802+0000","last_undegraded":"2023-10-11T16:30:41.189802+0000","last_fullsized":"2023-10-11T16:30:41.189802+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:04:31.832644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182446+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:30:41.182446+0000","last_peered":"2023-10-11T16:30:41.182446+0000","last_clean":"2023-10-11T16:30:41.182446+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:30:41.182446+0000","last_undegraded":"2023-10-11T16:30:41.182446+0000","last_fullsized":"2023-10-11T16:30:41.182446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:15:28.747024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403438+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:30:41.403438+0000","last_peered":"2023-10-11T16:30:41.403438+0000","last_clean":"2023-10-11T16:30:41.403438+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:30:41.403438+0000","last_undegraded":"2023-10-11T16:30:41.403438+0000","last_fullsized":"2023-10-11T16:30:41.403438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:20:51.806951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185872+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:30:41.185872+0000","last_peered":"2023-10-11T16:30:41.185872+0000","last_clean":"2023-10-11T16:30:41.185872+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:30:41.185872+0000","last_undegraded":"2023-10-11T16:30:41.185872+0000","last_fullsized":"2023-10-11T16:30:41.185872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:07:38.837524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190769+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:30:41.190769+0000","last_peered":"2023-10-11T16:30:41.190769+0000","last_clean":"2023-10-11T16:30:41.190769+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:30:41.190769+0000","last_undegraded":"2023-10-11T16:30:41.190769+0000","last_fullsized":"2023-10-11T16:30:41.190769+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:17:25.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182922+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:30:41.182922+0000","last_peered":"2023-10-11T16:30:41.182922+0000","last_clean":"2023-10-11T16:30:41.182922+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:30:41.182922+0000","last_undegraded":"2023-10-11T16:30:41.182922+0000","last_fullsized":"2023-10-11T16:30:41.182922+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:16:24.814604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188720+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:30:41.188720+0000","last_peered":"2023-10-11T16:30:41.188720+0000","last_clean":"2023-10-11T16:30:41.188720+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:30:41.188720+0000","last_undegraded":"2023-10-11T16:30:41.188720+0000","last_fullsized":"2023-10-11T16:30:41.188720+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:33:23.932678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198123+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:30:41.198123+0000","last_peered":"2023-10-11T16:30:41.198123+0000","last_clean":"2023-10-11T16:30:41.198123+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:30:41.198123+0000","last_undegraded":"2023-10-11T16:30:41.198123+0000","last_fullsized":"2023-10-11T16:30:41.198123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:09:09.086960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184893+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:30:41.184893+0000","last_peered":"2023-10-11T16:30:41.184893+0000","last_clean":"2023-10-11T16:30:41.184893+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:30:41.184893+0000","last_undegraded":"2023-10-11T16:30:41.184893+0000","last_fullsized":"2023-10-11T16:30:41.184893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:06.789653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185441+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:30:41.185441+0000","last_peered":"2023-10-11T16:30:41.185441+0000","last_clean":"2023-10-11T16:30:41.185441+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:30:41.185441+0000","last_undegraded":"2023-10-11T16:30:41.185441+0000","last_fullsized":"2023-10-11T16:30:41.185441+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:22:47.667568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184863+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:30:41.184863+0000","last_peered":"2023-10-11T16:30:41.184863+0000","last_clean":"2023-10-11T16:30:41.184863+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:30:41.184863+0000","last_undegraded":"2023-10-11T16:30:41.184863+0000","last_fullsized":"2023-10-11T16:30:41.184863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:51:56.240736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185599+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:30:41.185599+0000","last_peered":"2023-10-11T16:30:41.185599+0000","last_clean":"2023-10-11T16:30:41.185599+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:30:41.185599+0000","last_undegraded":"2023-10-11T16:30:41.185599+0000","last_fullsized":"2023-10-11T16:30:41.185599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:34:17.358317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182794+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:30:41.182794+0000","last_peered":"2023-10-11T16:30:41.182794+0000","last_clean":"2023-10-11T16:30:41.182794+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:30:41.182794+0000","last_undegraded":"2023-10-11T16:30:41.182794+0000","last_fullsized":"2023-10-11T16:30:41.182794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:09:27.215051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185311+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:30:41.185311+0000","last_peered":"2023-10-11T16:30:41.185311+0000","last_clean":"2023-10-11T16:30:41.185311+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:30:41.185311+0000","last_undegraded":"2023-10-11T16:30:41.185311+0000","last_fullsized":"2023-10-11T16:30:41.185311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:46:35.027109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186180+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.186180+0000","last_peered":"2023-10-11T16:30:41.186180+0000","last_clean":"2023-10-11T16:30:41.186180+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:30:41.186180+0000","last_undegraded":"2023-10-11T16:30:41.186180+0000","last_fullsized":"2023-10-11T16:30:41.186180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:05.403679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198854+0000","last_change":"2023-10-11T16:30:41.198854+0000","last_active":"2023-10-11T16:30:41.198854+0000","last_peered":"2023-10-11T16:30:41.198854+0000","last_clean":"2023-10-11T16:30:41.198854+0000","last_became_active":"2023-10-11T16:30:41.189677+0000","last_became_peered":"2023-10-11T16:30:41.189677+0000","last_unstale":"2023-10-11T16:30:41.198854+0000","last_undegraded":"2023-10-11T16:30:41.198854+0000","last_fullsized":"2023-10-11T16:30:41.198854+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:58:40.952333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403443+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:30:41.403443+0000","last_peered":"2023-10-11T16:30:41.403443+0000","last_clean":"2023-10-11T16:30:41.403443+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:30:41.403443+0000","last_undegraded":"2023-10-11T16:30:41.403443+0000","last_fullsized":"2023-10-11T16:30:41.403443+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:16:44.616541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185309+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:30:41.185309+0000","last_peered":"2023-10-11T16:30:41.185309+0000","last_clean":"2023-10-11T16:30:41.185309+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:30:41.185309+0000","last_undegraded":"2023-10-11T16:30:41.185309+0000","last_fullsized":"2023-10-11T16:30:41.185309+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:33:41.484467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403809+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:30:41.403809+0000","last_peered":"2023-10-11T16:30:41.403809+0000","last_clean":"2023-10-11T16:30:41.403809+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:30:41.403809+0000","last_undegraded":"2023-10-11T16:30:41.403809+0000","last_fullsized":"2023-10-11T16:30:41.403809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:05:06.232815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185766+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:30:41.185766+0000","last_peered":"2023-10-11T16:30:41.185766+0000","last_clean":"2023-10-11T16:30:41.185766+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:30:41.185766+0000","last_undegraded":"2023-10-11T16:30:41.185766+0000","last_fullsized":"2023-10-11T16:30:41.185766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:16.994395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188044+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:30:41.188044+0000","last_peered":"2023-10-11T16:30:41.188044+0000","last_clean":"2023-10-11T16:30:41.188044+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:30:41.188044+0000","last_undegraded":"2023-10-11T16:30:41.188044+0000","last_fullsized":"2023-10-11T16:30:41.188044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:30:57.536276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186101+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:30:41.186101+0000","last_peered":"2023-10-11T16:30:41.186101+0000","last_clean":"2023-10-11T16:30:41.186101+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:30:41.186101+0000","last_undegraded":"2023-10-11T16:30:41.186101+0000","last_fullsized":"2023-10-11T16:30:41.186101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:11:55.996038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185713+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:30:41.185713+0000","last_peered":"2023-10-11T16:30:41.185713+0000","last_clean":"2023-10-11T16:30:41.185713+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:30:41.185713+0000","last_undegraded":"2023-10-11T16:30:41.185713+0000","last_fullsized":"2023-10-11T16:30:41.185713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:31:18.914747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184425+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:30:41.184425+0000","last_peered":"2023-10-11T16:30:41.184425+0000","last_clean":"2023-10-11T16:30:41.184425+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:30:41.184425+0000","last_undegraded":"2023-10-11T16:30:41.184425+0000","last_fullsized":"2023-10-11T16:30:41.184425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:57:53.652251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186025+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:30:41.186025+0000","last_peered":"2023-10-11T16:30:41.186025+0000","last_clean":"2023-10-11T16:30:41.186025+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:30:41.186025+0000","last_undegraded":"2023-10-11T16:30:41.186025+0000","last_fullsized":"2023-10-11T16:30:41.186025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:46:36.549662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.195736+0000","last_change":"2023-10-11T16:30:41.195736+0000","last_active":"2023-10-11T16:30:41.195736+0000","last_peered":"2023-10-11T16:30:41.195736+0000","last_clean":"2023-10-11T16:30:41.195736+0000","last_became_active":"2023-10-11T16:30:41.195352+0000","last_became_peered":"2023-10-11T16:30:41.195352+0000","last_unstale":"2023-10-11T16:30:41.195736+0000","last_undegraded":"2023-10-11T16:30:41.195736+0000","last_fullsized":"2023-10-11T16:30:41.195736+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:00:25.607770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185067+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:30:41.185067+0000","last_peered":"2023-10-11T16:30:41.185067+0000","last_clean":"2023-10-11T16:30:41.185067+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:30:41.185067+0000","last_undegraded":"2023-10-11T16:30:41.185067+0000","last_fullsized":"2023-10-11T16:30:41.185067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:48.316789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:44.197856+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:30:44.197856+0000","last_peered":"2023-10-11T16:30:44.197856+0000","last_clean":"2023-10-11T16:30:44.197856+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:30:44.197856+0000","last_undegraded":"2023-10-11T16:30:44.197856+0000","last_fullsized":"2023-10-11T16:30:44.197856+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:32:24.986353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186484+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:30:41.186484+0000","last_peered":"2023-10-11T16:30:41.186484+0000","last_clean":"2023-10-11T16:30:41.186484+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:30:41.186484+0000","last_undegraded":"2023-10-11T16:30:41.186484+0000","last_fullsized":"2023-10-11T16:30:41.186484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:39:49.981983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186478+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:30:41.186478+0000","last_peered":"2023-10-11T16:30:41.186478+0000","last_clean":"2023-10-11T16:30:41.186478+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:30:41.186478+0000","last_undegraded":"2023-10-11T16:30:41.186478+0000","last_fullsized":"2023-10-11T16:30:41.186478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:20:03.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189794+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:30:41.189794+0000","last_peered":"2023-10-11T16:30:41.189794+0000","last_clean":"2023-10-11T16:30:41.189794+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:30:41.189794+0000","last_undegraded":"2023-10-11T16:30:41.189794+0000","last_fullsized":"2023-10-11T16:30:41.189794+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:08.968090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189977+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:30:41.189977+0000","last_peered":"2023-10-11T16:30:41.189977+0000","last_clean":"2023-10-11T16:30:41.189977+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:30:41.189977+0000","last_undegraded":"2023-10-11T16:30:41.189977+0000","last_fullsized":"2023-10-11T16:30:41.189977+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:41:46.444179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185463+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:30:41.185463+0000","last_peered":"2023-10-11T16:30:41.185463+0000","last_clean":"2023-10-11T16:30:41.185463+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:30:41.185463+0000","last_undegraded":"2023-10-11T16:30:41.185463+0000","last_fullsized":"2023-10-11T16:30:41.185463+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:16:11.533226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.197973+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:30:41.197973+0000","last_peered":"2023-10-11T16:30:41.197973+0000","last_clean":"2023-10-11T16:30:41.197973+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:30:41.197973+0000","last_undegraded":"2023-10-11T16:30:41.197973+0000","last_fullsized":"2023-10-11T16:30:41.197973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:49:04.403007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403246+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:30:41.403246+0000","last_peered":"2023-10-11T16:30:41.403246+0000","last_clean":"2023-10-11T16:30:41.403246+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:30:41.403246+0000","last_undegraded":"2023-10-11T16:30:41.403246+0000","last_fullsized":"2023-10-11T16:30:41.403246+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:18:58.866162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185429+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:30:41.185429+0000","last_peered":"2023-10-11T16:30:41.185429+0000","last_clean":"2023-10-11T16:30:41.185429+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:30:41.185429+0000","last_undegraded":"2023-10-11T16:30:41.185429+0000","last_fullsized":"2023-10-11T16:30:41.185429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:49.007168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189537+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:30:41.189537+0000","last_peered":"2023-10-11T16:30:41.189537+0000","last_clean":"2023-10-11T16:30:41.189537+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:30:41.189537+0000","last_undegraded":"2023-10-11T16:30:41.189537+0000","last_fullsized":"2023-10-11T16:30:41.189537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:46:32.350328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181702+0000","last_change":"2023-10-11T16:30:11.662850+0000","last_active":"2023-10-11T16:30:41.181702+0000","last_peered":"2023-10-11T16:30:41.181702+0000","last_clean":"2023-10-11T16:30:41.181702+0000","last_became_active":"2023-10-11T16:30:11.661052+0000","last_became_peered":"2023-10-11T16:30:11.661052+0000","last_unstale":"2023-10-11T16:30:41.181702+0000","last_undegraded":"2023-10-11T16:30:41.181702+0000","last_fullsized":"2023-10-11T16:30:41.181702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:23.324421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403234+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:30:41.403234+0000","last_peered":"2023-10-11T16:30:41.403234+0000","last_clean":"2023-10-11T16:30:41.403234+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:30:41.403234+0000","last_undegraded":"2023-10-11T16:30:41.403234+0000","last_fullsized":"2023-10-11T16:30:41.403234+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:53:00.713151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.192155+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:30:41.192155+0000","last_peered":"2023-10-11T16:30:41.192155+0000","last_clean":"2023-10-11T16:30:41.192155+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:30:41.192155+0000","last_undegraded":"2023-10-11T16:30:41.192155+0000","last_fullsized":"2023-10-11T16:30:41.192155+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:21:03.094344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185994+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:30:41.185994+0000","last_peered":"2023-10-11T16:30:41.185994+0000","last_clean":"2023-10-11T16:30:41.185994+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:30:41.185994+0000","last_undegraded":"2023-10-11T16:30:41.185994+0000","last_fullsized":"2023-10-11T16:30:41.185994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:51:15.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181694+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:30:41.181694+0000","last_peered":"2023-10-11T16:30:41.181694+0000","last_clean":"2023-10-11T16:30:41.181694+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:30:41.181694+0000","last_undegraded":"2023-10-11T16:30:41.181694+0000","last_fullsized":"2023-10-11T16:30:41.181694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:32.776111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183006+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:30:41.183006+0000","last_peered":"2023-10-11T16:30:41.183006+0000","last_clean":"2023-10-11T16:30:41.183006+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:30:41.183006+0000","last_undegraded":"2023-10-11T16:30:41.183006+0000","last_fullsized":"2023-10-11T16:30:41.183006+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:50:47.408054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185791+0000","last_change":"2023-10-11T16:30:11.666124+0000","last_active":"2023-10-11T16:30:41.185791+0000","last_peered":"2023-10-11T16:30:41.185791+0000","last_clean":"2023-10-11T16:30:41.185791+0000","last_became_active":"2023-10-11T16:30:11.665786+0000","last_became_peered":"2023-10-11T16:30:11.665786+0000","last_unstale":"2023-10-11T16:30:41.185791+0000","last_undegraded":"2023-10-11T16:30:41.185791+0000","last_fullsized":"2023-10-11T16:30:41.185791+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:16:08.621915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188345+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:30:41.188345+0000","last_peered":"2023-10-11T16:30:41.188345+0000","last_clean":"2023-10-11T16:30:41.188345+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:30:41.188345+0000","last_undegraded":"2023-10-11T16:30:41.188345+0000","last_fullsized":"2023-10-11T16:30:41.188345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:49.162491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190222+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:30:41.190222+0000","last_peered":"2023-10-11T16:30:41.190222+0000","last_clean":"2023-10-11T16:30:41.190222+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:30:41.190222+0000","last_undegraded":"2023-10-11T16:30:41.190222+0000","last_fullsized":"2023-10-11T16:30:41.190222+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:23:11.290409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186265+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:30:41.186265+0000","last_peered":"2023-10-11T16:30:41.186265+0000","last_clean":"2023-10-11T16:30:41.186265+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:30:41.186265+0000","last_undegraded":"2023-10-11T16:30:41.186265+0000","last_fullsized":"2023-10-11T16:30:41.186265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:59:10.188855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181930+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:30:41.181930+0000","last_peered":"2023-10-11T16:30:41.181930+0000","last_clean":"2023-10-11T16:30:41.181930+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:30:41.181930+0000","last_undegraded":"2023-10-11T16:30:41.181930+0000","last_fullsized":"2023-10-11T16:30:41.181930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:27:45.123842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'23","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:42.199445+0000","last_change":"2023-10-11T16:30:41.788994+0000","last_active":"2023-10-11T16:30:42.199445+0000","last_peered":"2023-10-11T16:30:42.199445+0000","last_clean":"2023-10-11T16:30:42.199445+0000","last_became_active":"2023-10-11T16:30:41.189949+0000","last_became_peered":"2023-10-11T16:30:41.189949+0000","last_unstale":"2023-10-11T16:30:42.199445+0000","last_undegraded":"2023-10-11T16:30:42.199445+0000","last_fullsized":"2023-10-11T16:30:42.199445+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:36:50.095047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43128,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":48,"num_write":24,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38515,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186506+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:30:41.186506+0000","last_peered":"2023-10-11T16:30:41.186506+0000","last_clean":"2023-10-11T16:30:41.186506+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:30:41.186506+0000","last_undegraded":"2023-10-11T16:30:41.186506+0000","last_fullsized":"2023-10-11T16:30:41.186506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:26:09.244377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190731+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:30:41.190731+0000","last_peered":"2023-10-11T16:30:41.190731+0000","last_clean":"2023-10-11T16:30:41.190731+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:30:41.190731+0000","last_undegraded":"2023-10-11T16:30:41.190731+0000","last_fullsized":"2023-10-11T16:30:41.190731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:10:05.160429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185142+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:30:41.185142+0000","last_peered":"2023-10-11T16:30:41.185142+0000","last_clean":"2023-10-11T16:30:41.185142+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:30:41.185142+0000","last_undegraded":"2023-10-11T16:30:41.185142+0000","last_fullsized":"2023-10-11T16:30:41.185142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:25:53.623192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198771+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:30:41.198771+0000","last_peered":"2023-10-11T16:30:41.198771+0000","last_clean":"2023-10-11T16:30:41.198771+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:30:41.198771+0000","last_undegraded":"2023-10-11T16:30:41.198771+0000","last_fullsized":"2023-10-11T16:30:41.198771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:09:22.911180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184864+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:30:41.184864+0000","last_peered":"2023-10-11T16:30:41.184864+0000","last_clean":"2023-10-11T16:30:41.184864+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:30:41.184864+0000","last_undegraded":"2023-10-11T16:30:41.184864+0000","last_fullsized":"2023-10-11T16:30:41.184864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:28:35.220071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190012+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:30:41.190012+0000","last_peered":"2023-10-11T16:30:41.190012+0000","last_clean":"2023-10-11T16:30:41.190012+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:30:41.190012+0000","last_undegraded":"2023-10-11T16:30:41.190012+0000","last_fullsized":"2023-10-11T16:30:41.190012+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:18:02.905266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183098+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:30:41.183098+0000","last_peered":"2023-10-11T16:30:41.183098+0000","last_clean":"2023-10-11T16:30:41.183098+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:30:41.183098+0000","last_undegraded":"2023-10-11T16:30:41.183098+0000","last_fullsized":"2023-10-11T16:30:41.183098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:13:30.957186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182281+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.182281+0000","last_peered":"2023-10-11T16:30:41.182281+0000","last_clean":"2023-10-11T16:30:41.182281+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:30:41.182281+0000","last_undegraded":"2023-10-11T16:30:41.182281+0000","last_fullsized":"2023-10-11T16:30:41.182281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:35:47.449805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:43.201255+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:30:43.201255+0000","last_peered":"2023-10-11T16:30:43.201255+0000","last_clean":"2023-10-11T16:30:43.201255+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:30:43.201255+0000","last_undegraded":"2023-10-11T16:30:43.201255+0000","last_fullsized":"2023-10-11T16:30:43.201255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:49:34.210642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182403+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:30:41.182403+0000","last_peered":"2023-10-11T16:30:41.182403+0000","last_clean":"2023-10-11T16:30:41.182403+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:30:41.182403+0000","last_undegraded":"2023-10-11T16:30:41.182403+0000","last_fullsized":"2023-10-11T16:30:41.182403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:49:06.683312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198932+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:30:41.198932+0000","last_peered":"2023-10-11T16:30:41.198932+0000","last_clean":"2023-10-11T16:30:41.198932+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:30:41.198932+0000","last_undegraded":"2023-10-11T16:30:41.198932+0000","last_fullsized":"2023-10-11T16:30:41.198932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:38:08.422304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.187119+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:30:41.187119+0000","last_peered":"2023-10-11T16:30:41.187119+0000","last_clean":"2023-10-11T16:30:41.187119+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:30:41.187119+0000","last_undegraded":"2023-10-11T16:30:41.187119+0000","last_fullsized":"2023-10-11T16:30:41.187119+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:22:31.437710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189360+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:30:41.189360+0000","last_peered":"2023-10-11T16:30:41.189360+0000","last_clean":"2023-10-11T16:30:41.189360+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:30:41.189360+0000","last_undegraded":"2023-10-11T16:30:41.189360+0000","last_fullsized":"2023-10-11T16:30:41.189360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:58.907731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189214+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:30:41.189214+0000","last_peered":"2023-10-11T16:30:41.189214+0000","last_clean":"2023-10-11T16:30:41.189214+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:30:41.189214+0000","last_undegraded":"2023-10-11T16:30:41.189214+0000","last_fullsized":"2023-10-11T16:30:41.189214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:29:15.740818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.187919+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:30:41.187919+0000","last_peered":"2023-10-11T16:30:41.187919+0000","last_clean":"2023-10-11T16:30:41.187919+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:30:41.187919+0000","last_undegraded":"2023-10-11T16:30:41.187919+0000","last_fullsized":"2023-10-11T16:30:41.187919+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:49:28.444855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185524+0000","last_change":"2023-10-11T16:30:11.670547+0000","last_active":"2023-10-11T16:30:41.185524+0000","last_peered":"2023-10-11T16:30:41.185524+0000","last_clean":"2023-10-11T16:30:41.185524+0000","last_became_active":"2023-10-11T16:30:11.669842+0000","last_became_peered":"2023-10-11T16:30:11.669842+0000","last_unstale":"2023-10-11T16:30:41.185524+0000","last_undegraded":"2023-10-11T16:30:41.185524+0000","last_fullsized":"2023-10-11T16:30:41.185524+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:00:14.369445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186325+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:30:41.186325+0000","last_peered":"2023-10-11T16:30:41.186325+0000","last_clean":"2023-10-11T16:30:41.186325+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:30:41.186325+0000","last_undegraded":"2023-10-11T16:30:41.186325+0000","last_fullsized":"2023-10-11T16:30:41.186325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:54:10.393416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182101+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:30:41.182101+0000","last_peered":"2023-10-11T16:30:41.182101+0000","last_clean":"2023-10-11T16:30:41.182101+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:30:41.182101+0000","last_undegraded":"2023-10-11T16:30:41.182101+0000","last_fullsized":"2023-10-11T16:30:41.182101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:36.264713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190141+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:30:41.190141+0000","last_peered":"2023-10-11T16:30:41.190141+0000","last_clean":"2023-10-11T16:30:41.190141+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:30:41.190141+0000","last_undegraded":"2023-10-11T16:30:41.190141+0000","last_fullsized":"2023-10-11T16:30:41.190141+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:23:03.581102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198172+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:30:41.198172+0000","last_peered":"2023-10-11T16:30:41.198172+0000","last_clean":"2023-10-11T16:30:41.198172+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:30:41.198172+0000","last_undegraded":"2023-10-11T16:30:41.198172+0000","last_fullsized":"2023-10-11T16:30:41.198172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:24:19.511542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182736+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:30:41.182736+0000","last_peered":"2023-10-11T16:30:41.182736+0000","last_clean":"2023-10-11T16:30:41.182736+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:30:41.182736+0000","last_undegraded":"2023-10-11T16:30:41.182736+0000","last_fullsized":"2023-10-11T16:30:41.182736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:10:16.893854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190364+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:30:41.190364+0000","last_peered":"2023-10-11T16:30:41.190364+0000","last_clean":"2023-10-11T16:30:41.190364+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:30:41.190364+0000","last_undegraded":"2023-10-11T16:30:41.190364+0000","last_fullsized":"2023-10-11T16:30:41.190364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:38:35.761085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198257+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:30:41.198257+0000","last_peered":"2023-10-11T16:30:41.198257+0000","last_clean":"2023-10-11T16:30:41.198257+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:30:41.198257+0000","last_undegraded":"2023-10-11T16:30:41.198257+0000","last_fullsized":"2023-10-11T16:30:41.198257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:21:50.405889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189800+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:30:41.189800+0000","last_peered":"2023-10-11T16:30:41.189800+0000","last_clean":"2023-10-11T16:30:41.189800+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:30:41.189800+0000","last_undegraded":"2023-10-11T16:30:41.189800+0000","last_fullsized":"2023-10-11T16:30:41.189800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:41:33.198602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198160+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:30:41.198160+0000","last_peered":"2023-10-11T16:30:41.198160+0000","last_clean":"2023-10-11T16:30:41.198160+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:30:41.198160+0000","last_undegraded":"2023-10-11T16:30:41.198160+0000","last_fullsized":"2023-10-11T16:30:41.198160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:37:23.548227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182645+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:30:41.182645+0000","last_peered":"2023-10-11T16:30:41.182645+0000","last_clean":"2023-10-11T16:30:41.182645+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:30:41.182645+0000","last_undegraded":"2023-10-11T16:30:41.182645+0000","last_fullsized":"2023-10-11T16:30:41.182645+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:40:48.239885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403663+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:30:41.403663+0000","last_peered":"2023-10-11T16:30:41.403663+0000","last_clean":"2023-10-11T16:30:41.403663+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:30:41.403663+0000","last_undegraded":"2023-10-11T16:30:41.403663+0000","last_fullsized":"2023-10-11T16:30:41.403663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:58:23.479875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188194+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:30:41.188194+0000","last_peered":"2023-10-11T16:30:41.188194+0000","last_clean":"2023-10-11T16:30:41.188194+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:30:41.188194+0000","last_undegraded":"2023-10-11T16:30:41.188194+0000","last_fullsized":"2023-10-11T16:30:41.188194+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:03.654864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182242+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:30:41.182242+0000","last_peered":"2023-10-11T16:30:41.182242+0000","last_clean":"2023-10-11T16:30:41.182242+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:30:41.182242+0000","last_undegraded":"2023-10-11T16:30:41.182242+0000","last_fullsized":"2023-10-11T16:30:41.182242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:11:53.750545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189957+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:30:41.189957+0000","last_peered":"2023-10-11T16:30:41.189957+0000","last_clean":"2023-10-11T16:30:41.189957+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:30:41.189957+0000","last_undegraded":"2023-10-11T16:30:41.189957+0000","last_fullsized":"2023-10-11T16:30:41.189957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:27:51.812285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.197783+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:30:41.197783+0000","last_peered":"2023-10-11T16:30:41.197783+0000","last_clean":"2023-10-11T16:30:41.197783+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:30:41.197783+0000","last_undegraded":"2023-10-11T16:30:41.197783+0000","last_fullsized":"2023-10-11T16:30:41.197783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:40:50.386941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.938949+0000","last_change":"2023-10-11T16:30:41.938949+0000","last_active":"2023-10-11T16:30:41.938949+0000","last_peered":"2023-10-11T16:30:41.938949+0000","last_clean":"2023-10-11T16:30:41.938949+0000","last_became_active":"2023-10-11T16:30:41.192902+0000","last_became_peered":"2023-10-11T16:30:41.192902+0000","last_unstale":"2023-10-11T16:30:41.938949+0000","last_undegraded":"2023-10-11T16:30:41.938949+0000","last_fullsized":"2023-10-11T16:30:41.938949+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:08:59.943869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186174+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:30:41.186174+0000","last_peered":"2023-10-11T16:30:41.186174+0000","last_clean":"2023-10-11T16:30:41.186174+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:30:41.186174+0000","last_undegraded":"2023-10-11T16:30:41.186174+0000","last_fullsized":"2023-10-11T16:30:41.186174+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:51:47.458050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.205885+0000","last_change":"2023-10-11T16:30:41.205550+0000","last_active":"2023-10-11T16:30:41.205885+0000","last_peered":"2023-10-11T16:30:41.205885+0000","last_clean":"2023-10-11T16:30:41.205885+0000","last_became_active":"2023-10-11T16:30:41.190345+0000","last_became_peered":"2023-10-11T16:30:41.190345+0000","last_unstale":"2023-10-11T16:30:41.205885+0000","last_undegraded":"2023-10-11T16:30:41.205885+0000","last_fullsized":"2023-10-11T16:30:41.205885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:21:53.506003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198393+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:30:41.198393+0000","last_peered":"2023-10-11T16:30:41.198393+0000","last_clean":"2023-10-11T16:30:41.198393+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:30:41.198393+0000","last_undegraded":"2023-10-11T16:30:41.198393+0000","last_fullsized":"2023-10-11T16:30:41.198393+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:25:27.458254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403257+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:30:41.403257+0000","last_peered":"2023-10-11T16:30:41.403257+0000","last_clean":"2023-10-11T16:30:41.403257+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:30:41.403257+0000","last_undegraded":"2023-10-11T16:30:41.403257+0000","last_fullsized":"2023-10-11T16:30:41.403257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:32:39.048475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182666+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:30:41.182666+0000","last_peered":"2023-10-11T16:30:41.182666+0000","last_clean":"2023-10-11T16:30:41.182666+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:30:41.182666+0000","last_undegraded":"2023-10-11T16:30:41.182666+0000","last_fullsized":"2023-10-11T16:30:41.182666+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:35.861737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403786+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:30:41.403786+0000","last_peered":"2023-10-11T16:30:41.403786+0000","last_clean":"2023-10-11T16:30:41.403786+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:30:41.403786+0000","last_undegraded":"2023-10-11T16:30:41.403786+0000","last_fullsized":"2023-10-11T16:30:41.403786+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:10:56.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185784+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:30:41.185784+0000","last_peered":"2023-10-11T16:30:41.185784+0000","last_clean":"2023-10-11T16:30:41.185784+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:30:41.185784+0000","last_undegraded":"2023-10-11T16:30:41.185784+0000","last_fullsized":"2023-10-11T16:30:41.185784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:57:36.382181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189399+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:30:41.189399+0000","last_peered":"2023-10-11T16:30:41.189399+0000","last_clean":"2023-10-11T16:30:41.189399+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:30:41.189399+0000","last_undegraded":"2023-10-11T16:30:41.189399+0000","last_fullsized":"2023-10-11T16:30:41.189399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:06.520839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185604+0000","last_change":"2023-10-11T16:30:11.661388+0000","last_active":"2023-10-11T16:30:41.185604+0000","last_peered":"2023-10-11T16:30:41.185604+0000","last_clean":"2023-10-11T16:30:41.185604+0000","last_became_active":"2023-10-11T16:30:11.661073+0000","last_became_peered":"2023-10-11T16:30:11.661073+0000","last_unstale":"2023-10-11T16:30:41.185604+0000","last_undegraded":"2023-10-11T16:30:41.185604+0000","last_fullsized":"2023-10-11T16:30:41.185604+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:12.390368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.193489+0000","last_change":"2023-10-11T16:30:41.193489+0000","last_active":"2023-10-11T16:30:41.193489+0000","last_peered":"2023-10-11T16:30:41.193489+0000","last_clean":"2023-10-11T16:30:41.193489+0000","last_became_active":"2023-10-11T16:30:41.193060+0000","last_became_peered":"2023-10-11T16:30:41.193060+0000","last_unstale":"2023-10-11T16:30:41.193489+0000","last_undegraded":"2023-10-11T16:30:41.193489+0000","last_fullsized":"2023-10-11T16:30:41.193489+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:52:34.859402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185001+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:30:41.185001+0000","last_peered":"2023-10-11T16:30:41.185001+0000","last_clean":"2023-10-11T16:30:41.185001+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:30:41.185001+0000","last_undegraded":"2023-10-11T16:30:41.185001+0000","last_fullsized":"2023-10-11T16:30:41.185001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:01:22.571040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185793+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:30:41.185793+0000","last_peered":"2023-10-11T16:30:41.185793+0000","last_clean":"2023-10-11T16:30:41.185793+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:30:41.185793+0000","last_undegraded":"2023-10-11T16:30:41.185793+0000","last_fullsized":"2023-10-11T16:30:41.185793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:22:41.435586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.405455+0000","last_change":"2023-10-11T16:30:41.404891+0000","last_active":"2023-10-11T16:30:41.405455+0000","last_peered":"2023-10-11T16:30:41.405455+0000","last_clean":"2023-10-11T16:30:41.405455+0000","last_became_active":"2023-10-11T16:30:41.404396+0000","last_became_peered":"2023-10-11T16:30:41.404396+0000","last_unstale":"2023-10-11T16:30:41.405455+0000","last_undegraded":"2023-10-11T16:30:41.405455+0000","last_fullsized":"2023-10-11T16:30:41.405455+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:39:56.108444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403349+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:30:41.403349+0000","last_peered":"2023-10-11T16:30:41.403349+0000","last_clean":"2023-10-11T16:30:41.403349+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:30:41.403349+0000","last_undegraded":"2023-10-11T16:30:41.403349+0000","last_fullsized":"2023-10-11T16:30:41.403349+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:47:20.053030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184635+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:30:41.184635+0000","last_peered":"2023-10-11T16:30:41.184635+0000","last_clean":"2023-10-11T16:30:41.184635+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:30:41.184635+0000","last_undegraded":"2023-10-11T16:30:41.184635+0000","last_fullsized":"2023-10-11T16:30:41.184635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:48:42.041910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189125+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:30:41.189125+0000","last_peered":"2023-10-11T16:30:41.189125+0000","last_clean":"2023-10-11T16:30:41.189125+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:30:41.189125+0000","last_undegraded":"2023-10-11T16:30:41.189125+0000","last_fullsized":"2023-10-11T16:30:41.189125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:21.935947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185977+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:30:41.185977+0000","last_peered":"2023-10-11T16:30:41.185977+0000","last_clean":"2023-10-11T16:30:41.185977+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:30:41.185977+0000","last_undegraded":"2023-10-11T16:30:41.185977+0000","last_fullsized":"2023-10-11T16:30:41.185977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:32:39.882844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403694+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:30:41.403694+0000","last_peered":"2023-10-11T16:30:41.403694+0000","last_clean":"2023-10-11T16:30:41.403694+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:30:41.403694+0000","last_undegraded":"2023-10-11T16:30:41.403694+0000","last_fullsized":"2023-10-11T16:30:41.403694+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:23:58.685801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188391+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:30:41.188391+0000","last_peered":"2023-10-11T16:30:41.188391+0000","last_clean":"2023-10-11T16:30:41.188391+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:30:41.188391+0000","last_undegraded":"2023-10-11T16:30:41.188391+0000","last_fullsized":"2023-10-11T16:30:41.188391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:04.067909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185574+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:30:41.185574+0000","last_peered":"2023-10-11T16:30:41.185574+0000","last_clean":"2023-10-11T16:30:41.185574+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:30:41.185574+0000","last_undegraded":"2023-10-11T16:30:41.185574+0000","last_fullsized":"2023-10-11T16:30:41.185574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:34.746546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182953+0000","last_change":"2023-10-11T16:30:11.648776+0000","last_active":"2023-10-11T16:30:41.182953+0000","last_peered":"2023-10-11T16:30:41.182953+0000","last_clean":"2023-10-11T16:30:41.182953+0000","last_became_active":"2023-10-11T16:30:11.648465+0000","last_became_peered":"2023-10-11T16:30:11.648465+0000","last_unstale":"2023-10-11T16:30:41.182953+0000","last_undegraded":"2023-10-11T16:30:41.182953+0000","last_fullsized":"2023-10-11T16:30:41.182953+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:44:56.663827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.200104+0000","last_change":"2023-10-11T16:30:41.200104+0000","last_active":"2023-10-11T16:30:41.200104+0000","last_peered":"2023-10-11T16:30:41.200104+0000","last_clean":"2023-10-11T16:30:41.200104+0000","last_became_active":"2023-10-11T16:30:41.199637+0000","last_became_peered":"2023-10-11T16:30:41.199637+0000","last_unstale":"2023-10-11T16:30:41.200104+0000","last_undegraded":"2023-10-11T16:30:41.200104+0000","last_fullsized":"2023-10-11T16:30:41.200104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:08:40.301726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.203024+0000","last_change":"2023-10-11T16:30:41.202718+0000","last_active":"2023-10-11T16:30:41.203024+0000","last_peered":"2023-10-11T16:30:41.203024+0000","last_clean":"2023-10-11T16:30:41.203024+0000","last_became_active":"2023-10-11T16:30:41.190741+0000","last_became_peered":"2023-10-11T16:30:41.190741+0000","last_unstale":"2023-10-11T16:30:41.203024+0000","last_undegraded":"2023-10-11T16:30:41.203024+0000","last_fullsized":"2023-10-11T16:30:41.203024+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:37:17.174287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183059+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:30:41.183059+0000","last_peered":"2023-10-11T16:30:41.183059+0000","last_clean":"2023-10-11T16:30:41.183059+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:30:41.183059+0000","last_undegraded":"2023-10-11T16:30:41.183059+0000","last_fullsized":"2023-10-11T16:30:41.183059+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:46:18.410074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190477+0000","last_change":"2023-10-11T16:30:41.190477+0000","last_active":"2023-10-11T16:30:41.190477+0000","last_peered":"2023-10-11T16:30:41.190477+0000","last_clean":"2023-10-11T16:30:41.190477+0000","last_became_active":"2023-10-11T16:30:41.189929+0000","last_became_peered":"2023-10-11T16:30:41.189929+0000","last_unstale":"2023-10-11T16:30:41.190477+0000","last_undegraded":"2023-10-11T16:30:41.190477+0000","last_fullsized":"2023-10-11T16:30:41.190477+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:01:59.888888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190434+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:30:41.190434+0000","last_peered":"2023-10-11T16:30:41.190434+0000","last_clean":"2023-10-11T16:30:41.190434+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:30:41.190434+0000","last_undegraded":"2023-10-11T16:30:41.190434+0000","last_fullsized":"2023-10-11T16:30:41.190434+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:14:41.192364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182589+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:30:41.182589+0000","last_peered":"2023-10-11T16:30:41.182589+0000","last_clean":"2023-10-11T16:30:41.182589+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:30:41.182589+0000","last_undegraded":"2023-10-11T16:30:41.182589+0000","last_fullsized":"2023-10-11T16:30:41.182589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:05:17.909356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189311+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:30:41.189311+0000","last_peered":"2023-10-11T16:30:41.189311+0000","last_clean":"2023-10-11T16:30:41.189311+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:30:41.189311+0000","last_undegraded":"2023-10-11T16:30:41.189311+0000","last_fullsized":"2023-10-11T16:30:41.189311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:47.760490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198308+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:30:41.198308+0000","last_peered":"2023-10-11T16:30:41.198308+0000","last_clean":"2023-10-11T16:30:41.198308+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:30:41.198308+0000","last_undegraded":"2023-10-11T16:30:41.198308+0000","last_fullsized":"2023-10-11T16:30:41.198308+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:33:35.561074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182863+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:30:41.182863+0000","last_peered":"2023-10-11T16:30:41.182863+0000","last_clean":"2023-10-11T16:30:41.182863+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:30:41.182863+0000","last_undegraded":"2023-10-11T16:30:41.182863+0000","last_fullsized":"2023-10-11T16:30:41.182863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:56:12.549504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186098+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:30:41.186098+0000","last_peered":"2023-10-11T16:30:41.186098+0000","last_clean":"2023-10-11T16:30:41.186098+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:30:41.186098+0000","last_undegraded":"2023-10-11T16:30:41.186098+0000","last_fullsized":"2023-10-11T16:30:41.186098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:53:26.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186048+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:30:41.186048+0000","last_peered":"2023-10-11T16:30:41.186048+0000","last_clean":"2023-10-11T16:30:41.186048+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:30:41.186048+0000","last_undegraded":"2023-10-11T16:30:41.186048+0000","last_fullsized":"2023-10-11T16:30:41.186048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:50.699880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185837+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:30:41.185837+0000","last_peered":"2023-10-11T16:30:41.185837+0000","last_clean":"2023-10-11T16:30:41.185837+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:30:41.185837+0000","last_undegraded":"2023-10-11T16:30:41.185837+0000","last_fullsized":"2023-10-11T16:30:41.185837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:03:23.234106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185679+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:30:41.185679+0000","last_peered":"2023-10-11T16:30:41.185679+0000","last_clean":"2023-10-11T16:30:41.185679+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:30:41.185679+0000","last_undegraded":"2023-10-11T16:30:41.185679+0000","last_fullsized":"2023-10-11T16:30:41.185679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:10:44.726491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198414+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:30:41.198414+0000","last_peered":"2023-10-11T16:30:41.198414+0000","last_clean":"2023-10-11T16:30:41.198414+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:30:41.198414+0000","last_undegraded":"2023-10-11T16:30:41.198414+0000","last_fullsized":"2023-10-11T16:30:41.198414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:14.252893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186179+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:30:41.186179+0000","last_peered":"2023-10-11T16:30:41.186179+0000","last_clean":"2023-10-11T16:30:41.186179+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:30:41.186179+0000","last_undegraded":"2023-10-11T16:30:41.186179+0000","last_fullsized":"2023-10-11T16:30:41.186179+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:16:58.345842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185560+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:30:41.185560+0000","last_peered":"2023-10-11T16:30:41.185560+0000","last_clean":"2023-10-11T16:30:41.185560+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:30:41.185560+0000","last_undegraded":"2023-10-11T16:30:41.185560+0000","last_fullsized":"2023-10-11T16:30:41.185560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:19:06.908283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403856+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:30:41.403856+0000","last_peered":"2023-10-11T16:30:41.403856+0000","last_clean":"2023-10-11T16:30:41.403856+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:30:41.403856+0000","last_undegraded":"2023-10-11T16:30:41.403856+0000","last_fullsized":"2023-10-11T16:30:41.403856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:52:27.929431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189673+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:30:41.189673+0000","last_peered":"2023-10-11T16:30:41.189673+0000","last_clean":"2023-10-11T16:30:41.189673+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:30:41.189673+0000","last_undegraded":"2023-10-11T16:30:41.189673+0000","last_fullsized":"2023-10-11T16:30:41.189673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:47:07.501466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403440+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:30:41.403440+0000","last_peered":"2023-10-11T16:30:41.403440+0000","last_clean":"2023-10-11T16:30:41.403440+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:30:41.403440+0000","last_undegraded":"2023-10-11T16:30:41.403440+0000","last_fullsized":"2023-10-11T16:30:41.403440+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:24:21.537533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185688+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:30:41.185688+0000","last_peered":"2023-10-11T16:30:41.185688+0000","last_clean":"2023-10-11T16:30:41.185688+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:30:41.185688+0000","last_undegraded":"2023-10-11T16:30:41.185688+0000","last_fullsized":"2023-10-11T16:30:41.185688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:25:15.131318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198464+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:30:41.198464+0000","last_peered":"2023-10-11T16:30:41.198464+0000","last_clean":"2023-10-11T16:30:41.198464+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:30:41.198464+0000","last_undegraded":"2023-10-11T16:30:41.198464+0000","last_fullsized":"2023-10-11T16:30:41.198464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:10.823288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185010+0000","last_change":"2023-10-11T16:30:11.668522+0000","last_active":"2023-10-11T16:30:41.185010+0000","last_peered":"2023-10-11T16:30:41.185010+0000","last_clean":"2023-10-11T16:30:41.185010+0000","last_became_active":"2023-10-11T16:30:11.654192+0000","last_became_peered":"2023-10-11T16:30:11.654192+0000","last_unstale":"2023-10-11T16:30:41.185010+0000","last_undegraded":"2023-10-11T16:30:41.185010+0000","last_fullsized":"2023-10-11T16:30:41.185010+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:26:57.163089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":46642,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":52,"num_read_kb":73,"num_write":141,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":39701,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389120,"data_stored":178441,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449106,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":978944,"data_stored":488446,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.51600000000000001}]},{"osd":1,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.63}]},{"osd":2,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.70399999999999996}]},{"osd":5,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.78600000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.82299999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":9,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.83999999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:30:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.92100000000000004}]}]},{"osd":10,"up_from":71,"seq":304942678038,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":1072,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":1097728,"data_stored":1118190,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":1,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.49399999999999999}]},{"osd":3,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73499999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.39600000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.504},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.55500000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.91500000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":0.64800000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.93799999999999994}]},{"osd":11,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57099999999999995}]}]},{"osd":9,"up_from":66,"seq":283467841561,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6276,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":978944,"data_stored":482592,"data_compressed":14600,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.622},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.42499999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.77300000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.495}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.59199999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.85099999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.754},{"interface":"front","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.70199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":8,"up_from":58,"seq":249108103200,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6468,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738684,"statfs":{"total":95995035648,"available":95988412416,"internally_reserved":0,"allocated":978944,"data_stored":487645,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.622}]},{"osd":1,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56599999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.71799999999999997}]},{"osd":2,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.443,"15min":0.44},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.55200000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.55900000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.78100000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60899999999999999,"15min":0.61399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.42299999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.65500000000000003}]},{"osd":5,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.52900000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.65000000000000002,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67300000000000004,"15min":0.67100000000000004},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.63600000000000001}]},{"osd":7,"last update":"Wed Oct 11 16:30:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67800000000000005},"min":{"1min":0.497,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.86499999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:30:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.51800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.71999999999999997,"15min":0.71699999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.72999999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:30:05 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.69299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":7,"up_from":51,"seq":219043332132,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6584,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738568,"statfs":{"total":95995035648,"available":95988293632,"internally_reserved":0,"allocated":1032192,"data_stored":1082838,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.33800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.44800000000000001},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.629,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.39400000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59199999999999997,"15min":0.59099999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81499999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.42999999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85499999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.50900000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42599999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78000000000000003,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.63100000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.748,"5min":1.103,"15min":1.103},"last":0.54700000000000004}]},{"osd":4,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77100000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.375}]},{"osd":5,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63,"15min":0.628},"min":{"1min":0.45400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79500000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.64000000000000001,"15min":0.63400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.624}]},{"osd":6,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.70699999999999996,"15min":0.70999999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65100000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69799999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.71799999999999997,"15min":0.72599999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.69699999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:30:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77400000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.995,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76800000000000002,"15min":0.77200000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.99199999999999999,"5min":1.042,"15min":1.042},"last":0.68500000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":966656,"data_stored":483254,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63900000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63900000000000001}]},{"osd":1,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.56100000000000005},"min":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.60699999999999998,"15min":0.58499999999999996},"min":{"1min":0.44400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85299999999999998,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.68799999999999994}]},{"osd":2,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.75700000000000001,"5min":1.002,"15min":1.002},"last":0.749},{"interface":"front","average":{"1min":0.622,"5min":0.64200000000000002,"15min":0.63400000000000001},"min":{"1min":0.47299999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.83599999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83599999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56699999999999995,"15min":0.55500000000000005},"min":{"1min":0.38400000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79700000000000004,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74099999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53700000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.70399999999999996,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90400000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66800000000000004,"15min":0.65200000000000002},"min":{"1min":0.52600000000000002,"5min":0.502,"15min":0.502},"max":{"1min":0.89100000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67800000000000005,"15min":0.65900000000000003},"min":{"1min":0.52500000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.871,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.5,"5min":0.434,"15min":0.434},"max":{"1min":0.877,"5min":1.133,"15min":1.133},"last":0.78400000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:29:55 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.77300000000000002,"15min":0.75700000000000001},"min":{"1min":0.66300000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73199999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.66000000000000003}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.74099999999999999,"15min":0.749},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.84799999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.85699999999999998,"15min":0.86799999999999999},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0569999999999999,"5min":1.254,"15min":1.254},"last":0.89300000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.92800000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:06 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.875},{"interface":"front","average":{"1min":0.877,"5min":0.877,"15min":0.877},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.91300000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7464,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737688,"statfs":{"total":95995035648,"available":95987392512,"internally_reserved":0,"allocated":1015808,"data_stored":526543,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57399999999999995,"15min":0.621},"min":{"1min":0.371,"5min":0.251,"15min":0.251},"max":{"1min":0.754,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.53500000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76600000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.505,"15min":0.501},"min":{"1min":0.442,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.748},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.50900000000000001,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.307,"15min":0.307},"max":{"1min":0.68600000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.76000000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.438,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.68200000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:29:53 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41999999999999998,"15min":0.40899999999999997},"min":{"1min":0.33300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.46100000000000002,"15min":0.439},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.65800000000000003}]},{"osd":5,"last update":"Wed Oct 11 16:30:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.55100000000000005,"15min":0.52100000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.80300000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.61899999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.504,"15min":0.48999999999999999},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.70299999999999996}]},{"osd":8,"last update":"Wed Oct 11 16:30:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65300000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.72399999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.82899999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.86099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7376,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737776,"statfs":{"total":95995035648,"available":95987482624,"internally_reserved":0,"allocated":991232,"data_stored":484642,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.40899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.55500000000000005,"15min":0.52600000000000002},"min":{"1min":0.441,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.55900000000000005}]},{"osd":2,"last update":"Wed Oct 11 16:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60099999999999998,"15min":0.57799999999999996},"min":{"1min":0.38500000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.622,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.72199999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:28:27 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.44500000000000001,"15min":0.433},"min":{"1min":0.35799999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.437},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59399999999999997}]},{"osd":4,"last update":"Wed Oct 11 16:29:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57599999999999996,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.628}]},{"osd":5,"last update":"Wed Oct 11 16:30:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64700000000000002,"15min":0.63400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.67400000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:30:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63500000000000001,"15min":0.61899999999999999},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.755,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.74299999999999999,"5min":0.999,"15min":0.999},"last":0.57699999999999996}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.74399999999999999,"15min":0.73599999999999999},"min":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.61599999999999999}]},{"osd":8,"last update":"Wed Oct 11 16:30:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.73199999999999998,"15min":0.73899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75900000000000001},"min":{"1min":0.56599999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91800000000000004,"5min":1.133,"15min":1.133},"last":0.83899999999999997}]},{"osd":9,"last update":"Wed Oct 11 16:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.63800000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.751},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70799999999999996}]},{"osd":11,"last update":"Wed Oct 11 16:30:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.85599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7472,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737680,"statfs":{"total":95995035648,"available":95987384320,"internally_reserved":0,"allocated":1024000,"data_stored":513399,"data_compressed":14600,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.79800000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.38300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91800000000000004,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78300000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53900000000000003,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.83599999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.84199999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56699999999999995,"15min":0.54200000000000004},"min":{"1min":0.49399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59499999999999997,"15min":0.56299999999999994},"min":{"1min":0.47299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68600000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74399999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57899999999999996,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.75900000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.875}]},{"osd":5,"last update":"Wed Oct 11 16:30:10 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.45800000000000002},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66600000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.49099999999999999,"15min":0.47799999999999998},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.61499999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.64200000000000002,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86599999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.65600000000000003,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76100000000000001}]},{"osd":7,"last update":"Wed Oct 11 16:30:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77900000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67500000000000004,"15min":0.67600000000000005},"min":{"1min":0.46000000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.80700000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":1.0149999999999999}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.877,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.627,"15min":0.61599999999999999},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92200000000000004,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.91500000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.52600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":1},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.73299999999999998,"15min":0.71999999999999997},"min":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.92800000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:04 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":1.036}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6720,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738432,"statfs":{"total":95995035648,"available":95988154368,"internally_reserved":0,"allocated":974848,"data_stored":483117,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.38200000000000001},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69999999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.40699999999999997},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60599999999999998,"15min":0.56999999999999995},"min":{"1min":0.433,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.76400000000000001}]},{"osd":2,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63500000000000001,"15min":0.58899999999999997},"min":{"1min":0.48199999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69699999999999995}]},{"osd":4,"last update":"Wed Oct 11 16:29:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64500000000000002,"15min":0.63400000000000001},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90900000000000003,"5min":1.101,"15min":1.101},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.69399999999999995,"15min":0.68000000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0509999999999999,"5min":1.194,"15min":1.194},"last":0.83299999999999996}]},{"osd":5,"last update":"Wed Oct 11 16:30:12 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.63,"15min":0.63400000000000001},"min":{"1min":0.443,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85399999999999998,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.69199999999999995,"15min":0.68899999999999995},"min":{"1min":0.51300000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91600000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.70999999999999996}]},{"osd":6,"last update":"Wed Oct 11 16:30:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91500000000000004,"5min":1.268,"15min":1.268},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.40300000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:29:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.84599999999999997,"5min":1.119,"15min":1.119},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.071,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.79800000000000004}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:30:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.98699999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.496},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.79700000000000004,"15min":0.80600000000000005},"min":{"1min":0.53200000000000003,"5min":0.502,"15min":0.502},"max":{"1min":0.92100000000000004,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56499999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.81100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1003520,"data_stored":530984,"data_compressed":24251,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.379,"15min":0.374},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.38,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40400000000000003}]},{"osd":2,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.55200000000000005,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79600000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.622}]},{"osd":3,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.38,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.53500000000000003}]},{"osd":5,"last update":"Wed Oct 11 16:30:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.71999999999999997,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":4.2560000000000002,"5min":4.5,"15min":4.5},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68899999999999995,"15min":0.72999999999999998},"min":{"1min":0.372,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.998,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.73999999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.64300000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:30:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48999999999999999,"15min":0.48199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.69099999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:30:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.60999999999999999,"15min":0.59499999999999997},"min":{"1min":0.47799999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.45500000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55700000000000005}]},{"osd":9,"last update":"Wed Oct 11 16:30:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.55000000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91900000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.55400000000000005,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.75600000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6724,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738428,"statfs":{"total":95995035648,"available":95988150272,"internally_reserved":0,"allocated":1044480,"data_stored":1079056,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.745,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.78700000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.52700000000000002},"min":{"1min":0.38100000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.873,"5min":0.874,"15min":0.874},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77700000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.70599999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.438,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69599999999999995,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63200000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55700000000000005,"15min":0.55600000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.71399999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.55400000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.90600000000000003,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.76500000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58899999999999997,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85299999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.99299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78100000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.83899999999999997}]},{"osd":7,"last update":"Wed Oct 11 16:29:58 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.70799999999999996,"15min":0.71499999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69299999999999995,"15min":0.69599999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.89200000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:30:21 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40799999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64600000000000002,"15min":0.64800000000000002},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84299999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.79000000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.93600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.97499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":43710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":39235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":43724,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":44658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:30:46.929 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-11T16:30:46.930 INFO:tasks.cephadm:Setting up client nodes... 2023-10-11T16:30:46.930 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-10-11T16:30:47.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:47 smithi130 ceph-mon[131859]: pgmap v292: 129 pgs: 129 active+clean; 622 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.2 KiB/s wr, 15 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:47.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:47 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:47.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:47 smithi111 ceph-mon[98527]: pgmap v292: 129 pgs: 129 active+clean; 622 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.2 KiB/s wr, 15 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:47.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:47 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:47 smithi150 ceph-mon[132217]: pgmap v292: 129 pgs: 129 active+clean; 622 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.2 KiB/s wr, 15 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:47 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:47.807 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:30:48.300 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:48 smithi111 ceph-mon[98527]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:48.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:48 smithi130 ceph-mon[131859]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:48.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:48 smithi150 ceph-mon[132217]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:30:49.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 -- 172.21.15.111:0/4250064853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17f4102330 msgr2=0x7f17f4102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:49.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 --2- 172.21.15.111:0/4250064853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17f4102330 0x7f17f4102740 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009990 tx=0x7f17f0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 -- 172.21.15.111:0/4250064853 shutdown_connections 2023-10-11T16:30:49.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 --2- 172.21.15.111:0/4250064853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17f4103ec0 0x7f17f410a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 --2- 172.21.15.111:0/4250064853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17f4102330 0x7f17f4102740 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 --2- 172.21.15.111:0/4250064853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f4103530 0x7f17f4103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 -- 172.21.15.111:0/4250064853 >> 172.21.15.111:0/4250064853 conn(0x7f17f40fd9a0 msgr2=0x7f17f40ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 -- 172.21.15.111:0/4250064853 shutdown_connections 2023-10-11T16:30:49.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.229+0000 7f17faff8700 1 -- 172.21.15.111:0/4250064853 wait complete. 2023-10-11T16:30:49.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.230+0000 7f17faff8700 1 Processor -- start 2023-10-11T16:30:49.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.230+0000 7f17faff8700 1 -- start start 2023-10-11T16:30:49.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17faff8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17f4102330 0x7f17f41980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:49.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17faff8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f4103530 0x7f17f41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:49.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17faff8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17f4103ec0 0x7f17f419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:49.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17faff8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f419fc30 con 0x7f17f4103530 2023-10-11T16:30:49.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17faff8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f419fd70 con 0x7f17f4103ec0 2023-10-11T16:30:49.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17faff8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f419ff00 con 0x7f17f4102330 2023-10-11T16:30:49.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f9ff6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17f4102330 0x7f17f41980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:49.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f9ff6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17f4102330 0x7f17f41980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36066/0 (socket says 172.21.15.111:36066) 2023-10-11T16:30:49.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f97f5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f4103530 0x7f17f41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:49.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f9ff6700 1 -- 172.21.15.111:0/1194020135 learned_addr learned my addr 172.21.15.111:0/1194020135 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:49.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17fa7f7700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17f4103ec0 0x7f17f419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:49.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f9ff6700 1 -- 172.21.15.111:0/1194020135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17f4103ec0 msgr2=0x7f17f419f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:49.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f9ff6700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17f4103ec0 0x7f17f419f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f9ff6700 1 -- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f4103530 msgr2=0x7f17f41985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:49.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f9ff6700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f4103530 0x7f17f41985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f9ff6700 1 -- 172.21.15.111:0/1194020135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e4009520 con 0x7f17f4102330 2023-10-11T16:30:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.231+0000 7f17f97f5700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f4103530 0x7f17f41985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.232+0000 7f17f9ff6700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17f4102330 0x7f17f41980a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17f0004740 tx=0x7f17f0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.232+0000 7f17eaffd700 1 -- 172.21.15.111:0/1194020135 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f001d070 con 0x7f17f4102330 2023-10-11T16:30:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.232+0000 7f17eaffd700 1 -- 172.21.15.111:0/1194020135 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17f0021410 con 0x7f17f4102330 2023-10-11T16:30:49.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.232+0000 7f17eaffd700 1 -- 172.21.15.111:0/1194020135 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0017650 con 0x7f17f4102330 2023-10-11T16:30:49.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.232+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f00095f0 con 0x7f17f4102330 2023-10-11T16:30:49.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.232+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f410dcf0 con 0x7f17f4102330 2023-10-11T16:30:49.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.233+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f404e8c0 con 0x7f17f4102330 2023-10-11T16:30:49.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.235+0000 7f17eaffd700 1 -- 172.21.15.111:0/1194020135 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f17f0021580 con 0x7f17f4102330 2023-10-11T16:30:49.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.236+0000 7f17eaffd700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f17e00a1260 0x7f17e00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:49.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.236+0000 7f17eaffd700 1 -- 172.21.15.111:0/1194020135 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f17f00ce280 con 0x7f17f4102330 2023-10-11T16:30:49.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.237+0000 7f17f97f5700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f17e00a1260 0x7f17e00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:49.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.237+0000 7f17f97f5700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f17e00a1260 0x7f17e00a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f17e4009fd0 tx=0x7f17e4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:49.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.239+0000 7f17eaffd700 1 -- 172.21.15.111:0/1194020135 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f17f0097590 con 0x7f17f4102330 2023-10-11T16:30:49.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:49 smithi111 ceph-mon[98527]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 13 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:49.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.514+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f17f4109240 con 0x7f17f4102330 2023-10-11T16:30:49.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.523+0000 7f17eaffd700 1 -- 172.21.15.111:0/1194020135 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f17f0026050 con 0x7f17f4102330 2023-10-11T16:30:49.525 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2023-10-11T16:30:49.525 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQC5zSZlPHfYHhAAsrExhN8cOK8+1tUPbpSIpw== 2023-10-11T16:30:49.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f17e00a1260 msgr2=0x7f17e00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:49.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f17e00a1260 0x7f17e00a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f17e4009fd0 tx=0x7f17e4009260 comp rx=0 tx=0).stop 2023-10-11T16:30:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17f4102330 msgr2=0x7f17f41980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17f4102330 0x7f17f41980a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17f0004740 tx=0x7f17f0009580 comp rx=0 tx=0).stop 2023-10-11T16:30:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 shutdown_connections 2023-10-11T16:30:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f17e00a1260 0x7f17e00a3710 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17f4103ec0 0x7f17f419f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17f4102330 0x7f17f41980a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 --2- 172.21.15.111:0/1194020135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17f4103530 0x7f17f41985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:49.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 >> 172.21.15.111:0/1194020135 conn(0x7f17f40fd9a0 msgr2=0x7f17f40ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:49.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 shutdown_connections 2023-10-11T16:30:49.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:49.526+0000 7f17faff8700 1 -- 172.21.15.111:0/1194020135 wait complete. 2023-10-11T16:30:49.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:49 smithi130 ceph-mon[131859]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 13 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:49.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:49 smithi150 ceph-mon[132217]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 13 op/s; 4.8 KiB/s, 1 objects/s recovering 2023-10-11T16:30:50.012 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:30:50.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-10-11T16:30:50.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-11T16:30:50.113 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-11T16:30:50.115 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-10-11T16:30:50.115 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph mgr dump --format=json 2023-10-11T16:30:50.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:50 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1194020135' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-11T16:30:50.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:50 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-11T16:30:50.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:50 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-11T16:30:50.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:50 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1194020135' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-11T16:30:50.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:50 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-11T16:30:50.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:50 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-11T16:30:50.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:50 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1194020135' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-11T16:30:50.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:50 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-11T16:30:50.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:50 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-11T16:30:50.999 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:30:51.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:51 smithi130 ceph-mon[131859]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.4 KiB/s wr, 11 op/s; 3.9 KiB/s, 1 objects/s recovering 2023-10-11T16:30:51.609 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:51 smithi111 ceph-mon[98527]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.4 KiB/s wr, 11 op/s; 3.9 KiB/s, 1 objects/s recovering 2023-10-11T16:30:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:51 smithi150 ceph-mon[132217]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.4 KiB/s wr, 11 op/s; 3.9 KiB/s, 1 objects/s recovering 2023-10-11T16:30:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.575+0000 7f9f165a4700 1 -- 172.21.15.111:0/3211331882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100ee0 msgr2=0x7f9f10104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.575+0000 7f9f165a4700 1 --2- 172.21.15.111:0/3211331882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100ee0 0x7f9f10104eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9f040099c0 tx=0x7f9f04009cd0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.575+0000 7f9f165a4700 1 -- 172.21.15.111:0/3211331882 shutdown_connections 2023-10-11T16:30:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.576+0000 7f9f165a4700 1 --2- 172.21.15.111:0/3211331882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f101055e0 0x7f9f1010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.576+0000 7f9f165a4700 1 --2- 172.21.15.111:0/3211331882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f10100540 0x7f9f10100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.576+0000 7f9f165a4700 1 --2- 172.21.15.111:0/3211331882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100ee0 0x7f9f10104eb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.576+0000 7f9f165a4700 1 -- 172.21.15.111:0/3211331882 >> 172.21.15.111:0/3211331882 conn(0x7f9f100fbdc0 msgr2=0x7f9f100fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:52.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.576+0000 7f9f165a4700 1 -- 172.21.15.111:0/3211331882 shutdown_connections 2023-10-11T16:30:52.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.576+0000 7f9f165a4700 1 -- 172.21.15.111:0/3211331882 wait complete. 2023-10-11T16:30:52.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f165a4700 1 Processor -- start 2023-10-11T16:30:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f165a4700 1 -- start start 2023-10-11T16:30:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f165a4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100540 0x7f9f10198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f165a4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f10100ee0 0x7f9f10198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f165a4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f101055e0 0x7f9f1019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f165a4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1019d4f0 con 0x7f9f10100540 2023-10-11T16:30:52.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f165a4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1019d660 con 0x7f9f101055e0 2023-10-11T16:30:52.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f165a4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1019d820 con 0x7f9f10100ee0 2023-10-11T16:30:52.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f0ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100540 0x7f9f10198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:52.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f0ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100540 0x7f9f10198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43026/0 (socket says 172.21.15.111:43026) 2023-10-11T16:30:52.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.577+0000 7f9f0ffff700 1 -- 172.21.15.111:0/85282151 learned_addr learned my addr 172.21.15.111:0/85282151 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:52.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0f7fe700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f10100ee0 0x7f9f10198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:52.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f14b41700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f101055e0 0x7f9f1019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:52.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0ffff700 1 -- 172.21.15.111:0/85282151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f10100ee0 msgr2=0x7f9f10198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:52.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0ffff700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f10100ee0 0x7f9f10198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0ffff700 1 -- 172.21.15.111:0/85282151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f101055e0 msgr2=0x7f9f1019cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0ffff700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f101055e0 0x7f9f1019cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0ffff700 1 -- 172.21.15.111:0/85282151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f040095f0 con 0x7f9f10100540 2023-10-11T16:30:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f14b41700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f101055e0 0x7f9f1019cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:30:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0ffff700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100540 0x7f9f10198530 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000c9b0 tx=0x7f9f0000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0d7fa700 1 -- 172.21.15.111:0/85282151 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00007b00 con 0x7f9f10100540 2023-10-11T16:30:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0d7fa700 1 -- 172.21.15.111:0/85282151 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f00007c60 con 0x7f9f10100540 2023-10-11T16:30:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f10110310 con 0x7f9f10100540 2023-10-11T16:30:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.578+0000 7f9f0d7fa700 1 -- 172.21.15.111:0/85282151 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00018660 con 0x7f9f10100540 2023-10-11T16:30:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.579+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f10110180 con 0x7f9f10100540 2023-10-11T16:30:52.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.580+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f101104a0 con 0x7f9f10100540 2023-10-11T16:30:52.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:52 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:52.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.582+0000 7f9f0d7fa700 1 -- 172.21.15.111:0/85282151 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f9f0000f450 con 0x7f9f10100540 2023-10-11T16:30:52.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.583+0000 7f9f0d7fa700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ef80a11e0 0x7f9ef80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:52.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.583+0000 7f9f0d7fa700 1 -- 172.21.15.111:0/85282151 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9f000cd0a0 con 0x7f9f10100540 2023-10-11T16:30:52.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.583+0000 7f9f0f7fe700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ef80a11e0 0x7f9ef80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:52.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.584+0000 7f9f0f7fe700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ef80a11e0 0x7f9ef80a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400b580 tx=0x7f9f04003680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:52.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.585+0000 7f9f0d7fa700 1 -- 172.21.15.111:0/85282151 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9f000963b0 con 0x7f9f10100540 2023-10-11T16:30:52.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:52 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:52.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:52 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:30:52.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.838+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f9f10066ea0 con 0x7f9f10100540 2023-10-11T16:30:52.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.844+0000 7f9f0d7fa700 1 -- 172.21.15.111:0/85282151 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260133 (secure 0 0 0) 0x7f9f000999d0 con 0x7f9f10100540 2023-10-11T16:30:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:52.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ef80a11e0 msgr2=0x7f9ef80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:52.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ef80a11e0 0x7f9ef80a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400b580 tx=0x7f9f04003680 comp rx=0 tx=0).stop 2023-10-11T16:30:52.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100540 msgr2=0x7f9f10198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:52.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100540 0x7f9f10198530 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9f0000c9b0 tx=0x7f9f0000ccc0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 shutdown_connections 2023-10-11T16:30:52.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9ef80a11e0 0x7f9ef80a3690 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f101055e0 0x7f9f1019cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f10100540 0x7f9f10198530 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 --2- 172.21.15.111:0/85282151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f10100ee0 0x7f9f10198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:52.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 >> 172.21.15.111:0/85282151 conn(0x7f9f100fbdc0 msgr2=0x7f9f1010b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:52.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 shutdown_connections 2023-10-11T16:30:52.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:52.848+0000 7f9f165a4700 1 -- 172.21.15.111:0/85282151 wait complete. 2023-10-11T16:30:53.305 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":17,"active_gid":14148,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2838247405},{"type":"v1","addr":"172.21.15.111:6801","nonce":2838247405}]},"active_addr":"172.21.15.111:6801/2838247405","active_change":"2023-10-11T16:23:39.970770+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.111:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":3816963652}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":444576470}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":1856545817}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":803529843}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":1217667995}]}]} 2023-10-11T16:30:53.312 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-10-11T16:30:53.312 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-10-11T16:30:53.312 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd dump --format=json 2023-10-11T16:30:53.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:53 smithi130 ceph-mon[131859]: pgmap v295: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 7.5 KiB/s wr, 10 op/s; 3.6 KiB/s, 1 objects/s recovering 2023-10-11T16:30:53.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:53 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/85282151' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-11T16:30:53.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:53 smithi111 ceph-mon[98527]: pgmap v295: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 7.5 KiB/s wr, 10 op/s; 3.6 KiB/s, 1 objects/s recovering 2023-10-11T16:30:53.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:53 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/85282151' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-11T16:30:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:53 smithi150 ceph-mon[132217]: pgmap v295: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 7.5 KiB/s wr, 10 op/s; 3.6 KiB/s, 1 objects/s recovering 2023-10-11T16:30:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:53 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/85282151' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-11T16:30:54.152 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:30:55.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:55 smithi130 ceph-mon[131859]: pgmap v296: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.7 KiB/s wr, 9 op/s; 3.2 KiB/s, 0 objects/s recovering 2023-10-11T16:30:55.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.598+0000 7f3f66e27700 1 -- 172.21.15.111:0/4036839078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f601055e0 msgr2=0x7f3f6010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:55.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.598+0000 7f3f66e27700 1 --2- 172.21.15.111:0/4036839078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f601055e0 0x7f3f6010d160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800b0d0 tx=0x7f3f5800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.598+0000 7f3f66e27700 1 -- 172.21.15.111:0/4036839078 shutdown_connections 2023-10-11T16:30:55.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.598+0000 7f3f66e27700 1 --2- 172.21.15.111:0/4036839078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f601055e0 0x7f3f6010d160 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.598+0000 7f3f66e27700 1 --2- 172.21.15.111:0/4036839078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f60100540 0x7f3f60100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.598+0000 7f3f66e27700 1 --2- 172.21.15.111:0/4036839078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f60100ee0 0x7f3f60104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.598+0000 7f3f66e27700 1 -- 172.21.15.111:0/4036839078 >> 172.21.15.111:0/4036839078 conn(0x7f3f600fbdc0 msgr2=0x7f3f600fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:55.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.599+0000 7f3f66e27700 1 -- 172.21.15.111:0/4036839078 shutdown_connections 2023-10-11T16:30:55.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.599+0000 7f3f66e27700 1 -- 172.21.15.111:0/4036839078 wait complete. 2023-10-11T16:30:55.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.600+0000 7f3f66e27700 1 Processor -- start 2023-10-11T16:30:55.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.600+0000 7f3f66e27700 1 -- start start 2023-10-11T16:30:55.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f66e27700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:55.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f66e27700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:55.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f66e27700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:55.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f66e27700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a4920 con 0x7f3f60100ee0 2023-10-11T16:30:55.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f66e27700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a4a90 con 0x7f3f601055e0 2023-10-11T16:30:55.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f66e27700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a4c50 con 0x7f3f60100540 2023-10-11T16:30:55.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f5ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:55.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f64bc3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:55.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f5ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46836/0 (socket says 172.21.15.111:46836) 2023-10-11T16:30:55.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f64bc3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:58118/0 (socket says 172.21.15.111:58118) 2023-10-11T16:30:55.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f5ffff700 1 -- 172.21.15.111:0/3161181279 learned_addr learned my addr 172.21.15.111:0/3161181279 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:55.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f653c4700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:55.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f64bc3700 1 -- 172.21.15.111:0/3161181279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f601055e0 msgr2=0x7f3f600738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:55.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.601+0000 7f3f64bc3700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f64bc3700 1 -- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f60100ee0 msgr2=0x7f3f60073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:55.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f64bc3700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f64bc3700 1 -- 172.21.15.111:0/3161181279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f58009d00 con 0x7f3f60100540 2023-10-11T16:30:55.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f5ffff700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:55.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f64bc3700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400e9f0 tx=0x7f3f5400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:55.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f5dffb700 1 -- 172.21.15.111:0/3161181279 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400c9c0 con 0x7f3f60100540 2023-10-11T16:30:55.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f5dffb700 1 -- 172.21.15.111:0/3161181279 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f54010430 con 0x7f3f60100540 2023-10-11T16:30:55.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f60110370 con 0x7f3f60100540 2023-10-11T16:30:55.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f5dffb700 1 -- 172.21.15.111:0/3161181279 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54018680 con 0x7f3f60100540 2023-10-11T16:30:55.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.602+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f601101e0 con 0x7f3f60100540 2023-10-11T16:30:55.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.603+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f60110500 con 0x7f3f60100540 2023-10-11T16:30:55.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.605+0000 7f3f5dffb700 1 -- 172.21.15.111:0/3161181279 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f3f54010860 con 0x7f3f60100540 2023-10-11T16:30:55.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.606+0000 7f3f5dffb700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:55.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.607+0000 7f3f5dffb700 1 -- 172.21.15.111:0/3161181279 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3f54014070 con 0x7f3f60100540 2023-10-11T16:30:55.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.607+0000 7f3f5ffff700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:55.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.607+0000 7f3f5ffff700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3f500a11e0 0x7f3f500a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f600745e0 tx=0x7f3f4c00b480 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:55.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.611+0000 7f3f5dffb700 1 -- 172.21.15.111:0/3161181279 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3f54051730 con 0x7f3f60100540 2023-10-11T16:30:55.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:55 smithi111 ceph-mon[98527]: pgmap v296: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.7 KiB/s wr, 9 op/s; 3.2 KiB/s, 0 objects/s recovering 2023-10-11T16:30:55.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:55 smithi150 ceph-mon[132217]: pgmap v296: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.7 KiB/s wr, 9 op/s; 3.2 KiB/s, 0 objects/s recovering 2023-10-11T16:30:55.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.809+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f60066ea0 con 0x7f3f60100540 2023-10-11T16:30:55.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.811+0000 7f3f5dffb700 1 -- 172.21.15.111:0/3161181279 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24555 (secure 0 0 0) 0x7f3f54096190 con 0x7f3f60100540 2023-10-11T16:30:55.814 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:55.814 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":89,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:41.177705+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:55.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.813+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3f500a11e0 msgr2=0x7f3f500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:55.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.813+0000 7f3f66e27700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3f500a11e0 0x7f3f500a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f600745e0 tx=0x7f3f4c00b480 comp rx=0 tx=0).stop 2023-10-11T16:30:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.813+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f60100540 msgr2=0x7f3f60072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.813+0000 7f3f66e27700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400e9f0 tx=0x7f3f5400ed00 comp rx=0 tx=0).stop 2023-10-11T16:30:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.814+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 shutdown_connections 2023-10-11T16:30:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.814+0000 7f3f66e27700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.814+0000 7f3f66e27700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.814+0000 7f3f66e27700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.814+0000 7f3f66e27700 1 --2- 172.21.15.111:0/3161181279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:55.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.814+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 >> 172.21.15.111:0/3161181279 conn(0x7f3f600fbdc0 msgr2=0x7f3f600fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:55.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.814+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 shutdown_connections 2023-10-11T16:30:55.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:55.814+0000 7f3f66e27700 1 -- 172.21.15.111:0/3161181279 wait complete. 2023-10-11T16:30:56.245 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-10-11T16:30:56.246 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd dump --format=json 2023-10-11T16:30:56.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:56 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3161181279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:56.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:56 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3161181279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:56.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:56 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3161181279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:30:57.532 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:57 smithi111 ceph-mon[98527]: pgmap v297: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 50 B/s, 0 objects/s recovering 2023-10-11T16:30:57.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:57 smithi130 ceph-mon[131859]: pgmap v297: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 50 B/s, 0 objects/s recovering 2023-10-11T16:30:57.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:57 smithi150 ceph-mon[132217]: pgmap v297: 129 pgs: 129 active+clean; 650 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.2 KiB/s wr, 7 op/s; 50 B/s, 0 objects/s recovering 2023-10-11T16:30:58.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/461686127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 msgr2=0x7fdf7810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:58.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/461686127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 0x7fdf7810d160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000b0d0 tx=0x7fdf7000b3e0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/461686127 shutdown_connections 2023-10-11T16:30:58.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/461686127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 0x7fdf7810d160 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/461686127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf78100540 0x7fdf78100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/461686127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf78100ee0 0x7fdf78104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/461686127 >> 172.21.15.111:0/461686127 conn(0x7fdf780fbdc0 msgr2=0x7fdf780fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:58.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/461686127 shutdown_connections 2023-10-11T16:30:58.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.633+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/461686127 wait complete. 2023-10-11T16:30:58.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.634+0000 7fdf7ec3f700 1 Processor -- start 2023-10-11T16:30:58.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.634+0000 7fdf7ec3f700 1 -- start start 2023-10-11T16:30:58.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7ec3f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf78100540 0x7fdf781986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:58.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7ec3f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf78100ee0 0x7fdf78198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:58.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7ec3f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 0x7fdf7819bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:58.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7ec3f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7819c5a0 con 0x7fdf781055e0 2023-10-11T16:30:58.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7ec3f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7819c710 con 0x7fdf78100540 2023-10-11T16:30:58.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7ec3f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7819c8d0 con 0x7fdf78100ee0 2023-10-11T16:30:58.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7d1dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 0x7fdf7819bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:58.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7c9db700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf78100540 0x7fdf781986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:58.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7d1dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 0x7fdf7819bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46854/0 (socket says 172.21.15.111:46854) 2023-10-11T16:30:58.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7d1dc700 1 -- 172.21.15.111:0/453505809 learned_addr learned my addr 172.21.15.111:0/453505809 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:30:58.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf77fff700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf78100ee0 0x7fdf78198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:58.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7d1dc700 1 -- 172.21.15.111:0/453505809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf78100ee0 msgr2=0x7fdf78198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:58.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7d1dc700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf78100ee0 0x7fdf78198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7d1dc700 1 -- 172.21.15.111:0/453505809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf78100540 msgr2=0x7fdf781986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:58.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7d1dc700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf78100540 0x7fdf781986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7d1dc700 1 -- 172.21.15.111:0/453505809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf70009d00 con 0x7fdf781055e0 2023-10-11T16:30:58.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.635+0000 7fdf7c9db700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf78100540 0x7fdf781986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:30:58.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.636+0000 7fdf7d1dc700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 0x7fdf7819bfa0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdf70007870 tx=0x7fdf700078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:58.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.636+0000 7fdf75ffb700 1 -- 172.21.15.111:0/453505809 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7000e040 con 0x7fdf781055e0 2023-10-11T16:30:58.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.636+0000 7fdf75ffb700 1 -- 172.21.15.111:0/453505809 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf70003e10 con 0x7fdf781055e0 2023-10-11T16:30:58.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.636+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf78110320 con 0x7fdf781055e0 2023-10-11T16:30:58.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.636+0000 7fdf75ffb700 1 -- 172.21.15.111:0/453505809 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7001ba10 con 0x7fdf781055e0 2023-10-11T16:30:58.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.636+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf78110190 con 0x7fdf781055e0 2023-10-11T16:30:58.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.637+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf781104b0 con 0x7fdf781055e0 2023-10-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.639+0000 7fdf75ffb700 1 -- 172.21.15.111:0/453505809 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fdf70024410 con 0x7fdf781055e0 2023-10-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.640+0000 7fdf75ffb700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf680a1260 0x7fdf680a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:30:58.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.640+0000 7fdf75ffb700 1 -- 172.21.15.111:0/453505809 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdf700ce350 con 0x7fdf781055e0 2023-10-11T16:30:58.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.641+0000 7fdf7c9db700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf680a1260 0x7fdf680a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:30:58.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.641+0000 7fdf7c9db700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf680a1260 0x7fdf680a3710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c005950 tx=0x7fdf6c00b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:30:58.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.644+0000 7fdf75ffb700 1 -- 172.21.15.111:0/453505809 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdf700976c0 con 0x7fdf781055e0 2023-10-11T16:30:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.843+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf781a5750 con 0x7fdf781055e0 2023-10-11T16:30:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.845+0000 7fdf75ffb700 1 -- 172.21.15.111:0/453505809 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24555 (secure 0 0 0) 0x7fdf7009ace0 con 0x7fdf781055e0 2023-10-11T16:30:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:30:58.846 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":89,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:41.177705+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:30:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf680a1260 msgr2=0x7fdf680a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf680a1260 0x7fdf680a3710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c005950 tx=0x7fdf6c00b410 comp rx=0 tx=0).stop 2023-10-11T16:30:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 msgr2=0x7fdf7819bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:30:58.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 0x7fdf7819bfa0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdf70007870 tx=0x7fdf700078a0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 shutdown_connections 2023-10-11T16:30:58.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf680a1260 0x7fdf680a3710 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf781055e0 0x7fdf7819bfa0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf78100540 0x7fdf781986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 --2- 172.21.15.111:0/453505809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf78100ee0 0x7fdf78198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:30:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 >> 172.21.15.111:0/453505809 conn(0x7fdf780fbdc0 msgr2=0x7fdf780fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:30:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.847+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 shutdown_connections 2023-10-11T16:30:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:30:58.848+0000 7fdf7ec3f700 1 -- 172.21.15.111:0/453505809 wait complete. 2023-10-11T16:30:59.289 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-10-11T16:30:59.290 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-10-11T16:30:59.291 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-10-11T16:30:59.291 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-10-11T16:30:59.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-10-11T16:30:59.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-10-11T16:30:59.293 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-10-11T16:30:59.293 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-10-11T16:30:59.294 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.8 flush_pg_stats 2023-10-11T16:30:59.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.9 flush_pg_stats 2023-10-11T16:30:59.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.10 flush_pg_stats 2023-10-11T16:30:59.296 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph tell osd.11 flush_pg_stats 2023-10-11T16:30:59.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:59 smithi111 ceph-mon[98527]: pgmap v298: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-10-11T16:30:59.514 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:30:59 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/453505809' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:59.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:59 smithi130 ceph-mon[131859]: pgmap v298: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-10-11T16:30:59.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:30:59 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/453505809' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:30:59.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:59 smithi150 ceph-mon[132217]: pgmap v298: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-10-11T16:30:59.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:30:59 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/453505809' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:01.288 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:01.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:01 smithi130 ceph-mon[131859]: pgmap v299: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-10-11T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:01 smithi111 ceph-mon[98527]: pgmap v299: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-10-11T16:31:01.429 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:01.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:01 smithi150 ceph-mon[132217]: pgmap v299: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-10-11T16:31:01.840 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:02.252 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:03.054 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:03.066 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:03.301 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:03.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:03 smithi130 ceph-mon[131859]: pgmap v300: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-10-11T16:31:03.333 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:03 smithi111 ceph-mon[98527]: pgmap v300: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-10-11T16:31:03.347 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:03.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:03 smithi150 ceph-mon[132217]: pgmap v300: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-10-11T16:31:03.606 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:03.900 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:05.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 -- 172.21.15.111:0/428360777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400101540 msgr2=0x7fb400101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 --2- 172.21.15.111:0/428360777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400101540 0x7fb400101910 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009990 tx=0x7fb3f0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 -- 172.21.15.111:0/428360777 shutdown_connections 2023-10-11T16:31:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 --2- 172.21.15.111:0/428360777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb40010a890 0x7fb40010ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 --2- 172.21.15.111:0/428360777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400101540 0x7fb400101910 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:05.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 --2- 172.21.15.111:0/428360777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb400101e50 0x7fb40010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 -- 172.21.15.111:0/428360777 >> 172.21.15.111:0/428360777 conn(0x7fb4000770a0 msgr2=0x7fb4000774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:05.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 -- 172.21.15.111:0/428360777 shutdown_connections 2023-10-11T16:31:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.055+0000 7fb404c2c700 1 -- 172.21.15.111:0/428360777 wait complete. 2023-10-11T16:31:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.056+0000 7fb404c2c700 1 Processor -- start 2023-10-11T16:31:05.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.056+0000 7fb404c2c700 1 -- start start 2023-10-11T16:31:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb404c2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400101540 0x7fb4001a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb404c2c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb400101e50 0x7fb4001a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:05.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb404c2c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40010a890 0x7fb40019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:05.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb404c2c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40019cf30 con 0x7fb400101540 2023-10-11T16:31:05.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb404c2c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40019d0a0 con 0x7fb400101e50 2023-10-11T16:31:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb404c2c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40019d260 con 0x7fb40010a890 2023-10-11T16:31:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb3fdd9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb400101e50 0x7fb4001a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:05.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb3fe59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400101540 0x7fb4001a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb3fdd9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb400101e50 0x7fb4001a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:53096/0 (socket says 172.21.15.111:53096) 2023-10-11T16:31:05.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb3fdd9b700 1 -- 172.21.15.111:0/1689021053 learned_addr learned my addr 172.21.15.111:0/1689021053 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.057+0000 7fb3fed9d700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40010a890 0x7fb40019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.058+0000 7fb3fe59c700 1 -- 172.21.15.111:0/1689021053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40010a890 msgr2=0x7fb40019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:05.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.058+0000 7fb3fe59c700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb40010a890 0x7fb40019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.058+0000 7fb3fe59c700 1 -- 172.21.15.111:0/1689021053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb400101e50 msgr2=0x7fb4001a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:05.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.058+0000 7fb3fe59c700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb400101e50 0x7fb4001a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.058+0000 7fb3fe59c700 1 -- 172.21.15.111:0/1689021053 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f00095f0 con 0x7fb400101540 2023-10-11T16:31:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.058+0000 7fb3fe59c700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400101540 0x7fb4001a2840 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0000c00 tx=0x7fb3f000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:05.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.059+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f001d070 con 0x7fb400101540 2023-10-11T16:31:05.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.059+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3f0005050 con 0x7fb400101540 2023-10-11T16:31:05.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.059+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0004a20 con 0x7fb400101540 2023-10-11T16:31:05.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.059+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40010e210 con 0x7fb400101540 2023-10-11T16:31:05.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.059+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40010e080 con 0x7fb400101540 2023-10-11T16:31:05.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.060+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb4000fc720 con 0x7fb400101540 2023-10-11T16:31:05.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.062+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fb3f00051c0 con 0x7fb400101540 2023-10-11T16:31:05.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.064+0000 7fb3ef7fe700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb3e80a11c0 0x7fb3e80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:05.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.064+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb3f00ce470 con 0x7fb400101540 2023-10-11T16:31:05.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.065+0000 7fb3fdd9b700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb3e80a11c0 0x7fb3e80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.065+0000 7fb3ef7fe700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] conn(0x7fb3e80ac1d0 0x7fb3e80ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.065+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 --> [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3e80aed30 con 0x7fb3e80ac1d0 2023-10-11T16:31:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.065+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3f00ce880 con 0x7fb400101540 2023-10-11T16:31:05.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.066+0000 7fb3fed9d700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] conn(0x7fb3e80ac1d0 0x7fb3e80ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.066+0000 7fb3fdd9b700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb3e80a11c0 0x7fb3e80a3670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb40010c480 tx=0x7fb3f4007480 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:05.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.066+0000 7fb3fed9d700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] conn(0x7fb3e80ac1d0 0x7fb3e80ae660 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.067+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 <== osd.11 v2:172.21.15.150:6824/2209861208 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb3e80aed30 con 0x7fb3e80ac1d0 2023-10-11T16:31:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.086+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 --> [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb40010e3a0 con 0x7fb3e80ac1d0 2023-10-11T16:31:05.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.087+0000 7fb3ef7fe700 1 -- 172.21.15.111:0/1689021053 <== osd.11 v2:172.21.15.150:6824/2209861208 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb40010e3a0 con 0x7fb3e80ac1d0 2023-10-11T16:31:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 >> [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] conn(0x7fb3e80ac1d0 msgr2=0x7fb3e80ae660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] conn(0x7fb3e80ac1d0 0x7fb3e80ae660 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb3e80a11c0 msgr2=0x7fb3e80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:05.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb3e80a11c0 0x7fb3e80a3670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb40010c480 tx=0x7fb3f4007480 comp rx=0 tx=0).stop 2023-10-11T16:31:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400101540 msgr2=0x7fb4001a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 --2- 172.21.15.111:0/1689021053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb400101540 0x7fb4001a2840 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0000c00 tx=0x7fb3f000f690 comp rx=0 tx=0).stop 2023-10-11T16:31:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb3fed9d700 1 -- 172.21.15.111:0/1689021053 reap_dead start 2023-10-11T16:31:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 shutdown_connections 2023-10-11T16:31:05.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 >> 172.21.15.111:0/1689021053 conn(0x7fb4000770a0 msgr2=0x7fb4000fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 shutdown_connections 2023-10-11T16:31:05.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.088+0000 7fb404c2c700 1 -- 172.21.15.111:0/1689021053 wait complete. 2023-10-11T16:31:05.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:05 smithi130 ceph-mon[131859]: pgmap v301: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:31:05.349 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:05 smithi111 ceph-mon[98527]: pgmap v301: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:31:05.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:05 smithi150 ceph-mon[132217]: pgmap v301: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:31:06.026 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:06.302 INFO:teuthology.orchestra.run.smithi111.stdout:335007449111 2023-10-11T16:31:06.303 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.11 2023-10-11T16:31:06.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 -- 172.21.15.111:0/251418905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73241055e0 msgr2=0x7f732410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:06.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 --2- 172.21.15.111:0/251418905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73241055e0 0x7f732410d160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f731c00b0d0 tx=0x7f731c00b3e0 comp rx=0 tx=0).stop 2023-10-11T16:31:06.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 -- 172.21.15.111:0/251418905 shutdown_connections 2023-10-11T16:31:06.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 --2- 172.21.15.111:0/251418905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73241055e0 0x7f732410d160 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:06.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 --2- 172.21.15.111:0/251418905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7324100540 0x7f7324100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:06.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 --2- 172.21.15.111:0/251418905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324100ee0 0x7f7324104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:06.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 -- 172.21.15.111:0/251418905 >> 172.21.15.111:0/251418905 conn(0x7f73240fbdc0 msgr2=0x7f73240fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:06.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 -- 172.21.15.111:0/251418905 shutdown_connections 2023-10-11T16:31:06.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.301+0000 7f732b856700 1 -- 172.21.15.111:0/251418905 wait complete. 2023-10-11T16:31:06.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.302+0000 7f732b856700 1 Processor -- start 2023-10-11T16:31:06.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.302+0000 7f732b856700 1 -- start start 2023-10-11T16:31:06.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f732b856700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7324100540 0x7f73241986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:06.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f732b856700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324100ee0 0x7f7324198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:06.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f732b856700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73241055e0 0x7f732419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:06.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f732b856700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732419d610 con 0x7f73241055e0 2023-10-11T16:31:06.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f732b856700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732419d780 con 0x7f7324100540 2023-10-11T16:31:06.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f732b856700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732419d940 con 0x7f7324100ee0 2023-10-11T16:31:06.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7329df3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73241055e0 0x7f732419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:06.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7328df1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324100ee0 0x7f7324198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7328df1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324100ee0 0x7f7324198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43052/0 (socket says 172.21.15.111:43052) 2023-10-11T16:31:06.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7328df1700 1 -- 172.21.15.111:0/1971389026 learned_addr learned my addr 172.21.15.111:0/1971389026 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f73295f2700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7324100540 0x7f73241986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:06.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7329df3700 1 -- 172.21.15.111:0/1971389026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324100ee0 msgr2=0x7f7324198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7329df3700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324100ee0 0x7f7324198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7329df3700 1 -- 172.21.15.111:0/1971389026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7324100540 msgr2=0x7f73241986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:06.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7329df3700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7324100540 0x7f73241986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:06.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7329df3700 1 -- 172.21.15.111:0/1971389026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c009d00 con 0x7f73241055e0 2023-10-11T16:31:06.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.303+0000 7f7328df1700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7324100ee0 0x7f7324198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:06.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.304+0000 7f7329df3700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73241055e0 0x7f732419d000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f731c009fd0 tx=0x7f731c0077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.304+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c00e040 con 0x7f73241055e0 2023-10-11T16:31:06.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.304+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f731c00bcb0 con 0x7f73241055e0 2023-10-11T16:31:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.304+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731c005840 con 0x7f73241055e0 2023-10-11T16:31:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.304+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7324110370 con 0x7f73241055e0 2023-10-11T16:31:06.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.304+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73241101e0 con 0x7f73241055e0 2023-10-11T16:31:06.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.305+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f732410bfd0 con 0x7f73241055e0 2023-10-11T16:31:06.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.307+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f731c004a50 con 0x7f73241055e0 2023-10-11T16:31:06.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.308+0000 7f731a7fc700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73100a1260 0x7f73100a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:06.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.308+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f731c0ce240 con 0x7f73241055e0 2023-10-11T16:31:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.308+0000 7f73295f2700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73100a1260 0x7f73100a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.309+0000 7f731a7fc700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] conn(0x7f73100ac270 0x7f73100ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:06.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.309+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 --> [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73100aedd0 con 0x7f73100ac270 2023-10-11T16:31:06.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.309+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f731c0ce5b0 con 0x7f73241055e0 2023-10-11T16:31:06.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.309+0000 7f7328df1700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] conn(0x7f73100ac270 0x7f73100ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:06.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.309+0000 7f73295f2700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73100a1260 0x7f73100a3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7320009c60 tx=0x7f7320009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:06.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.310+0000 7f7328df1700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] conn(0x7f73100ac270 0x7f73100ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:06.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.311+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 <== osd.7 v2:172.21.15.130:6816/2699897389 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f73100aedd0 con 0x7f73100ac270 2023-10-11T16:31:06.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.327+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 --> [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7324110500 con 0x7f73100ac270 2023-10-11T16:31:06.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f731a7fc700 1 -- 172.21.15.111:0/1971389026 <== osd.7 v2:172.21.15.130:6816/2699897389 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7324110500 con 0x7f73100ac270 2023-10-11T16:31:06.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 >> [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] conn(0x7f73100ac270 msgr2=0x7f73100ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:06.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] conn(0x7f73100ac270 0x7f73100ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:06.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73100a1260 msgr2=0x7f73100a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:06.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f73100a1260 0x7f73100a3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7320009c60 tx=0x7f7320009260 comp rx=0 tx=0).stop 2023-10-11T16:31:06.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73241055e0 msgr2=0x7f732419d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:06.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 --2- 172.21.15.111:0/1971389026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73241055e0 0x7f732419d000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f731c009fd0 tx=0x7f731c0077f0 comp rx=0 tx=0).stop 2023-10-11T16:31:06.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f7329df3700 1 -- 172.21.15.111:0/1971389026 reap_dead start 2023-10-11T16:31:06.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 shutdown_connections 2023-10-11T16:31:06.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 >> 172.21.15.111:0/1971389026 conn(0x7f73240fbdc0 msgr2=0x7f732410b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:06.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 shutdown_connections 2023-10-11T16:31:06.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:05.328+0000 7f732b856700 1 -- 172.21.15.111:0/1971389026 wait complete. 2023-10-11T16:31:07.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:07 smithi111 ceph-mon[98527]: pgmap v302: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:31:07.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.212+0000 7ff743e9f700 1 -- 172.21.15.111:0/3940769137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff73c100540 msgr2=0x7ff73c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:07.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.212+0000 7ff743e9f700 1 --2- 172.21.15.111:0/3940769137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff73c100540 0x7ff73c100910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff7380099c0 tx=0x7ff738009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:07.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.213+0000 7ff743e9f700 1 -- 172.21.15.111:0/3940769137 shutdown_connections 2023-10-11T16:31:07.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.213+0000 7ff743e9f700 1 --2- 172.21.15.111:0/3940769137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff73c1055e0 0x7ff73c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.213+0000 7ff743e9f700 1 --2- 172.21.15.111:0/3940769137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff73c100540 0x7ff73c100910 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.213+0000 7ff743e9f700 1 --2- 172.21.15.111:0/3940769137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c100ee0 0x7ff73c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.213+0000 7ff743e9f700 1 -- 172.21.15.111:0/3940769137 >> 172.21.15.111:0/3940769137 conn(0x7ff73c0fbdc0 msgr2=0x7ff73c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:07.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.213+0000 7ff743e9f700 1 -- 172.21.15.111:0/3940769137 shutdown_connections 2023-10-11T16:31:07.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.213+0000 7ff743e9f700 1 -- 172.21.15.111:0/3940769137 wait complete. 2023-10-11T16:31:07.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.214+0000 7ff743e9f700 1 Processor -- start 2023-10-11T16:31:07.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.214+0000 7ff743e9f700 1 -- start start 2023-10-11T16:31:07.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff743e9f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff73c100540 0x7ff73c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:07.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff743e9f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff73c100ee0 0x7ff73c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:07.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff743e9f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c1055e0 0x7ff73c0738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff743e9f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c1a4860 con 0x7ff73c100ee0 2023-10-11T16:31:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff743e9f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c1a49d0 con 0x7ff73c100540 2023-10-11T16:31:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff743e9f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c1a4b90 con 0x7ff73c1055e0 2023-10-11T16:31:07.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff741c3b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff73c100540 0x7ff73c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:07.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff741c3b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff73c100540 0x7ff73c072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:53128/0 (socket says 172.21.15.111:53128) 2023-10-11T16:31:07.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff741c3b700 1 -- 172.21.15.111:0/364837688 learned_addr learned my addr 172.21.15.111:0/364837688 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:07.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.215+0000 7ff74143a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff73c100ee0 0x7ff73c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:07.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff741c3b700 1 -- 172.21.15.111:0/364837688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c1055e0 msgr2=0x7ff73c0738b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-11T16:31:07.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff741c3b700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c1055e0 0x7ff73c0738b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:07.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff741c3b700 1 -- 172.21.15.111:0/364837688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff73c100ee0 msgr2=0x7ff73c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:07.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff741c3b700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff73c100ee0 0x7ff73c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:07.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff741c3b700 1 -- 172.21.15.111:0/364837688 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7380095f0 con 0x7ff73c100540 2023-10-11T16:31:07.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff741c3b700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff73c100540 0x7ff73c072df0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff73800b580 tx=0x7ff73800bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:07.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73801d070 con 0x7ff73c100540 2023-10-11T16:31:07.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73c110370 con 0x7ff73c100540 2023-10-11T16:31:07.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff738005050 con 0x7ff73c100540 2023-10-11T16:31:07.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.216+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738004ad0 con 0x7ff73c100540 2023-10-11T16:31:07.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.217+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73c1101e0 con 0x7ff73c100540 2023-10-11T16:31:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.218+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff73c0fd1e0 con 0x7ff73c100540 2023-10-11T16:31:07.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.220+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff7380051c0 con 0x7ff73c100540 2023-10-11T16:31:07.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.221+0000 7ff732ffd700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff7280a1260 0x7ff7280a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:07.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.221+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff7380cdce0 con 0x7ff73c100540 2023-10-11T16:31:07.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.221+0000 7ff74143a700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff7280a1260 0x7ff7280a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:07.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.221+0000 7ff732ffd700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] conn(0x7ff7280ac270 0x7ff7280ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:07.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.221+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 --> [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7280aedd0 con 0x7ff7280ac270 2023-10-11T16:31:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.221+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7380ce050 con 0x7ff73c100540 2023-10-11T16:31:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.222+0000 7ff74243c700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] conn(0x7ff7280ac270 0x7ff7280ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.223+0000 7ff74243c700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] conn(0x7ff7280ac270 0x7ff7280ae700 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:07.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.223+0000 7ff74143a700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff7280a1260 0x7ff7280a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff73c0745e0 tx=0x7ff72c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:07.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.224+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 <== osd.4 v2:172.21.15.130:6808/1737228093 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff7280aedd0 con 0x7ff7280ac270 2023-10-11T16:31:07.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.258+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 --> [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff73c110500 con 0x7ff7280ac270 2023-10-11T16:31:07.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.259+0000 7ff732ffd700 1 -- 172.21.15.111:0/364837688 <== osd.4 v2:172.21.15.130:6808/1737228093 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff73c110500 con 0x7ff7280ac270 2023-10-11T16:31:07.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.259+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 >> [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] conn(0x7ff7280ac270 msgr2=0x7ff7280ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:07.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.259+0000 7ff743e9f700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] conn(0x7ff7280ac270 0x7ff7280ae700 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:07.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff7280a1260 msgr2=0x7ff7280a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:07.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff743e9f700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff7280a1260 0x7ff7280a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff73c0745e0 tx=0x7ff72c009260 comp rx=0 tx=0).stop 2023-10-11T16:31:07.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff73c100540 msgr2=0x7ff73c072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:07.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff743e9f700 1 --2- 172.21.15.111:0/364837688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff73c100540 0x7ff73c072df0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff73800b580 tx=0x7ff73800bfd0 comp rx=0 tx=0).stop 2023-10-11T16:31:07.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff74243c700 1 -- 172.21.15.111:0/364837688 reap_dead start 2023-10-11T16:31:07.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 shutdown_connections 2023-10-11T16:31:07.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 >> 172.21.15.111:0/364837688 conn(0x7ff73c0fbdc0 msgr2=0x7ff73c0fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:07.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 shutdown_connections 2023-10-11T16:31:07.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.260+0000 7ff743e9f700 1 -- 172.21.15.111:0/364837688 wait complete. 2023-10-11T16:31:07.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:07 smithi130 ceph-mon[131859]: pgmap v302: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:31:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:07 smithi150 ceph-mon[132217]: pgmap v302: 129 pgs: 129 active+clean; 653 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:31:08.267 INFO:teuthology.orchestra.run.smithi111.stdout:141733920824 2023-10-11T16:31:08.268 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-10-11T16:31:08.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 -- 172.21.15.111:0/2795397942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5380ff0d0 msgr2=0x7fc5380ff4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 --2- 172.21.15.111:0/2795397942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5380ff0d0 0x7fc5380ff4a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc52c009990 tx=0x7fc52c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 -- 172.21.15.111:0/2795397942 shutdown_connections 2023-10-11T16:31:08.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 --2- 172.21.15.111:0/2795397942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc53810cc80 0x7fc53810f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 --2- 172.21.15.111:0/2795397942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5380ff0d0 0x7fc5380ff4a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 --2- 172.21.15.111:0/2795397942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5380ff9e0 0x7fc53810c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 -- 172.21.15.111:0/2795397942 >> 172.21.15.111:0/2795397942 conn(0x7fc5380770a0 msgr2=0x7fc5380774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:08.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 -- 172.21.15.111:0/2795397942 shutdown_connections 2023-10-11T16:31:08.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.410+0000 7fc53ef65700 1 -- 172.21.15.111:0/2795397942 wait complete. 2023-10-11T16:31:08.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.411+0000 7fc53ef65700 1 Processor -- start 2023-10-11T16:31:08.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.411+0000 7fc53ef65700 1 -- start start 2023-10-11T16:31:08.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.411+0000 7fc53ef65700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5380ff0d0 0x7fc53810c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.411+0000 7fc53ef65700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380ff9e0 0x7fc53810c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.411+0000 7fc53ef65700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc53810cc80 0x7fc5381093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.411+0000 7fc53ef65700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538109ba0 con 0x7fc5380ff9e0 2023-10-11T16:31:08.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.411+0000 7fc53ef65700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538109d10 con 0x7fc53810cc80 2023-10-11T16:31:08.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.411+0000 7fc53ef65700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538109ed0 con 0x7fc5380ff0d0 2023-10-11T16:31:08.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380ff9e0 0x7fc53810c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380ff9e0 0x7fc53810c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40462/0 (socket says 172.21.15.111:40462) 2023-10-11T16:31:08.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 -- 172.21.15.111:0/3527209913 learned_addr learned my addr 172.21.15.111:0/3527209913 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:08.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc53cd01700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5380ff0d0 0x7fc53810c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc53d502700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc53810cc80 0x7fc5381093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 -- 172.21.15.111:0/3527209913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5380ff0d0 msgr2=0x7fc53810c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5380ff0d0 0x7fc53810c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 -- 172.21.15.111:0/3527209913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc53810cc80 msgr2=0x7fc5381093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc53810cc80 0x7fc5381093d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 -- 172.21.15.111:0/3527209913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c0095f0 con 0x7fc5380ff9e0 2023-10-11T16:31:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.412+0000 7fc537fff700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380ff9e0 0x7fc53810c6c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc52400d950 tx=0x7fc52400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.413+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52400f890 con 0x7fc5380ff9e0 2023-10-11T16:31:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.413+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc524010460 con 0x7fc5380ff9e0 2023-10-11T16:31:08.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.413+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc538112560 con 0x7fc5380ff9e0 2023-10-11T16:31:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.413+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52400e610 con 0x7fc5380ff9e0 2023-10-11T16:31:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.413+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5381123d0 con 0x7fc5380ff9e0 2023-10-11T16:31:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.414+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc53810eae0 con 0x7fc5380ff9e0 2023-10-11T16:31:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.416+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fc5240108d0 con 0x7fc5380ff9e0 2023-10-11T16:31:08.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.417+0000 7fc535ffb700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5280a11e0 0x7fc5280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.417+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc5240ccbe0 con 0x7fc5380ff9e0 2023-10-11T16:31:08.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.418+0000 7fc53cd01700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5280a11e0 0x7fc5280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.418+0000 7fc535ffb700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] conn(0x7fc5280ac1f0 0x7fc5280ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.418+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 --> [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5280aed50 con 0x7fc5280ac1f0 2023-10-11T16:31:08.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.418+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5240ccff0 con 0x7fc5380ff9e0 2023-10-11T16:31:08.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.418+0000 7fc53d502700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] conn(0x7fc5280ac1f0 0x7fc5280ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.418+0000 7fc53cd01700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5280a11e0 0x7fc5280a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00b2e0 tx=0x7fc52c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:08.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.418+0000 7fc53d502700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] conn(0x7fc5280ac1f0 0x7fc5280ae680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:08.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.419+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 <== osd.9 v2:172.21.15.111:6826/3788746669 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc5280aed50 con 0x7fc5280ac1f0 2023-10-11T16:31:08.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.437+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 --> [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc5381126f0 con 0x7fc5280ac1f0 2023-10-11T16:31:08.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.437+0000 7fc535ffb700 1 -- 172.21.15.111:0/3527209913 <== osd.9 v2:172.21.15.111:6826/3788746669 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc5381126f0 con 0x7fc5280ac1f0 2023-10-11T16:31:08.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.437+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] conn(0x7fc5280ac1f0 msgr2=0x7fc5280ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.437+0000 7fc53ef65700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] conn(0x7fc5280ac1f0 0x7fc5280ae680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5280a11e0 msgr2=0x7fc5280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53ef65700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5280a11e0 0x7fc5280a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00b2e0 tx=0x7fc52c005fd0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380ff9e0 msgr2=0x7fc53810c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53ef65700 1 --2- 172.21.15.111:0/3527209913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5380ff9e0 0x7fc53810c6c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc52400d950 tx=0x7fc52400dc60 comp rx=0 tx=0).stop 2023-10-11T16:31:08.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53d502700 1 -- 172.21.15.111:0/3527209913 reap_dead start 2023-10-11T16:31:08.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 shutdown_connections 2023-10-11T16:31:08.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 >> 172.21.15.111:0/3527209913 conn(0x7fc5380770a0 msgr2=0x7fc53810e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:08.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 shutdown_connections 2023-10-11T16:31:08.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:07.438+0000 7fc53ef65700 1 -- 172.21.15.111:0/3527209913 wait complete. 2023-10-11T16:31:08.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.602+0000 7f54bbe73700 1 -- 172.21.15.111:0/2521568501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54b4100540 msgr2=0x7f54b4100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.602+0000 7f54bbe73700 1 --2- 172.21.15.111:0/2521568501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54b4100540 0x7f54b4100910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009990 tx=0x7f54b0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.602+0000 7f54bbe73700 1 -- 172.21.15.111:0/2521568501 shutdown_connections 2023-10-11T16:31:08.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.602+0000 7f54bbe73700 1 --2- 172.21.15.111:0/2521568501 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b41055e0 0x7f54b410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.602+0000 7f54bbe73700 1 --2- 172.21.15.111:0/2521568501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54b4100540 0x7f54b4100910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.602+0000 7f54bbe73700 1 --2- 172.21.15.111:0/2521568501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b4100ee0 0x7f54b4104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.602+0000 7f54bbe73700 1 -- 172.21.15.111:0/2521568501 >> 172.21.15.111:0/2521568501 conn(0x7f54b40fbdc0 msgr2=0x7f54b40fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.602+0000 7f54bbe73700 1 -- 172.21.15.111:0/2521568501 shutdown_connections 2023-10-11T16:31:08.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.603+0000 7f54bbe73700 1 -- 172.21.15.111:0/2521568501 wait complete. 2023-10-11T16:31:08.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.604+0000 7f54bbe73700 1 Processor -- start 2023-10-11T16:31:08.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.604+0000 7f54bbe73700 1 -- start start 2023-10-11T16:31:08.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54bbe73700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b4100540 0x7f54b4072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54bbe73700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54b4100ee0 0x7f54b4073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54bbe73700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b41055e0 0x7f54b40738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54bbe73700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41a4890 con 0x7f54b4100540 2023-10-11T16:31:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54bbe73700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41a4a00 con 0x7f54b4100ee0 2023-10-11T16:31:08.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54bbe73700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41a4bc0 con 0x7f54b41055e0 2023-10-11T16:31:08.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b9c0f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b4100540 0x7f54b4072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b940e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54b4100ee0 0x7f54b4073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b940e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54b4100ee0 0x7f54b4073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:53164/0 (socket says 172.21.15.111:53164) 2023-10-11T16:31:08.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b940e700 1 -- 172.21.15.111:0/742416699 learned_addr learned my addr 172.21.15.111:0/742416699 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:08.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54ba410700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b41055e0 0x7f54b40738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b9c0f700 1 -- 172.21.15.111:0/742416699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b41055e0 msgr2=0x7f54b40738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b9c0f700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b41055e0 0x7f54b40738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b9c0f700 1 -- 172.21.15.111:0/742416699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54b4100ee0 msgr2=0x7f54b4073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b9c0f700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54b4100ee0 0x7f54b4073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.605+0000 7f54b9c0f700 1 -- 172.21.15.111:0/742416699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b00095f0 con 0x7f54b4100540 2023-10-11T16:31:08.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.606+0000 7f54b9c0f700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b4100540 0x7f54b4072df0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f54b00059d0 tx=0x7f54b0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:08.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.606+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b001d070 con 0x7f54b4100540 2023-10-11T16:31:08.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.606+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b4110320 con 0x7f54b4100540 2023-10-11T16:31:08.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.606+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b4110190 con 0x7f54b4100540 2023-10-11T16:31:08.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.606+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54b0022470 con 0x7f54b4100540 2023-10-11T16:31:08.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.606+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0017650 con 0x7f54b4100540 2023-10-11T16:31:08.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.607+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54b41047e0 con 0x7f54b4100540 2023-10-11T16:31:08.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.609+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f54b0022870 con 0x7f54b4100540 2023-10-11T16:31:08.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.610+0000 7f54aaffd700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f54a00a1260 0x7f54a00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.611+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f54b00cdf10 con 0x7f54b4100540 2023-10-11T16:31:08.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.611+0000 7f54b940e700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f54a00a1260 0x7f54a00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.611+0000 7f54aaffd700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] conn(0x7f54a00ac270 0x7f54a00ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:08.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.611+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 --> [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54a00aedd0 con 0x7f54a00ac270 2023-10-11T16:31:08.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.611+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54b00d04f0 con 0x7f54b4100540 2023-10-11T16:31:08.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.611+0000 7f54ba410700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] conn(0x7f54a00ac270 0x7f54a00ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:08.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.611+0000 7f54b940e700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f54a00a1260 0x7f54a00a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f54b40745e0 tx=0x7f54a4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:08.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.612+0000 7f54ba410700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] conn(0x7f54a00ac270 0x7f54a00ae700 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:08.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.613+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 <== osd.2 v2:172.21.15.150:6800/1993692554 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f54a00aedd0 con 0x7f54a00ac270 2023-10-11T16:31:08.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.631+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 --> [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54b41104b0 con 0x7f54a00ac270 2023-10-11T16:31:08.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.632+0000 7f54aaffd700 1 -- 172.21.15.111:0/742416699 <== osd.2 v2:172.21.15.150:6800/1993692554 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f54b41104b0 con 0x7f54a00ac270 2023-10-11T16:31:08.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 >> [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] conn(0x7f54a00ac270 msgr2=0x7f54a00ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] conn(0x7f54a00ac270 0x7f54a00ae700 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:08.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f54a00a1260 msgr2=0x7f54a00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f54a00a1260 0x7f54a00a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f54b40745e0 tx=0x7f54a4009260 comp rx=0 tx=0).stop 2023-10-11T16:31:08.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b4100540 msgr2=0x7f54b4072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:08.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 --2- 172.21.15.111:0/742416699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b4100540 0x7f54b4072df0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f54b00059d0 tx=0x7f54b0005720 comp rx=0 tx=0).stop 2023-10-11T16:31:08.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54ba410700 1 -- 172.21.15.111:0/742416699 reap_dead start 2023-10-11T16:31:08.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 shutdown_connections 2023-10-11T16:31:08.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 >> 172.21.15.111:0/742416699 conn(0x7f54b40fbdc0 msgr2=0x7f54b4103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:08.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 shutdown_connections 2023-10-11T16:31:08.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:08.633+0000 7f54bbe73700 1 -- 172.21.15.111:0/742416699 wait complete. 2023-10-11T16:31:09.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.271+0000 7f4943bab700 1 -- 172.21.15.111:0/1320284599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493c100540 msgr2=0x7f493c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.271+0000 7f4943bab700 1 --2- 172.21.15.111:0/1320284599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493c100540 0x7f493c100910 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4938009920 tx=0x7f4938009c30 comp rx=0 tx=0).stop 2023-10-11T16:31:09.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.271+0000 7f4943bab700 1 -- 172.21.15.111:0/1320284599 shutdown_connections 2023-10-11T16:31:09.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.271+0000 7f4943bab700 1 --2- 172.21.15.111:0/1320284599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c1055e0 0x7f493c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.271+0000 7f4943bab700 1 --2- 172.21.15.111:0/1320284599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493c100540 0x7f493c100910 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.271+0000 7f4943bab700 1 --2- 172.21.15.111:0/1320284599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f493c100ee0 0x7f493c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.271+0000 7f4943bab700 1 -- 172.21.15.111:0/1320284599 >> 172.21.15.111:0/1320284599 conn(0x7f493c0fbdc0 msgr2=0x7f493c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:09.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.271+0000 7f4943bab700 1 -- 172.21.15.111:0/1320284599 shutdown_connections 2023-10-11T16:31:09.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.272+0000 7f4943bab700 1 -- 172.21.15.111:0/1320284599 wait complete. 2023-10-11T16:31:09.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.273+0000 7f4943bab700 1 Processor -- start 2023-10-11T16:31:09.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.273+0000 7f4943bab700 1 -- start start 2023-10-11T16:31:09.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.273+0000 7f4943bab700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c100540 0x7f493c19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.273+0000 7f4943bab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493c100ee0 0x7f493c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.273+0000 7f4943bab700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f493c1055e0 0x7f493c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4943bab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c198c70 con 0x7f493c100ee0 2023-10-11T16:31:09.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4943bab700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c198de0 con 0x7f493c1055e0 2023-10-11T16:31:09.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4943bab700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c198fa0 con 0x7f493c100540 2023-10-11T16:31:09.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941146700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493c100ee0 0x7f493c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941947700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c100540 0x7f493c19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941146700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493c100ee0 0x7f493c19ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40492/0 (socket says 172.21.15.111:40492) 2023-10-11T16:31:09.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941146700 1 -- 172.21.15.111:0/511231546 learned_addr learned my addr 172.21.15.111:0/511231546 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:09.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4942148700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f493c1055e0 0x7f493c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941947700 1 -- 172.21.15.111:0/511231546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f493c1055e0 msgr2=0x7f493c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941947700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f493c1055e0 0x7f493c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941947700 1 -- 172.21.15.111:0/511231546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493c100ee0 msgr2=0x7f493c19ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941947700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f493c100ee0 0x7f493c19ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.274+0000 7f4941947700 1 -- 172.21.15.111:0/511231546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0095f0 con 0x7f493c100540 2023-10-11T16:31:09.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.275+0000 7f4941947700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c100540 0x7f493c19e590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4938009230 tx=0x7f493800fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:09.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.275+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493801c070 con 0x7f493c100540 2023-10-11T16:31:09.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.275+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4938009520 con 0x7f493c100540 2023-10-11T16:31:09.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.275+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f493800baa0 con 0x7f493c100540 2023-10-11T16:31:09.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.275+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49380038e0 con 0x7f493c100540 2023-10-11T16:31:09.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.275+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c110370 con 0x7f493c100540 2023-10-11T16:31:09.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.276+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f493c0fd2d0 con 0x7f493c100540 2023-10-11T16:31:09.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.278+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f4938020030 con 0x7f493c100540 2023-10-11T16:31:09.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.280+0000 7f4932ffd700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49280a1260 0x7f49280a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.280+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f49380ce0b0 con 0x7f493c100540 2023-10-11T16:31:09.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.280+0000 7f4941146700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49280a1260 0x7f49280a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.281+0000 7f4932ffd700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] conn(0x7f49280ac140 0x7f49280ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.281+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 --> [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49280aeca0 con 0x7f49280ac140 2023-10-11T16:31:09.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.281+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49380ce420 con 0x7f493c100540 2023-10-11T16:31:09.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.281+0000 7f4941146700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49280a1260 0x7f49280a3710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f493c199de0 tx=0x7f492c009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:09.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.281+0000 7f4942148700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] conn(0x7f49280ac140 0x7f49280ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.282+0000 7f4942148700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] conn(0x7f49280ac140 0x7f49280ae5d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:09.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.283+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 <== osd.5 v2:172.21.15.150:6808/2672847721 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f49280aeca0 con 0x7f49280ac140 2023-10-11T16:31:09.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.300+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 --> [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f493c1101e0 con 0x7f49280ac140 2023-10-11T16:31:09.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.300+0000 7f4932ffd700 1 -- 172.21.15.111:0/511231546 <== osd.5 v2:172.21.15.150:6808/2672847721 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f493c1101e0 con 0x7f49280ac140 2023-10-11T16:31:09.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 >> [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] conn(0x7f49280ac140 msgr2=0x7f49280ae5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] conn(0x7f49280ac140 0x7f49280ae5d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49280a1260 msgr2=0x7f49280a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49280a1260 0x7f49280a3710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f493c199de0 tx=0x7f492c009310 comp rx=0 tx=0).stop 2023-10-11T16:31:09.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c100540 msgr2=0x7f493c19e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 --2- 172.21.15.111:0/511231546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f493c100540 0x7f493c19e590 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4938009230 tx=0x7f493800fa20 comp rx=0 tx=0).stop 2023-10-11T16:31:09.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4942148700 1 -- 172.21.15.111:0/511231546 reap_dead start 2023-10-11T16:31:09.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 shutdown_connections 2023-10-11T16:31:09.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 >> 172.21.15.111:0/511231546 conn(0x7f493c0fbdc0 msgr2=0x7f493c0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:09.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 shutdown_connections 2023-10-11T16:31:09.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.301+0000 7f4943bab700 1 -- 172.21.15.111:0/511231546 wait complete. 2023-10-11T16:31:09.363 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:09 smithi111 ceph-mon[98527]: pgmap v303: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-10-11T16:31:09.387 INFO:teuthology.orchestra.run.smithi111.stdout:219043332137 2023-10-11T16:31:09.388 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-10-11T16:31:09.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:09 smithi150 ceph-mon[132217]: pgmap v303: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-10-11T16:31:09.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:09 smithi130 ceph-mon[131859]: pgmap v303: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-10-11T16:31:09.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 -- 172.21.15.111:0/246928648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 msgr2=0x7ff818101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 --2- 172.21.15.111:0/246928648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 0x7ff818101910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff8080098f0 tx=0x7ff808009c00 comp rx=0 tx=0).stop 2023-10-11T16:31:09.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 -- 172.21.15.111:0/246928648 shutdown_connections 2023-10-11T16:31:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 --2- 172.21.15.111:0/246928648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff81810a890 0x7ff81810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 --2- 172.21.15.111:0/246928648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 0x7ff818101910 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 --2- 172.21.15.111:0/246928648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff818101e50 0x7ff81810a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 -- 172.21.15.111:0/246928648 >> 172.21.15.111:0/246928648 conn(0x7ff8180770a0 msgr2=0x7ff8180774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:09.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 -- 172.21.15.111:0/246928648 shutdown_connections 2023-10-11T16:31:09.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.855+0000 7ff81ca44700 1 -- 172.21.15.111:0/246928648 wait complete. 2023-10-11T16:31:09.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.856+0000 7ff81ca44700 1 Processor -- start 2023-10-11T16:31:09.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.856+0000 7ff81ca44700 1 -- start start 2023-10-11T16:31:09.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81ca44700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 0x7ff8180ff1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81ca44700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff818101e50 0x7ff8180ff6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81ca44700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff81810a890 0x7ff8180ffc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81ca44700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8181a4860 con 0x7ff818101e50 2023-10-11T16:31:09.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81ca44700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8181a49d0 con 0x7ff81810a890 2023-10-11T16:31:09.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81ca44700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8181a4b90 con 0x7ff818101540 2023-10-11T16:31:09.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 0x7ff8180ff1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 0x7ff8180ff1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43144/0 (socket says 172.21.15.111:43144) 2023-10-11T16:31:09.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81659c700 1 -- 172.21.15.111:0/2405210264 learned_addr learned my addr 172.21.15.111:0/2405210264 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:09.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff815d9b700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff818101e50 0x7ff8180ff6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff816d9d700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff81810a890 0x7ff8180ffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81659c700 1 -- 172.21.15.111:0/2405210264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff81810a890 msgr2=0x7ff8180ffc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81659c700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff81810a890 0x7ff8180ffc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81659c700 1 -- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff818101e50 msgr2=0x7ff8180ff6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81659c700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff818101e50 0x7ff8180ff6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.857+0000 7ff81659c700 1 -- 172.21.15.111:0/2405210264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0095f0 con 0x7ff818101540 2023-10-11T16:31:09.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.858+0000 7ff81659c700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 0x7ff8180ff1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff808009230 tx=0x7ff80800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:09.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.858+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80801c070 con 0x7ff818101540 2023-10-11T16:31:09.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.858+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff80800bba0 con 0x7ff818101540 2023-10-11T16:31:09.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.858+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8080038e0 con 0x7ff818101540 2023-10-11T16:31:09.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.859+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff808009520 con 0x7ff818101540 2023-10-11T16:31:09.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.859+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff818110310 con 0x7ff818101540 2023-10-11T16:31:09.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.860+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff8180fc6e0 con 0x7ff818101540 2023-10-11T16:31:09.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.862+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff808020030 con 0x7ff818101540 2023-10-11T16:31:09.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.864+0000 7ff8077fe700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8000a1260 0x7ff8000a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.864+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff8080cde90 con 0x7ff818101540 2023-10-11T16:31:09.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.864+0000 7ff815d9b700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8000a1260 0x7ff8000a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.864+0000 7ff8077fe700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] conn(0x7ff8000ac270 0x7ff8000ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:09.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.864+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 --> [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8000aedd0 con 0x7ff8000ac270 2023-10-11T16:31:09.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.864+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8080ce200 con 0x7ff818101540 2023-10-11T16:31:09.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.864+0000 7ff816d9d700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] conn(0x7ff8000ac270 0x7ff8000ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:09.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.865+0000 7ff815d9b700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8000a1260 0x7ff8000a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff818100990 tx=0x7ff80c009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:09.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.865+0000 7ff816d9d700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] conn(0x7ff8000ac270 0x7ff8000ae700 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:09.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.866+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 <== osd.6 v2:172.21.15.111:6818/2248469058 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff8000aedd0 con 0x7ff8000ac270 2023-10-11T16:31:09.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.884+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 --> [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff818110180 con 0x7ff8000ac270 2023-10-11T16:31:09.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.884+0000 7ff8077fe700 1 -- 172.21.15.111:0/2405210264 <== osd.6 v2:172.21.15.111:6818/2248469058 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff818110180 con 0x7ff8000ac270 2023-10-11T16:31:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] conn(0x7ff8000ac270 msgr2=0x7ff8000ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] conn(0x7ff8000ac270 0x7ff8000ae700 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:09.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8000a1260 msgr2=0x7ff8000a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8000a1260 0x7ff8000a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff818100990 tx=0x7ff80c009310 comp rx=0 tx=0).stop 2023-10-11T16:31:09.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 msgr2=0x7ff8180ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:09.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 --2- 172.21.15.111:0/2405210264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff818101540 0x7ff8180ff1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff808009230 tx=0x7ff80800fa00 comp rx=0 tx=0).stop 2023-10-11T16:31:09.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff816d9d700 1 -- 172.21.15.111:0/2405210264 reap_dead start 2023-10-11T16:31:09.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 shutdown_connections 2023-10-11T16:31:09.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 >> 172.21.15.111:0/2405210264 conn(0x7ff8180770a0 msgr2=0x7ff8180fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:09.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 shutdown_connections 2023-10-11T16:31:09.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:09.885+0000 7ff81ca44700 1 -- 172.21.15.111:0/2405210264 wait complete. 2023-10-11T16:31:10.371 INFO:teuthology.orchestra.run.smithi111.stdout:283467841567 2023-10-11T16:31:10.371 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.9 2023-10-11T16:31:11.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.066+0000 7f5f47791700 1 -- 172.21.15.111:0/2183437751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f401055e0 msgr2=0x7f5f4010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.066+0000 7f5f47791700 1 --2- 172.21.15.111:0/2183437751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f401055e0 0x7f5f4010d160 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f3800b0d0 tx=0x7f5f3800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.066+0000 7f5f47791700 1 -- 172.21.15.111:0/2183437751 shutdown_connections 2023-10-11T16:31:11.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.066+0000 7f5f47791700 1 --2- 172.21.15.111:0/2183437751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f401055e0 0x7f5f4010d160 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.066+0000 7f5f47791700 1 --2- 172.21.15.111:0/2183437751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f40100540 0x7f5f40100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.066+0000 7f5f47791700 1 --2- 172.21.15.111:0/2183437751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f40100ee0 0x7f5f40104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.066+0000 7f5f47791700 1 -- 172.21.15.111:0/2183437751 >> 172.21.15.111:0/2183437751 conn(0x7f5f400fbdc0 msgr2=0x7f5f400fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:11.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.066+0000 7f5f47791700 1 -- 172.21.15.111:0/2183437751 shutdown_connections 2023-10-11T16:31:11.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.067+0000 7f5f47791700 1 -- 172.21.15.111:0/2183437751 wait complete. 2023-10-11T16:31:11.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.068+0000 7f5f47791700 1 Processor -- start 2023-10-11T16:31:11.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.069+0000 7f5f47791700 1 -- start start 2023-10-11T16:31:11.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.069+0000 7f5f47791700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f40100540 0x7f5f401985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.069+0000 7f5f47791700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f40100ee0 0x7f5f40198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.069+0000 7f5f47791700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f401055e0 0x7f5f4019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.069+0000 7f5f47791700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4019d500 con 0x7f5f40100ee0 2023-10-11T16:31:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.069+0000 7f5f47791700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4019d670 con 0x7f5f40100540 2023-10-11T16:31:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.069+0000 7f5f47791700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4019d830 con 0x7f5f401055e0 2023-10-11T16:31:11.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.069+0000 7f5f4552d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f40100540 0x7f5f401985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f4552d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f40100540 0x7f5f401985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:53236/0 (socket says 172.21.15.111:53236) 2023-10-11T16:31:11.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f4552d700 1 -- 172.21.15.111:0/3824884236 learned_addr learned my addr 172.21.15.111:0/3824884236 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:11.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f44d2c700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f40100ee0 0x7f5f40198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f45d2e700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f401055e0 0x7f5f4019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f4552d700 1 -- 172.21.15.111:0/3824884236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f401055e0 msgr2=0x7f5f4019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f4552d700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f401055e0 0x7f5f4019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f4552d700 1 -- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f40100ee0 msgr2=0x7f5f40198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f4552d700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f40100ee0 0x7f5f40198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f4552d700 1 -- 172.21.15.111:0/3824884236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f38009d00 con 0x7f5f40100540 2023-10-11T16:31:11.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.070+0000 7f5f44d2c700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f40100ee0 0x7f5f40198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:11.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.071+0000 7f5f4552d700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f40100540 0x7f5f401985e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00e9a0 tx=0x7f5f3c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.071+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c009790 con 0x7f5f40100540 2023-10-11T16:31:11.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.071+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40110370 con 0x7f5f40100540 2023-10-11T16:31:11.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.071+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f3c010430 con 0x7f5f40100540 2023-10-11T16:31:11.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.071+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c018680 con 0x7f5f40100540 2023-10-11T16:31:11.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.071+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f401101e0 con 0x7f5f40100540 2023-10-11T16:31:11.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.072+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f400fda50 con 0x7f5f40100540 2023-10-11T16:31:11.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.075+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f5f3c0105a0 con 0x7f5f40100540 2023-10-11T16:31:11.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.076+0000 7f5f367fc700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f2c0a11e0 0x7f5f2c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.077+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5f3c014070 con 0x7f5f40100540 2023-10-11T16:31:11.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.077+0000 7f5f44d2c700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f2c0a11e0 0x7f5f2c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.077+0000 7f5f367fc700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] conn(0x7f5f2c0ac1f0 0x7f5f2c0ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.077+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 --> [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f2c0aed50 con 0x7f5f2c0ac1f0 2023-10-11T16:31:11.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.077+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f3c0cd130 con 0x7f5f40100540 2023-10-11T16:31:11.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.077+0000 7f5f45d2e700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] conn(0x7f5f2c0ac1f0 0x7f5f2c0ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.078+0000 7f5f44d2c700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f2c0a11e0 0x7f5f2c0a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5f40199b60 tx=0x7f5f30009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.078+0000 7f5f45d2e700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] conn(0x7f5f2c0ac1f0 0x7f5f2c0ae680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.079+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 <== osd.3 v2:172.21.15.111:6810/4229058554 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5f2c0aed50 con 0x7f5f2c0ac1f0 2023-10-11T16:31:11.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.108+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 --> [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f40110500 con 0x7f5f2c0ac1f0 2023-10-11T16:31:11.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.109+0000 7f5f367fc700 1 -- 172.21.15.111:0/3824884236 <== osd.3 v2:172.21.15.111:6810/4229058554 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5f40110500 con 0x7f5f2c0ac1f0 2023-10-11T16:31:11.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.109+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] conn(0x7f5f2c0ac1f0 msgr2=0x7f5f2c0ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.109+0000 7f5f47791700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] conn(0x7f5f2c0ac1f0 0x7f5f2c0ae680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.109+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f2c0a11e0 msgr2=0x7f5f2c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.109+0000 7f5f47791700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f2c0a11e0 0x7f5f2c0a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5f40199b60 tx=0x7f5f30009260 comp rx=0 tx=0).stop 2023-10-11T16:31:11.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.110+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f40100540 msgr2=0x7f5f401985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.110+0000 7f5f47791700 1 --2- 172.21.15.111:0/3824884236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f40100540 0x7f5f401985e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00e9a0 tx=0x7f5f3c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.110+0000 7f5f45d2e700 1 -- 172.21.15.111:0/3824884236 reap_dead start 2023-10-11T16:31:11.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.110+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 shutdown_connections 2023-10-11T16:31:11.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.110+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 >> 172.21.15.111:0/3824884236 conn(0x7f5f400fbdc0 msgr2=0x7f5f4010b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:11.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.110+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 shutdown_connections 2023-10-11T16:31:11.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.110+0000 7f5f47791700 1 -- 172.21.15.111:0/3824884236 wait complete. 2023-10-11T16:31:11.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:11 smithi130 ceph-mon[131859]: pgmap v304: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:11.350 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:11 smithi111 ceph-mon[98527]: pgmap v304: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:11.449 INFO:teuthology.orchestra.run.smithi111.stdout:163208757299 2023-10-11T16:31:11.449 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-10-11T16:31:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:11 smithi150 ceph-mon[132217]: pgmap v304: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 -- 172.21.15.111:0/3019937378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd58100ee0 msgr2=0x7fdd58104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 --2- 172.21.15.111:0/3019937378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd58100ee0 0x7fdd58104eb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdd480099c0 tx=0x7fdd48009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 -- 172.21.15.111:0/3019937378 shutdown_connections 2023-10-11T16:31:11.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 --2- 172.21.15.111:0/3019937378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd581055e0 0x7fdd5810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 --2- 172.21.15.111:0/3019937378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd58100540 0x7fdd58100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 --2- 172.21.15.111:0/3019937378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd58100ee0 0x7fdd58104eb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 -- 172.21.15.111:0/3019937378 >> 172.21.15.111:0/3019937378 conn(0x7fdd580fbdc0 msgr2=0x7fdd580fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:11.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 -- 172.21.15.111:0/3019937378 shutdown_connections 2023-10-11T16:31:11.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.456+0000 7fdd6007e700 1 -- 172.21.15.111:0/3019937378 wait complete. 2023-10-11T16:31:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd6007e700 1 Processor -- start 2023-10-11T16:31:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd6007e700 1 -- start start 2023-10-11T16:31:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd6007e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd58100540 0x7fdd581985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd6007e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd58100ee0 0x7fdd58198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd6007e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd581055e0 0x7fdd5819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd6007e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5819d580 con 0x7fdd581055e0 2023-10-11T16:31:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd6007e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5819d6f0 con 0x7fdd58100ee0 2023-10-11T16:31:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd6007e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5819d8b0 con 0x7fdd58100540 2023-10-11T16:31:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.458+0000 7fdd5e61b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd581055e0 0x7fdd5819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5de1a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd58100540 0x7fdd581985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5e61b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd581055e0 0x7fdd5819cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40568/0 (socket says 172.21.15.111:40568) 2023-10-11T16:31:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5e61b700 1 -- 172.21.15.111:0/479482834 learned_addr learned my addr 172.21.15.111:0/479482834 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5d619700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd58100ee0 0x7fdd58198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5e61b700 1 -- 172.21.15.111:0/479482834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd58100540 msgr2=0x7fdd581985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5e61b700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd58100540 0x7fdd581985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5e61b700 1 -- 172.21.15.111:0/479482834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd58100ee0 msgr2=0x7fdd58198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5e61b700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd58100ee0 0x7fdd58198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5e61b700 1 -- 172.21.15.111:0/479482834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480095f0 con 0x7fdd581055e0 2023-10-11T16:31:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd5e61b700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd581055e0 0x7fdd5819cf70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000ef30 tx=0x7fdd5000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50018070 con 0x7fdd581055e0 2023-10-11T16:31:11.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd5000f040 con 0x7fdd581055e0 2023-10-11T16:31:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.459+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5000cd00 con 0x7fdd581055e0 2023-10-11T16:31:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.460+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd58110360 con 0x7fdd581055e0 2023-10-11T16:31:11.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.460+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd581101d0 con 0x7fdd581055e0 2023-10-11T16:31:11.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.461+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdd58074070 con 0x7fdd581055e0 2023-10-11T16:31:11.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.463+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fdd50003a00 con 0x7fdd581055e0 2023-10-11T16:31:11.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.464+0000 7fdd4effd700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd440a1260 0x7fdd440a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.464+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdd500cc860 con 0x7fdd581055e0 2023-10-11T16:31:11.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.465+0000 7fdd5de1a700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd440a1260 0x7fdd440a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.465+0000 7fdd4effd700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] conn(0x7fdd440ac270 0x7fdd440ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.465+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 --> [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd440aedd0 con 0x7fdd440ac270 2023-10-11T16:31:11.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.465+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd500cee40 con 0x7fdd581055e0 2023-10-11T16:31:11.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.465+0000 7fdd5de1a700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd440a1260 0x7fdd440a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009c60 tx=0x7fdd54009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.465+0000 7fdd5d619700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] conn(0x7fdd440ac270 0x7fdd440ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.466+0000 7fdd5d619700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] conn(0x7fdd440ac270 0x7fdd440ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.467+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 <== osd.1 v2:172.21.15.130:6800/1526012165 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fdd440aedd0 con 0x7fdd440ac270 2023-10-11T16:31:11.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.483+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 --> [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdd581104f0 con 0x7fdd440ac270 2023-10-11T16:31:11.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.484+0000 7fdd4effd700 1 -- 172.21.15.111:0/479482834 <== osd.1 v2:172.21.15.130:6800/1526012165 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdd581104f0 con 0x7fdd440ac270 2023-10-11T16:31:11.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.484+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 >> [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] conn(0x7fdd440ac270 msgr2=0x7fdd440ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.484+0000 7fdd6007e700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] conn(0x7fdd440ac270 0x7fdd440ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.484+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd440a1260 msgr2=0x7fdd440a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.484+0000 7fdd6007e700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd440a1260 0x7fdd440a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009c60 tx=0x7fdd54009260 comp rx=0 tx=0).stop 2023-10-11T16:31:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.485+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd581055e0 msgr2=0x7fdd5819cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.485+0000 7fdd6007e700 1 --2- 172.21.15.111:0/479482834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd581055e0 0x7fdd5819cf70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000ef30 tx=0x7fdd5000c3a0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.485+0000 7fdd5e61b700 1 -- 172.21.15.111:0/479482834 reap_dead start 2023-10-11T16:31:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.485+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 shutdown_connections 2023-10-11T16:31:11.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.485+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 >> 172.21.15.111:0/479482834 conn(0x7fdd580fbdc0 msgr2=0x7fdd580fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.485+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 shutdown_connections 2023-10-11T16:31:11.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.485+0000 7fdd6007e700 1 -- 172.21.15.111:0/479482834 wait complete. 2023-10-11T16:31:11.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 -- 172.21.15.111:0/2654026645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda801055e0 msgr2=0x7fda8010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 --2- 172.21.15.111:0/2654026645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda801055e0 0x7fda8010d160 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fda7800b0d0 tx=0x7fda7800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 -- 172.21.15.111:0/2654026645 shutdown_connections 2023-10-11T16:31:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 --2- 172.21.15.111:0/2654026645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda801055e0 0x7fda8010d160 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 --2- 172.21.15.111:0/2654026645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda80100540 0x7fda80100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 --2- 172.21.15.111:0/2654026645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda80100ee0 0x7fda80104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 -- 172.21.15.111:0/2654026645 >> 172.21.15.111:0/2654026645 conn(0x7fda800fbdc0 msgr2=0x7fda800fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 -- 172.21.15.111:0/2654026645 shutdown_connections 2023-10-11T16:31:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.747+0000 7fda8546f700 1 -- 172.21.15.111:0/2654026645 wait complete. 2023-10-11T16:31:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.748+0000 7fda8546f700 1 Processor -- start 2023-10-11T16:31:11.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.748+0000 7fda8546f700 1 -- start start 2023-10-11T16:31:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.748+0000 7fda8546f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda80100540 0x7fda8019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.748+0000 7fda8546f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda80100ee0 0x7fda8019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.748+0000 7fda8546f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda801055e0 0x7fda801985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.748+0000 7fda8546f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80198c20 con 0x7fda80100540 2023-10-11T16:31:11.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.748+0000 7fda8546f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80198d90 con 0x7fda80100ee0 2023-10-11T16:31:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.748+0000 7fda8546f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80198f50 con 0x7fda801055e0 2023-10-11T16:31:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda80100540 0x7fda8019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda80100540 0x7fda8019e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40580/0 (socket says 172.21.15.111:40580) 2023-10-11T16:31:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 -- 172.21.15.111:0/1976094733 learned_addr learned my addr 172.21.15.111:0/1976094733 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:11.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7e7fc700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda80100ee0 0x7fda8019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7f7fe700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda801055e0 0x7fda801985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 -- 172.21.15.111:0/1976094733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda801055e0 msgr2=0x7fda801985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda801055e0 0x7fda801985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 -- 172.21.15.111:0/1976094733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda80100ee0 msgr2=0x7fda8019ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda80100ee0 0x7fda8019ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 -- 172.21.15.111:0/1976094733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda78009d00 con 0x7fda80100540 2023-10-11T16:31:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.749+0000 7fda7effd700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda80100540 0x7fda8019e530 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fda7000e9a0 tx=0x7fda7000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.750+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70009790 con 0x7fda80100540 2023-10-11T16:31:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.750+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda70010430 con 0x7fda80100540 2023-10-11T16:31:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.750+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda70018680 con 0x7fda80100540 2023-10-11T16:31:11.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.750+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda80110370 con 0x7fda80100540 2023-10-11T16:31:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.750+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda801101e0 con 0x7fda80100540 2023-10-11T16:31:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.751+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda8010c030 con 0x7fda80100540 2023-10-11T16:31:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.753+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fda700105a0 con 0x7fda80100540 2023-10-11T16:31:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.755+0000 7fda67fff700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fda680a1260 0x7fda680a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.755+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fda70014070 con 0x7fda80100540 2023-10-11T16:31:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.755+0000 7fda7e7fc700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fda680a1260 0x7fda680a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.756+0000 7fda67fff700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] conn(0x7fda680ac270 0x7fda680ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.756+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 --> [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda680aedd0 con 0x7fda680ac270 2023-10-11T16:31:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.756+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda700cdc70 con 0x7fda80100540 2023-10-11T16:31:11.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.756+0000 7fda7e7fc700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fda680a1260 0x7fda680a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fda800fc460 tx=0x7fda740073d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.756+0000 7fda7f7fe700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] conn(0x7fda680ac270 0x7fda680ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.756+0000 7fda7f7fe700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] conn(0x7fda680ac270 0x7fda680ae700 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.757+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 <== osd.8 v2:172.21.15.150:6816/410436517 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fda680aedd0 con 0x7fda680ac270 2023-10-11T16:31:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.774+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 --> [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fda80110500 con 0x7fda680ac270 2023-10-11T16:31:11.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.775+0000 7fda67fff700 1 -- 172.21.15.111:0/1976094733 <== osd.8 v2:172.21.15.150:6816/410436517 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fda80110500 con 0x7fda680ac270 2023-10-11T16:31:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.775+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 >> [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] conn(0x7fda680ac270 msgr2=0x7fda680ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.775+0000 7fda8546f700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] conn(0x7fda680ac270 0x7fda680ae700 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.775+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fda680a1260 msgr2=0x7fda680a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.775+0000 7fda8546f700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fda680a1260 0x7fda680a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fda800fc460 tx=0x7fda740073d0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.775+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda80100540 msgr2=0x7fda8019e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.775+0000 7fda8546f700 1 --2- 172.21.15.111:0/1976094733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fda80100540 0x7fda8019e530 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fda7000e9a0 tx=0x7fda7000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.775+0000 7fda7f7fe700 1 -- 172.21.15.111:0/1976094733 reap_dead start 2023-10-11T16:31:11.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.776+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 shutdown_connections 2023-10-11T16:31:11.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.776+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 >> 172.21.15.111:0/1976094733 conn(0x7fda800fbdc0 msgr2=0x7fda8010b660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:11.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.776+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 shutdown_connections 2023-10-11T16:31:11.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:11.776+0000 7fda8546f700 1 -- 172.21.15.111:0/1976094733 wait complete. 2023-10-11T16:31:12.125 INFO:teuthology.orchestra.run.smithi111.stdout:77309411394 2023-10-11T16:31:12.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-10-11T16:31:12.471 INFO:teuthology.orchestra.run.smithi111.stdout:249108103206 2023-10-11T16:31:12.472 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.8 2023-10-11T16:31:12.728 INFO:teuthology.orchestra.run.smithi111.stdout:107374182460 2023-10-11T16:31:12.729 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-10-11T16:31:13.340 INFO:teuthology.orchestra.run.smithi111.stdout:197568495662 2023-10-11T16:31:13.341 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-10-11T16:31:13.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:13 smithi111 ceph-mon[98527]: pgmap v305: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2023-10-11T16:31:13.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:13 smithi150 ceph-mon[132217]: pgmap v305: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2023-10-11T16:31:13.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:13 smithi130 ceph-mon[131859]: pgmap v305: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2023-10-11T16:31:13.614 INFO:teuthology.orchestra.run.smithi111.stdout:55834574917 2023-10-11T16:31:13.615 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-10-11T16:31:13.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 -- 172.21.15.111:0/960572181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5578100ee0 msgr2=0x7f5578104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 --2- 172.21.15.111:0/960572181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5578100ee0 0x7f5578104eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f556c009990 tx=0x7f556c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 -- 172.21.15.111:0/960572181 shutdown_connections 2023-10-11T16:31:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 --2- 172.21.15.111:0/960572181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55781055e0 0x7f557810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 --2- 172.21.15.111:0/960572181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5578100540 0x7f5578100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 --2- 172.21.15.111:0/960572181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5578100ee0 0x7f5578104eb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 -- 172.21.15.111:0/960572181 >> 172.21.15.111:0/960572181 conn(0x7f55780fbdc0 msgr2=0x7f55780fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:13.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 -- 172.21.15.111:0/960572181 shutdown_connections 2023-10-11T16:31:13.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.878+0000 7f557e89e700 1 -- 172.21.15.111:0/960572181 wait complete. 2023-10-11T16:31:13.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.879+0000 7f557e89e700 1 Processor -- start 2023-10-11T16:31:13.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f557e89e700 1 -- start start 2023-10-11T16:31:13.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f557e89e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5578100540 0x7f55781985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f557e89e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5578100ee0 0x7f5578198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f557e89e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55781055e0 0x7f557819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f557e89e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557819d560 con 0x7f5578100540 2023-10-11T16:31:13.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f557e89e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557819d6d0 con 0x7f55781055e0 2023-10-11T16:31:13.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f557e89e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557819d890 con 0x7f5578100ee0 2023-10-11T16:31:13.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f5577fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5578100540 0x7f55781985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.880+0000 7f5577fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5578100540 0x7f55781985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40598/0 (socket says 172.21.15.111:40598) 2023-10-11T16:31:13.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f5577fff700 1 -- 172.21.15.111:0/1680317575 learned_addr learned my addr 172.21.15.111:0/1680317575 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:13.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f55777fe700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5578100ee0 0x7f5578198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f557ce3b700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55781055e0 0x7f557819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f5577fff700 1 -- 172.21.15.111:0/1680317575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5578100ee0 msgr2=0x7f5578198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f5577fff700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5578100ee0 0x7f5578198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f5577fff700 1 -- 172.21.15.111:0/1680317575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55781055e0 msgr2=0x7f557819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f5577fff700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55781055e0 0x7f557819cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f5577fff700 1 -- 172.21.15.111:0/1680317575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c0095f0 con 0x7f5578100540 2023-10-11T16:31:13.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.881+0000 7f55777fe700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5578100ee0 0x7f5578198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:13.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.882+0000 7f5577fff700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5578100540 0x7f55781985a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f556800e9a0 tx=0x7f556800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:13.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.882+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568009790 con 0x7f5578100540 2023-10-11T16:31:13.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.882+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f556800cbd0 con 0x7f5578100540 2023-10-11T16:31:13.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.882+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568018680 con 0x7f5578100540 2023-10-11T16:31:13.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.882+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5578110310 con 0x7f5578100540 2023-10-11T16:31:13.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.882+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5578110180 con 0x7f5578100540 2023-10-11T16:31:13.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.883+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5578074070 con 0x7f5578100540 2023-10-11T16:31:13.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.886+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f556800cd40 con 0x7f5578100540 2023-10-11T16:31:13.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.889+0000 7f55757fa700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f55600a1260 0x7f55600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.889+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5568014070 con 0x7f5578100540 2023-10-11T16:31:13.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.889+0000 7f55757fa700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] conn(0x7f55600ac270 0x7f55600ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.889+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 --> [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55600aedd0 con 0x7f55600ac270 2023-10-11T16:31:13.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.889+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55680cd780 con 0x7f5578100540 2023-10-11T16:31:13.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.890+0000 7f55777fe700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f55600a1260 0x7f55600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.890+0000 7f557ce3b700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] conn(0x7f55600ac270 0x7f55600ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.891+0000 7f557ce3b700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] conn(0x7f55600ac270 0x7f55600ae700 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:13.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.891+0000 7f55777fe700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f55600a1260 0x7f55600a3710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f556c006010 tx=0x7f556c005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:13.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.892+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 <== osd.10 v2:172.21.15.130:6824/2844086684 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f55600aedd0 con 0x7f55600ac270 2023-10-11T16:31:13.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 -- 172.21.15.111:0/476938364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd31c101540 msgr2=0x7fd31c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 --2- 172.21.15.111:0/476938364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd31c101540 0x7fd31c101910 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd318009990 tx=0x7fd318009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 -- 172.21.15.111:0/476938364 shutdown_connections 2023-10-11T16:31:13.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 --2- 172.21.15.111:0/476938364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd31c10a890 0x7fd31c10ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 --2- 172.21.15.111:0/476938364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd31c101540 0x7fd31c101910 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 --2- 172.21.15.111:0/476938364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd31c101e50 0x7fd31c10a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 -- 172.21.15.111:0/476938364 >> 172.21.15.111:0/476938364 conn(0x7fd31c0770a0 msgr2=0x7fd31c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:13.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 -- 172.21.15.111:0/476938364 shutdown_connections 2023-10-11T16:31:13.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.899+0000 7fd323a87700 1 -- 172.21.15.111:0/476938364 wait complete. 2023-10-11T16:31:13.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.901+0000 7fd323a87700 1 Processor -- start 2023-10-11T16:31:13.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.901+0000 7fd323a87700 1 -- start start 2023-10-11T16:31:13.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd323a87700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd31c101540 0x7fd31c196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd323a87700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd31c101e50 0x7fd31c1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd323a87700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd31c10a890 0x7fd31c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd323a87700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c19b420 con 0x7fd31c101e50 2023-10-11T16:31:13.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd323a87700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c19b590 con 0x7fd31c10a890 2023-10-11T16:31:13.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd323a87700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c19b750 con 0x7fd31c101540 2023-10-11T16:31:13.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd321022700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd31c101e50 0x7fd31c1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd321022700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd31c101e50 0x7fd31c1969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40620/0 (socket says 172.21.15.111:40620) 2023-10-11T16:31:13.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd321022700 1 -- 172.21.15.111:0/3621046968 learned_addr learned my addr 172.21.15.111:0/3621046968 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:13.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd321823700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd31c101540 0x7fd31c196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.902+0000 7fd322024700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd31c10a890 0x7fd31c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd321022700 1 -- 172.21.15.111:0/3621046968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd31c101540 msgr2=0x7fd31c196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd321022700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd31c101540 0x7fd31c196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd321022700 1 -- 172.21.15.111:0/3621046968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd31c10a890 msgr2=0x7fd31c19ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd321022700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd31c10a890 0x7fd31c19ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd321022700 1 -- 172.21.15.111:0/3621046968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3180095f0 con 0x7fd31c101e50 2023-10-11T16:31:13.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd321022700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd31c101e50 0x7fd31c1969a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00e9a0 tx=0x7fd30c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:13.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c009790 con 0x7fd31c101e50 2023-10-11T16:31:13.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd30c00cbd0 con 0x7fd31c101e50 2023-10-11T16:31:13.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c110360 con 0x7fd31c101e50 2023-10-11T16:31:13.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c018680 con 0x7fd31c101e50 2023-10-11T16:31:13.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.903+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c1101d0 con 0x7fd31c101e50 2023-10-11T16:31:13.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.904+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd31c0fc6e0 con 0x7fd31c101e50 2023-10-11T16:31:13.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.908+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fd30c00cd40 con 0x7fd31c101e50 2023-10-11T16:31:13.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.910+0000 7fd312ffd700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd3080a11e0 0x7fd3080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.911+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd30c014070 con 0x7fd31c101e50 2023-10-11T16:31:13.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.911+0000 7fd321823700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd3080a11e0 0x7fd3080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.911+0000 7fd312ffd700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] conn(0x7fd3080ac1f0 0x7fd3080ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:13.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.911+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 --> [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3080aed50 con 0x7fd3080ac1f0 2023-10-11T16:31:13.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.911+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd30c0cd7c0 con 0x7fd31c101e50 2023-10-11T16:31:13.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.912+0000 7fd321823700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd3080a11e0 0x7fd3080a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd31800b2e0 tx=0x7fd318005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:13.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.912+0000 7fd322024700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] conn(0x7fd3080ac1f0 0x7fd3080ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:13.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.912+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 --> [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f55781104a0 con 0x7f55600ac270 2023-10-11T16:31:13.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.912+0000 7fd322024700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] conn(0x7fd3080ac1f0 0x7fd3080ae680 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:13.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.913+0000 7f55757fa700 1 -- 172.21.15.111:0/1680317575 <== osd.10 v2:172.21.15.130:6824/2844086684 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f55781104a0 con 0x7f55600ac270 2023-10-11T16:31:13.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.913+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 <== osd.0 v2:172.21.15.111:6802/937472257 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd3080aed50 con 0x7fd3080ac1f0 2023-10-11T16:31:13.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.913+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 >> [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] conn(0x7f55600ac270 msgr2=0x7f55600ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.913+0000 7f557e89e700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] conn(0x7f55600ac270 0x7f55600ae700 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f55600a1260 msgr2=0x7f55600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557e89e700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f55600a1260 0x7f55600a3710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f556c006010 tx=0x7f556c005960 comp rx=0 tx=0).stop 2023-10-11T16:31:13.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5578100540 msgr2=0x7f55781985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557e89e700 1 --2- 172.21.15.111:0/1680317575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5578100540 0x7f55781985a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f556800e9a0 tx=0x7f556800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557ce3b700 1 -- 172.21.15.111:0/1680317575 reap_dead start 2023-10-11T16:31:13.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 shutdown_connections 2023-10-11T16:31:13.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 >> 172.21.15.111:0/1680317575 conn(0x7f55780fbdc0 msgr2=0x7f55780fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:13.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 shutdown_connections 2023-10-11T16:31:13.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.914+0000 7f557e89e700 1 -- 172.21.15.111:0/1680317575 wait complete. 2023-10-11T16:31:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.934+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 --> [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd31c1104f0 con 0x7fd3080ac1f0 2023-10-11T16:31:13.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.935+0000 7fd312ffd700 1 -- 172.21.15.111:0/3621046968 <== osd.0 v2:172.21.15.111:6802/937472257 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd31c1104f0 con 0x7fd3080ac1f0 2023-10-11T16:31:13.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.935+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] conn(0x7fd3080ac1f0 msgr2=0x7fd3080ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.935+0000 7fd323a87700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] conn(0x7fd3080ac1f0 0x7fd3080ae680 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.936+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd3080a11e0 msgr2=0x7fd3080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.936+0000 7fd323a87700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd3080a11e0 0x7fd3080a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd31800b2e0 tx=0x7fd318005fd0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.936+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd31c101e50 msgr2=0x7fd31c1969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:13.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.936+0000 7fd323a87700 1 --2- 172.21.15.111:0/3621046968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd31c101e50 0x7fd31c1969a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00e9a0 tx=0x7fd30c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:13.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.936+0000 7fd322024700 1 -- 172.21.15.111:0/3621046968 reap_dead start 2023-10-11T16:31:13.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.936+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 shutdown_connections 2023-10-11T16:31:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.936+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 >> 172.21.15.111:0/3621046968 conn(0x7fd31c0770a0 msgr2=0x7fd31c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.937+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 shutdown_connections 2023-10-11T16:31:13.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:13.937+0000 7fd323a87700 1 -- 172.21.15.111:0/3621046968 wait complete. 2023-10-11T16:31:13.958 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:13.963 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:14.074 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:14.130 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:14.761 INFO:teuthology.orchestra.run.smithi111.stdout:304942678045 2023-10-11T16:31:14.762 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.10 2023-10-11T16:31:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:14.950 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:15.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:15 smithi111 ceph-mon[98527]: pgmap v306: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2023-10-11T16:31:15.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:15 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:31:15.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:15 smithi150 ceph-mon[132217]: pgmap v306: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2023-10-11T16:31:15.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:15 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:31:15.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:15 smithi130 ceph-mon[131859]: pgmap v306: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 7 op/s 2023-10-11T16:31:15.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:15 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:31:16.160 INFO:teuthology.orchestra.run.smithi111.stdout:34359738443 2023-10-11T16:31:16.161 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-10-11T16:31:16.863 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:17.014 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:17.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:17 smithi111 ceph-mon[98527]: pgmap v307: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:17.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:17 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:17 smithi150 ceph-mon[132217]: pgmap v307: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:17 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:17 smithi130 ceph-mon[131859]: pgmap v307: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:17 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:18.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.308+0000 7f1241d78700 1 -- 172.21.15.111:0/4097460518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100ee0 msgr2=0x7f123c104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:18.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.308+0000 7f1241d78700 1 --2- 172.21.15.111:0/4097460518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100ee0 0x7f123c104eb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f12240099c0 tx=0x7f1224009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.308+0000 7f1241d78700 1 -- 172.21.15.111:0/4097460518 shutdown_connections 2023-10-11T16:31:18.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.308+0000 7f1241d78700 1 --2- 172.21.15.111:0/4097460518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f123c1055e0 0x7f123c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.308+0000 7f1241d78700 1 --2- 172.21.15.111:0/4097460518 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f123c100540 0x7f123c100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.308+0000 7f1241d78700 1 --2- 172.21.15.111:0/4097460518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100ee0 0x7f123c104eb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.308+0000 7f1241d78700 1 -- 172.21.15.111:0/4097460518 >> 172.21.15.111:0/4097460518 conn(0x7f123c0fbdc0 msgr2=0x7f123c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:18.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.309+0000 7f1241d78700 1 -- 172.21.15.111:0/4097460518 shutdown_connections 2023-10-11T16:31:18.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.309+0000 7f1241d78700 1 -- 172.21.15.111:0/4097460518 wait complete. 2023-10-11T16:31:18.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.310+0000 7f1241d78700 1 Processor -- start 2023-10-11T16:31:18.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.310+0000 7f1241d78700 1 -- start start 2023-10-11T16:31:18.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f1241d78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100540 0x7f123c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:18.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f1241d78700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f123c100ee0 0x7f123c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:18.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f1241d78700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f123c1055e0 0x7f123c073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:18.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f1241d78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c1a8d80 con 0x7f123c100540 2023-10-11T16:31:18.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f1241d78700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c19a540 con 0x7f123c1055e0 2023-10-11T16:31:18.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f1241d78700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c19a700 con 0x7f123c100ee0 2023-10-11T16:31:18.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f123b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100540 0x7f123c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:18.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f123bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f123c1055e0 0x7f123c073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:18.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f123b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100540 0x7f123c072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57796/0 (socket says 172.21.15.111:57796) 2023-10-11T16:31:18.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.311+0000 7f123b7fe700 1 -- 172.21.15.111:0/1851982299 learned_addr learned my addr 172.21.15.111:0/1851982299 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:18.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.312+0000 7f1233fff700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f123c100ee0 0x7f123c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:18.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.312+0000 7f123b7fe700 1 -- 172.21.15.111:0/1851982299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f123c100ee0 msgr2=0x7f123c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:18.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.312+0000 7f123b7fe700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f123c100ee0 0x7f123c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.312+0000 7f123b7fe700 1 -- 172.21.15.111:0/1851982299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f123c1055e0 msgr2=0x7f123c073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:18.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.312+0000 7f123b7fe700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f123c1055e0 0x7f123c073870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.312+0000 7f123b7fe700 1 -- 172.21.15.111:0/1851982299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12240095f0 con 0x7f123c100540 2023-10-11T16:31:18.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.312+0000 7f123b7fe700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100540 0x7f123c072df0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f122c00d980 tx=0x7f122c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:18.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.313+0000 7f12397fa700 1 -- 172.21.15.111:0/1851982299 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c004220 con 0x7f123c100540 2023-10-11T16:31:18.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.313+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123c110360 con 0x7f123c100540 2023-10-11T16:31:18.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.313+0000 7f12397fa700 1 -- 172.21.15.111:0/1851982299 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f122c007640 con 0x7f123c100540 2023-10-11T16:31:18.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.313+0000 7f12397fa700 1 -- 172.21.15.111:0/1851982299 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c005610 con 0x7f123c100540 2023-10-11T16:31:18.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.313+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123c1101d0 con 0x7f123c100540 2023-10-11T16:31:18.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.314+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123c1104f0 con 0x7f123c100540 2023-10-11T16:31:18.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.316+0000 7f12397fa700 1 -- 172.21.15.111:0/1851982299 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f122c004750 con 0x7f123c100540 2023-10-11T16:31:18.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.318+0000 7f12397fa700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12280a1260 0x7f12280a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:18.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.319+0000 7f12397fa700 1 -- 172.21.15.111:0/1851982299 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f122c0ccbc0 con 0x7f123c100540 2023-10-11T16:31:18.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.319+0000 7f1233fff700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12280a1260 0x7f12280a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:18.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.319+0000 7f1233fff700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12280a1260 0x7f12280a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1224000c00 tx=0x7f1224005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:18.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.322+0000 7f12397fa700 1 -- 172.21.15.111:0/1851982299 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f122c095e40 con 0x7f123c100540 2023-10-11T16:31:18.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.535+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f123c066ea0 con 0x7f123c100540 2023-10-11T16:31:18.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.535+0000 7f12397fa700 1 -- 172.21.15.111:0/1851982299 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f122c016020 con 0x7f123c100540 2023-10-11T16:31:18.538 INFO:teuthology.orchestra.run.smithi111.stdout:163208757301 2023-10-11T16:31:18.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12280a1260 msgr2=0x7f12280a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:18.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12280a1260 0x7f12280a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1224000c00 tx=0x7f1224005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100540 msgr2=0x7f123c072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:18.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100540 0x7f123c072df0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f122c00d980 tx=0x7f122c00dc90 comp rx=0 tx=0).stop 2023-10-11T16:31:18.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 shutdown_connections 2023-10-11T16:31:18.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f12280a1260 0x7f12280a3710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f123c1055e0 0x7f123c073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f123c100540 0x7f123c072df0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 --2- 172.21.15.111:0/1851982299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f123c100ee0 0x7f123c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:18.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 >> 172.21.15.111:0/1851982299 conn(0x7f123c0fbdc0 msgr2=0x7f123c0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:18.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 shutdown_connections 2023-10-11T16:31:18.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:18.538+0000 7f1241d78700 1 -- 172.21.15.111:0/1851982299 wait complete. 2023-10-11T16:31:19.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:19 smithi111 ceph-mon[98527]: pgmap v308: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:19.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:19 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1851982299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-11T16:31:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:19 smithi150 ceph-mon[132217]: pgmap v308: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:19 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1851982299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-11T16:31:19.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:19 smithi130 ceph-mon[131859]: pgmap v308: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-10-11T16:31:19.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:19 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1851982299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-11T16:31:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 -- 172.21.15.111:0/3172374712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100ee0 msgr2=0x7f03f4104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 --2- 172.21.15.111:0/3172374712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100ee0 0x7f03f4104eb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009990 tx=0x7f03e8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 -- 172.21.15.111:0/3172374712 shutdown_connections 2023-10-11T16:31:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 --2- 172.21.15.111:0/3172374712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f41055e0 0x7f03f410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 --2- 172.21.15.111:0/3172374712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f4100540 0x7f03f4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 --2- 172.21.15.111:0/3172374712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100ee0 0x7f03f4104eb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 -- 172.21.15.111:0/3172374712 >> 172.21.15.111:0/3172374712 conn(0x7f03f40fbdc0 msgr2=0x7f03f40fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:19.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 -- 172.21.15.111:0/3172374712 shutdown_connections 2023-10-11T16:31:19.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.797+0000 7f03fa687700 1 -- 172.21.15.111:0/3172374712 wait complete. 2023-10-11T16:31:19.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.798+0000 7f03fa687700 1 Processor -- start 2023-10-11T16:31:19.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.798+0000 7f03fa687700 1 -- start start 2023-10-11T16:31:19.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03fa687700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100540 0x7f03f4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:19.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03fa687700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f4100ee0 0x7f03f4198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:19.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03fa687700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f41055e0 0x7f03f419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:19.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03fa687700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f419d5a0 con 0x7f03f4100ee0 2023-10-11T16:31:19.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03fa687700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f419d710 con 0x7f03f4100540 2023-10-11T16:31:19.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03fa687700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f419d8d0 con 0x7f03f41055e0 2023-10-11T16:31:19.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f3fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100540 0x7f03f4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:19.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f4100ee0 0x7f03f4198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:19.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f3fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100540 0x7f03f4198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:39098/0 (socket says 172.21.15.111:39098) 2023-10-11T16:31:19.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f3fff700 1 -- 172.21.15.111:0/2254977895 learned_addr learned my addr 172.21.15.111:0/2254977895 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:19.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f8c24700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f41055e0 0x7f03f419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:19.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f3fff700 1 -- 172.21.15.111:0/2254977895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f41055e0 msgr2=0x7f03f419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:19.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f3fff700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f41055e0 0x7f03f419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f3fff700 1 -- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f4100ee0 msgr2=0x7f03f4198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:19.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f3fff700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f4100ee0 0x7f03f4198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.799+0000 7f03f3fff700 1 -- 172.21.15.111:0/2254977895 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec009c40 con 0x7f03f4100540 2023-10-11T16:31:19.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.800+0000 7f03f3fff700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100540 0x7f03f4198680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f03e400e9a0 tx=0x7f03e400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:19.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.800+0000 7f03f17fa700 1 -- 172.21.15.111:0/2254977895 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4009790 con 0x7f03f4100540 2023-10-11T16:31:19.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.800+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e80095f0 con 0x7f03f4100540 2023-10-11T16:31:19.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.800+0000 7f03f17fa700 1 -- 172.21.15.111:0/2254977895 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03e400cbd0 con 0x7f03f4100540 2023-10-11T16:31:19.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.800+0000 7f03f17fa700 1 -- 172.21.15.111:0/2254977895 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4018680 con 0x7f03f4100540 2023-10-11T16:31:19.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.800+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f4110360 con 0x7f03f4100540 2023-10-11T16:31:19.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.801+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f41101d0 con 0x7f03f4100540 2023-10-11T16:31:19.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.803+0000 7f03f17fa700 1 -- 172.21.15.111:0/2254977895 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f03e400cd40 con 0x7f03f4100540 2023-10-11T16:31:19.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.804+0000 7f03f17fa700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03dc0a11e0 0x7f03dc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:19.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.805+0000 7f03f17fa700 1 -- 172.21.15.111:0/2254977895 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f03e4014070 con 0x7f03f4100540 2023-10-11T16:31:19.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.805+0000 7f03f37fe700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03dc0a11e0 0x7f03dc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:19.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.805+0000 7f03f37fe700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03dc0a11e0 0x7f03dc0a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f03f4199ca0 tx=0x7f03e8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:19.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.807+0000 7f03f17fa700 1 -- 172.21.15.111:0/2254977895 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f03e40965c0 con 0x7f03f4100540 2023-10-11T16:31:19.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 -- 172.21.15.111:0/4038633260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 msgr2=0x7f6604100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:19.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 --2- 172.21.15.111:0/4038633260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 0x7f6604100910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f66000099c0 tx=0x7f6600009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 -- 172.21.15.111:0/4038633260 shutdown_connections 2023-10-11T16:31:19.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 --2- 172.21.15.111:0/4038633260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66041055e0 0x7f660410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 --2- 172.21.15.111:0/4038633260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 0x7f6604100910 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 --2- 172.21.15.111:0/4038633260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6604100ee0 0x7f6604104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 -- 172.21.15.111:0/4038633260 >> 172.21.15.111:0/4038633260 conn(0x7f66040fbdc0 msgr2=0x7f66040fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:19.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 -- 172.21.15.111:0/4038633260 shutdown_connections 2023-10-11T16:31:19.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.909+0000 7f660b9e5700 1 -- 172.21.15.111:0/4038633260 wait complete. 2023-10-11T16:31:19.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.910+0000 7f660b9e5700 1 Processor -- start 2023-10-11T16:31:19.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.910+0000 7f660b9e5700 1 -- start start 2023-10-11T16:31:19.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.910+0000 7f660b9e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 0x7f6604198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:19.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.910+0000 7f660b9e5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6604100ee0 0x7f6604198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:19.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.910+0000 7f660b9e5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66041055e0 0x7f660419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:19.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.910+0000 7f660b9e5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660419d620 con 0x7f6604100540 2023-10-11T16:31:19.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.910+0000 7f660b9e5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660419d790 con 0x7f66041055e0 2023-10-11T16:31:19.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.910+0000 7f660b9e5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660419d950 con 0x7f6604100ee0 2023-10-11T16:31:19.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6609781700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 0x7f6604198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6608f80700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6604100ee0 0x7f6604198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6608f80700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6604100ee0 0x7f6604198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48250/0 (socket says 172.21.15.111:48250) 2023-10-11T16:31:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6608f80700 1 -- 172.21.15.111:0/1299302397 learned_addr learned my addr 172.21.15.111:0/1299302397 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6609f82700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66041055e0 0x7f660419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:19.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6608f80700 1 -- 172.21.15.111:0/1299302397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66041055e0 msgr2=0x7f660419d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6608f80700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66041055e0 0x7f660419d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6608f80700 1 -- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 msgr2=0x7f6604198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6608f80700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 0x7f6604198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6608f80700 1 -- 172.21.15.111:0/1299302397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66000095f0 con 0x7f6604100ee0 2023-10-11T16:31:19.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.911+0000 7f6609781700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 0x7f6604198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.912+0000 7f6608f80700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6604100ee0 0x7f6604198ba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f65f400e970 tx=0x7f65f400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.912+0000 7f65fa7fc700 1 -- 172.21.15.111:0/1299302397 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4009790 con 0x7f6604100ee0 2023-10-11T16:31:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.912+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6604110370 con 0x7f6604100ee0 2023-10-11T16:31:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.912+0000 7f65fa7fc700 1 -- 172.21.15.111:0/1299302397 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65f400c7e0 con 0x7f6604100ee0 2023-10-11T16:31:19.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.912+0000 7f65fa7fc700 1 -- 172.21.15.111:0/1299302397 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40054e0 con 0x7f6604100ee0 2023-10-11T16:31:19.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.912+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66041101e0 con 0x7f6604100ee0 2023-10-11T16:31:19.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.913+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6604110500 con 0x7f6604100ee0 2023-10-11T16:31:19.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.917+0000 7f65fa7fc700 1 -- 172.21.15.111:0/1299302397 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f65f400c950 con 0x7f6604100ee0 2023-10-11T16:31:19.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.918+0000 7f65fa7fc700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f65f00a11e0 0x7f65f00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:19.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.918+0000 7f65fa7fc700 1 -- 172.21.15.111:0/1299302397 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f65f4014070 con 0x7f6604100ee0 2023-10-11T16:31:19.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.918+0000 7f6609781700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f65f00a11e0 0x7f65f00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:19.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.919+0000 7f65fa7fc700 1 -- 172.21.15.111:0/1299302397 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f65f4099fa0 con 0x7f6604100ee0 2023-10-11T16:31:19.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:19.919+0000 7f6609781700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f65f00a11e0 0x7f65f00a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6600009990 tx=0x7f6600005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:20.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.033+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f03f41104f0 con 0x7f03f4100540 2023-10-11T16:31:20.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.034+0000 7f03f17fa700 1 -- 172.21.15.111:0/2254977895 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f03e4099be0 con 0x7f03f4100540 2023-10-11T16:31:20.036 INFO:teuthology.orchestra.run.smithi111.stdout:249108103207 2023-10-11T16:31:20.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03dc0a11e0 msgr2=0x7f03dc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03dc0a11e0 0x7f03dc0a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f03f4199ca0 tx=0x7f03e8005fd0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100540 msgr2=0x7f03f4198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100540 0x7f03f4198680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f03e400e9a0 tx=0x7f03e400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 shutdown_connections 2023-10-11T16:31:20.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03dc0a11e0 0x7f03dc0a3690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03f41055e0 0x7f03f419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03f4100540 0x7f03f4198680 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 --2- 172.21.15.111:0/2254977895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f4100ee0 0x7f03f4198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 >> 172.21.15.111:0/2254977895 conn(0x7f03f40fbdc0 msgr2=0x7f03f40fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:20.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 shutdown_connections 2023-10-11T16:31:20.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.036+0000 7f03fa687700 1 -- 172.21.15.111:0/2254977895 wait complete. 2023-10-11T16:31:20.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.152+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f66041a57a0 con 0x7f6604100ee0 2023-10-11T16:31:20.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.153+0000 7f65fa7fc700 1 -- 172.21.15.111:0/1299302397 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65f4099b30 con 0x7f6604100ee0 2023-10-11T16:31:20.155 INFO:teuthology.orchestra.run.smithi111.stdout:77309411396 2023-10-11T16:31:20.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f65f00a11e0 msgr2=0x7f65f00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f65f00a11e0 0x7f65f00a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6600009990 tx=0x7f6600005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6604100ee0 msgr2=0x7f6604198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6604100ee0 0x7f6604198ba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f65f400e970 tx=0x7f65f400ec80 comp rx=0 tx=0).stop 2023-10-11T16:31:20.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 shutdown_connections 2023-10-11T16:31:20.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f65f00a11e0 0x7f65f00a3690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66041055e0 0x7f660419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6604100540 0x7f6604198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 --2- 172.21.15.111:0/1299302397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6604100ee0 0x7f6604198ba0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f65f400e970 tx=0x7f65f400ec80 comp rx=0 tx=0).stop 2023-10-11T16:31:20.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.156+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 >> 172.21.15.111:0/1299302397 conn(0x7f66040fbdc0 msgr2=0x7f66040fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:20.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.157+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 shutdown_connections 2023-10-11T16:31:20.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.157+0000 7f660b9e5700 1 -- 172.21.15.111:0/1299302397 wait complete. 2023-10-11T16:31:20.330 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757299 got 163208757301 for osd.5 2023-10-11T16:31:20.331 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:20.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:20 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2254977895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-10-11T16:31:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:20 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2254977895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-10-11T16:31:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 -- 172.21.15.111:0/2259578355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff694100540 msgr2=0x7ff694100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 --2- 172.21.15.111:0/2259578355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff694100540 0x7ff694100910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff6840099c0 tx=0x7ff684009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 -- 172.21.15.111:0/2259578355 shutdown_connections 2023-10-11T16:31:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 --2- 172.21.15.111:0/2259578355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6941055e0 0x7ff69410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 --2- 172.21.15.111:0/2259578355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff694100540 0x7ff694100910 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 --2- 172.21.15.111:0/2259578355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff694100ee0 0x7ff694104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 -- 172.21.15.111:0/2259578355 >> 172.21.15.111:0/2259578355 conn(0x7ff6940fbdc0 msgr2=0x7ff6940fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 -- 172.21.15.111:0/2259578355 shutdown_connections 2023-10-11T16:31:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.488+0000 7ff699ab9700 1 -- 172.21.15.111:0/2259578355 wait complete. 2023-10-11T16:31:20.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.489+0000 7ff699ab9700 1 Processor -- start 2023-10-11T16:31:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff699ab9700 1 -- start start 2023-10-11T16:31:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff699ab9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100540 0x7ff694072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff699ab9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff694100ee0 0x7ff694073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:20.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff699ab9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6941055e0 0x7ff6940738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff699ab9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941a4810 con 0x7ff6941055e0 2023-10-11T16:31:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff699ab9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941a4980 con 0x7ff694100ee0 2023-10-11T16:31:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff699ab9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941a4b40 con 0x7ff694100540 2023-10-11T16:31:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff693fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6941055e0 0x7ff6940738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:20.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff693fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6941055e0 0x7ff6940738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57856/0 (socket says 172.21.15.111:57856) 2023-10-11T16:31:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff693fff700 1 -- 172.21.15.111:0/3528400731 learned_addr learned my addr 172.21.15.111:0/3528400731 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff692ffd700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff694100ee0 0x7ff694073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.490+0000 7ff6937fe700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100540 0x7ff694072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:20.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff693fff700 1 -- 172.21.15.111:0/3528400731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100540 msgr2=0x7ff694072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff693fff700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100540 0x7ff694072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff693fff700 1 -- 172.21.15.111:0/3528400731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff694100ee0 msgr2=0x7ff694073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff693fff700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff694100ee0 0x7ff694073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff693fff700 1 -- 172.21.15.111:0/3528400731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840095f0 con 0x7ff6941055e0 2023-10-11T16:31:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff693fff700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6941055e0 0x7ff6940738b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff68c007ee0 tx=0x7ff68c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff690ff9700 1 -- 172.21.15.111:0/3528400731 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c017070 con 0x7ff6941055e0 2023-10-11T16:31:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff690ff9700 1 -- 172.21.15.111:0/3528400731 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff68c00f040 con 0x7ff6941055e0 2023-10-11T16:31:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff694110320 con 0x7ff6941055e0 2023-10-11T16:31:20.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff690ff9700 1 -- 172.21.15.111:0/3528400731 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c005580 con 0x7ff6941055e0 2023-10-11T16:31:20.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.491+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff694110190 con 0x7ff6941055e0 2023-10-11T16:31:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.492+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6941104b0 con 0x7ff6941055e0 2023-10-11T16:31:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.494+0000 7ff690ff9700 1 -- 172.21.15.111:0/3528400731 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff68c00db20 con 0x7ff6941055e0 2023-10-11T16:31:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.496+0000 7ff690ff9700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff67c0a11e0 0x7ff67c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:20.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.497+0000 7ff690ff9700 1 -- 172.21.15.111:0/3528400731 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff68c0cc480 con 0x7ff6941055e0 2023-10-11T16:31:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.497+0000 7ff6937fe700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff67c0a11e0 0x7ff67c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:20.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.497+0000 7ff6937fe700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff67c0a11e0 0x7ff67c0a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff684009990 tx=0x7ff684003680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:20.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.501+0000 7ff690ff9700 1 -- 172.21.15.111:0/3528400731 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff68c095700 con 0x7ff6941055e0 2023-10-11T16:31:20.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:20 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2254977895' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-10-11T16:31:20.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.760+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff694074320 con 0x7ff6941055e0 2023-10-11T16:31:20.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.761+0000 7ff690ff9700 1 -- 172.21.15.111:0/3528400731 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff68c098d20 con 0x7ff6941055e0 2023-10-11T16:31:20.763 INFO:teuthology.orchestra.run.smithi111.stdout:219043332139 2023-10-11T16:31:20.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff67c0a11e0 msgr2=0x7ff67c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff67c0a11e0 0x7ff67c0a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff684009990 tx=0x7ff684003680 comp rx=0 tx=0).stop 2023-10-11T16:31:20.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6941055e0 msgr2=0x7ff6940738b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6941055e0 0x7ff6940738b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff68c007ee0 tx=0x7ff68c00d3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 shutdown_connections 2023-10-11T16:31:20.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff67c0a11e0 0x7ff67c0a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6941055e0 0x7ff6940738b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff694100540 0x7ff694072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 --2- 172.21.15.111:0/3528400731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff694100ee0 0x7ff694073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.763+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 >> 172.21.15.111:0/3528400731 conn(0x7ff6940fbdc0 msgr2=0x7ff6940fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:20.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.764+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 shutdown_connections 2023-10-11T16:31:20.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.765+0000 7ff699ab9700 1 -- 172.21.15.111:0/3528400731 wait complete. 2023-10-11T16:31:20.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.807+0000 7f283e58f700 1 -- 172.21.15.111:0/3587711062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f283810a890 msgr2=0x7f283810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.807+0000 7f283e58f700 1 --2- 172.21.15.111:0/3587711062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f283810a890 0x7f283810ccd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2830009f90 tx=0x7f283000b330 comp rx=0 tx=0).stop 2023-10-11T16:31:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.808+0000 7f283e58f700 1 -- 172.21.15.111:0/3587711062 shutdown_connections 2023-10-11T16:31:20.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.808+0000 7f283e58f700 1 --2- 172.21.15.111:0/3587711062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f283810a890 0x7f283810ccd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.808+0000 7f283e58f700 1 --2- 172.21.15.111:0/3587711062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2838101540 0x7f2838101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.808+0000 7f283e58f700 1 --2- 172.21.15.111:0/3587711062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838101e50 0x7f283810a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.808+0000 7f283e58f700 1 -- 172.21.15.111:0/3587711062 >> 172.21.15.111:0/3587711062 conn(0x7f28380770a0 msgr2=0x7f28380774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:20.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.808+0000 7f283e58f700 1 -- 172.21.15.111:0/3587711062 shutdown_connections 2023-10-11T16:31:20.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.808+0000 7f283e58f700 1 -- 172.21.15.111:0/3587711062 wait complete. 2023-10-11T16:31:20.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.809+0000 7f283e58f700 1 Processor -- start 2023-10-11T16:31:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.809+0000 7f283e58f700 1 -- start start 2023-10-11T16:31:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.809+0000 7f283e58f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838101540 0x7f2838196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.809+0000 7f283e58f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101e50 0x7f2838196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.809+0000 7f283e58f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f283810a890 0x7f283819aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.809+0000 7f283e58f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283819b4e0 con 0x7f283810a890 2023-10-11T16:31:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.809+0000 7f283e58f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283819b650 con 0x7f2838101e50 2023-10-11T16:31:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.809+0000 7f283e58f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283819b810 con 0x7f2838101540 2023-10-11T16:31:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f28377fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101e50 0x7f2838196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:20.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f283cb2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f283810a890 0x7f283819aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:20.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f28377fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101e50 0x7f2838196a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:39146/0 (socket says 172.21.15.111:39146) 2023-10-11T16:31:20.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f2837fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838101540 0x7f2838196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:20.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f28377fe700 1 -- 172.21.15.111:0/3458616841 learned_addr learned my addr 172.21.15.111:0/3458616841 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:20.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f28377fe700 1 -- 172.21.15.111:0/3458616841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838101540 msgr2=0x7f2838196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f28377fe700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838101540 0x7f2838196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f28377fe700 1 -- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f283810a890 msgr2=0x7f283819aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f28377fe700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f283810a890 0x7f283819aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f28377fe700 1 -- 172.21.15.111:0/3458616841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28280095f0 con 0x7f2838101e50 2023-10-11T16:31:20.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.810+0000 7f283cb2c700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f283810a890 0x7f283819aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.811+0000 7f28377fe700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101e50 0x7f2838196a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f282c00e9a0 tx=0x7f282c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.811+0000 7f28357fa700 1 -- 172.21.15.111:0/3458616841 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c009790 con 0x7f2838101e50 2023-10-11T16:31:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.811+0000 7f28357fa700 1 -- 172.21.15.111:0/3458616841 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f282c010430 con 0x7f2838101e50 2023-10-11T16:31:20.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.811+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2830009c40 con 0x7f2838101e50 2023-10-11T16:31:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.811+0000 7f28357fa700 1 -- 172.21.15.111:0/3458616841 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c018680 con 0x7f2838101e50 2023-10-11T16:31:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.811+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2838110350 con 0x7f2838101e50 2023-10-11T16:31:20.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.812+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28381101c0 con 0x7f2838101e50 2023-10-11T16:31:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.814+0000 7f28357fa700 1 -- 172.21.15.111:0/3458616841 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f282c0105a0 con 0x7f2838101e50 2023-10-11T16:31:20.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.815+0000 7f28357fa700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f28200a11e0 0x7f28200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.816+0000 7f28357fa700 1 -- 172.21.15.111:0/3458616841 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f282c014070 con 0x7f2838101e50 2023-10-11T16:31:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.816+0000 7f2837fff700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f28200a11e0 0x7f28200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.816+0000 7f2837fff700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f28200a11e0 0x7f28200a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2828005fd0 tx=0x7f2828009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:20.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:20.818+0000 7f28357fa700 1 -- 172.21.15.111:0/3458616841 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f282c096390 con 0x7f2838101e50 2023-10-11T16:31:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.035+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f28381104e0 con 0x7f2838101e50 2023-10-11T16:31:21.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.036+0000 7f28357fa700 1 -- 172.21.15.111:0/3458616841 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f282c0999b0 con 0x7f2838101e50 2023-10-11T16:31:21.038 INFO:teuthology.orchestra.run.smithi111.stdout:283467841569 2023-10-11T16:31:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f28200a11e0 msgr2=0x7f28200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f28200a11e0 0x7f28200a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2828005fd0 tx=0x7f2828009310 comp rx=0 tx=0).stop 2023-10-11T16:31:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101e50 msgr2=0x7f2838196a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:21.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101e50 0x7f2838196a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f282c00e9a0 tx=0x7f282c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 shutdown_connections 2023-10-11T16:31:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f28200a11e0 0x7f28200a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f283810a890 0x7f283819aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838101540 0x7f2838196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 --2- 172.21.15.111:0/3458616841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101e50 0x7f2838196a60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 >> 172.21.15.111:0/3458616841 conn(0x7f28380770a0 msgr2=0x7f28380fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 shutdown_connections 2023-10-11T16:31:21.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.038+0000 7f283e58f700 1 -- 172.21.15.111:0/3458616841 wait complete. 2023-10-11T16:31:21.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411396 for osd.2 2023-10-11T16:31:21.162 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:21.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:21 smithi111 ceph-mon[98527]: pgmap v309: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:21.313 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:21 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1299302397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-11T16:31:21.314 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:21 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3528400731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-11T16:31:21.314 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:21 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3458616841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-10-11T16:31:21.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:21 smithi150 ceph-mon[132217]: pgmap v309: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:21.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:21 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1299302397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-11T16:31:21.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:21 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3528400731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-11T16:31:21.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:21 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3458616841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-10-11T16:31:21.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.511+0000 7f57b6ce9700 1 -- 172.21.15.111:0/1874879865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 msgr2=0x7f57b010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:21.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.511+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/1874879865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 0x7f57b010d160 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f57a800b0d0 tx=0x7f57a800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.511+0000 7f57b6ce9700 1 -- 172.21.15.111:0/1874879865 shutdown_connections 2023-10-11T16:31:21.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.511+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/1874879865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 0x7f57b010d160 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.511+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/1874879865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57b0100540 0x7f57b0100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.511+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/1874879865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57b0100ee0 0x7f57b0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.511+0000 7f57b6ce9700 1 -- 172.21.15.111:0/1874879865 >> 172.21.15.111:0/1874879865 conn(0x7f57b00fbdc0 msgr2=0x7f57b00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:21.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.512+0000 7f57b6ce9700 1 -- 172.21.15.111:0/1874879865 shutdown_connections 2023-10-11T16:31:21.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.512+0000 7f57b6ce9700 1 -- 172.21.15.111:0/1874879865 wait complete. 2023-10-11T16:31:21.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.513+0000 7f57b6ce9700 1 Processor -- start 2023-10-11T16:31:21.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.513+0000 7f57b6ce9700 1 -- start start 2023-10-11T16:31:21.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b6ce9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57b0100540 0x7f57b0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:21.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b6ce9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57b0100ee0 0x7f57b0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:21.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b6ce9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 0x7f57b01a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:21.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b6ce9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0073c20 con 0x7f57b01055e0 2023-10-11T16:31:21.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b6ce9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b01a4ba0 con 0x7f57b0100540 2023-10-11T16:31:21.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b6ce9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b01a4d10 con 0x7f57b0100ee0 2023-10-11T16:31:21.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b5286700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 0x7f57b01a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:21.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57affff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57b0100ee0 0x7f57b0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:21.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b4a85700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57b0100540 0x7f57b0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:21.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57affff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57b0100ee0 0x7f57b0073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48294/0 (socket says 172.21.15.111:48294) 2023-10-11T16:31:21.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57b5286700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 0x7f57b01a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57892/0 (socket says 172.21.15.111:57892) 2023-10-11T16:31:21.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.514+0000 7f57affff700 1 -- 172.21.15.111:0/292542181 learned_addr learned my addr 172.21.15.111:0/292542181 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:21.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b5286700 1 -- 172.21.15.111:0/292542181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57b0100ee0 msgr2=0x7f57b0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:21.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b5286700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57b0100ee0 0x7f57b0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b5286700 1 -- 172.21.15.111:0/292542181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57b0100540 msgr2=0x7f57b0072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:21.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b5286700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57b0100540 0x7f57b0072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b5286700 1 -- 172.21.15.111:0/292542181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a8009d00 con 0x7f57b01055e0 2023-10-11T16:31:21.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57affff700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57b0100ee0 0x7f57b0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:21.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b4a85700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57b0100540 0x7f57b0072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:21.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b5286700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 0x7f57b01a4660 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f57a800b770 tx=0x7f57a8007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:21.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57adffb700 1 -- 172.21.15.111:0/292542181 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a800e040 con 0x7f57b01055e0 2023-10-11T16:31:21.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b0110360 con 0x7f57b01055e0 2023-10-11T16:31:21.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57adffb700 1 -- 172.21.15.111:0/292542181 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57a801d930 con 0x7f57b01055e0 2023-10-11T16:31:21.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57adffb700 1 -- 172.21.15.111:0/292542181 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a801b820 con 0x7f57b01055e0 2023-10-11T16:31:21.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.515+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b01101d0 con 0x7f57b01055e0 2023-10-11T16:31:21.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.517+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b01104f0 con 0x7f57b01055e0 2023-10-11T16:31:21.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.518+0000 7f57adffb700 1 -- 172.21.15.111:0/292542181 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f57a801d470 con 0x7f57b01055e0 2023-10-11T16:31:21.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.520+0000 7f57adffb700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57a00a11e0 0x7f57a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:21.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.520+0000 7f57adffb700 1 -- 172.21.15.111:0/292542181 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f57a80ce090 con 0x7f57b01055e0 2023-10-11T16:31:21.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.520+0000 7f57b4a85700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57a00a11e0 0x7f57a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:21.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.521+0000 7f57b4a85700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57a00a11e0 0x7f57a00a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f57a4011490 tx=0x7f57a4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:21.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.528+0000 7f57adffb700 1 -- 172.21.15.111:0/292542181 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f57a80973a0 con 0x7f57b01055e0 2023-10-11T16:31:21.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:21 smithi130 ceph-mon[131859]: pgmap v309: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:21.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:21 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1299302397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-11T16:31:21.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:21 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3528400731' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-11T16:31:21.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:21 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3458616841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-10-11T16:31:21.695 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332137 got 219043332139 for osd.7 2023-10-11T16:31:21.696 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:21.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.744+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f57b01a5690 con 0x7f57b01055e0 2023-10-11T16:31:21.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.745+0000 7f57adffb700 1 -- 172.21.15.111:0/292542181 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f57a809a9c0 con 0x7f57b01055e0 2023-10-11T16:31:21.747 INFO:teuthology.orchestra.run.smithi111.stdout:335007449114 2023-10-11T16:31:21.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57a00a11e0 msgr2=0x7f57a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:21.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57a00a11e0 0x7f57a00a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f57a4011490 tx=0x7f57a4009260 comp rx=0 tx=0).stop 2023-10-11T16:31:21.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 msgr2=0x7f57b01a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:21.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 0x7f57b01a4660 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f57a800b770 tx=0x7f57a8007870 comp rx=0 tx=0).stop 2023-10-11T16:31:21.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 shutdown_connections 2023-10-11T16:31:21.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f57a00a11e0 0x7f57a00a3690 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b01055e0 0x7f57b01a4660 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57b0100540 0x7f57b0072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 --2- 172.21.15.111:0/292542181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57b0100ee0 0x7f57b0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:21.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.747+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 >> 172.21.15.111:0/292542181 conn(0x7f57b00fbdc0 msgr2=0x7f57b00fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:21.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.748+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 shutdown_connections 2023-10-11T16:31:21.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.748+0000 7f57b6ce9700 1 -- 172.21.15.111:0/292542181 wait complete. 2023-10-11T16:31:22.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 -- 172.21.15.111:0/1915971024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8105410 msgr2=0x7f46d810cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 --2- 172.21.15.111:0/1915971024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8105410 0x7f46d810cf90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f46d000b0d0 tx=0x7f46d000b3e0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 -- 172.21.15.111:0/1915971024 shutdown_connections 2023-10-11T16:31:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 --2- 172.21.15.111:0/1915971024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8105410 0x7f46d810cf90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 --2- 172.21.15.111:0/1915971024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d8100370 0x7f46d8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 --2- 172.21.15.111:0/1915971024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46d8100d10 0x7f46d8104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 -- 172.21.15.111:0/1915971024 >> 172.21.15.111:0/1915971024 conn(0x7f46d80fbbd0 msgr2=0x7f46d80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:22.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 -- 172.21.15.111:0/1915971024 shutdown_connections 2023-10-11T16:31:22.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.577+0000 7f46dd362700 1 -- 172.21.15.111:0/1915971024 wait complete. 2023-10-11T16:31:22.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.578+0000 7f46dd362700 1 Processor -- start 2023-10-11T16:31:22.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.578+0000 7f46dd362700 1 -- start start 2023-10-11T16:31:22.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.578+0000 7f46dd362700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d8100370 0x7f46d8196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:22.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.578+0000 7f46dd362700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8100d10 0x7f46d8196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:22.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.578+0000 7f46dd362700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46d8105410 0x7f46d819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:22.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.578+0000 7f46dd362700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d819b2f0 con 0x7f46d8100d10 2023-10-11T16:31:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.578+0000 7f46dd362700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d819b460 con 0x7f46d8105410 2023-10-11T16:31:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.578+0000 7f46dd362700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d819b620 con 0x7f46d8100370 2023-10-11T16:31:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8100d10 0x7f46d8196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8100d10 0x7f46d8196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57918/0 (socket says 172.21.15.111:57918) 2023-10-11T16:31:22.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 -- 172.21.15.111:0/2995003314 learned_addr learned my addr 172.21.15.111:0/2995003314 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:22.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d7fff700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d8100370 0x7f46d8196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:22.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46dcb61700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46d8105410 0x7f46d819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:22.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 -- 172.21.15.111:0/2995003314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d8100370 msgr2=0x7f46d8196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:22.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d8100370 0x7f46d8196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 -- 172.21.15.111:0/2995003314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46d8105410 msgr2=0x7f46d819ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:22.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46d8105410 0x7f46d819ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 -- 172.21.15.111:0/2995003314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d0009d00 con 0x7f46d8100d10 2023-10-11T16:31:22.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.579+0000 7f46d77fe700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8100d10 0x7f46d8196870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00e970 tx=0x7f46cc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:22.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.580+0000 7f46d57fa700 1 -- 172.21.15.111:0/2995003314 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc009790 con 0x7f46d8100d10 2023-10-11T16:31:22.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.580+0000 7f46d57fa700 1 -- 172.21.15.111:0/2995003314 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46cc00c7e0 con 0x7f46d8100d10 2023-10-11T16:31:22.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.580+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d81101a0 con 0x7f46d8100d10 2023-10-11T16:31:22.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.580+0000 7f46d57fa700 1 -- 172.21.15.111:0/2995003314 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc0054e0 con 0x7f46d8100d10 2023-10-11T16:31:22.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.580+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d8110010 con 0x7f46d8100d10 2023-10-11T16:31:22.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.581+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d8110330 con 0x7f46d8100d10 2023-10-11T16:31:22.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.583+0000 7f46d57fa700 1 -- 172.21.15.111:0/2995003314 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f46cc00c950 con 0x7f46d8100d10 2023-10-11T16:31:22.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.584+0000 7f46d57fa700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f46c00a1260 0x7f46c00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:22.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.584+0000 7f46d57fa700 1 -- 172.21.15.111:0/2995003314 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f46cc014070 con 0x7f46d8100d10 2023-10-11T16:31:22.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.585+0000 7f46d7fff700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f46c00a1260 0x7f46c00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:22.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.585+0000 7f46d7fff700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f46c00a1260 0x7f46c00a3710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009c60 tx=0x7f46c8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:22.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.588+0000 7f46d57fa700 1 -- 172.21.15.111:0/2995003314 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f46cc0961d0 con 0x7f46d8100d10 2023-10-11T16:31:22.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.805+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f46d804e8c0 con 0x7f46d8100d10 2023-10-11T16:31:22.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.805+0000 7f46d57fa700 1 -- 172.21.15.111:0/2995003314 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f46cc0997f0 con 0x7f46d8100d10 2023-10-11T16:31:22.020 INFO:teuthology.orchestra.run.smithi111.stdout:141733920827 2023-10-11T16:31:22.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f46c00a1260 msgr2=0x7f46c00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:22.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f46c00a1260 0x7f46c00a3710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009c60 tx=0x7f46c8009260 comp rx=0 tx=0).stop 2023-10-11T16:31:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8100d10 msgr2=0x7f46d8196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8100d10 0x7f46d8196870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00e970 tx=0x7f46cc00ec80 comp rx=0 tx=0).stop 2023-10-11T16:31:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 shutdown_connections 2023-10-11T16:31:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f46c00a1260 0x7f46c00a3710 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46d8105410 0x7f46d819ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d8100370 0x7f46d8196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 --2- 172.21.15.111:0/2995003314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46d8100d10 0x7f46d8196870 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.807+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 >> 172.21.15.111:0/2995003314 conn(0x7f46d80fbbd0 msgr2=0x7f46d80fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:22.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.808+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 shutdown_connections 2023-10-11T16:31:22.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:21.808+0000 7f46dd362700 1 -- 172.21.15.111:0/2995003314 wait complete. 2023-10-11T16:31:22.033 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841567 got 283467841569 for osd.9 2023-10-11T16:31:22.033 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:22.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:22 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/292542181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-10-11T16:31:22.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:22 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2995003314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-11T16:31:22.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:22 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/292542181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-10-11T16:31:22.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:22 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2995003314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-11T16:31:22.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:22 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/292542181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-10-11T16:31:22.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:22 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2995003314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-11T16:31:22.629 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449111 got 335007449114 for osd.11 2023-10-11T16:31:22.629 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:22.658 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:22.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.976+0000 7fbe37fff700 1 -- 172.21.15.111:0/2614257850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101c40 msgr2=0x7fbe3810a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:22.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.976+0000 7fbe37fff700 1 --2- 172.21.15.111:0/2614257850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101c40 0x7fbe3810a140 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009990 tx=0x7fbe28009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.977+0000 7fbe37fff700 1 -- 172.21.15.111:0/2614257850 shutdown_connections 2023-10-11T16:31:22.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.977+0000 7fbe37fff700 1 --2- 172.21.15.111:0/2614257850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe3810a680 0x7fbe3810cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.977+0000 7fbe37fff700 1 --2- 172.21.15.111:0/2614257850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe38101330 0x7fbe38101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.977+0000 7fbe37fff700 1 --2- 172.21.15.111:0/2614257850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101c40 0x7fbe3810a140 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.977+0000 7fbe37fff700 1 -- 172.21.15.111:0/2614257850 >> 172.21.15.111:0/2614257850 conn(0x7fbe380761a0 msgr2=0x7fbe380765a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.977+0000 7fbe37fff700 1 -- 172.21.15.111:0/2614257850 shutdown_connections 2023-10-11T16:31:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.977+0000 7fbe37fff700 1 -- 172.21.15.111:0/2614257850 wait complete. 2023-10-11T16:31:22.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.978+0000 7fbe37fff700 1 Processor -- start 2023-10-11T16:31:22.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.978+0000 7fbe37fff700 1 -- start start 2023-10-11T16:31:22.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.978+0000 7fbe37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101330 0x7fbe38198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:22.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.978+0000 7fbe37fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe38101c40 0x7fbe38198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:22.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.978+0000 7fbe37fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe3810a680 0x7fbe3819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:22.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.978+0000 7fbe37fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3819d400 con 0x7fbe38101330 2023-10-11T16:31:22.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.978+0000 7fbe37fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3819d570 con 0x7fbe3810a680 2023-10-11T16:31:22.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.978+0000 7fbe37fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3819d730 con 0x7fbe38101c40 2023-10-11T16:31:22.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe36ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101330 0x7fbe38198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:22.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe36ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101330 0x7fbe38198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57930/0 (socket says 172.21.15.111:57930) 2023-10-11T16:31:22.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe36ffd700 1 -- 172.21.15.111:0/11386542 learned_addr learned my addr 172.21.15.111:0/11386542 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:22.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe367fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe38101c40 0x7fbe38198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:22.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe377fe700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe3810a680 0x7fbe3819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:22.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe36ffd700 1 -- 172.21.15.111:0/11386542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe38101c40 msgr2=0x7fbe38198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:22.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe36ffd700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe38101c40 0x7fbe38198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe36ffd700 1 -- 172.21.15.111:0/11386542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe3810a680 msgr2=0x7fbe3819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:22.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe36ffd700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe3810a680 0x7fbe3819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:22.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.979+0000 7fbe36ffd700 1 -- 172.21.15.111:0/11386542 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe280095f0 con 0x7fbe38101330 2023-10-11T16:31:22.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.980+0000 7fbe36ffd700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101330 0x7fbe38198440 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000e9a0 tx=0x7fbe2000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:22.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.980+0000 7fbe2ffff700 1 -- 172.21.15.111:0/11386542 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20009790 con 0x7fbe38101330 2023-10-11T16:31:22.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.980+0000 7fbe2ffff700 1 -- 172.21.15.111:0/11386542 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe20010430 con 0x7fbe38101330 2023-10-11T16:31:22.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.980+0000 7fbe37fff700 1 -- 172.21.15.111:0/11386542 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe38110170 con 0x7fbe38101330 2023-10-11T16:31:22.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.980+0000 7fbe2ffff700 1 -- 172.21.15.111:0/11386542 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20018680 con 0x7fbe38101330 2023-10-11T16:31:22.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.980+0000 7fbe37fff700 1 -- 172.21.15.111:0/11386542 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3810ffe0 con 0x7fbe38101330 2023-10-11T16:31:22.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.981+0000 7fbe2dffb700 1 -- 172.21.15.111:0/11386542 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe38110300 con 0x7fbe38101330 2023-10-11T16:31:22.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.984+0000 7fbe2ffff700 1 -- 172.21.15.111:0/11386542 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fbe200105a0 con 0x7fbe38101330 2023-10-11T16:31:22.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.985+0000 7fbe2ffff700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbe240a1260 0x7fbe240a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.986+0000 7fbe2ffff700 1 -- 172.21.15.111:0/11386542 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbe20014070 con 0x7fbe38101330 2023-10-11T16:31:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.986+0000 7fbe367fc700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbe240a1260 0x7fbe240a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.987+0000 7fbe367fc700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbe240a1260 0x7fbe240a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbe38199a60 tx=0x7fbe28005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:22.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:22.990+0000 7fbe2ffff700 1 -- 172.21.15.111:0/11386542 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbe20051730 con 0x7fbe38101330 2023-10-11T16:31:23.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103206 got 249108103207 for osd.8 2023-10-11T16:31:23.012 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:23.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.228+0000 7fbe2dffb700 1 -- 172.21.15.111:0/11386542 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbe3804e8c0 con 0x7fbe38101330 2023-10-11T16:31:23.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.229+0000 7fbe2ffff700 1 -- 172.21.15.111:0/11386542 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbe20096200 con 0x7fbe38101330 2023-10-11T16:31:23.231 INFO:teuthology.orchestra.run.smithi111.stdout:107374182462 2023-10-11T16:31:23.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 -- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbe240a1260 msgr2=0x7fbe240a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:23.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbe240a1260 0x7fbe240a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbe38199a60 tx=0x7fbe28005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:23.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 -- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101330 msgr2=0x7fbe38198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:23.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101330 0x7fbe38198440 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000e9a0 tx=0x7fbe2000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:23.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 -- 172.21.15.111:0/11386542 shutdown_connections 2023-10-11T16:31:23.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fbe240a1260 0x7fbe240a3710 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:23.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe3810a680 0x7fbe3819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:23.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe38101330 0x7fbe38198440 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:23.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 --2- 172.21.15.111:0/11386542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe38101c40 0x7fbe38198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:23.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 -- 172.21.15.111:0/11386542 >> 172.21.15.111:0/11386542 conn(0x7fbe380761a0 msgr2=0x7fbe380ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:23.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 -- 172.21.15.111:0/11386542 shutdown_connections 2023-10-11T16:31:23.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:23.231+0000 7fbe2dffb700 1 -- 172.21.15.111:0/11386542 wait complete. 2023-10-11T16:31:23.439 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:23 smithi111 ceph-mon[98527]: pgmap v310: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2023-10-11T16:31:23.443 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920824 got 141733920827 for osd.4 2023-10-11T16:31:23.443 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:23.446 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:23.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:23 smithi150 ceph-mon[132217]: pgmap v310: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2023-10-11T16:31:23.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:23 smithi130 ceph-mon[131859]: pgmap v310: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2023-10-11T16:31:23.726 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:23.924 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:24.262 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182460 got 107374182462 for osd.3 2023-10-11T16:31:24.263 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:24.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:24 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/11386542' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-11T16:31:24.463 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:24 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/11386542' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-11T16:31:24.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:24 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/11386542' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-11T16:31:25.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:25 smithi111 ceph-mon[98527]: pgmap v311: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:31:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 -- 172.21.15.111:0/3539421980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c100ee0 msgr2=0x7fa86c104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3539421980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c100ee0 0x7fa86c104eb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009990 tx=0x7fa85c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 -- 172.21.15.111:0/3539421980 shutdown_connections 2023-10-11T16:31:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3539421980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86c1055e0 0x7fa86c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3539421980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa86c100540 0x7fa86c100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3539421980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c100ee0 0x7fa86c104eb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 -- 172.21.15.111:0/3539421980 >> 172.21.15.111:0/3539421980 conn(0x7fa86c0fbdc0 msgr2=0x7fa86c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:25.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 -- 172.21.15.111:0/3539421980 shutdown_connections 2023-10-11T16:31:25.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.397+0000 7fa873ed9700 1 -- 172.21.15.111:0/3539421980 wait complete. 2023-10-11T16:31:25.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.398+0000 7fa873ed9700 1 Processor -- start 2023-10-11T16:31:25.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.398+0000 7fa873ed9700 1 -- start start 2023-10-11T16:31:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.398+0000 7fa873ed9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c100540 0x7fa86c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.398+0000 7fa873ed9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa86c100ee0 0x7fa86c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.398+0000 7fa873ed9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86c1055e0 0x7fa86c0738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.398+0000 7fa873ed9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1a4850 con 0x7fa86c1055e0 2023-10-11T16:31:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.398+0000 7fa873ed9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1a49c0 con 0x7fa86c100540 2023-10-11T16:31:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa873ed9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1a4b80 con 0x7fa86c100ee0 2023-10-11T16:31:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa872476700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86c1055e0 0x7fa86c0738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa872476700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86c1055e0 0x7fa86c0738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53450/0 (socket says 172.21.15.111:53450) 2023-10-11T16:31:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa872476700 1 -- 172.21.15.111:0/3301197506 learned_addr learned my addr 172.21.15.111:0/3301197506 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa871474700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa86c100ee0 0x7fa86c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa871c75700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c100540 0x7fa86c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa872476700 1 -- 172.21.15.111:0/3301197506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa86c100ee0 msgr2=0x7fa86c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa872476700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa86c100ee0 0x7fa86c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa872476700 1 -- 172.21.15.111:0/3301197506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c100540 msgr2=0x7fa86c072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa872476700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c100540 0x7fa86c072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.399+0000 7fa872476700 1 -- 172.21.15.111:0/3301197506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0095f0 con 0x7fa86c1055e0 2023-10-11T16:31:25.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.400+0000 7fa872476700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86c1055e0 0x7fa86c0738b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa86400ef30 tx=0x7fa86400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:25.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.400+0000 7fa862ffd700 1 -- 172.21.15.111:0/3301197506 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864018070 con 0x7fa86c1055e0 2023-10-11T16:31:25.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.400+0000 7fa862ffd700 1 -- 172.21.15.111:0/3301197506 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa86400f040 con 0x7fa86c1055e0 2023-10-11T16:31:25.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.400+0000 7fa862ffd700 1 -- 172.21.15.111:0/3301197506 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86400cd00 con 0x7fa86c1055e0 2023-10-11T16:31:25.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.400+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c110360 con 0x7fa86c1055e0 2023-10-11T16:31:25.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.400+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c1101d0 con 0x7fa86c1055e0 2023-10-11T16:31:25.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.402+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c1104f0 con 0x7fa86c1055e0 2023-10-11T16:31:25.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.404+0000 7fa862ffd700 1 -- 172.21.15.111:0/3301197506 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fa864003a00 con 0x7fa86c1055e0 2023-10-11T16:31:25.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.405+0000 7fa862ffd700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa8580a1260 0x7fa8580a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:25.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.405+0000 7fa862ffd700 1 -- 172.21.15.111:0/3301197506 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa8640cc820 con 0x7fa86c1055e0 2023-10-11T16:31:25.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.406+0000 7fa871c75700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa8580a1260 0x7fa8580a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:25.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.407+0000 7fa871c75700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa8580a1260 0x7fa8580a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa868007be0 tx=0x7fa8680073d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:25.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.410+0000 7fa862ffd700 1 -- 172.21.15.111:0/3301197506 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa864095b30 con 0x7fa86c1055e0 2023-10-11T16:31:25.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:25 smithi150 ceph-mon[132217]: pgmap v311: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:31:25.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:25 smithi130 ceph-mon[131859]: pgmap v311: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:31:25.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.650+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa86c1a5570 con 0x7fa86c1055e0 2023-10-11T16:31:25.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.650+0000 7fa862ffd700 1 -- 172.21.15.111:0/3301197506 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa864099150 con 0x7fa86c1055e0 2023-10-11T16:31:25.652 INFO:teuthology.orchestra.run.smithi111.stdout:197568495665 2023-10-11T16:31:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa8580a1260 msgr2=0x7fa8580a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa8580a1260 0x7fa8580a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa868007be0 tx=0x7fa8680073d0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86c1055e0 msgr2=0x7fa86c0738b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86c1055e0 0x7fa86c0738b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa86400ef30 tx=0x7fa86400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 shutdown_connections 2023-10-11T16:31:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa8580a1260 0x7fa8580a3710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa86c1055e0 0x7fa86c0738b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa86c100540 0x7fa86c072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 --2- 172.21.15.111:0/3301197506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa86c100ee0 0x7fa86c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:25.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 >> 172.21.15.111:0/3301197506 conn(0x7fa86c0fbdc0 msgr2=0x7fa86c0fc840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:25.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 shutdown_connections 2023-10-11T16:31:25.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:25.653+0000 7fa873ed9700 1 -- 172.21.15.111:0/3301197506 wait complete. 2023-10-11T16:31:26.300 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:26 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3301197506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-11T16:31:26.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:26 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3301197506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-11T16:31:26.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:26 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3301197506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-11T16:31:26.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495662 got 197568495665 for osd.6 2023-10-11T16:31:26.739 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:26.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.990+0000 7fcd32227700 1 -- 172.21.15.111:0/2977155445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100540 msgr2=0x7fcd2c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:26.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.990+0000 7fcd32227700 1 --2- 172.21.15.111:0/2977155445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100540 0x7fcd2c100910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0099c0 tx=0x7fcd1c009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:26.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.991+0000 7fcd32227700 1 -- 172.21.15.111:0/2977155445 shutdown_connections 2023-10-11T16:31:26.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.991+0000 7fcd32227700 1 --2- 172.21.15.111:0/2977155445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd2c1055e0 0x7fcd2c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:26.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.991+0000 7fcd32227700 1 --2- 172.21.15.111:0/2977155445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100540 0x7fcd2c100910 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:26.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.991+0000 7fcd32227700 1 --2- 172.21.15.111:0/2977155445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2c100ee0 0x7fcd2c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.991+0000 7fcd32227700 1 -- 172.21.15.111:0/2977155445 >> 172.21.15.111:0/2977155445 conn(0x7fcd2c0fbdc0 msgr2=0x7fcd2c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.991+0000 7fcd32227700 1 -- 172.21.15.111:0/2977155445 shutdown_connections 2023-10-11T16:31:26.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.991+0000 7fcd32227700 1 -- 172.21.15.111:0/2977155445 wait complete. 2023-10-11T16:31:26.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd32227700 1 Processor -- start 2023-10-11T16:31:26.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd32227700 1 -- start start 2023-10-11T16:31:26.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd32227700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2c100540 0x7fcd2c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:26.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd32227700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100ee0 0x7fcd2c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:26.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd32227700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd2c1055e0 0x7fcd2c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:26.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd32227700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c19d5d0 con 0x7fcd2c100ee0 2023-10-11T16:31:26.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd32227700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c19d740 con 0x7fcd2c1055e0 2023-10-11T16:31:26.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd32227700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c19d900 con 0x7fcd2c100540 2023-10-11T16:31:26.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd2affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100ee0 0x7fcd2c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:26.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd2affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100ee0 0x7fcd2c198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53466/0 (socket says 172.21.15.111:53466) 2023-10-11T16:31:26.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.992+0000 7fcd2affd700 1 -- 172.21.15.111:0/1525337444 learned_addr learned my addr 172.21.15.111:0/1525337444 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:26.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2b7fe700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2c100540 0x7fcd2c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:26.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2bfff700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd2c1055e0 0x7fcd2c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:26.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2affd700 1 -- 172.21.15.111:0/1525337444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2c100540 msgr2=0x7fcd2c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:26.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2affd700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2c100540 0x7fcd2c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:26.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2affd700 1 -- 172.21.15.111:0/1525337444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd2c1055e0 msgr2=0x7fcd2c19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2affd700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd2c1055e0 0x7fcd2c19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2affd700 1 -- 172.21.15.111:0/1525337444 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c0095f0 con 0x7fcd2c100ee0 2023-10-11T16:31:27.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2bfff700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd2c1055e0 0x7fcd2c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:31:27.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd2affd700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100ee0 0x7fcd2c198b50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000e970 tx=0x7fcd2000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:27.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd28ff9700 1 -- 172.21.15.111:0/1525337444 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20009790 con 0x7fcd2c100ee0 2023-10-11T16:31:27.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd28ff9700 1 -- 172.21.15.111:0/1525337444 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd2000eec0 con 0x7fcd2c100ee0 2023-10-11T16:31:27.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c110370 con 0x7fcd2c100ee0 2023-10-11T16:31:27.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.993+0000 7fcd28ff9700 1 -- 172.21.15.111:0/1525337444 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20005590 con 0x7fcd2c100ee0 2023-10-11T16:31:27.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.994+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c1101e0 con 0x7fcd2c100ee0 2023-10-11T16:31:27.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.995+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c110500 con 0x7fcd2c100ee0 2023-10-11T16:31:27.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.996+0000 7fcd28ff9700 1 -- 172.21.15.111:0/1525337444 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fcd2000cbd0 con 0x7fcd2c100ee0 2023-10-11T16:31:27.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.998+0000 7fcd28ff9700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcd140a11e0 0x7fcd140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.998+0000 7fcd28ff9700 1 -- 172.21.15.111:0/1525337444 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcd20014070 con 0x7fcd2c100ee0 2023-10-11T16:31:27.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.998+0000 7fcd2b7fe700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcd140a11e0 0x7fcd140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:27.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:26.999+0000 7fcd2b7fe700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcd140a11e0 0x7fcd140a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009990 tx=0x7fcd1c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:27.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.002+0000 7fcd28ff9700 1 -- 172.21.15.111:0/1525337444 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcd200962d0 con 0x7fcd2c100ee0 2023-10-11T16:31:27.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.256+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fcd2c1a5750 con 0x7fcd2c100ee0 2023-10-11T16:31:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.257+0000 7fcd28ff9700 1 -- 172.21.15.111:0/1525337444 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fcd20004ef0 con 0x7fcd2c100ee0 2023-10-11T16:31:27.259 INFO:teuthology.orchestra.run.smithi111.stdout:304942678047 2023-10-11T16:31:27.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcd140a11e0 msgr2=0x7fcd140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcd140a11e0 0x7fcd140a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009990 tx=0x7fcd1c005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100ee0 msgr2=0x7fcd2c198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100ee0 0x7fcd2c198b50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000e970 tx=0x7fcd2000ec80 comp rx=0 tx=0).stop 2023-10-11T16:31:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 shutdown_connections 2023-10-11T16:31:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcd140a11e0 0x7fcd140a3690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcd2c1055e0 0x7fcd2c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2c100540 0x7fcd2c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 --2- 172.21.15.111:0/1525337444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd2c100ee0 0x7fcd2c198b50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 >> 172.21.15.111:0/1525337444 conn(0x7fcd2c0fbdc0 msgr2=0x7fcd2c0fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 shutdown_connections 2023-10-11T16:31:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.260+0000 7fcd32227700 1 -- 172.21.15.111:0/1525337444 wait complete. 2023-10-11T16:31:27.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:27 smithi111 ceph-mon[98527]: pgmap v312: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2023-10-11T16:31:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 -- 172.21.15.111:0/618154207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb880ff330 msgr2=0x7feb880ff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 --2- 172.21.15.111:0/618154207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb880ff330 0x7feb880ff700 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb840098f0 tx=0x7feb84009c00 comp rx=0 tx=0).stop 2023-10-11T16:31:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 -- 172.21.15.111:0/618154207 shutdown_connections 2023-10-11T16:31:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 --2- 172.21.15.111:0/618154207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb88104660 0x7feb88104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 --2- 172.21.15.111:0/618154207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb880ff330 0x7feb880ff700 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 --2- 172.21.15.111:0/618154207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb880ffc40 0x7feb88104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 -- 172.21.15.111:0/618154207 >> 172.21.15.111:0/618154207 conn(0x7feb880770a0 msgr2=0x7feb880774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:27.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 -- 172.21.15.111:0/618154207 shutdown_connections 2023-10-11T16:31:27.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7feb90390700 1 -- 172.21.15.111:0/618154207 wait complete. 2023-10-11T16:31:27.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7feb90390700 1 Processor -- start 2023-10-11T16:31:27.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7feb90390700 1 -- start start 2023-10-11T16:31:27.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7feb90390700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb880ff330 0x7feb881a2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7feb90390700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb880ffc40 0x7feb881a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7feb90390700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb88104660 0x7feb8819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7feb90390700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8819cf10 con 0x7feb880ff330 2023-10-11T16:31:27.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb90390700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8819d080 con 0x7feb88104660 2023-10-11T16:31:27.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb90390700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8819d240 con 0x7feb880ffc40 2023-10-11T16:31:27.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e92d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb88104660 0x7feb8819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:27.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e92d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb88104660 0x7feb8819c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:47284/0 (socket says 172.21.15.111:47284) 2023-10-11T16:31:27.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e92d700 1 -- 172.21.15.111:0/2127823319 learned_addr learned my addr 172.21.15.111:0/2127823319 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:27.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e12c700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb880ff330 0x7feb881a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:27.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e92d700 1 -- 172.21.15.111:0/2127823319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb880ffc40 msgr2=0x7feb881a2d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-11T16:31:27.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e92d700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb880ffc40 0x7feb881a2d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e92d700 1 -- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb880ff330 msgr2=0x7feb881a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:27 smithi150 ceph-mon[132217]: pgmap v312: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2023-10-11T16:31:27.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e92d700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb880ff330 0x7feb881a2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7feb8e92d700 1 -- 172.21.15.111:0/2127823319 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80009c40 con 0x7feb88104660 2023-10-11T16:31:27.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7feb8e92d700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb88104660 0x7feb8819c8a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb8000c370 tx=0x7feb8000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:27.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7feb7f7fe700 1 -- 172.21.15.111:0/2127823319 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80017070 con 0x7feb88104660 2023-10-11T16:31:27.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7feb7f7fe700 1 -- 172.21.15.111:0/2127823319 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb8000f040 con 0x7feb88104660 2023-10-11T16:31:27.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7feb7f7fe700 1 -- 172.21.15.111:0/2127823319 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000e050 con 0x7feb88104660 2023-10-11T16:31:27.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb84009520 con 0x7feb88104660 2023-10-11T16:31:27.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb88116950 con 0x7feb88104660 2023-10-11T16:31:27.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.449+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb881167c0 con 0x7feb88104660 2023-10-11T16:31:27.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.451+0000 7feb7f7fe700 1 -- 172.21.15.111:0/2127823319 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7feb800044e0 con 0x7feb88104660 2023-10-11T16:31:27.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.452+0000 7feb7f7fe700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feb740a1260 0x7feb740a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.452+0000 7feb7f7fe700 1 -- 172.21.15.111:0/2127823319 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feb800cc6f0 con 0x7feb88104660 2023-10-11T16:31:27.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.452+0000 7feb8e12c700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feb740a1260 0x7feb740a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:27.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.453+0000 7feb8e12c700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feb740a1260 0x7feb740a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feb84005e90 tx=0x7feb84019040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:27.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.456+0000 7feb7f7fe700 1 -- 172.21.15.111:0/2127823319 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feb80095970 con 0x7feb88104660 2023-10-11T16:31:27.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.443+0000 7f2dd3a86700 1 -- 172.21.15.111:0/461489144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100ee0 msgr2=0x7f2dcc104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.443+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/461489144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100ee0 0x7f2dcc104eb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0099c0 tx=0x7f2dbc009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.443+0000 7f2dd3a86700 1 -- 172.21.15.111:0/461489144 shutdown_connections 2023-10-11T16:31:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.443+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/461489144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dcc1055e0 0x7f2dcc10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.443+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/461489144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dcc100540 0x7f2dcc100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.443+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/461489144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100ee0 0x7f2dcc104eb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.443+0000 7f2dd3a86700 1 -- 172.21.15.111:0/461489144 >> 172.21.15.111:0/461489144 conn(0x7f2dcc0fbdc0 msgr2=0x7f2dcc0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:27.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7f2dd3a86700 1 -- 172.21.15.111:0/461489144 shutdown_connections 2023-10-11T16:31:27.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.444+0000 7f2dd3a86700 1 -- 172.21.15.111:0/461489144 wait complete. 2023-10-11T16:31:27.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7f2dd3a86700 1 Processor -- start 2023-10-11T16:31:27.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7f2dd3a86700 1 -- start start 2023-10-11T16:31:27.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7f2dd3a86700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100540 0x7f2dcc1071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.445+0000 7f2dd3a86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dcc100ee0 0x7f2dcc107720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd3a86700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dcc1055e0 0x7f2dcc107c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd3a86700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1a8d10 con 0x7f2dcc100ee0 2023-10-11T16:31:27.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd3a86700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1a8e80 con 0x7f2dcc1055e0 2023-10-11T16:31:27.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd3a86700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1a9040 con 0x7f2dcc100540 2023-10-11T16:31:27.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd1822700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100540 0x7f2dcc1071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:27.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd1822700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100540 0x7f2dcc1071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36020/0 (socket says 172.21.15.111:36020) 2023-10-11T16:31:27.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd1822700 1 -- 172.21.15.111:0/3930795823 learned_addr learned my addr 172.21.15.111:0/3930795823 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:27.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd1021700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dcc100ee0 0x7f2dcc107720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:27.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.446+0000 7f2dd2023700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dcc1055e0 0x7f2dcc107c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:27.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7f2dd1021700 1 -- 172.21.15.111:0/3930795823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100540 msgr2=0x7f2dcc1071e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7f2dd1021700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100540 0x7f2dcc1071e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7f2dd1021700 1 -- 172.21.15.111:0/3930795823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dcc1055e0 msgr2=0x7f2dcc107c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7f2dd1021700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dcc1055e0 0x7f2dcc107c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.447+0000 7f2dd1021700 1 -- 172.21.15.111:0/3930795823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0095f0 con 0x7f2dcc100ee0 2023-10-11T16:31:27.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.448+0000 7f2dd1021700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dcc100ee0 0x7f2dcc107720 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc000c00 tx=0x7f2dbc005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:27.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.448+0000 7f2dc2ffd700 1 -- 172.21.15.111:0/3930795823 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc00b790 con 0x7f2dcc100ee0 2023-10-11T16:31:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.448+0000 7f2dc2ffd700 1 -- 172.21.15.111:0/3930795823 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dbc02e070 con 0x7f2dcc100ee0 2023-10-11T16:31:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.448+0000 7f2dc2ffd700 1 -- 172.21.15.111:0/3930795823 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc033b60 con 0x7f2dcc100ee0 2023-10-11T16:31:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.448+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc110400 con 0x7f2dcc100ee0 2023-10-11T16:31:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.448+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc110270 con 0x7f2dcc100ee0 2023-10-11T16:31:27.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.450+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc110590 con 0x7f2dcc100ee0 2023-10-11T16:31:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.452+0000 7f2dc2ffd700 1 -- 172.21.15.111:0/3930795823 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f2dbc00b8f0 con 0x7f2dcc100ee0 2023-10-11T16:31:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.454+0000 7f2dc2ffd700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2db80a1260 0x7f2db80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.454+0000 7f2dc2ffd700 1 -- 172.21.15.111:0/3930795823 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2dbc0ded60 con 0x7f2dcc100ee0 2023-10-11T16:31:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.455+0000 7f2dd1822700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2db80a1260 0x7f2db80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:27.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.456+0000 7f2dd1822700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2db80a1260 0x7f2db80a3710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009d00 tx=0x7f2dc8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.458+0000 7f2dc2ffd700 1 -- 172.21.15.111:0/3930795823 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2dbc06d730 con 0x7f2dcc100ee0 2023-10-11T16:31:27.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:27 smithi130 ceph-mon[131859]: pgmap v312: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2023-10-11T16:31:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.682+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7feb88116ae0 con 0x7feb88104660 2023-10-11T16:31:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.683+0000 7feb7f7fe700 1 -- 172.21.15.111:0/2127823319 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb80098f90 con 0x7feb88104660 2023-10-11T16:31:27.685 INFO:teuthology.orchestra.run.smithi111.stdout:34359738445 2023-10-11T16:31:27.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feb740a1260 msgr2=0x7feb740a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feb740a1260 0x7feb740a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feb84005e90 tx=0x7feb84019040 comp rx=0 tx=0).stop 2023-10-11T16:31:27.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb88104660 msgr2=0x7feb8819c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb88104660 0x7feb8819c8a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb8000c370 tx=0x7feb8000c680 comp rx=0 tx=0).stop 2023-10-11T16:31:27.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 shutdown_connections 2023-10-11T16:31:27.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7feb740a1260 0x7feb740a3710 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb88104660 0x7feb8819c8a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb880ff330 0x7feb881a2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 --2- 172.21.15.111:0/2127823319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb880ffc40 0x7feb881a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.685+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 >> 172.21.15.111:0/2127823319 conn(0x7feb880770a0 msgr2=0x7feb88102250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:27.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.686+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 shutdown_connections 2023-10-11T16:31:27.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.686+0000 7feb90390700 1 -- 172.21.15.111:0/2127823319 wait complete. 2023-10-11T16:31:27.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.693+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2dcc066ea0 con 0x7f2dcc100ee0 2023-10-11T16:31:27.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.693+0000 7f2dc2ffd700 1 -- 172.21.15.111:0/3930795823 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2dbc0a8070 con 0x7f2dcc100ee0 2023-10-11T16:31:27.695 INFO:teuthology.orchestra.run.smithi111.stdout:55834574920 2023-10-11T16:31:27.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2db80a1260 msgr2=0x7f2db80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2db80a1260 0x7f2db80a3710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009d00 tx=0x7f2dc8009260 comp rx=0 tx=0).stop 2023-10-11T16:31:27.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dcc100ee0 msgr2=0x7f2dcc107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:27.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dcc100ee0 0x7f2dcc107720 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc000c00 tx=0x7f2dbc005370 comp rx=0 tx=0).stop 2023-10-11T16:31:27.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 shutdown_connections 2023-10-11T16:31:27.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f2db80a1260 0x7f2db80a3710 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dcc1055e0 0x7f2dcc107c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dcc100540 0x7f2dcc1071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 --2- 172.21.15.111:0/3930795823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dcc100ee0 0x7f2dcc107720 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:27.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 >> 172.21.15.111:0/3930795823 conn(0x7f2dcc0fbdc0 msgr2=0x7f2dcc0fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:27.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 shutdown_connections 2023-10-11T16:31:27.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:27.696+0000 7f2dd3a86700 1 -- 172.21.15.111:0/3930795823 wait complete. 2023-10-11T16:31:27.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678045 got 304942678047 for osd.10 2023-10-11T16:31:27.850 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:28.436 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:28 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1525337444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-10-11T16:31:28.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:28 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2127823319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-11T16:31:28.437 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:28 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3930795823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-11T16:31:28.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:28 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1525337444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-10-11T16:31:28.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:28 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2127823319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-11T16:31:28.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:28 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3930795823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-11T16:31:28.474 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738443 got 34359738445 for osd.0 2023-10-11T16:31:28.474 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:28.555 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574917 got 55834574920 for osd.1 2023-10-11T16:31:28.556 DEBUG:teuthology.parallel:result is None 2023-10-11T16:31:28.556 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-10-11T16:31:28.557 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph pg dump --format=json 2023-10-11T16:31:28.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1525337444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-10-11T16:31:28.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2127823319' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-11T16:31:28.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3930795823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-11T16:31:29.436 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:29.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:29 smithi150 ceph-mon[132217]: pgmap v313: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:31:29.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:29.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:29 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:29.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:29 smithi130 ceph-mon[131859]: pgmap v313: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:31:29.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:29.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:29 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:29.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:29 smithi111 ceph-mon[98527]: pgmap v313: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:31:29.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:29.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:29 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:30.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 -- 172.21.15.111:0/2160107972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100ee0 msgr2=0x7fafc8104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:30.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 --2- 172.21.15.111:0/2160107972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100ee0 0x7fafc8104eb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fafb80099c0 tx=0x7fafb8009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:30.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 -- 172.21.15.111:0/2160107972 shutdown_connections 2023-10-11T16:31:30.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 --2- 172.21.15.111:0/2160107972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc81055e0 0x7fafc810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:30.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 --2- 172.21.15.111:0/2160107972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafc8100540 0x7fafc8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:30.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 --2- 172.21.15.111:0/2160107972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100ee0 0x7fafc8104eb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:30.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 -- 172.21.15.111:0/2160107972 >> 172.21.15.111:0/2160107972 conn(0x7fafc80fbdc0 msgr2=0x7fafc80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:30.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 -- 172.21.15.111:0/2160107972 shutdown_connections 2023-10-11T16:31:30.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.835+0000 7fafcf912700 1 -- 172.21.15.111:0/2160107972 wait complete. 2023-10-11T16:31:30.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.837+0000 7fafcf912700 1 Processor -- start 2023-10-11T16:31:30.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.837+0000 7fafcf912700 1 -- start start 2023-10-11T16:31:30.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.837+0000 7fafcf912700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100540 0x7fafc81963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:30.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.837+0000 7fafcf912700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafc8100ee0 0x7fafc8196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:30.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcf912700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc81055e0 0x7fafc819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:30.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcf912700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc819b390 con 0x7fafc8100540 2023-10-11T16:31:30.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcf912700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc819b500 con 0x7fafc81055e0 2023-10-11T16:31:30.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcf912700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc819b6c0 con 0x7fafc8100ee0 2023-10-11T16:31:30.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcd6ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100540 0x7fafc81963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:30.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafccead700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafc8100ee0 0x7fafc8196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:30.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcd6ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100540 0x7fafc81963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53508/0 (socket says 172.21.15.111:53508) 2023-10-11T16:31:30.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcd6ae700 1 -- 172.21.15.111:0/3490708655 learned_addr learned my addr 172.21.15.111:0/3490708655 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:30.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcdeaf700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc81055e0 0x7fafc819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:30.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcd6ae700 1 -- 172.21.15.111:0/3490708655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafc8100ee0 msgr2=0x7fafc8196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:30.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcd6ae700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafc8100ee0 0x7fafc8196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:30.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcd6ae700 1 -- 172.21.15.111:0/3490708655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc81055e0 msgr2=0x7fafc819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:30.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcd6ae700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc81055e0 0x7fafc819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:30.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.838+0000 7fafcd6ae700 1 -- 172.21.15.111:0/3490708655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb80095f0 con 0x7fafc8100540 2023-10-11T16:31:30.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.839+0000 7fafcd6ae700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100540 0x7fafc81963d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e9a0 tx=0x7fafc400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:30.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.839+0000 7fafbe7fc700 1 -- 172.21.15.111:0/3490708655 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4009790 con 0x7fafc8100540 2023-10-11T16:31:30.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.839+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc8110360 con 0x7fafc8100540 2023-10-11T16:31:30.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.839+0000 7fafbe7fc700 1 -- 172.21.15.111:0/3490708655 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafc4010430 con 0x7fafc8100540 2023-10-11T16:31:30.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.839+0000 7fafbe7fc700 1 -- 172.21.15.111:0/3490708655 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc4018680 con 0x7fafc8100540 2023-10-11T16:31:30.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.839+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc81101d0 con 0x7fafc8100540 2023-10-11T16:31:30.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.840+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc81104f0 con 0x7fafc8100540 2023-10-11T16:31:30.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.842+0000 7fafbe7fc700 1 -- 172.21.15.111:0/3490708655 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fafc40105a0 con 0x7fafc8100540 2023-10-11T16:31:30.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.843+0000 7fafbe7fc700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafb40a1260 0x7fafb40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:30.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.843+0000 7fafbe7fc700 1 -- 172.21.15.111:0/3490708655 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fafc4014070 con 0x7fafc8100540 2023-10-11T16:31:30.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.843+0000 7fafccead700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafb40a1260 0x7fafb40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:30.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.844+0000 7fafccead700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafb40a1260 0x7fafb40a3710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fafb80098e0 tx=0x7fafb8000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:30.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:30.847+0000 7fafbe7fc700 1 -- 172.21.15.111:0/3490708655 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fafc4051730 con 0x7fafc8100540 2023-10-11T16:31:31.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.045+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafc80fcee0 con 0x7fafb40a1260 2023-10-11T16:31:31.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.059+0000 7fafbe7fc700 1 -- 172.21.15.111:0/3490708655 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427832 (secure 0 0 0) 0x7fafc80fcee0 con 0x7fafb40a1260 2023-10-11T16:31:31.067 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:31.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.064+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafb40a1260 msgr2=0x7fafb40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:31.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.064+0000 7fafcf912700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafb40a1260 0x7fafb40a3710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fafb80098e0 tx=0x7fafb8000bc0 comp rx=0 tx=0).stop 2023-10-11T16:31:31.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.064+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100540 msgr2=0x7fafc81963d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:31.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.064+0000 7fafcf912700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100540 0x7fafc81963d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e9a0 tx=0x7fafc400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:31.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.065+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 shutdown_connections 2023-10-11T16:31:31.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.065+0000 7fafcf912700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fafb40a1260 0x7fafb40a3710 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:31.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.065+0000 7fafcf912700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafc81055e0 0x7fafc819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:31.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.065+0000 7fafcf912700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fafc8100540 0x7fafc81963d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:31.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.065+0000 7fafcf912700 1 --2- 172.21.15.111:0/3490708655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafc8100ee0 0x7fafc8196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:31.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.065+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 >> 172.21.15.111:0/3490708655 conn(0x7fafc80fbdc0 msgr2=0x7fafc80fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:31.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.065+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 shutdown_connections 2023-10-11T16:31:31.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:31.065+0000 7fafcf912700 1 -- 172.21.15.111:0/3490708655 wait complete. 2023-10-11T16:31:31.073 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:31:31.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:31 smithi130 ceph-mon[131859]: pgmap v314: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:31.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:31 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:31:31.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:31 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:31:31.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:31 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:31:31.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:31 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:31.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:31 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:31:31.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:31 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:31:31.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:31 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:31:31.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:31 smithi111 ceph-mon[98527]: pgmap v314: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:31.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:31 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:31:31.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:31 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:31:31.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:31 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:31:31.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:31 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:31.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:31 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:31:31.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:31 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:31:31.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:31 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:31:31.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:31 smithi150 ceph-mon[132217]: pgmap v314: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:31.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:31 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:31:31.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:31 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:31:31.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:31 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:31:31.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:31 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' 2023-10-11T16:31:31.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:31 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:31:31.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:31 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:31:31.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:31 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:31:31.533 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2023-10-11T16:31:30.072513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670226,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":458,"num_read_kb":362,"num_write":347,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1220437,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81012,"kb_used_data":11860,"kb_used_omap":0,"kb_used_meta":69056,"kb_avail":1124860812,"statfs":{"total":1151940427776,"available":1151857471488,"internally_reserved":0,"allocated":12144640,"data_stored":7839683,"data_compressed":324562,"data_compressed_allocated":438272,"data_compressed_original":2396160,"omap_allocated":0,"internal_metadata":70713344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003124"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185713+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:30:41.185713+0000","last_peered":"2023-10-11T16:30:41.185713+0000","last_clean":"2023-10-11T16:30:41.185713+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:30:41.185713+0000","last_undegraded":"2023-10-11T16:30:41.185713+0000","last_fullsized":"2023-10-11T16:30:41.185713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:47:02.731639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185549+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:30:41.185549+0000","last_peered":"2023-10-11T16:30:41.185549+0000","last_clean":"2023-10-11T16:30:41.185549+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:30:41.185549+0000","last_undegraded":"2023-10-11T16:30:41.185549+0000","last_fullsized":"2023-10-11T16:30:41.185549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:04:44.769304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185196+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:30:41.185196+0000","last_peered":"2023-10-11T16:30:41.185196+0000","last_clean":"2023-10-11T16:30:41.185196+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:30:41.185196+0000","last_undegraded":"2023-10-11T16:30:41.185196+0000","last_fullsized":"2023-10-11T16:30:41.185196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:22:01.830634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188320+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:30:41.188320+0000","last_peered":"2023-10-11T16:30:41.188320+0000","last_clean":"2023-10-11T16:30:41.188320+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:30:41.188320+0000","last_undegraded":"2023-10-11T16:30:41.188320+0000","last_fullsized":"2023-10-11T16:30:41.188320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:21:55.362917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185730+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:30:41.185730+0000","last_peered":"2023-10-11T16:30:41.185730+0000","last_clean":"2023-10-11T16:30:41.185730+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:30:41.185730+0000","last_undegraded":"2023-10-11T16:30:41.185730+0000","last_fullsized":"2023-10-11T16:30:41.185730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:47.175695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189802+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:30:41.189802+0000","last_peered":"2023-10-11T16:30:41.189802+0000","last_clean":"2023-10-11T16:30:41.189802+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:30:41.189802+0000","last_undegraded":"2023-10-11T16:30:41.189802+0000","last_fullsized":"2023-10-11T16:30:41.189802+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:04:31.832644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182446+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:30:41.182446+0000","last_peered":"2023-10-11T16:30:41.182446+0000","last_clean":"2023-10-11T16:30:41.182446+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:30:41.182446+0000","last_undegraded":"2023-10-11T16:30:41.182446+0000","last_fullsized":"2023-10-11T16:30:41.182446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:15:28.747024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403438+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:30:41.403438+0000","last_peered":"2023-10-11T16:30:41.403438+0000","last_clean":"2023-10-11T16:30:41.403438+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:30:41.403438+0000","last_undegraded":"2023-10-11T16:30:41.403438+0000","last_fullsized":"2023-10-11T16:30:41.403438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:20:51.806951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185872+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:30:41.185872+0000","last_peered":"2023-10-11T16:30:41.185872+0000","last_clean":"2023-10-11T16:30:41.185872+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:30:41.185872+0000","last_undegraded":"2023-10-11T16:30:41.185872+0000","last_fullsized":"2023-10-11T16:30:41.185872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:07:38.837524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190769+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:30:41.190769+0000","last_peered":"2023-10-11T16:30:41.190769+0000","last_clean":"2023-10-11T16:30:41.190769+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:30:41.190769+0000","last_undegraded":"2023-10-11T16:30:41.190769+0000","last_fullsized":"2023-10-11T16:30:41.190769+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:17:25.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182922+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:30:41.182922+0000","last_peered":"2023-10-11T16:30:41.182922+0000","last_clean":"2023-10-11T16:30:41.182922+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:30:41.182922+0000","last_undegraded":"2023-10-11T16:30:41.182922+0000","last_fullsized":"2023-10-11T16:30:41.182922+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:16:24.814604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188720+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:30:41.188720+0000","last_peered":"2023-10-11T16:30:41.188720+0000","last_clean":"2023-10-11T16:30:41.188720+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:30:41.188720+0000","last_undegraded":"2023-10-11T16:30:41.188720+0000","last_fullsized":"2023-10-11T16:30:41.188720+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:33:23.932678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198123+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:30:41.198123+0000","last_peered":"2023-10-11T16:30:41.198123+0000","last_clean":"2023-10-11T16:30:41.198123+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:30:41.198123+0000","last_undegraded":"2023-10-11T16:30:41.198123+0000","last_fullsized":"2023-10-11T16:30:41.198123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:09:09.086960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184893+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:30:41.184893+0000","last_peered":"2023-10-11T16:30:41.184893+0000","last_clean":"2023-10-11T16:30:41.184893+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:30:41.184893+0000","last_undegraded":"2023-10-11T16:30:41.184893+0000","last_fullsized":"2023-10-11T16:30:41.184893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:06.789653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185441+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:30:41.185441+0000","last_peered":"2023-10-11T16:30:41.185441+0000","last_clean":"2023-10-11T16:30:41.185441+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:30:41.185441+0000","last_undegraded":"2023-10-11T16:30:41.185441+0000","last_fullsized":"2023-10-11T16:30:41.185441+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:22:47.667568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184863+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:30:41.184863+0000","last_peered":"2023-10-11T16:30:41.184863+0000","last_clean":"2023-10-11T16:30:41.184863+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:30:41.184863+0000","last_undegraded":"2023-10-11T16:30:41.184863+0000","last_fullsized":"2023-10-11T16:30:41.184863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:51:56.240736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185599+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:30:41.185599+0000","last_peered":"2023-10-11T16:30:41.185599+0000","last_clean":"2023-10-11T16:30:41.185599+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:30:41.185599+0000","last_undegraded":"2023-10-11T16:30:41.185599+0000","last_fullsized":"2023-10-11T16:30:41.185599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:34:17.358317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182794+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:30:41.182794+0000","last_peered":"2023-10-11T16:30:41.182794+0000","last_clean":"2023-10-11T16:30:41.182794+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:30:41.182794+0000","last_undegraded":"2023-10-11T16:30:41.182794+0000","last_fullsized":"2023-10-11T16:30:41.182794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:09:27.215051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185311+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:30:41.185311+0000","last_peered":"2023-10-11T16:30:41.185311+0000","last_clean":"2023-10-11T16:30:41.185311+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:30:41.185311+0000","last_undegraded":"2023-10-11T16:30:41.185311+0000","last_fullsized":"2023-10-11T16:30:41.185311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:46:35.027109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186180+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.186180+0000","last_peered":"2023-10-11T16:30:41.186180+0000","last_clean":"2023-10-11T16:30:41.186180+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:30:41.186180+0000","last_undegraded":"2023-10-11T16:30:41.186180+0000","last_fullsized":"2023-10-11T16:30:41.186180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:05.403679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198854+0000","last_change":"2023-10-11T16:30:41.198854+0000","last_active":"2023-10-11T16:30:41.198854+0000","last_peered":"2023-10-11T16:30:41.198854+0000","last_clean":"2023-10-11T16:30:41.198854+0000","last_became_active":"2023-10-11T16:30:41.189677+0000","last_became_peered":"2023-10-11T16:30:41.189677+0000","last_unstale":"2023-10-11T16:30:41.198854+0000","last_undegraded":"2023-10-11T16:30:41.198854+0000","last_fullsized":"2023-10-11T16:30:41.198854+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:58:40.952333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403443+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:30:41.403443+0000","last_peered":"2023-10-11T16:30:41.403443+0000","last_clean":"2023-10-11T16:30:41.403443+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:30:41.403443+0000","last_undegraded":"2023-10-11T16:30:41.403443+0000","last_fullsized":"2023-10-11T16:30:41.403443+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:16:44.616541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185309+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:30:41.185309+0000","last_peered":"2023-10-11T16:30:41.185309+0000","last_clean":"2023-10-11T16:30:41.185309+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:30:41.185309+0000","last_undegraded":"2023-10-11T16:30:41.185309+0000","last_fullsized":"2023-10-11T16:30:41.185309+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:33:41.484467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403809+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:30:41.403809+0000","last_peered":"2023-10-11T16:30:41.403809+0000","last_clean":"2023-10-11T16:30:41.403809+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:30:41.403809+0000","last_undegraded":"2023-10-11T16:30:41.403809+0000","last_fullsized":"2023-10-11T16:30:41.403809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:05:06.232815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185766+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:30:41.185766+0000","last_peered":"2023-10-11T16:30:41.185766+0000","last_clean":"2023-10-11T16:30:41.185766+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:30:41.185766+0000","last_undegraded":"2023-10-11T16:30:41.185766+0000","last_fullsized":"2023-10-11T16:30:41.185766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:16.994395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188044+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:30:41.188044+0000","last_peered":"2023-10-11T16:30:41.188044+0000","last_clean":"2023-10-11T16:30:41.188044+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:30:41.188044+0000","last_undegraded":"2023-10-11T16:30:41.188044+0000","last_fullsized":"2023-10-11T16:30:41.188044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:30:57.536276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186101+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:30:41.186101+0000","last_peered":"2023-10-11T16:30:41.186101+0000","last_clean":"2023-10-11T16:30:41.186101+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:30:41.186101+0000","last_undegraded":"2023-10-11T16:30:41.186101+0000","last_fullsized":"2023-10-11T16:30:41.186101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:11:55.996038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185713+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:30:41.185713+0000","last_peered":"2023-10-11T16:30:41.185713+0000","last_clean":"2023-10-11T16:30:41.185713+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:30:41.185713+0000","last_undegraded":"2023-10-11T16:30:41.185713+0000","last_fullsized":"2023-10-11T16:30:41.185713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:31:18.914747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184425+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:30:41.184425+0000","last_peered":"2023-10-11T16:30:41.184425+0000","last_clean":"2023-10-11T16:30:41.184425+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:30:41.184425+0000","last_undegraded":"2023-10-11T16:30:41.184425+0000","last_fullsized":"2023-10-11T16:30:41.184425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:57:53.652251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186025+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:30:41.186025+0000","last_peered":"2023-10-11T16:30:41.186025+0000","last_clean":"2023-10-11T16:30:41.186025+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:30:41.186025+0000","last_undegraded":"2023-10-11T16:30:41.186025+0000","last_fullsized":"2023-10-11T16:30:41.186025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:46:36.549662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.195736+0000","last_change":"2023-10-11T16:30:41.195736+0000","last_active":"2023-10-11T16:30:41.195736+0000","last_peered":"2023-10-11T16:30:41.195736+0000","last_clean":"2023-10-11T16:30:41.195736+0000","last_became_active":"2023-10-11T16:30:41.195352+0000","last_became_peered":"2023-10-11T16:30:41.195352+0000","last_unstale":"2023-10-11T16:30:41.195736+0000","last_undegraded":"2023-10-11T16:30:41.195736+0000","last_fullsized":"2023-10-11T16:30:41.195736+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:00:25.607770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185067+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:30:41.185067+0000","last_peered":"2023-10-11T16:30:41.185067+0000","last_clean":"2023-10-11T16:30:41.185067+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:30:41.185067+0000","last_undegraded":"2023-10-11T16:30:41.185067+0000","last_fullsized":"2023-10-11T16:30:41.185067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:48.316789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'8","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:24.259045+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:31:24.259045+0000","last_peered":"2023-10-11T16:31:24.259045+0000","last_clean":"2023-10-11T16:31:24.259045+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:31:24.259045+0000","last_undegraded":"2023-10-11T16:31:24.259045+0000","last_fullsized":"2023-10-11T16:31:24.259045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:32:24.986353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10572,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":12,"num_write":5,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186484+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:30:41.186484+0000","last_peered":"2023-10-11T16:30:41.186484+0000","last_clean":"2023-10-11T16:30:41.186484+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:30:41.186484+0000","last_undegraded":"2023-10-11T16:30:41.186484+0000","last_fullsized":"2023-10-11T16:30:41.186484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:39:49.981983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186478+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:30:41.186478+0000","last_peered":"2023-10-11T16:30:41.186478+0000","last_clean":"2023-10-11T16:30:41.186478+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:30:41.186478+0000","last_undegraded":"2023-10-11T16:30:41.186478+0000","last_fullsized":"2023-10-11T16:30:41.186478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:20:03.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189794+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:30:41.189794+0000","last_peered":"2023-10-11T16:30:41.189794+0000","last_clean":"2023-10-11T16:30:41.189794+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:30:41.189794+0000","last_undegraded":"2023-10-11T16:30:41.189794+0000","last_fullsized":"2023-10-11T16:30:41.189794+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:08.968090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189977+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:30:41.189977+0000","last_peered":"2023-10-11T16:30:41.189977+0000","last_clean":"2023-10-11T16:30:41.189977+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:30:41.189977+0000","last_undegraded":"2023-10-11T16:30:41.189977+0000","last_fullsized":"2023-10-11T16:30:41.189977+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:41:46.444179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185463+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:30:41.185463+0000","last_peered":"2023-10-11T16:30:41.185463+0000","last_clean":"2023-10-11T16:30:41.185463+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:30:41.185463+0000","last_undegraded":"2023-10-11T16:30:41.185463+0000","last_fullsized":"2023-10-11T16:30:41.185463+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:16:11.533226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.197973+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:30:41.197973+0000","last_peered":"2023-10-11T16:30:41.197973+0000","last_clean":"2023-10-11T16:30:41.197973+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:30:41.197973+0000","last_undegraded":"2023-10-11T16:30:41.197973+0000","last_fullsized":"2023-10-11T16:30:41.197973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:49:04.403007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403246+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:30:41.403246+0000","last_peered":"2023-10-11T16:30:41.403246+0000","last_clean":"2023-10-11T16:30:41.403246+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:30:41.403246+0000","last_undegraded":"2023-10-11T16:30:41.403246+0000","last_fullsized":"2023-10-11T16:30:41.403246+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:18:58.866162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185429+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:30:41.185429+0000","last_peered":"2023-10-11T16:30:41.185429+0000","last_clean":"2023-10-11T16:30:41.185429+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:30:41.185429+0000","last_undegraded":"2023-10-11T16:30:41.185429+0000","last_fullsized":"2023-10-11T16:30:41.185429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:49.007168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189537+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:30:41.189537+0000","last_peered":"2023-10-11T16:30:41.189537+0000","last_clean":"2023-10-11T16:30:41.189537+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:30:41.189537+0000","last_undegraded":"2023-10-11T16:30:41.189537+0000","last_fullsized":"2023-10-11T16:30:41.189537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:46:32.350328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181702+0000","last_change":"2023-10-11T16:30:11.662850+0000","last_active":"2023-10-11T16:30:41.181702+0000","last_peered":"2023-10-11T16:30:41.181702+0000","last_clean":"2023-10-11T16:30:41.181702+0000","last_became_active":"2023-10-11T16:30:11.661052+0000","last_became_peered":"2023-10-11T16:30:11.661052+0000","last_unstale":"2023-10-11T16:30:41.181702+0000","last_undegraded":"2023-10-11T16:30:41.181702+0000","last_fullsized":"2023-10-11T16:30:41.181702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:23.324421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403234+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:30:41.403234+0000","last_peered":"2023-10-11T16:30:41.403234+0000","last_clean":"2023-10-11T16:30:41.403234+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:30:41.403234+0000","last_undegraded":"2023-10-11T16:30:41.403234+0000","last_fullsized":"2023-10-11T16:30:41.403234+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:53:00.713151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:26.261157+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:31:26.261157+0000","last_peered":"2023-10-11T16:31:26.261157+0000","last_clean":"2023-10-11T16:31:26.261157+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:31:26.261157+0000","last_undegraded":"2023-10-11T16:31:26.261157+0000","last_fullsized":"2023-10-11T16:31:26.261157+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:21:03.094344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185994+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:30:41.185994+0000","last_peered":"2023-10-11T16:30:41.185994+0000","last_clean":"2023-10-11T16:30:41.185994+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:30:41.185994+0000","last_undegraded":"2023-10-11T16:30:41.185994+0000","last_fullsized":"2023-10-11T16:30:41.185994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:51:15.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181694+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:30:41.181694+0000","last_peered":"2023-10-11T16:30:41.181694+0000","last_clean":"2023-10-11T16:30:41.181694+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:30:41.181694+0000","last_undegraded":"2023-10-11T16:30:41.181694+0000","last_fullsized":"2023-10-11T16:30:41.181694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:32.776111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183006+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:30:41.183006+0000","last_peered":"2023-10-11T16:30:41.183006+0000","last_clean":"2023-10-11T16:30:41.183006+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:30:41.183006+0000","last_undegraded":"2023-10-11T16:30:41.183006+0000","last_fullsized":"2023-10-11T16:30:41.183006+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:50:47.408054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185791+0000","last_change":"2023-10-11T16:30:11.666124+0000","last_active":"2023-10-11T16:30:41.185791+0000","last_peered":"2023-10-11T16:30:41.185791+0000","last_clean":"2023-10-11T16:30:41.185791+0000","last_became_active":"2023-10-11T16:30:11.665786+0000","last_became_peered":"2023-10-11T16:30:11.665786+0000","last_unstale":"2023-10-11T16:30:41.185791+0000","last_undegraded":"2023-10-11T16:30:41.185791+0000","last_fullsized":"2023-10-11T16:30:41.185791+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:16:08.621915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188345+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:30:41.188345+0000","last_peered":"2023-10-11T16:30:41.188345+0000","last_clean":"2023-10-11T16:30:41.188345+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:30:41.188345+0000","last_undegraded":"2023-10-11T16:30:41.188345+0000","last_fullsized":"2023-10-11T16:30:41.188345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:49.162491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190222+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:30:41.190222+0000","last_peered":"2023-10-11T16:30:41.190222+0000","last_clean":"2023-10-11T16:30:41.190222+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:30:41.190222+0000","last_undegraded":"2023-10-11T16:30:41.190222+0000","last_fullsized":"2023-10-11T16:30:41.190222+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:23:11.290409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186265+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:30:41.186265+0000","last_peered":"2023-10-11T16:30:41.186265+0000","last_clean":"2023-10-11T16:30:41.186265+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:30:41.186265+0000","last_undegraded":"2023-10-11T16:30:41.186265+0000","last_fullsized":"2023-10-11T16:30:41.186265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:59:10.188855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181930+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:30:41.181930+0000","last_peered":"2023-10-11T16:30:41.181930+0000","last_clean":"2023-10-11T16:30:41.181930+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:30:41.181930+0000","last_undegraded":"2023-10-11T16:30:41.181930+0000","last_fullsized":"2023-10-11T16:30:41.181930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:27:45.123842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":185,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:27.262734+0000","last_change":"2023-10-11T16:30:41.788994+0000","last_active":"2023-10-11T16:31:27.262734+0000","last_peered":"2023-10-11T16:31:27.262734+0000","last_clean":"2023-10-11T16:31:27.262734+0000","last_became_active":"2023-10-11T16:30:41.189949+0000","last_became_peered":"2023-10-11T16:30:41.189949+0000","last_unstale":"2023-10-11T16:31:27.262734+0000","last_undegraded":"2023-10-11T16:31:27.262734+0000","last_fullsized":"2023-10-11T16:31:27.262734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:36:50.095047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66100,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":118,"num_write":36,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38515,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186506+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:30:41.186506+0000","last_peered":"2023-10-11T16:30:41.186506+0000","last_clean":"2023-10-11T16:30:41.186506+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:30:41.186506+0000","last_undegraded":"2023-10-11T16:30:41.186506+0000","last_fullsized":"2023-10-11T16:30:41.186506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:26:09.244377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190731+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:30:41.190731+0000","last_peered":"2023-10-11T16:30:41.190731+0000","last_clean":"2023-10-11T16:30:41.190731+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:30:41.190731+0000","last_undegraded":"2023-10-11T16:30:41.190731+0000","last_fullsized":"2023-10-11T16:30:41.190731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:10:05.160429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185142+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:30:41.185142+0000","last_peered":"2023-10-11T16:30:41.185142+0000","last_clean":"2023-10-11T16:30:41.185142+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:30:41.185142+0000","last_undegraded":"2023-10-11T16:30:41.185142+0000","last_fullsized":"2023-10-11T16:30:41.185142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:25:53.623192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198771+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:30:41.198771+0000","last_peered":"2023-10-11T16:30:41.198771+0000","last_clean":"2023-10-11T16:30:41.198771+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:30:41.198771+0000","last_undegraded":"2023-10-11T16:30:41.198771+0000","last_fullsized":"2023-10-11T16:30:41.198771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:09:22.911180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184864+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:30:41.184864+0000","last_peered":"2023-10-11T16:30:41.184864+0000","last_clean":"2023-10-11T16:30:41.184864+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:30:41.184864+0000","last_undegraded":"2023-10-11T16:30:41.184864+0000","last_fullsized":"2023-10-11T16:30:41.184864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:28:35.220071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190012+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:30:41.190012+0000","last_peered":"2023-10-11T16:30:41.190012+0000","last_clean":"2023-10-11T16:30:41.190012+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:30:41.190012+0000","last_undegraded":"2023-10-11T16:30:41.190012+0000","last_fullsized":"2023-10-11T16:30:41.190012+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:18:02.905266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183098+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:30:41.183098+0000","last_peered":"2023-10-11T16:30:41.183098+0000","last_clean":"2023-10-11T16:30:41.183098+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:30:41.183098+0000","last_undegraded":"2023-10-11T16:30:41.183098+0000","last_fullsized":"2023-10-11T16:30:41.183098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:13:30.957186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182281+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.182281+0000","last_peered":"2023-10-11T16:30:41.182281+0000","last_clean":"2023-10-11T16:30:41.182281+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:30:41.182281+0000","last_undegraded":"2023-10-11T16:30:41.182281+0000","last_fullsized":"2023-10-11T16:30:41.182281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:35:47.449805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:28.264672+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:31:28.264672+0000","last_peered":"2023-10-11T16:31:28.264672+0000","last_clean":"2023-10-11T16:31:28.264672+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:31:28.264672+0000","last_undegraded":"2023-10-11T16:31:28.264672+0000","last_fullsized":"2023-10-11T16:31:28.264672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:49:34.210642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182403+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:30:41.182403+0000","last_peered":"2023-10-11T16:30:41.182403+0000","last_clean":"2023-10-11T16:30:41.182403+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:30:41.182403+0000","last_undegraded":"2023-10-11T16:30:41.182403+0000","last_fullsized":"2023-10-11T16:30:41.182403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:49:06.683312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198932+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:30:41.198932+0000","last_peered":"2023-10-11T16:30:41.198932+0000","last_clean":"2023-10-11T16:30:41.198932+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:30:41.198932+0000","last_undegraded":"2023-10-11T16:30:41.198932+0000","last_fullsized":"2023-10-11T16:30:41.198932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:38:08.422304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.187119+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:30:41.187119+0000","last_peered":"2023-10-11T16:30:41.187119+0000","last_clean":"2023-10-11T16:30:41.187119+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:30:41.187119+0000","last_undegraded":"2023-10-11T16:30:41.187119+0000","last_fullsized":"2023-10-11T16:30:41.187119+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:22:31.437710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189360+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:30:41.189360+0000","last_peered":"2023-10-11T16:30:41.189360+0000","last_clean":"2023-10-11T16:30:41.189360+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:30:41.189360+0000","last_undegraded":"2023-10-11T16:30:41.189360+0000","last_fullsized":"2023-10-11T16:30:41.189360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:58.907731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189214+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:30:41.189214+0000","last_peered":"2023-10-11T16:30:41.189214+0000","last_clean":"2023-10-11T16:30:41.189214+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:30:41.189214+0000","last_undegraded":"2023-10-11T16:30:41.189214+0000","last_fullsized":"2023-10-11T16:30:41.189214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:29:15.740818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.187919+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:30:41.187919+0000","last_peered":"2023-10-11T16:30:41.187919+0000","last_clean":"2023-10-11T16:30:41.187919+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:30:41.187919+0000","last_undegraded":"2023-10-11T16:30:41.187919+0000","last_fullsized":"2023-10-11T16:30:41.187919+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:49:28.444855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185524+0000","last_change":"2023-10-11T16:30:11.670547+0000","last_active":"2023-10-11T16:30:41.185524+0000","last_peered":"2023-10-11T16:30:41.185524+0000","last_clean":"2023-10-11T16:30:41.185524+0000","last_became_active":"2023-10-11T16:30:11.669842+0000","last_became_peered":"2023-10-11T16:30:11.669842+0000","last_unstale":"2023-10-11T16:30:41.185524+0000","last_undegraded":"2023-10-11T16:30:41.185524+0000","last_fullsized":"2023-10-11T16:30:41.185524+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:00:14.369445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186325+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:30:41.186325+0000","last_peered":"2023-10-11T16:30:41.186325+0000","last_clean":"2023-10-11T16:30:41.186325+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:30:41.186325+0000","last_undegraded":"2023-10-11T16:30:41.186325+0000","last_fullsized":"2023-10-11T16:30:41.186325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:54:10.393416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182101+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:30:41.182101+0000","last_peered":"2023-10-11T16:30:41.182101+0000","last_clean":"2023-10-11T16:30:41.182101+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:30:41.182101+0000","last_undegraded":"2023-10-11T16:30:41.182101+0000","last_fullsized":"2023-10-11T16:30:41.182101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:36.264713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190141+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:30:41.190141+0000","last_peered":"2023-10-11T16:30:41.190141+0000","last_clean":"2023-10-11T16:30:41.190141+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:30:41.190141+0000","last_undegraded":"2023-10-11T16:30:41.190141+0000","last_fullsized":"2023-10-11T16:30:41.190141+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:23:03.581102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198172+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:30:41.198172+0000","last_peered":"2023-10-11T16:30:41.198172+0000","last_clean":"2023-10-11T16:30:41.198172+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:30:41.198172+0000","last_undegraded":"2023-10-11T16:30:41.198172+0000","last_fullsized":"2023-10-11T16:30:41.198172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:24:19.511542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182736+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:30:41.182736+0000","last_peered":"2023-10-11T16:30:41.182736+0000","last_clean":"2023-10-11T16:30:41.182736+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:30:41.182736+0000","last_undegraded":"2023-10-11T16:30:41.182736+0000","last_fullsized":"2023-10-11T16:30:41.182736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:10:16.893854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190364+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:30:41.190364+0000","last_peered":"2023-10-11T16:30:41.190364+0000","last_clean":"2023-10-11T16:30:41.190364+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:30:41.190364+0000","last_undegraded":"2023-10-11T16:30:41.190364+0000","last_fullsized":"2023-10-11T16:30:41.190364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:38:35.761085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198257+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:30:41.198257+0000","last_peered":"2023-10-11T16:30:41.198257+0000","last_clean":"2023-10-11T16:30:41.198257+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:30:41.198257+0000","last_undegraded":"2023-10-11T16:30:41.198257+0000","last_fullsized":"2023-10-11T16:30:41.198257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:21:50.405889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189800+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:30:41.189800+0000","last_peered":"2023-10-11T16:30:41.189800+0000","last_clean":"2023-10-11T16:30:41.189800+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:30:41.189800+0000","last_undegraded":"2023-10-11T16:30:41.189800+0000","last_fullsized":"2023-10-11T16:30:41.189800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:41:33.198602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:26.261259+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:31:26.261259+0000","last_peered":"2023-10-11T16:31:26.261259+0000","last_clean":"2023-10-11T16:31:26.261259+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:31:26.261259+0000","last_undegraded":"2023-10-11T16:31:26.261259+0000","last_fullsized":"2023-10-11T16:31:26.261259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:37:23.548227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182645+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:30:41.182645+0000","last_peered":"2023-10-11T16:30:41.182645+0000","last_clean":"2023-10-11T16:30:41.182645+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:30:41.182645+0000","last_undegraded":"2023-10-11T16:30:41.182645+0000","last_fullsized":"2023-10-11T16:30:41.182645+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:40:48.239885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403663+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:30:41.403663+0000","last_peered":"2023-10-11T16:30:41.403663+0000","last_clean":"2023-10-11T16:30:41.403663+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:30:41.403663+0000","last_undegraded":"2023-10-11T16:30:41.403663+0000","last_fullsized":"2023-10-11T16:30:41.403663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:58:23.479875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188194+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:30:41.188194+0000","last_peered":"2023-10-11T16:30:41.188194+0000","last_clean":"2023-10-11T16:30:41.188194+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:30:41.188194+0000","last_undegraded":"2023-10-11T16:30:41.188194+0000","last_fullsized":"2023-10-11T16:30:41.188194+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:03.654864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182242+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:30:41.182242+0000","last_peered":"2023-10-11T16:30:41.182242+0000","last_clean":"2023-10-11T16:30:41.182242+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:30:41.182242+0000","last_undegraded":"2023-10-11T16:30:41.182242+0000","last_fullsized":"2023-10-11T16:30:41.182242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:11:53.750545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189957+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:30:41.189957+0000","last_peered":"2023-10-11T16:30:41.189957+0000","last_clean":"2023-10-11T16:30:41.189957+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:30:41.189957+0000","last_undegraded":"2023-10-11T16:30:41.189957+0000","last_fullsized":"2023-10-11T16:30:41.189957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:27:51.812285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.197783+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:30:41.197783+0000","last_peered":"2023-10-11T16:30:41.197783+0000","last_clean":"2023-10-11T16:30:41.197783+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:30:41.197783+0000","last_undegraded":"2023-10-11T16:30:41.197783+0000","last_fullsized":"2023-10-11T16:30:41.197783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:40:50.386941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.938949+0000","last_change":"2023-10-11T16:30:41.938949+0000","last_active":"2023-10-11T16:30:41.938949+0000","last_peered":"2023-10-11T16:30:41.938949+0000","last_clean":"2023-10-11T16:30:41.938949+0000","last_became_active":"2023-10-11T16:30:41.192902+0000","last_became_peered":"2023-10-11T16:30:41.192902+0000","last_unstale":"2023-10-11T16:30:41.938949+0000","last_undegraded":"2023-10-11T16:30:41.938949+0000","last_fullsized":"2023-10-11T16:30:41.938949+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:08:59.943869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186174+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:30:41.186174+0000","last_peered":"2023-10-11T16:30:41.186174+0000","last_clean":"2023-10-11T16:30:41.186174+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:30:41.186174+0000","last_undegraded":"2023-10-11T16:30:41.186174+0000","last_fullsized":"2023-10-11T16:30:41.186174+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:51:47.458050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.205885+0000","last_change":"2023-10-11T16:30:41.205550+0000","last_active":"2023-10-11T16:30:41.205885+0000","last_peered":"2023-10-11T16:30:41.205885+0000","last_clean":"2023-10-11T16:30:41.205885+0000","last_became_active":"2023-10-11T16:30:41.190345+0000","last_became_peered":"2023-10-11T16:30:41.190345+0000","last_unstale":"2023-10-11T16:30:41.205885+0000","last_undegraded":"2023-10-11T16:30:41.205885+0000","last_fullsized":"2023-10-11T16:30:41.205885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:21:53.506003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198393+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:30:41.198393+0000","last_peered":"2023-10-11T16:30:41.198393+0000","last_clean":"2023-10-11T16:30:41.198393+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:30:41.198393+0000","last_undegraded":"2023-10-11T16:30:41.198393+0000","last_fullsized":"2023-10-11T16:30:41.198393+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:25:27.458254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403257+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:30:41.403257+0000","last_peered":"2023-10-11T16:30:41.403257+0000","last_clean":"2023-10-11T16:30:41.403257+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:30:41.403257+0000","last_undegraded":"2023-10-11T16:30:41.403257+0000","last_fullsized":"2023-10-11T16:30:41.403257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:32:39.048475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:45.562548+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:30:45.562548+0000","last_peered":"2023-10-11T16:30:45.562548+0000","last_clean":"2023-10-11T16:30:45.562548+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:30:45.562548+0000","last_undegraded":"2023-10-11T16:30:45.562548+0000","last_fullsized":"2023-10-11T16:30:45.562548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:35.861737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403786+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:30:41.403786+0000","last_peered":"2023-10-11T16:30:41.403786+0000","last_clean":"2023-10-11T16:30:41.403786+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:30:41.403786+0000","last_undegraded":"2023-10-11T16:30:41.403786+0000","last_fullsized":"2023-10-11T16:30:41.403786+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:10:56.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185784+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:30:41.185784+0000","last_peered":"2023-10-11T16:30:41.185784+0000","last_clean":"2023-10-11T16:30:41.185784+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:30:41.185784+0000","last_undegraded":"2023-10-11T16:30:41.185784+0000","last_fullsized":"2023-10-11T16:30:41.185784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:57:36.382181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189399+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:30:41.189399+0000","last_peered":"2023-10-11T16:30:41.189399+0000","last_clean":"2023-10-11T16:30:41.189399+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:30:41.189399+0000","last_undegraded":"2023-10-11T16:30:41.189399+0000","last_fullsized":"2023-10-11T16:30:41.189399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:06.520839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185604+0000","last_change":"2023-10-11T16:30:11.661388+0000","last_active":"2023-10-11T16:30:41.185604+0000","last_peered":"2023-10-11T16:30:41.185604+0000","last_clean":"2023-10-11T16:30:41.185604+0000","last_became_active":"2023-10-11T16:30:11.661073+0000","last_became_peered":"2023-10-11T16:30:11.661073+0000","last_unstale":"2023-10-11T16:30:41.185604+0000","last_undegraded":"2023-10-11T16:30:41.185604+0000","last_fullsized":"2023-10-11T16:30:41.185604+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:12.390368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.193489+0000","last_change":"2023-10-11T16:30:41.193489+0000","last_active":"2023-10-11T16:30:41.193489+0000","last_peered":"2023-10-11T16:30:41.193489+0000","last_clean":"2023-10-11T16:30:41.193489+0000","last_became_active":"2023-10-11T16:30:41.193060+0000","last_became_peered":"2023-10-11T16:30:41.193060+0000","last_unstale":"2023-10-11T16:30:41.193489+0000","last_undegraded":"2023-10-11T16:30:41.193489+0000","last_fullsized":"2023-10-11T16:30:41.193489+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:52:34.859402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185001+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:30:41.185001+0000","last_peered":"2023-10-11T16:30:41.185001+0000","last_clean":"2023-10-11T16:30:41.185001+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:30:41.185001+0000","last_undegraded":"2023-10-11T16:30:41.185001+0000","last_fullsized":"2023-10-11T16:30:41.185001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:01:22.571040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185793+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:30:41.185793+0000","last_peered":"2023-10-11T16:30:41.185793+0000","last_clean":"2023-10-11T16:30:41.185793+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:30:41.185793+0000","last_undegraded":"2023-10-11T16:30:41.185793+0000","last_fullsized":"2023-10-11T16:30:41.185793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:22:41.435586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.405455+0000","last_change":"2023-10-11T16:30:41.404891+0000","last_active":"2023-10-11T16:30:41.405455+0000","last_peered":"2023-10-11T16:30:41.405455+0000","last_clean":"2023-10-11T16:30:41.405455+0000","last_became_active":"2023-10-11T16:30:41.404396+0000","last_became_peered":"2023-10-11T16:30:41.404396+0000","last_unstale":"2023-10-11T16:30:41.405455+0000","last_undegraded":"2023-10-11T16:30:41.405455+0000","last_fullsized":"2023-10-11T16:30:41.405455+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:39:56.108444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403349+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:30:41.403349+0000","last_peered":"2023-10-11T16:30:41.403349+0000","last_clean":"2023-10-11T16:30:41.403349+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:30:41.403349+0000","last_undegraded":"2023-10-11T16:30:41.403349+0000","last_fullsized":"2023-10-11T16:30:41.403349+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:47:20.053030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184635+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:30:41.184635+0000","last_peered":"2023-10-11T16:30:41.184635+0000","last_clean":"2023-10-11T16:30:41.184635+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:30:41.184635+0000","last_undegraded":"2023-10-11T16:30:41.184635+0000","last_fullsized":"2023-10-11T16:30:41.184635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:48:42.041910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189125+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:30:41.189125+0000","last_peered":"2023-10-11T16:30:41.189125+0000","last_clean":"2023-10-11T16:30:41.189125+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:30:41.189125+0000","last_undegraded":"2023-10-11T16:30:41.189125+0000","last_fullsized":"2023-10-11T16:30:41.189125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:21.935947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:42.144229+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:30:42.144229+0000","last_peered":"2023-10-11T16:30:42.144229+0000","last_clean":"2023-10-11T16:30:42.144229+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:30:42.144229+0000","last_undegraded":"2023-10-11T16:30:42.144229+0000","last_fullsized":"2023-10-11T16:30:42.144229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:32:39.882844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403694+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:30:41.403694+0000","last_peered":"2023-10-11T16:30:41.403694+0000","last_clean":"2023-10-11T16:30:41.403694+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:30:41.403694+0000","last_undegraded":"2023-10-11T16:30:41.403694+0000","last_fullsized":"2023-10-11T16:30:41.403694+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:23:58.685801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188391+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:30:41.188391+0000","last_peered":"2023-10-11T16:30:41.188391+0000","last_clean":"2023-10-11T16:30:41.188391+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:30:41.188391+0000","last_undegraded":"2023-10-11T16:30:41.188391+0000","last_fullsized":"2023-10-11T16:30:41.188391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:04.067909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185574+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:30:41.185574+0000","last_peered":"2023-10-11T16:30:41.185574+0000","last_clean":"2023-10-11T16:30:41.185574+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:30:41.185574+0000","last_undegraded":"2023-10-11T16:30:41.185574+0000","last_fullsized":"2023-10-11T16:30:41.185574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:34.746546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182953+0000","last_change":"2023-10-11T16:30:11.648776+0000","last_active":"2023-10-11T16:30:41.182953+0000","last_peered":"2023-10-11T16:30:41.182953+0000","last_clean":"2023-10-11T16:30:41.182953+0000","last_became_active":"2023-10-11T16:30:11.648465+0000","last_became_peered":"2023-10-11T16:30:11.648465+0000","last_unstale":"2023-10-11T16:30:41.182953+0000","last_undegraded":"2023-10-11T16:30:41.182953+0000","last_fullsized":"2023-10-11T16:30:41.182953+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:44:56.663827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.200104+0000","last_change":"2023-10-11T16:30:41.200104+0000","last_active":"2023-10-11T16:30:41.200104+0000","last_peered":"2023-10-11T16:30:41.200104+0000","last_clean":"2023-10-11T16:30:41.200104+0000","last_became_active":"2023-10-11T16:30:41.199637+0000","last_became_peered":"2023-10-11T16:30:41.199637+0000","last_unstale":"2023-10-11T16:30:41.200104+0000","last_undegraded":"2023-10-11T16:30:41.200104+0000","last_fullsized":"2023-10-11T16:30:41.200104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:08:40.301726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.203024+0000","last_change":"2023-10-11T16:30:41.202718+0000","last_active":"2023-10-11T16:30:41.203024+0000","last_peered":"2023-10-11T16:30:41.203024+0000","last_clean":"2023-10-11T16:30:41.203024+0000","last_became_active":"2023-10-11T16:30:41.190741+0000","last_became_peered":"2023-10-11T16:30:41.190741+0000","last_unstale":"2023-10-11T16:30:41.203024+0000","last_undegraded":"2023-10-11T16:30:41.203024+0000","last_fullsized":"2023-10-11T16:30:41.203024+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:37:17.174287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183059+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:30:41.183059+0000","last_peered":"2023-10-11T16:30:41.183059+0000","last_clean":"2023-10-11T16:30:41.183059+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:30:41.183059+0000","last_undegraded":"2023-10-11T16:30:41.183059+0000","last_fullsized":"2023-10-11T16:30:41.183059+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:46:18.410074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190477+0000","last_change":"2023-10-11T16:30:41.190477+0000","last_active":"2023-10-11T16:30:41.190477+0000","last_peered":"2023-10-11T16:30:41.190477+0000","last_clean":"2023-10-11T16:30:41.190477+0000","last_became_active":"2023-10-11T16:30:41.189929+0000","last_became_peered":"2023-10-11T16:30:41.189929+0000","last_unstale":"2023-10-11T16:30:41.190477+0000","last_undegraded":"2023-10-11T16:30:41.190477+0000","last_fullsized":"2023-10-11T16:30:41.190477+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:01:59.888888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190434+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:30:41.190434+0000","last_peered":"2023-10-11T16:30:41.190434+0000","last_clean":"2023-10-11T16:30:41.190434+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:30:41.190434+0000","last_undegraded":"2023-10-11T16:30:41.190434+0000","last_fullsized":"2023-10-11T16:30:41.190434+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:14:41.192364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182589+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:30:41.182589+0000","last_peered":"2023-10-11T16:30:41.182589+0000","last_clean":"2023-10-11T16:30:41.182589+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:30:41.182589+0000","last_undegraded":"2023-10-11T16:30:41.182589+0000","last_fullsized":"2023-10-11T16:30:41.182589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:05:17.909356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189311+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:30:41.189311+0000","last_peered":"2023-10-11T16:30:41.189311+0000","last_clean":"2023-10-11T16:30:41.189311+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:30:41.189311+0000","last_undegraded":"2023-10-11T16:30:41.189311+0000","last_fullsized":"2023-10-11T16:30:41.189311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:47.760490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198308+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:30:41.198308+0000","last_peered":"2023-10-11T16:30:41.198308+0000","last_clean":"2023-10-11T16:30:41.198308+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:30:41.198308+0000","last_undegraded":"2023-10-11T16:30:41.198308+0000","last_fullsized":"2023-10-11T16:30:41.198308+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:33:35.561074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182863+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:30:41.182863+0000","last_peered":"2023-10-11T16:30:41.182863+0000","last_clean":"2023-10-11T16:30:41.182863+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:30:41.182863+0000","last_undegraded":"2023-10-11T16:30:41.182863+0000","last_fullsized":"2023-10-11T16:30:41.182863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:56:12.549504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186098+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:30:41.186098+0000","last_peered":"2023-10-11T16:30:41.186098+0000","last_clean":"2023-10-11T16:30:41.186098+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:30:41.186098+0000","last_undegraded":"2023-10-11T16:30:41.186098+0000","last_fullsized":"2023-10-11T16:30:41.186098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:53:26.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186048+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:30:41.186048+0000","last_peered":"2023-10-11T16:30:41.186048+0000","last_clean":"2023-10-11T16:30:41.186048+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:30:41.186048+0000","last_undegraded":"2023-10-11T16:30:41.186048+0000","last_fullsized":"2023-10-11T16:30:41.186048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:50.699880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185837+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:30:41.185837+0000","last_peered":"2023-10-11T16:30:41.185837+0000","last_clean":"2023-10-11T16:30:41.185837+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:30:41.185837+0000","last_undegraded":"2023-10-11T16:30:41.185837+0000","last_fullsized":"2023-10-11T16:30:41.185837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:03:23.234106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185679+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:30:41.185679+0000","last_peered":"2023-10-11T16:30:41.185679+0000","last_clean":"2023-10-11T16:30:41.185679+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:30:41.185679+0000","last_undegraded":"2023-10-11T16:30:41.185679+0000","last_fullsized":"2023-10-11T16:30:41.185679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:10:44.726491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198414+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:30:41.198414+0000","last_peered":"2023-10-11T16:30:41.198414+0000","last_clean":"2023-10-11T16:30:41.198414+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:30:41.198414+0000","last_undegraded":"2023-10-11T16:30:41.198414+0000","last_fullsized":"2023-10-11T16:30:41.198414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:14.252893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186179+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:30:41.186179+0000","last_peered":"2023-10-11T16:30:41.186179+0000","last_clean":"2023-10-11T16:30:41.186179+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:30:41.186179+0000","last_undegraded":"2023-10-11T16:30:41.186179+0000","last_fullsized":"2023-10-11T16:30:41.186179+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:16:58.345842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185560+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:30:41.185560+0000","last_peered":"2023-10-11T16:30:41.185560+0000","last_clean":"2023-10-11T16:30:41.185560+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:30:41.185560+0000","last_undegraded":"2023-10-11T16:30:41.185560+0000","last_fullsized":"2023-10-11T16:30:41.185560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:19:06.908283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403856+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:30:41.403856+0000","last_peered":"2023-10-11T16:30:41.403856+0000","last_clean":"2023-10-11T16:30:41.403856+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:30:41.403856+0000","last_undegraded":"2023-10-11T16:30:41.403856+0000","last_fullsized":"2023-10-11T16:30:41.403856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:52:27.929431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189673+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:30:41.189673+0000","last_peered":"2023-10-11T16:30:41.189673+0000","last_clean":"2023-10-11T16:30:41.189673+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:30:41.189673+0000","last_undegraded":"2023-10-11T16:30:41.189673+0000","last_fullsized":"2023-10-11T16:30:41.189673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:47:07.501466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403440+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:30:41.403440+0000","last_peered":"2023-10-11T16:30:41.403440+0000","last_clean":"2023-10-11T16:30:41.403440+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:30:41.403440+0000","last_undegraded":"2023-10-11T16:30:41.403440+0000","last_fullsized":"2023-10-11T16:30:41.403440+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:24:21.537533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185688+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:30:41.185688+0000","last_peered":"2023-10-11T16:30:41.185688+0000","last_clean":"2023-10-11T16:30:41.185688+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:30:41.185688+0000","last_undegraded":"2023-10-11T16:30:41.185688+0000","last_fullsized":"2023-10-11T16:30:41.185688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:25:15.131318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198464+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:30:41.198464+0000","last_peered":"2023-10-11T16:30:41.198464+0000","last_clean":"2023-10-11T16:30:41.198464+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:30:41.198464+0000","last_undegraded":"2023-10-11T16:30:41.198464+0000","last_fullsized":"2023-10-11T16:30:41.198464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:10.823288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185010+0000","last_change":"2023-10-11T16:30:11.668522+0000","last_active":"2023-10-11T16:30:41.185010+0000","last_peered":"2023-10-11T16:30:41.185010+0000","last_clean":"2023-10-11T16:30:41.185010+0000","last_became_active":"2023-10-11T16:30:11.654192+0000","last_became_peered":"2023-10-11T16:30:11.654192+0000","last_unstale":"2023-10-11T16:30:41.185010+0000","last_undegraded":"2023-10-11T16:30:41.185010+0000","last_fullsized":"2023-10-11T16:30:41.185010+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:26:57.163089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":79612,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":322,"num_read_kb":243,"num_write":164,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":39701,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":238836,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449116,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":978944,"data_stored":488446,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":1.204,"15min":1.204},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73699999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.61099999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57099999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.46999999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79200000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70399999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.73599999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.39400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.627}]},{"osd":6,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.50600000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76500000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.70799999999999996},"min":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.006,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83199999999999996,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73599999999999999,"15min":0.74299999999999999},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.86099999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.78000000000000003}]},{"osd":8,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.749}]},{"osd":9,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78700000000000003,"15min":0.76900000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.96,"5min":2.96,"15min":2.96},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.79700000000000004,"15min":0.80100000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.86499999999999999,"5min":1.286,"15min":1.286},"last":0.79300000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.86599999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76500000000000001,"15min":0.77100000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92100000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.76500000000000001}]}]},{"osd":10,"up_from":71,"seq":304942678048,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6604,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738548,"statfs":{"total":95995035648,"available":95988273152,"internally_reserved":0,"allocated":1118208,"data_stored":1141162,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.62},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48699999999999999,"15min":0.48999999999999999},"min":{"1min":0.34599999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72699999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.44400000000000001}]},{"osd":2,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.66200000000000003,"15min":0.66800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77700000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70899999999999996}]},{"osd":3,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.71599999999999997,"5min":0.995,"15min":0.995},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65800000000000003,"15min":0.66700000000000004},"min":{"1min":0.46100000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.55700000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.36599999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.53200000000000003}]},{"osd":5,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89000000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72399999999999998,"15min":0.72599999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.92100000000000004,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.76200000000000001}]},{"osd":6,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85699999999999998,"5min":1.085,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.93300000000000005,"5min":1.03,"15min":1.03},"last":0.747}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76200000000000001,"15min":0.76300000000000001},"min":{"1min":0.57699999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97099999999999997,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.872,"15min":0.88800000000000001},"min":{"1min":0.63200000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.95899999999999996,"5min":2.46,"15min":2.46},"last":0.79500000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.81299999999999994,"15min":0.82399999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.98199999999999998,"5min":1.286,"15min":1.286},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.81100000000000005,"15min":0.81899999999999995},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.93799999999999994,"5min":1.347,"15min":1.347},"last":0.83199999999999996}]},{"osd":11,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.80900000000000005}]}]},{"osd":9,"up_from":66,"seq":283467841571,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":991232,"data_stored":492174,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.54200000000000004,"15min":0.55400000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.70199999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.40699999999999997}]},{"osd":1,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.54300000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71199999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.86599999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.495}]},{"osd":2,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.79300000000000004,"15min":0.76500000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":7.056,"5min":7.056,"15min":7.056},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.81299999999999994,"15min":0.78500000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":7.1529999999999996,"5min":7.1529999999999996,"15min":7.1529999999999996},"last":0.68100000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.70499999999999996,"15min":0.73099999999999998},"min":{"1min":0.38600000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.79200000000000004,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69699999999999995,"15min":0.71199999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.878,"5min":1.018,"15min":1.018},"last":0.42899999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.80400000000000005,"15min":0.81499999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.622},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.78700000000000003,"15min":0.79900000000000004},"min":{"1min":0.58099999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.224,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.54500000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.82199999999999995,"15min":0.84699999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.036,"5min":1.875,"15min":1.875},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.82499999999999996,"15min":0.84799999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.99299999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.71499999999999997}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.89100000000000001,"15min":0.90300000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88500000000000001,"15min":0.89200000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":0.63500000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":0.97299999999999998,"15min":0.95499999999999996},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":7.1070000000000002,"5min":7.1070000000000002,"15min":7.1070000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.075,"5min":0.98599999999999999,"15min":0.97099999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":7.1260000000000003,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.755}]},{"osd":10,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.88400000000000001,"15min":0.89500000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.96199999999999997,"15min":0.96999999999999997},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.66200000000000003}]},{"osd":11,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.7320000000000002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.73199999999999998}]}]},{"osd":8,"up_from":58,"seq":249108103210,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6468,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738684,"statfs":{"total":95995035648,"available":95988412416,"internally_reserved":0,"allocated":978944,"data_stored":487645,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91600000000000004,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":0.996,"15min":0.996},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82599999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.44600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.59799999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.77400000000000002,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0649999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.70199999999999996}]},{"osd":4,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82599999999999996,"5min":1.028,"15min":1.028},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90900000000000003,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.622}]},{"osd":5,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82499999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55900000000000005,"15min":0.54500000000000004},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.89400000000000002}]},{"osd":6,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.179,"15min":1.179},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.499,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.75900000000000001}]},{"osd":7,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.68899999999999995,"15min":0.68100000000000005},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.95299999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.69099999999999995,"15min":0.68300000000000005},"min":{"1min":0.56599999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.99399999999999999,"5min":1.101,"15min":1.101},"last":0.93600000000000005}]},{"osd":9,"last update":"Wed Oct 11 16:30:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.51800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.71999999999999997,"15min":0.71699999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.92000000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.71199999999999997,"15min":0.70099999999999996},"min":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.69599999999999995,"15min":0.68700000000000006},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.86199999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.876},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.98199999999999998}]}]},{"osd":7,"up_from":51,"seq":219043332142,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6592,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738560,"statfs":{"total":95995035648,"available":95988285440,"internally_reserved":0,"allocated":1040384,"data_stored":1092836,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.438,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68000000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.439},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.46300000000000002,"15min":0.45300000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.41499999999999998}]},{"osd":2,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.42899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76100000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83299999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55100000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.39100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94999999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.875},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80200000000000005,"5min":1.103,"15min":1.103},"last":0.53700000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.502,"15min":0.5},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.40000000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65000000000000002,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.66200000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70199999999999996,"15min":0.70799999999999996},"min":{"1min":0.496,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89000000000000001,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.67100000000000004,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97799999999999998,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64700000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71199999999999997,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72599999999999998,"15min":0.72799999999999998},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.68300000000000005}]},{"osd":9,"last update":"Wed Oct 11 16:30:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77400000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.995,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76800000000000002,"15min":0.77200000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.99199999999999999,"5min":1.042,"15min":1.042},"last":0.69899999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.70199999999999996,"15min":0.69399999999999995},"min":{"1min":0.436,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.73499999999999999,"15min":0.72999999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0169999999999999,"5min":1.099,"15min":1.099},"last":0.78600000000000003}]},{"osd":11,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.80700000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":966656,"data_stored":483254,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63900000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.88700000000000001}]},{"osd":1,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.56100000000000005},"min":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.626},{"interface":"front","average":{"1min":0.63,"5min":0.60699999999999998,"15min":0.58499999999999996},"min":{"1min":0.44400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85299999999999998,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.75700000000000001,"5min":1.002,"15min":1.002},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.64200000000000002,"15min":0.63400000000000001},"min":{"1min":0.47299999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.83599999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56699999999999995,"15min":0.55500000000000005},"min":{"1min":0.38400000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79700000000000004,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74099999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.71899999999999997}]},{"osd":4,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.70399999999999996,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90400000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66800000000000004,"15min":0.65200000000000002},"min":{"1min":0.52600000000000002,"5min":0.502,"15min":0.502},"max":{"1min":0.89100000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67800000000000005,"15min":0.65900000000000003},"min":{"1min":0.52500000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.871,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.5,"5min":0.434,"15min":0.434},"max":{"1min":0.877,"5min":1.133,"15min":1.133},"last":1.0149999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:30:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76700000000000002,"15min":0.755},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72699999999999998,"15min":0.71999999999999997},"min":{"1min":0.58099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.878,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.80500000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.752},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.875,"15min":0.874},"min":{"1min":0.60099999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.999}]},{"osd":9,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.85799999999999998,"15min":0.876},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0409999999999999,"5min":1.655,"15min":1.655},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.179,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.93999999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:31:07 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.81899999999999995,"15min":0.82899999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0740000000000001,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.873,"15min":0.875},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.2889999999999999,"5min":1.752,"15min":1.752},"last":0.92600000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.5640000000000001,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":0.95699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7548,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737604,"statfs":{"total":95995035648,"available":95987306496,"internally_reserved":0,"allocated":1036288,"data_stored":549515,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56200000000000006,"15min":0.61699999999999999},"min":{"1min":0.29499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.84899999999999998,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.872,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.626}]},{"osd":2,"last update":"Wed Oct 11 16:31:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.53100000000000003,"15min":0.51500000000000001},"min":{"1min":0.34499999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.71699999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.438,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.80700000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.443,"15min":0.41699999999999998},"min":{"1min":0.31,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.82799999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.498,"15min":0.45100000000000001},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.73999999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.55200000000000005,"15min":0.498},"min":{"1min":0.38800000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.59999999999999998,"15min":0.53800000000000003},"min":{"1min":0.51900000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.82799999999999996}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.874}]},{"osd":7,"last update":"Wed Oct 11 16:31:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.53800000000000003,"15min":0.502},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.626,"15min":0.59499999999999997},"min":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.84199999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.69499999999999995,"15min":0.66600000000000004},"min":{"1min":0.52300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.66200000000000003,"15min":0.63},"min":{"1min":0.59599999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.99199999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.72199999999999998,"15min":0.71399999999999997},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.76700000000000002,"15min":0.76100000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.93600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.68400000000000005,"15min":0.66800000000000004},"min":{"1min":0.42799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.86199999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":1.024},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.90700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7376,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737776,"statfs":{"total":95995035648,"available":95987482624,"internally_reserved":0,"allocated":991232,"data_stored":484642,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:30:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.33100000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.495},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57599999999999996,"15min":0.53300000000000003},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.82499999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.62,"15min":0.58399999999999996},"min":{"1min":0.39300000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64300000000000002,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66300000000000003}]},{"osd":3,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.307}]},{"osd":4,"last update":"Wed Oct 11 16:30:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59499999999999997,"15min":0.55500000000000005},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86599999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.628,"15min":0.58999999999999997},"min":{"1min":0.47099999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90000000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72199999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.60699999999999998},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66300000000000003,"15min":0.63900000000000001},"min":{"1min":0.502,"5min":0.436,"15min":0.436},"max":{"1min":0.94699999999999995,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68899999999999995}]},{"osd":6,"last update":"Wed Oct 11 16:30:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63500000000000001,"15min":0.61899999999999999},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.755,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.74299999999999999,"5min":0.999,"15min":0.999},"last":0.52500000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.81100000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.77300000000000002,"15min":0.753},"min":{"1min":0.63500000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":3.073,"5min":3.073,"15min":3.073},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75800000000000001,"15min":0.76200000000000001},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.60399999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:30:47 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.82599999999999996,"15min":0.84599999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.04,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.81699999999999995,"15min":0.83299999999999996},"min":{"1min":0.47599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93700000000000006,"5min":1.179,"15min":1.179},"last":0.74399999999999999}]},{"osd":10,"last update":"Wed Oct 11 16:31:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.60899999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0649999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.82199999999999995,"15min":0.82599999999999996},"min":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.624}]},{"osd":11,"last update":"Wed Oct 11 16:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.64100000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.70399999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.70399999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7440,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737712,"statfs":{"total":95995035648,"available":95987417088,"internally_reserved":0,"allocated":991232,"data_stored":493882,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.41699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.82899999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39900000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71099999999999997,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.52700000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:31:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56499999999999995,"15min":0.51900000000000002},"min":{"1min":0.36499999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.82499999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56799999999999995,"15min":0.51900000000000002},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.625}]},{"osd":3,"last update":"Wed Oct 11 16:30:25 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56699999999999995,"15min":0.54200000000000004},"min":{"1min":0.49399999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59499999999999997,"15min":0.56299999999999994},"min":{"1min":0.47299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68000000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:31:13 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76100000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:31:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.502,"15min":0.46800000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.78600000000000003,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.503,"15min":0.48199999999999998},"min":{"1min":0.39600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.85099999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.64200000000000002,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86599999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.65600000000000003,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.80700000000000005}]},{"osd":7,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.93400000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:31:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.64400000000000002,"15min":0.61499999999999999},"min":{"1min":0.48299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.64800000000000002,"15min":0.623},"min":{"1min":0.49199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.95899999999999996}]},{"osd":9,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.52600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":1.03},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.73299999999999998,"15min":0.71999999999999997},"min":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.90100000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.748,"15min":0.73699999999999999},"min":{"1min":0.56999999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.78500000000000003,"15min":0.77200000000000002},"min":{"1min":0.61299999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.99199999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:31:13 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":1.054}]}]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6720,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738432,"statfs":{"total":95995035648,"available":95988154368,"internally_reserved":0,"allocated":974848,"data_stored":483117,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.38200000000000001},"min":{"1min":0.33100000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69999999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.42699999999999999,"15min":0.40699999999999997},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.44}]},{"osd":1,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60599999999999998,"15min":0.56999999999999995},"min":{"1min":0.433,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.55400000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.751,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.47099999999999997}]},{"osd":2,"last update":"Wed Oct 11 16:30:26 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.60599999999999998},"min":{"1min":0.46000000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.63500000000000001,"15min":0.58899999999999997},"min":{"1min":0.48199999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Wed Oct 11 16:30:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.65700000000000003,"15min":0.63800000000000001},"min":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.83899999999999997,"5min":1.101,"15min":1.101},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.70699999999999996,"15min":0.68400000000000005},"min":{"1min":0.55900000000000005,"5min":0.498,"15min":0.498},"max":{"1min":0.97199999999999998,"5min":1.194,"15min":1.194},"last":0.67900000000000005}]},{"osd":5,"last update":"Wed Oct 11 16:31:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.625,"15min":0.63200000000000001},"min":{"1min":0.41999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76100000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68600000000000005,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81899999999999995,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.749}]},{"osd":6,"last update":"Wed Oct 11 16:30:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91500000000000004,"5min":1.268,"15min":1.268},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.51300000000000001}]},{"osd":7,"last update":"Wed Oct 11 16:31:03 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.85499999999999998,"5min":1.119,"15min":1.119},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.84199999999999997,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.81499999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:30:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.98699999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.79700000000000004,"15min":0.80600000000000005},"min":{"1min":0.53200000000000003,"5min":0.502,"15min":0.502},"max":{"1min":0.92100000000000004,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.73999999999999999}]},{"osd":10,"last update":"Wed Oct 11 16:31:08 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.81699999999999995,"15min":0.82699999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.94499999999999995,"5min":1.137,"15min":1.137},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.82899999999999996,"15min":0.83499999999999996},"min":{"1min":0.67300000000000004,"5min":0.627,"15min":0.627},"max":{"1min":1.0009999999999999,"5min":1.165,"15min":1.165},"last":0.77700000000000002}]},{"osd":11,"last update":"Wed Oct 11 16:31:14 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.76300000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6640,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738512,"statfs":{"total":95995035648,"available":95988236288,"internally_reserved":0,"allocated":1024000,"data_stored":553956,"data_compressed":24251,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:30:54 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.375},"min":{"1min":0.28799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.54600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.377},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38500000000000001},"min":{"1min":0.311,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57599999999999996,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:30:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.57599999999999996,"15min":0.55300000000000005},"min":{"1min":0.442,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.86099999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.81100000000000005,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.495}]},{"osd":3,"last update":"Wed Oct 11 16:30:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.53900000000000003},"min":{"1min":0.439,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68300000000000005,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.52800000000000002},"min":{"1min":0.44,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71599999999999997,"5min":0.875,"15min":0.875},"last":0.51000000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.69599999999999995,"15min":0.74399999999999999},"min":{"1min":0.38200000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86399999999999999,"5min":4.5,"15min":4.5},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66700000000000004,"15min":0.72299999999999998},"min":{"1min":0.36699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.873,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.65300000000000002}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.55500000000000005}]},{"osd":7,"last update":"Wed Oct 11 16:31:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49199999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67500000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.56499999999999995,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.58599999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.48999999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76800000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.61599999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:30:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.55000000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91900000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.55400000000000005,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.69299999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.54400000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.873,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72699999999999998,"15min":0.71899999999999997},"min":{"1min":0.58499999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.96499999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.80100000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.84999999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6732,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738420,"statfs":{"total":95995035648,"available":95988142080,"internally_reserved":0,"allocated":1052672,"data_stored":1089054,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53200000000000003,"15min":0.52400000000000002},"min":{"1min":0.375,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.40200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67800000000000005,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.52200000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.56299999999999994,"15min":0.53400000000000003},"min":{"1min":0.42899999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76400000000000001,"5min":0.874,"15min":0.874},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81999999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57299999999999995}]},{"osd":2,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.42999999999999999,"15min":0.40400000000000003},"min":{"1min":0.35499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.63100000000000001,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.42399999999999999},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.48699999999999999}]},{"osd":3,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56899999999999995,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.628}]},{"osd":4,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.61399999999999999,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.61399999999999999,"15min":0.58499999999999996},"min":{"1min":0.47799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65500000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:30:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78100000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67100000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.496,"5min":0.44,"15min":0.44},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.70499999999999996,"15min":0.69999999999999996},"min":{"1min":0.502,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.70899999999999996}]},{"osd":8,"last update":"Wed Oct 11 16:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68999999999999995},"min":{"1min":0.502,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83699999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.69999999999999996}]},{"osd":9,"last update":"Wed Oct 11 16:30:45 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.749,"15min":0.75},"min":{"1min":0.50700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93899999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.76100000000000001,"15min":0.76400000000000001},"min":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.95899999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.77600000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.73099999999999998,"15min":0.72099999999999997},"min":{"1min":0.53800000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.74099999999999999,"15min":0.73199999999999998},"min":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.74299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":10718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":66696,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":11426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":11112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:31:31.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph pg dump --format=json 2023-10-11T16:31:32.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:32 smithi130 ceph-mon[131859]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:31:32.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:32 smithi111 ceph-mon[98527]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:31:32.399 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:32.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:32 smithi150 ceph-mon[132217]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:31:33.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:33 smithi111 ceph-mon[98527]: pgmap v315: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:33.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:33 smithi150 ceph-mon[132217]: pgmap v315: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:33.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:33 smithi130 ceph-mon[131859]: pgmap v315: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:33.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 -- 172.21.15.111:0/878824611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100540 msgr2=0x7f722c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 --2- 172.21.15.111:0/878824611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100540 0x7f722c100910 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f721c0099c0 tx=0x7f721c009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 -- 172.21.15.111:0/878824611 shutdown_connections 2023-10-11T16:31:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 --2- 172.21.15.111:0/878824611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f722c1055e0 0x7f722c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 --2- 172.21.15.111:0/878824611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100540 0x7f722c100910 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:33.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 --2- 172.21.15.111:0/878824611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f722c100ee0 0x7f722c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 -- 172.21.15.111:0/878824611 >> 172.21.15.111:0/878824611 conn(0x7f722c0fbdc0 msgr2=0x7f722c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 -- 172.21.15.111:0/878824611 shutdown_connections 2023-10-11T16:31:33.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.936+0000 7f7230ae5700 1 -- 172.21.15.111:0/878824611 wait complete. 2023-10-11T16:31:33.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.938+0000 7f7230ae5700 1 Processor -- start 2023-10-11T16:31:33.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.938+0000 7f7230ae5700 1 -- start start 2023-10-11T16:31:33.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.938+0000 7f7230ae5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f722c100540 0x7f722c1963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:33.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.938+0000 7f7230ae5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100ee0 0x7f722c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:33.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.938+0000 7f7230ae5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f722c1055e0 0x7f722c19ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:33.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.938+0000 7f7230ae5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c19b390 con 0x7f722c100ee0 2023-10-11T16:31:33.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.938+0000 7f7230ae5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c19b500 con 0x7f722c100540 2023-10-11T16:31:33.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7230ae5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722c19b6c0 con 0x7f722c1055e0 2023-10-11T16:31:33.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100ee0 0x7f722c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:33.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f722a59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f722c100540 0x7f722c1963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:33.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100ee0 0x7f722c196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53520/0 (socket says 172.21.15.111:53520) 2023-10-11T16:31:33.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 -- 172.21.15.111:0/1699790268 learned_addr learned my addr 172.21.15.111:0/1699790268 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:33.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f722ad9d700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f722c1055e0 0x7f722c19ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:33.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 -- 172.21.15.111:0/1699790268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f722c1055e0 msgr2=0x7f722c19ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:33.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f722c1055e0 0x7f722c19ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 -- 172.21.15.111:0/1699790268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f722c100540 msgr2=0x7f722c1963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f722c100540 0x7f722c1963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 -- 172.21.15.111:0/1699790268 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c0095f0 con 0x7f722c100ee0 2023-10-11T16:31:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f722a59c700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f722c100540 0x7f722c1963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:33.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.939+0000 7f7229d9b700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100ee0 0x7f722c196910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f722000e970 tx=0x7f722000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:33.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.940+0000 7f721b7fe700 1 -- 172.21.15.111:0/1699790268 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7220009790 con 0x7f722c100ee0 2023-10-11T16:31:33.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.940+0000 7f721b7fe700 1 -- 172.21.15.111:0/1699790268 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f722000c7e0 con 0x7f722c100ee0 2023-10-11T16:31:33.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.940+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722c110370 con 0x7f722c100ee0 2023-10-11T16:31:33.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.940+0000 7f721b7fe700 1 -- 172.21.15.111:0/1699790268 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72200054e0 con 0x7f722c100ee0 2023-10-11T16:31:33.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.940+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722c1101e0 con 0x7f722c100ee0 2023-10-11T16:31:33.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.941+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722c110500 con 0x7f722c100ee0 2023-10-11T16:31:33.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.943+0000 7f721b7fe700 1 -- 172.21.15.111:0/1699790268 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f722000c950 con 0x7f722c100ee0 2023-10-11T16:31:33.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.944+0000 7f721b7fe700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72140a1260 0x7f72140a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:33.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.944+0000 7f721b7fe700 1 -- 172.21.15.111:0/1699790268 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7220014070 con 0x7f722c100ee0 2023-10-11T16:31:33.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.944+0000 7f722a59c700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72140a1260 0x7f72140a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:33.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.945+0000 7f722a59c700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72140a1260 0x7f72140a3710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f721c009990 tx=0x7f721c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:33.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:33.948+0000 7f721b7fe700 1 -- 172.21.15.111:0/1699790268 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7220096290 con 0x7f722c100ee0 2023-10-11T16:31:34.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.141+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f722c0fcf10 con 0x7f72140a1260 2023-10-11T16:31:34.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.154+0000 7f721b7fe700 1 -- 172.21.15.111:0/1699790268 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427775 (secure 0 0 0) 0x7f722c0fcf10 con 0x7f72140a1260 2023-10-11T16:31:34.161 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72140a1260 msgr2=0x7f72140a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72140a1260 0x7f72140a3710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f721c009990 tx=0x7f721c005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100ee0 msgr2=0x7f722c196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100ee0 0x7f722c196910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f722000e970 tx=0x7f722000ec80 comp rx=0 tx=0).stop 2023-10-11T16:31:34.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 shutdown_connections 2023-10-11T16:31:34.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72140a1260 0x7f72140a3710 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:34.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f722c1055e0 0x7f722c19ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:34.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f722c100540 0x7f722c1963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:34.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 --2- 172.21.15.111:0/1699790268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f722c100ee0 0x7f722c196910 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:34.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.159+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 >> 172.21.15.111:0/1699790268 conn(0x7f722c0fbdc0 msgr2=0x7f722c0fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:34.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.160+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 shutdown_connections 2023-10-11T16:31:34.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:34.160+0000 7f7230ae5700 1 -- 172.21.15.111:0/1699790268 wait complete. 2023-10-11T16:31:34.167 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:31:34.592 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2023-10-11T16:31:34.073523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670226,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":483,"num_read_kb":377,"num_write":347,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1220437,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81012,"kb_used_data":11860,"kb_used_omap":0,"kb_used_meta":69056,"kb_avail":1124860812,"statfs":{"total":1151940427776,"available":1151857471488,"internally_reserved":0,"allocated":12144640,"data_stored":7839683,"data_compressed":324562,"data_compressed_allocated":438272,"data_compressed_original":2396160,"omap_allocated":0,"internal_metadata":70713344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003155"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185713+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:30:41.185713+0000","last_peered":"2023-10-11T16:30:41.185713+0000","last_clean":"2023-10-11T16:30:41.185713+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:30:41.185713+0000","last_undegraded":"2023-10-11T16:30:41.185713+0000","last_fullsized":"2023-10-11T16:30:41.185713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:47:02.731639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185549+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:30:41.185549+0000","last_peered":"2023-10-11T16:30:41.185549+0000","last_clean":"2023-10-11T16:30:41.185549+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:30:41.185549+0000","last_undegraded":"2023-10-11T16:30:41.185549+0000","last_fullsized":"2023-10-11T16:30:41.185549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:04:44.769304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185196+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:30:41.185196+0000","last_peered":"2023-10-11T16:30:41.185196+0000","last_clean":"2023-10-11T16:30:41.185196+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:30:41.185196+0000","last_undegraded":"2023-10-11T16:30:41.185196+0000","last_fullsized":"2023-10-11T16:30:41.185196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:22:01.830634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188320+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:30:41.188320+0000","last_peered":"2023-10-11T16:30:41.188320+0000","last_clean":"2023-10-11T16:30:41.188320+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:30:41.188320+0000","last_undegraded":"2023-10-11T16:30:41.188320+0000","last_fullsized":"2023-10-11T16:30:41.188320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:21:55.362917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185730+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:30:41.185730+0000","last_peered":"2023-10-11T16:30:41.185730+0000","last_clean":"2023-10-11T16:30:41.185730+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:30:41.185730+0000","last_undegraded":"2023-10-11T16:30:41.185730+0000","last_fullsized":"2023-10-11T16:30:41.185730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:47.175695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189802+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:30:41.189802+0000","last_peered":"2023-10-11T16:30:41.189802+0000","last_clean":"2023-10-11T16:30:41.189802+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:30:41.189802+0000","last_undegraded":"2023-10-11T16:30:41.189802+0000","last_fullsized":"2023-10-11T16:30:41.189802+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:04:31.832644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182446+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:30:41.182446+0000","last_peered":"2023-10-11T16:30:41.182446+0000","last_clean":"2023-10-11T16:30:41.182446+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:30:41.182446+0000","last_undegraded":"2023-10-11T16:30:41.182446+0000","last_fullsized":"2023-10-11T16:30:41.182446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:15:28.747024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403438+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:30:41.403438+0000","last_peered":"2023-10-11T16:30:41.403438+0000","last_clean":"2023-10-11T16:30:41.403438+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:30:41.403438+0000","last_undegraded":"2023-10-11T16:30:41.403438+0000","last_fullsized":"2023-10-11T16:30:41.403438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:20:51.806951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185872+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:30:41.185872+0000","last_peered":"2023-10-11T16:30:41.185872+0000","last_clean":"2023-10-11T16:30:41.185872+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:30:41.185872+0000","last_undegraded":"2023-10-11T16:30:41.185872+0000","last_fullsized":"2023-10-11T16:30:41.185872+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:07:38.837524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190769+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:30:41.190769+0000","last_peered":"2023-10-11T16:30:41.190769+0000","last_clean":"2023-10-11T16:30:41.190769+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:30:41.190769+0000","last_undegraded":"2023-10-11T16:30:41.190769+0000","last_fullsized":"2023-10-11T16:30:41.190769+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:17:25.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182922+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:30:41.182922+0000","last_peered":"2023-10-11T16:30:41.182922+0000","last_clean":"2023-10-11T16:30:41.182922+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:30:41.182922+0000","last_undegraded":"2023-10-11T16:30:41.182922+0000","last_fullsized":"2023-10-11T16:30:41.182922+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:16:24.814604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188720+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:30:41.188720+0000","last_peered":"2023-10-11T16:30:41.188720+0000","last_clean":"2023-10-11T16:30:41.188720+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:30:41.188720+0000","last_undegraded":"2023-10-11T16:30:41.188720+0000","last_fullsized":"2023-10-11T16:30:41.188720+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:33:23.932678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198123+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:30:41.198123+0000","last_peered":"2023-10-11T16:30:41.198123+0000","last_clean":"2023-10-11T16:30:41.198123+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:30:41.198123+0000","last_undegraded":"2023-10-11T16:30:41.198123+0000","last_fullsized":"2023-10-11T16:30:41.198123+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:09:09.086960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184893+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:30:41.184893+0000","last_peered":"2023-10-11T16:30:41.184893+0000","last_clean":"2023-10-11T16:30:41.184893+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:30:41.184893+0000","last_undegraded":"2023-10-11T16:30:41.184893+0000","last_fullsized":"2023-10-11T16:30:41.184893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:06.789653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185441+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:30:41.185441+0000","last_peered":"2023-10-11T16:30:41.185441+0000","last_clean":"2023-10-11T16:30:41.185441+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:30:41.185441+0000","last_undegraded":"2023-10-11T16:30:41.185441+0000","last_fullsized":"2023-10-11T16:30:41.185441+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:22:47.667568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184863+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:30:41.184863+0000","last_peered":"2023-10-11T16:30:41.184863+0000","last_clean":"2023-10-11T16:30:41.184863+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:30:41.184863+0000","last_undegraded":"2023-10-11T16:30:41.184863+0000","last_fullsized":"2023-10-11T16:30:41.184863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:51:56.240736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185599+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:30:41.185599+0000","last_peered":"2023-10-11T16:30:41.185599+0000","last_clean":"2023-10-11T16:30:41.185599+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:30:41.185599+0000","last_undegraded":"2023-10-11T16:30:41.185599+0000","last_fullsized":"2023-10-11T16:30:41.185599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:34:17.358317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182794+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:30:41.182794+0000","last_peered":"2023-10-11T16:30:41.182794+0000","last_clean":"2023-10-11T16:30:41.182794+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:30:41.182794+0000","last_undegraded":"2023-10-11T16:30:41.182794+0000","last_fullsized":"2023-10-11T16:30:41.182794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:09:27.215051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185311+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:30:41.185311+0000","last_peered":"2023-10-11T16:30:41.185311+0000","last_clean":"2023-10-11T16:30:41.185311+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:30:41.185311+0000","last_undegraded":"2023-10-11T16:30:41.185311+0000","last_fullsized":"2023-10-11T16:30:41.185311+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:46:35.027109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186180+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.186180+0000","last_peered":"2023-10-11T16:30:41.186180+0000","last_clean":"2023-10-11T16:30:41.186180+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:30:41.186180+0000","last_undegraded":"2023-10-11T16:30:41.186180+0000","last_fullsized":"2023-10-11T16:30:41.186180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:08:05.403679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198854+0000","last_change":"2023-10-11T16:30:41.198854+0000","last_active":"2023-10-11T16:30:41.198854+0000","last_peered":"2023-10-11T16:30:41.198854+0000","last_clean":"2023-10-11T16:30:41.198854+0000","last_became_active":"2023-10-11T16:30:41.189677+0000","last_became_peered":"2023-10-11T16:30:41.189677+0000","last_unstale":"2023-10-11T16:30:41.198854+0000","last_undegraded":"2023-10-11T16:30:41.198854+0000","last_fullsized":"2023-10-11T16:30:41.198854+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:58:40.952333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403443+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:30:41.403443+0000","last_peered":"2023-10-11T16:30:41.403443+0000","last_clean":"2023-10-11T16:30:41.403443+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:30:41.403443+0000","last_undegraded":"2023-10-11T16:30:41.403443+0000","last_fullsized":"2023-10-11T16:30:41.403443+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:16:44.616541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185309+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:30:41.185309+0000","last_peered":"2023-10-11T16:30:41.185309+0000","last_clean":"2023-10-11T16:30:41.185309+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:30:41.185309+0000","last_undegraded":"2023-10-11T16:30:41.185309+0000","last_fullsized":"2023-10-11T16:30:41.185309+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:33:41.484467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403809+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:30:41.403809+0000","last_peered":"2023-10-11T16:30:41.403809+0000","last_clean":"2023-10-11T16:30:41.403809+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:30:41.403809+0000","last_undegraded":"2023-10-11T16:30:41.403809+0000","last_fullsized":"2023-10-11T16:30:41.403809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:05:06.232815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185766+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:30:41.185766+0000","last_peered":"2023-10-11T16:30:41.185766+0000","last_clean":"2023-10-11T16:30:41.185766+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:30:41.185766+0000","last_undegraded":"2023-10-11T16:30:41.185766+0000","last_fullsized":"2023-10-11T16:30:41.185766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:16.994395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188044+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:30:41.188044+0000","last_peered":"2023-10-11T16:30:41.188044+0000","last_clean":"2023-10-11T16:30:41.188044+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:30:41.188044+0000","last_undegraded":"2023-10-11T16:30:41.188044+0000","last_fullsized":"2023-10-11T16:30:41.188044+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:30:57.536276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186101+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:30:41.186101+0000","last_peered":"2023-10-11T16:30:41.186101+0000","last_clean":"2023-10-11T16:30:41.186101+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:30:41.186101+0000","last_undegraded":"2023-10-11T16:30:41.186101+0000","last_fullsized":"2023-10-11T16:30:41.186101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:11:55.996038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185713+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:30:41.185713+0000","last_peered":"2023-10-11T16:30:41.185713+0000","last_clean":"2023-10-11T16:30:41.185713+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:30:41.185713+0000","last_undegraded":"2023-10-11T16:30:41.185713+0000","last_fullsized":"2023-10-11T16:30:41.185713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:31:18.914747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184425+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:30:41.184425+0000","last_peered":"2023-10-11T16:30:41.184425+0000","last_clean":"2023-10-11T16:30:41.184425+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:30:41.184425+0000","last_undegraded":"2023-10-11T16:30:41.184425+0000","last_fullsized":"2023-10-11T16:30:41.184425+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:57:53.652251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186025+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:30:41.186025+0000","last_peered":"2023-10-11T16:30:41.186025+0000","last_clean":"2023-10-11T16:30:41.186025+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:30:41.186025+0000","last_undegraded":"2023-10-11T16:30:41.186025+0000","last_fullsized":"2023-10-11T16:30:41.186025+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:46:36.549662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.195736+0000","last_change":"2023-10-11T16:30:41.195736+0000","last_active":"2023-10-11T16:30:41.195736+0000","last_peered":"2023-10-11T16:30:41.195736+0000","last_clean":"2023-10-11T16:30:41.195736+0000","last_became_active":"2023-10-11T16:30:41.195352+0000","last_became_peered":"2023-10-11T16:30:41.195352+0000","last_unstale":"2023-10-11T16:30:41.195736+0000","last_undegraded":"2023-10-11T16:30:41.195736+0000","last_fullsized":"2023-10-11T16:30:41.195736+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:00:25.607770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185067+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:30:41.185067+0000","last_peered":"2023-10-11T16:30:41.185067+0000","last_clean":"2023-10-11T16:30:41.185067+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:30:41.185067+0000","last_undegraded":"2023-10-11T16:30:41.185067+0000","last_fullsized":"2023-10-11T16:30:41.185067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:48:48.316789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'8","reported_seq":90,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:29.266305+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:31:29.266305+0000","last_peered":"2023-10-11T16:31:29.266305+0000","last_clean":"2023-10-11T16:31:29.266305+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:31:29.266305+0000","last_undegraded":"2023-10-11T16:31:29.266305+0000","last_fullsized":"2023-10-11T16:31:29.266305+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:32:24.986353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10572,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":12,"num_write":5,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186484+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:30:41.186484+0000","last_peered":"2023-10-11T16:30:41.186484+0000","last_clean":"2023-10-11T16:30:41.186484+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:30:41.186484+0000","last_undegraded":"2023-10-11T16:30:41.186484+0000","last_fullsized":"2023-10-11T16:30:41.186484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:39:49.981983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186478+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:30:41.186478+0000","last_peered":"2023-10-11T16:30:41.186478+0000","last_clean":"2023-10-11T16:30:41.186478+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:30:41.186478+0000","last_undegraded":"2023-10-11T16:30:41.186478+0000","last_fullsized":"2023-10-11T16:30:41.186478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:20:03.910243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189794+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:30:41.189794+0000","last_peered":"2023-10-11T16:30:41.189794+0000","last_clean":"2023-10-11T16:30:41.189794+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:30:41.189794+0000","last_undegraded":"2023-10-11T16:30:41.189794+0000","last_fullsized":"2023-10-11T16:30:41.189794+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:08.968090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189977+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:30:41.189977+0000","last_peered":"2023-10-11T16:30:41.189977+0000","last_clean":"2023-10-11T16:30:41.189977+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:30:41.189977+0000","last_undegraded":"2023-10-11T16:30:41.189977+0000","last_fullsized":"2023-10-11T16:30:41.189977+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:41:46.444179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185463+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:30:41.185463+0000","last_peered":"2023-10-11T16:30:41.185463+0000","last_clean":"2023-10-11T16:30:41.185463+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:30:41.185463+0000","last_undegraded":"2023-10-11T16:30:41.185463+0000","last_fullsized":"2023-10-11T16:30:41.185463+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:16:11.533226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.197973+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:30:41.197973+0000","last_peered":"2023-10-11T16:30:41.197973+0000","last_clean":"2023-10-11T16:30:41.197973+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:30:41.197973+0000","last_undegraded":"2023-10-11T16:30:41.197973+0000","last_fullsized":"2023-10-11T16:30:41.197973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:49:04.403007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403246+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:30:41.403246+0000","last_peered":"2023-10-11T16:30:41.403246+0000","last_clean":"2023-10-11T16:30:41.403246+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:30:41.403246+0000","last_undegraded":"2023-10-11T16:30:41.403246+0000","last_fullsized":"2023-10-11T16:30:41.403246+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:18:58.866162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185429+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:30:41.185429+0000","last_peered":"2023-10-11T16:30:41.185429+0000","last_clean":"2023-10-11T16:30:41.185429+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:30:41.185429+0000","last_undegraded":"2023-10-11T16:30:41.185429+0000","last_fullsized":"2023-10-11T16:30:41.185429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:49.007168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189537+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:30:41.189537+0000","last_peered":"2023-10-11T16:30:41.189537+0000","last_clean":"2023-10-11T16:30:41.189537+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:30:41.189537+0000","last_undegraded":"2023-10-11T16:30:41.189537+0000","last_fullsized":"2023-10-11T16:30:41.189537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:46:32.350328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181702+0000","last_change":"2023-10-11T16:30:11.662850+0000","last_active":"2023-10-11T16:30:41.181702+0000","last_peered":"2023-10-11T16:30:41.181702+0000","last_clean":"2023-10-11T16:30:41.181702+0000","last_became_active":"2023-10-11T16:30:11.661052+0000","last_became_peered":"2023-10-11T16:30:11.661052+0000","last_unstale":"2023-10-11T16:30:41.181702+0000","last_undegraded":"2023-10-11T16:30:41.181702+0000","last_fullsized":"2023-10-11T16:30:41.181702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:23.324421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403234+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:30:41.403234+0000","last_peered":"2023-10-11T16:30:41.403234+0000","last_clean":"2023-10-11T16:30:41.403234+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:30:41.403234+0000","last_undegraded":"2023-10-11T16:30:41.403234+0000","last_fullsized":"2023-10-11T16:30:41.403234+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:53:00.713151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:31.268734+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:31:31.268734+0000","last_peered":"2023-10-11T16:31:31.268734+0000","last_clean":"2023-10-11T16:31:31.268734+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:31:31.268734+0000","last_undegraded":"2023-10-11T16:31:31.268734+0000","last_fullsized":"2023-10-11T16:31:31.268734+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:21:03.094344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185994+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:30:41.185994+0000","last_peered":"2023-10-11T16:30:41.185994+0000","last_clean":"2023-10-11T16:30:41.185994+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:30:41.185994+0000","last_undegraded":"2023-10-11T16:30:41.185994+0000","last_fullsized":"2023-10-11T16:30:41.185994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:51:15.389807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181694+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:30:41.181694+0000","last_peered":"2023-10-11T16:30:41.181694+0000","last_clean":"2023-10-11T16:30:41.181694+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:30:41.181694+0000","last_undegraded":"2023-10-11T16:30:41.181694+0000","last_fullsized":"2023-10-11T16:30:41.181694+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:23:32.776111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183006+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:30:41.183006+0000","last_peered":"2023-10-11T16:30:41.183006+0000","last_clean":"2023-10-11T16:30:41.183006+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:30:41.183006+0000","last_undegraded":"2023-10-11T16:30:41.183006+0000","last_fullsized":"2023-10-11T16:30:41.183006+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:50:47.408054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185791+0000","last_change":"2023-10-11T16:30:11.666124+0000","last_active":"2023-10-11T16:30:41.185791+0000","last_peered":"2023-10-11T16:30:41.185791+0000","last_clean":"2023-10-11T16:30:41.185791+0000","last_became_active":"2023-10-11T16:30:11.665786+0000","last_became_peered":"2023-10-11T16:30:11.665786+0000","last_unstale":"2023-10-11T16:30:41.185791+0000","last_undegraded":"2023-10-11T16:30:41.185791+0000","last_fullsized":"2023-10-11T16:30:41.185791+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:16:08.621915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188345+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:30:41.188345+0000","last_peered":"2023-10-11T16:30:41.188345+0000","last_clean":"2023-10-11T16:30:41.188345+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:30:41.188345+0000","last_undegraded":"2023-10-11T16:30:41.188345+0000","last_fullsized":"2023-10-11T16:30:41.188345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:49.162491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190222+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:30:41.190222+0000","last_peered":"2023-10-11T16:30:41.190222+0000","last_clean":"2023-10-11T16:30:41.190222+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:30:41.190222+0000","last_undegraded":"2023-10-11T16:30:41.190222+0000","last_fullsized":"2023-10-11T16:30:41.190222+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:23:11.290409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186265+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:30:41.186265+0000","last_peered":"2023-10-11T16:30:41.186265+0000","last_clean":"2023-10-11T16:30:41.186265+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:30:41.186265+0000","last_undegraded":"2023-10-11T16:30:41.186265+0000","last_fullsized":"2023-10-11T16:30:41.186265+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:59:10.188855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.181930+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:30:41.181930+0000","last_peered":"2023-10-11T16:30:41.181930+0000","last_clean":"2023-10-11T16:30:41.181930+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:30:41.181930+0000","last_undegraded":"2023-10-11T16:30:41.181930+0000","last_fullsized":"2023-10-11T16:30:41.181930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:27:45.123842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":195,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:32.269638+0000","last_change":"2023-10-11T16:30:41.788994+0000","last_active":"2023-10-11T16:31:32.269638+0000","last_peered":"2023-10-11T16:31:32.269638+0000","last_clean":"2023-10-11T16:31:32.269638+0000","last_became_active":"2023-10-11T16:30:41.189949+0000","last_became_peered":"2023-10-11T16:30:41.189949+0000","last_unstale":"2023-10-11T16:31:32.269638+0000","last_undegraded":"2023-10-11T16:31:32.269638+0000","last_fullsized":"2023-10-11T16:31:32.269638+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:36:50.095047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66100,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":116,"num_read_kb":123,"num_write":36,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38515,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186506+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:30:41.186506+0000","last_peered":"2023-10-11T16:30:41.186506+0000","last_clean":"2023-10-11T16:30:41.186506+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:30:41.186506+0000","last_undegraded":"2023-10-11T16:30:41.186506+0000","last_fullsized":"2023-10-11T16:30:41.186506+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:26:09.244377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190731+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:30:41.190731+0000","last_peered":"2023-10-11T16:30:41.190731+0000","last_clean":"2023-10-11T16:30:41.190731+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:30:41.190731+0000","last_undegraded":"2023-10-11T16:30:41.190731+0000","last_fullsized":"2023-10-11T16:30:41.190731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:10:05.160429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185142+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:30:41.185142+0000","last_peered":"2023-10-11T16:30:41.185142+0000","last_clean":"2023-10-11T16:30:41.185142+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:30:41.185142+0000","last_undegraded":"2023-10-11T16:30:41.185142+0000","last_fullsized":"2023-10-11T16:30:41.185142+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:25:53.623192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198771+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:30:41.198771+0000","last_peered":"2023-10-11T16:30:41.198771+0000","last_clean":"2023-10-11T16:30:41.198771+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:30:41.198771+0000","last_undegraded":"2023-10-11T16:30:41.198771+0000","last_fullsized":"2023-10-11T16:30:41.198771+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:09:22.911180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184864+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:30:41.184864+0000","last_peered":"2023-10-11T16:30:41.184864+0000","last_clean":"2023-10-11T16:30:41.184864+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:30:41.184864+0000","last_undegraded":"2023-10-11T16:30:41.184864+0000","last_fullsized":"2023-10-11T16:30:41.184864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:28:35.220071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190012+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:30:41.190012+0000","last_peered":"2023-10-11T16:30:41.190012+0000","last_clean":"2023-10-11T16:30:41.190012+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:30:41.190012+0000","last_undegraded":"2023-10-11T16:30:41.190012+0000","last_fullsized":"2023-10-11T16:30:41.190012+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:18:02.905266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183098+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:30:41.183098+0000","last_peered":"2023-10-11T16:30:41.183098+0000","last_clean":"2023-10-11T16:30:41.183098+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:30:41.183098+0000","last_undegraded":"2023-10-11T16:30:41.183098+0000","last_fullsized":"2023-10-11T16:30:41.183098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:13:30.957186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182281+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:30:41.182281+0000","last_peered":"2023-10-11T16:30:41.182281+0000","last_clean":"2023-10-11T16:30:41.182281+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:30:41.182281+0000","last_undegraded":"2023-10-11T16:30:41.182281+0000","last_fullsized":"2023-10-11T16:30:41.182281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:35:47.449805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:28.264672+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:31:28.264672+0000","last_peered":"2023-10-11T16:31:28.264672+0000","last_clean":"2023-10-11T16:31:28.264672+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:31:28.264672+0000","last_undegraded":"2023-10-11T16:31:28.264672+0000","last_fullsized":"2023-10-11T16:31:28.264672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:49:34.210642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182403+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:30:41.182403+0000","last_peered":"2023-10-11T16:30:41.182403+0000","last_clean":"2023-10-11T16:30:41.182403+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:30:41.182403+0000","last_undegraded":"2023-10-11T16:30:41.182403+0000","last_fullsized":"2023-10-11T16:30:41.182403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:49:06.683312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198932+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:30:41.198932+0000","last_peered":"2023-10-11T16:30:41.198932+0000","last_clean":"2023-10-11T16:30:41.198932+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:30:41.198932+0000","last_undegraded":"2023-10-11T16:30:41.198932+0000","last_fullsized":"2023-10-11T16:30:41.198932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:38:08.422304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.187119+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:30:41.187119+0000","last_peered":"2023-10-11T16:30:41.187119+0000","last_clean":"2023-10-11T16:30:41.187119+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:30:41.187119+0000","last_undegraded":"2023-10-11T16:30:41.187119+0000","last_fullsized":"2023-10-11T16:30:41.187119+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:22:31.437710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189360+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:30:41.189360+0000","last_peered":"2023-10-11T16:30:41.189360+0000","last_clean":"2023-10-11T16:30:41.189360+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:30:41.189360+0000","last_undegraded":"2023-10-11T16:30:41.189360+0000","last_fullsized":"2023-10-11T16:30:41.189360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:58.907731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189214+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:30:41.189214+0000","last_peered":"2023-10-11T16:30:41.189214+0000","last_clean":"2023-10-11T16:30:41.189214+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:30:41.189214+0000","last_undegraded":"2023-10-11T16:30:41.189214+0000","last_fullsized":"2023-10-11T16:30:41.189214+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:29:15.740818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.187919+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:30:41.187919+0000","last_peered":"2023-10-11T16:30:41.187919+0000","last_clean":"2023-10-11T16:30:41.187919+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:30:41.187919+0000","last_undegraded":"2023-10-11T16:30:41.187919+0000","last_fullsized":"2023-10-11T16:30:41.187919+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:49:28.444855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185524+0000","last_change":"2023-10-11T16:30:11.670547+0000","last_active":"2023-10-11T16:30:41.185524+0000","last_peered":"2023-10-11T16:30:41.185524+0000","last_clean":"2023-10-11T16:30:41.185524+0000","last_became_active":"2023-10-11T16:30:11.669842+0000","last_became_peered":"2023-10-11T16:30:11.669842+0000","last_unstale":"2023-10-11T16:30:41.185524+0000","last_undegraded":"2023-10-11T16:30:41.185524+0000","last_fullsized":"2023-10-11T16:30:41.185524+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:00:14.369445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186325+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:30:41.186325+0000","last_peered":"2023-10-11T16:30:41.186325+0000","last_clean":"2023-10-11T16:30:41.186325+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:30:41.186325+0000","last_undegraded":"2023-10-11T16:30:41.186325+0000","last_fullsized":"2023-10-11T16:30:41.186325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:54:10.393416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182101+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:30:41.182101+0000","last_peered":"2023-10-11T16:30:41.182101+0000","last_clean":"2023-10-11T16:30:41.182101+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:30:41.182101+0000","last_undegraded":"2023-10-11T16:30:41.182101+0000","last_fullsized":"2023-10-11T16:30:41.182101+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:23:36.264713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190141+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:30:41.190141+0000","last_peered":"2023-10-11T16:30:41.190141+0000","last_clean":"2023-10-11T16:30:41.190141+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:30:41.190141+0000","last_undegraded":"2023-10-11T16:30:41.190141+0000","last_fullsized":"2023-10-11T16:30:41.190141+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:23:03.581102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198172+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:30:41.198172+0000","last_peered":"2023-10-11T16:30:41.198172+0000","last_clean":"2023-10-11T16:30:41.198172+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:30:41.198172+0000","last_undegraded":"2023-10-11T16:30:41.198172+0000","last_fullsized":"2023-10-11T16:30:41.198172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:24:19.511542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182736+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:30:41.182736+0000","last_peered":"2023-10-11T16:30:41.182736+0000","last_clean":"2023-10-11T16:30:41.182736+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:30:41.182736+0000","last_undegraded":"2023-10-11T16:30:41.182736+0000","last_fullsized":"2023-10-11T16:30:41.182736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:10:16.893854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190364+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:30:41.190364+0000","last_peered":"2023-10-11T16:30:41.190364+0000","last_clean":"2023-10-11T16:30:41.190364+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:30:41.190364+0000","last_undegraded":"2023-10-11T16:30:41.190364+0000","last_fullsized":"2023-10-11T16:30:41.190364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:38:35.761085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198257+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:30:41.198257+0000","last_peered":"2023-10-11T16:30:41.198257+0000","last_clean":"2023-10-11T16:30:41.198257+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:30:41.198257+0000","last_undegraded":"2023-10-11T16:30:41.198257+0000","last_fullsized":"2023-10-11T16:30:41.198257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:21:50.405889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189800+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:30:41.189800+0000","last_peered":"2023-10-11T16:30:41.189800+0000","last_clean":"2023-10-11T16:30:41.189800+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:30:41.189800+0000","last_undegraded":"2023-10-11T16:30:41.189800+0000","last_fullsized":"2023-10-11T16:30:41.189800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:41:33.198602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:31:31.268109+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:31:31.268109+0000","last_peered":"2023-10-11T16:31:31.268109+0000","last_clean":"2023-10-11T16:31:31.268109+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:31:31.268109+0000","last_undegraded":"2023-10-11T16:31:31.268109+0000","last_fullsized":"2023-10-11T16:31:31.268109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:37:23.548227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182645+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:30:41.182645+0000","last_peered":"2023-10-11T16:30:41.182645+0000","last_clean":"2023-10-11T16:30:41.182645+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:30:41.182645+0000","last_undegraded":"2023-10-11T16:30:41.182645+0000","last_fullsized":"2023-10-11T16:30:41.182645+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:40:48.239885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403663+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:30:41.403663+0000","last_peered":"2023-10-11T16:30:41.403663+0000","last_clean":"2023-10-11T16:30:41.403663+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:30:41.403663+0000","last_undegraded":"2023-10-11T16:30:41.403663+0000","last_fullsized":"2023-10-11T16:30:41.403663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:58:23.479875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188194+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:30:41.188194+0000","last_peered":"2023-10-11T16:30:41.188194+0000","last_clean":"2023-10-11T16:30:41.188194+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:30:41.188194+0000","last_undegraded":"2023-10-11T16:30:41.188194+0000","last_fullsized":"2023-10-11T16:30:41.188194+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:03.654864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182242+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:30:41.182242+0000","last_peered":"2023-10-11T16:30:41.182242+0000","last_clean":"2023-10-11T16:30:41.182242+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:30:41.182242+0000","last_undegraded":"2023-10-11T16:30:41.182242+0000","last_fullsized":"2023-10-11T16:30:41.182242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:11:53.750545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189957+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:30:41.189957+0000","last_peered":"2023-10-11T16:30:41.189957+0000","last_clean":"2023-10-11T16:30:41.189957+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:30:41.189957+0000","last_undegraded":"2023-10-11T16:30:41.189957+0000","last_fullsized":"2023-10-11T16:30:41.189957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:27:51.812285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.197783+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:30:41.197783+0000","last_peered":"2023-10-11T16:30:41.197783+0000","last_clean":"2023-10-11T16:30:41.197783+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:30:41.197783+0000","last_undegraded":"2023-10-11T16:30:41.197783+0000","last_fullsized":"2023-10-11T16:30:41.197783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:40:50.386941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.938949+0000","last_change":"2023-10-11T16:30:41.938949+0000","last_active":"2023-10-11T16:30:41.938949+0000","last_peered":"2023-10-11T16:30:41.938949+0000","last_clean":"2023-10-11T16:30:41.938949+0000","last_became_active":"2023-10-11T16:30:41.192902+0000","last_became_peered":"2023-10-11T16:30:41.192902+0000","last_unstale":"2023-10-11T16:30:41.938949+0000","last_undegraded":"2023-10-11T16:30:41.938949+0000","last_fullsized":"2023-10-11T16:30:41.938949+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:08:59.943869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186174+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:30:41.186174+0000","last_peered":"2023-10-11T16:30:41.186174+0000","last_clean":"2023-10-11T16:30:41.186174+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:30:41.186174+0000","last_undegraded":"2023-10-11T16:30:41.186174+0000","last_fullsized":"2023-10-11T16:30:41.186174+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:51:47.458050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.205885+0000","last_change":"2023-10-11T16:30:41.205550+0000","last_active":"2023-10-11T16:30:41.205885+0000","last_peered":"2023-10-11T16:30:41.205885+0000","last_clean":"2023-10-11T16:30:41.205885+0000","last_became_active":"2023-10-11T16:30:41.190345+0000","last_became_peered":"2023-10-11T16:30:41.190345+0000","last_unstale":"2023-10-11T16:30:41.205885+0000","last_undegraded":"2023-10-11T16:30:41.205885+0000","last_fullsized":"2023-10-11T16:30:41.205885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:21:53.506003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198393+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:30:41.198393+0000","last_peered":"2023-10-11T16:30:41.198393+0000","last_clean":"2023-10-11T16:30:41.198393+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:30:41.198393+0000","last_undegraded":"2023-10-11T16:30:41.198393+0000","last_fullsized":"2023-10-11T16:30:41.198393+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:25:27.458254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403257+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:30:41.403257+0000","last_peered":"2023-10-11T16:30:41.403257+0000","last_clean":"2023-10-11T16:30:41.403257+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:30:41.403257+0000","last_undegraded":"2023-10-11T16:30:41.403257+0000","last_fullsized":"2023-10-11T16:30:41.403257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:32:39.048475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:45.562548+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:30:45.562548+0000","last_peered":"2023-10-11T16:30:45.562548+0000","last_clean":"2023-10-11T16:30:45.562548+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:30:45.562548+0000","last_undegraded":"2023-10-11T16:30:45.562548+0000","last_fullsized":"2023-10-11T16:30:45.562548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:01:35.861737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403786+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:30:41.403786+0000","last_peered":"2023-10-11T16:30:41.403786+0000","last_clean":"2023-10-11T16:30:41.403786+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:30:41.403786+0000","last_undegraded":"2023-10-11T16:30:41.403786+0000","last_fullsized":"2023-10-11T16:30:41.403786+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:10:56.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185784+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:30:41.185784+0000","last_peered":"2023-10-11T16:30:41.185784+0000","last_clean":"2023-10-11T16:30:41.185784+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:30:41.185784+0000","last_undegraded":"2023-10-11T16:30:41.185784+0000","last_fullsized":"2023-10-11T16:30:41.185784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:57:36.382181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189399+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:30:41.189399+0000","last_peered":"2023-10-11T16:30:41.189399+0000","last_clean":"2023-10-11T16:30:41.189399+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:30:41.189399+0000","last_undegraded":"2023-10-11T16:30:41.189399+0000","last_fullsized":"2023-10-11T16:30:41.189399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:06.520839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185604+0000","last_change":"2023-10-11T16:30:11.661388+0000","last_active":"2023-10-11T16:30:41.185604+0000","last_peered":"2023-10-11T16:30:41.185604+0000","last_clean":"2023-10-11T16:30:41.185604+0000","last_became_active":"2023-10-11T16:30:11.661073+0000","last_became_peered":"2023-10-11T16:30:11.661073+0000","last_unstale":"2023-10-11T16:30:41.185604+0000","last_undegraded":"2023-10-11T16:30:41.185604+0000","last_fullsized":"2023-10-11T16:30:41.185604+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:59:12.390368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.193489+0000","last_change":"2023-10-11T16:30:41.193489+0000","last_active":"2023-10-11T16:30:41.193489+0000","last_peered":"2023-10-11T16:30:41.193489+0000","last_clean":"2023-10-11T16:30:41.193489+0000","last_became_active":"2023-10-11T16:30:41.193060+0000","last_became_peered":"2023-10-11T16:30:41.193060+0000","last_unstale":"2023-10-11T16:30:41.193489+0000","last_undegraded":"2023-10-11T16:30:41.193489+0000","last_fullsized":"2023-10-11T16:30:41.193489+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:52:34.859402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185001+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:30:41.185001+0000","last_peered":"2023-10-11T16:30:41.185001+0000","last_clean":"2023-10-11T16:30:41.185001+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:30:41.185001+0000","last_undegraded":"2023-10-11T16:30:41.185001+0000","last_fullsized":"2023-10-11T16:30:41.185001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:01:22.571040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185793+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:30:41.185793+0000","last_peered":"2023-10-11T16:30:41.185793+0000","last_clean":"2023-10-11T16:30:41.185793+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:30:41.185793+0000","last_undegraded":"2023-10-11T16:30:41.185793+0000","last_fullsized":"2023-10-11T16:30:41.185793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:22:41.435586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.405455+0000","last_change":"2023-10-11T16:30:41.404891+0000","last_active":"2023-10-11T16:30:41.405455+0000","last_peered":"2023-10-11T16:30:41.405455+0000","last_clean":"2023-10-11T16:30:41.405455+0000","last_became_active":"2023-10-11T16:30:41.404396+0000","last_became_peered":"2023-10-11T16:30:41.404396+0000","last_unstale":"2023-10-11T16:30:41.405455+0000","last_undegraded":"2023-10-11T16:30:41.405455+0000","last_fullsized":"2023-10-11T16:30:41.405455+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:39:56.108444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403349+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:30:41.403349+0000","last_peered":"2023-10-11T16:30:41.403349+0000","last_clean":"2023-10-11T16:30:41.403349+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:30:41.403349+0000","last_undegraded":"2023-10-11T16:30:41.403349+0000","last_fullsized":"2023-10-11T16:30:41.403349+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:47:20.053030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.184635+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:30:41.184635+0000","last_peered":"2023-10-11T16:30:41.184635+0000","last_clean":"2023-10-11T16:30:41.184635+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:30:41.184635+0000","last_undegraded":"2023-10-11T16:30:41.184635+0000","last_fullsized":"2023-10-11T16:30:41.184635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:48:42.041910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189125+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:30:41.189125+0000","last_peered":"2023-10-11T16:30:41.189125+0000","last_clean":"2023-10-11T16:30:41.189125+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:30:41.189125+0000","last_undegraded":"2023-10-11T16:30:41.189125+0000","last_fullsized":"2023-10-11T16:30:41.189125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:21.935947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:42.144229+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:30:42.144229+0000","last_peered":"2023-10-11T16:30:42.144229+0000","last_clean":"2023-10-11T16:30:42.144229+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:30:42.144229+0000","last_undegraded":"2023-10-11T16:30:42.144229+0000","last_fullsized":"2023-10-11T16:30:42.144229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:32:39.882844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403694+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:30:41.403694+0000","last_peered":"2023-10-11T16:30:41.403694+0000","last_clean":"2023-10-11T16:30:41.403694+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:30:41.403694+0000","last_undegraded":"2023-10-11T16:30:41.403694+0000","last_fullsized":"2023-10-11T16:30:41.403694+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:23:58.685801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.188391+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:30:41.188391+0000","last_peered":"2023-10-11T16:30:41.188391+0000","last_clean":"2023-10-11T16:30:41.188391+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:30:41.188391+0000","last_undegraded":"2023-10-11T16:30:41.188391+0000","last_fullsized":"2023-10-11T16:30:41.188391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:04.067909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185574+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:30:41.185574+0000","last_peered":"2023-10-11T16:30:41.185574+0000","last_clean":"2023-10-11T16:30:41.185574+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:30:41.185574+0000","last_undegraded":"2023-10-11T16:30:41.185574+0000","last_fullsized":"2023-10-11T16:30:41.185574+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:34.746546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182953+0000","last_change":"2023-10-11T16:30:11.648776+0000","last_active":"2023-10-11T16:30:41.182953+0000","last_peered":"2023-10-11T16:30:41.182953+0000","last_clean":"2023-10-11T16:30:41.182953+0000","last_became_active":"2023-10-11T16:30:11.648465+0000","last_became_peered":"2023-10-11T16:30:11.648465+0000","last_unstale":"2023-10-11T16:30:41.182953+0000","last_undegraded":"2023-10-11T16:30:41.182953+0000","last_fullsized":"2023-10-11T16:30:41.182953+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:44:56.663827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.200104+0000","last_change":"2023-10-11T16:30:41.200104+0000","last_active":"2023-10-11T16:30:41.200104+0000","last_peered":"2023-10-11T16:30:41.200104+0000","last_clean":"2023-10-11T16:30:41.200104+0000","last_became_active":"2023-10-11T16:30:41.199637+0000","last_became_peered":"2023-10-11T16:30:41.199637+0000","last_unstale":"2023-10-11T16:30:41.200104+0000","last_undegraded":"2023-10-11T16:30:41.200104+0000","last_fullsized":"2023-10-11T16:30:41.200104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:08:40.301726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.203024+0000","last_change":"2023-10-11T16:30:41.202718+0000","last_active":"2023-10-11T16:30:41.203024+0000","last_peered":"2023-10-11T16:30:41.203024+0000","last_clean":"2023-10-11T16:30:41.203024+0000","last_became_active":"2023-10-11T16:30:41.190741+0000","last_became_peered":"2023-10-11T16:30:41.190741+0000","last_unstale":"2023-10-11T16:30:41.203024+0000","last_undegraded":"2023-10-11T16:30:41.203024+0000","last_fullsized":"2023-10-11T16:30:41.203024+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:37:17.174287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.183059+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:30:41.183059+0000","last_peered":"2023-10-11T16:30:41.183059+0000","last_clean":"2023-10-11T16:30:41.183059+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:30:41.183059+0000","last_undegraded":"2023-10-11T16:30:41.183059+0000","last_fullsized":"2023-10-11T16:30:41.183059+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:46:18.410074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190477+0000","last_change":"2023-10-11T16:30:41.190477+0000","last_active":"2023-10-11T16:30:41.190477+0000","last_peered":"2023-10-11T16:30:41.190477+0000","last_clean":"2023-10-11T16:30:41.190477+0000","last_became_active":"2023-10-11T16:30:41.189929+0000","last_became_peered":"2023-10-11T16:30:41.189929+0000","last_unstale":"2023-10-11T16:30:41.190477+0000","last_undegraded":"2023-10-11T16:30:41.190477+0000","last_fullsized":"2023-10-11T16:30:41.190477+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:01:59.888888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.190434+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:30:41.190434+0000","last_peered":"2023-10-11T16:30:41.190434+0000","last_clean":"2023-10-11T16:30:41.190434+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:30:41.190434+0000","last_undegraded":"2023-10-11T16:30:41.190434+0000","last_fullsized":"2023-10-11T16:30:41.190434+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:14:41.192364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182589+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:30:41.182589+0000","last_peered":"2023-10-11T16:30:41.182589+0000","last_clean":"2023-10-11T16:30:41.182589+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:30:41.182589+0000","last_undegraded":"2023-10-11T16:30:41.182589+0000","last_fullsized":"2023-10-11T16:30:41.182589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:05:17.909356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189311+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:30:41.189311+0000","last_peered":"2023-10-11T16:30:41.189311+0000","last_clean":"2023-10-11T16:30:41.189311+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:30:41.189311+0000","last_undegraded":"2023-10-11T16:30:41.189311+0000","last_fullsized":"2023-10-11T16:30:41.189311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:47.760490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198308+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:30:41.198308+0000","last_peered":"2023-10-11T16:30:41.198308+0000","last_clean":"2023-10-11T16:30:41.198308+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:30:41.198308+0000","last_undegraded":"2023-10-11T16:30:41.198308+0000","last_fullsized":"2023-10-11T16:30:41.198308+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:33:35.561074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.182863+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:30:41.182863+0000","last_peered":"2023-10-11T16:30:41.182863+0000","last_clean":"2023-10-11T16:30:41.182863+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:30:41.182863+0000","last_undegraded":"2023-10-11T16:30:41.182863+0000","last_fullsized":"2023-10-11T16:30:41.182863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:56:12.549504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186098+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:30:41.186098+0000","last_peered":"2023-10-11T16:30:41.186098+0000","last_clean":"2023-10-11T16:30:41.186098+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:30:41.186098+0000","last_undegraded":"2023-10-11T16:30:41.186098+0000","last_fullsized":"2023-10-11T16:30:41.186098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:53:26.667324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186048+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:30:41.186048+0000","last_peered":"2023-10-11T16:30:41.186048+0000","last_clean":"2023-10-11T16:30:41.186048+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:30:41.186048+0000","last_undegraded":"2023-10-11T16:30:41.186048+0000","last_fullsized":"2023-10-11T16:30:41.186048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:08:50.699880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185837+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:30:41.185837+0000","last_peered":"2023-10-11T16:30:41.185837+0000","last_clean":"2023-10-11T16:30:41.185837+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:30:41.185837+0000","last_undegraded":"2023-10-11T16:30:41.185837+0000","last_fullsized":"2023-10-11T16:30:41.185837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:03:23.234106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185679+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:30:41.185679+0000","last_peered":"2023-10-11T16:30:41.185679+0000","last_clean":"2023-10-11T16:30:41.185679+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:30:41.185679+0000","last_undegraded":"2023-10-11T16:30:41.185679+0000","last_fullsized":"2023-10-11T16:30:41.185679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:10:44.726491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198414+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:30:41.198414+0000","last_peered":"2023-10-11T16:30:41.198414+0000","last_clean":"2023-10-11T16:30:41.198414+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:30:41.198414+0000","last_undegraded":"2023-10-11T16:30:41.198414+0000","last_fullsized":"2023-10-11T16:30:41.198414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:40:14.252893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.186179+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:30:41.186179+0000","last_peered":"2023-10-11T16:30:41.186179+0000","last_clean":"2023-10-11T16:30:41.186179+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:30:41.186179+0000","last_undegraded":"2023-10-11T16:30:41.186179+0000","last_fullsized":"2023-10-11T16:30:41.186179+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:16:58.345842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185560+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:30:41.185560+0000","last_peered":"2023-10-11T16:30:41.185560+0000","last_clean":"2023-10-11T16:30:41.185560+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:30:41.185560+0000","last_undegraded":"2023-10-11T16:30:41.185560+0000","last_fullsized":"2023-10-11T16:30:41.185560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:19:06.908283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403856+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:30:41.403856+0000","last_peered":"2023-10-11T16:30:41.403856+0000","last_clean":"2023-10-11T16:30:41.403856+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:30:41.403856+0000","last_undegraded":"2023-10-11T16:30:41.403856+0000","last_fullsized":"2023-10-11T16:30:41.403856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:52:27.929431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.189673+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:30:41.189673+0000","last_peered":"2023-10-11T16:30:41.189673+0000","last_clean":"2023-10-11T16:30:41.189673+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:30:41.189673+0000","last_undegraded":"2023-10-11T16:30:41.189673+0000","last_fullsized":"2023-10-11T16:30:41.189673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:47:07.501466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.403440+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:30:41.403440+0000","last_peered":"2023-10-11T16:30:41.403440+0000","last_clean":"2023-10-11T16:30:41.403440+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:30:41.403440+0000","last_undegraded":"2023-10-11T16:30:41.403440+0000","last_fullsized":"2023-10-11T16:30:41.403440+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:24:21.537533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185688+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:30:41.185688+0000","last_peered":"2023-10-11T16:30:41.185688+0000","last_clean":"2023-10-11T16:30:41.185688+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:30:41.185688+0000","last_undegraded":"2023-10-11T16:30:41.185688+0000","last_fullsized":"2023-10-11T16:30:41.185688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:25:15.131318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.198464+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:30:41.198464+0000","last_peered":"2023-10-11T16:30:41.198464+0000","last_clean":"2023-10-11T16:30:41.198464+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:30:41.198464+0000","last_undegraded":"2023-10-11T16:30:41.198464+0000","last_fullsized":"2023-10-11T16:30:41.198464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:13:10.823288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-11T16:30:41.185010+0000","last_change":"2023-10-11T16:30:11.668522+0000","last_active":"2023-10-11T16:30:41.185010+0000","last_peered":"2023-10-11T16:30:41.185010+0000","last_clean":"2023-10-11T16:30:41.185010+0000","last_became_active":"2023-10-11T16:30:11.654192+0000","last_became_peered":"2023-10-11T16:30:11.654192+0000","last_unstale":"2023-10-11T16:30:41.185010+0000","last_undegraded":"2023-10-11T16:30:41.185010+0000","last_fullsized":"2023-10-11T16:30:41.185010+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:26:57.163089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":79612,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":347,"num_read_kb":258,"num_write":164,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":39701,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":238836,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449116,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":978944,"data_stored":488446,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":1.204,"15min":1.204},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73699999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.61099999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57099999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.46999999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79200000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70399999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.73599999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.39400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.627}]},{"osd":6,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.50600000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76500000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.70799999999999996},"min":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.006,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83199999999999996,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73599999999999999,"15min":0.74299999999999999},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.86099999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.78000000000000003}]},{"osd":8,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.749}]},{"osd":9,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78700000000000003,"15min":0.76900000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.96,"5min":2.96,"15min":2.96},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.79700000000000004,"15min":0.80100000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.86499999999999999,"5min":1.286,"15min":1.286},"last":0.79300000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.86599999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76500000000000001,"15min":0.77100000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92100000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.76500000000000001}]}]},{"osd":10,"up_from":71,"seq":304942678049,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6604,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738548,"statfs":{"total":95995035648,"available":95988273152,"internally_reserved":0,"allocated":1118208,"data_stored":1141162,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.53100000000000003}]},{"osd":1,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48699999999999999,"15min":0.48999999999999999},"min":{"1min":0.34599999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72699999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.433},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51000000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.47299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.747},{"interface":"front","average":{"1min":0.627,"5min":0.66200000000000003,"15min":0.66800000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77700000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.66300000000000003}]},{"osd":3,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.65400000000000003,"15min":0.66600000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.71599999999999997,"5min":0.995,"15min":0.995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65800000000000003,"15min":0.66700000000000004},"min":{"1min":0.46100000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.80100000000000005,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56999999999999995}]},{"osd":4,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59999999999999998,"15min":0.60599999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.74099999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.36599999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.48899999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89000000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72399999999999998,"15min":0.72599999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.92100000000000004,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.71999999999999997}]},{"osd":6,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85699999999999998,"5min":1.085,"15min":1.085},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.93300000000000005,"5min":1.03,"15min":1.03},"last":0.63200000000000001}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.76200000000000001,"15min":0.76300000000000001},"min":{"1min":0.57699999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97099999999999997,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.872,"15min":0.88800000000000001},"min":{"1min":0.63200000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":0.95899999999999996,"5min":2.46,"15min":2.46},"last":0.79600000000000004}]},{"osd":9,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.81299999999999994,"15min":0.82399999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.98199999999999998,"5min":1.286,"15min":1.286},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.81100000000000005,"15min":0.81899999999999995},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.93799999999999994,"5min":1.347,"15min":1.347},"last":0.80700000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.876,"5min":1.073,"15min":1.073},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90100000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.73199999999999998}]}]},{"osd":9,"up_from":66,"seq":283467841572,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":991232,"data_stored":492174,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.54200000000000004,"15min":0.55400000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.70199999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.42399999999999999}]},{"osd":1,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.54300000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71199999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.86599999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.499}]},{"osd":2,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.79300000000000004,"15min":0.76500000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":7.056,"5min":7.056,"15min":7.056},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.81299999999999994,"15min":0.78500000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":7.1529999999999996,"5min":7.1529999999999996,"15min":7.1529999999999996},"last":0.52900000000000003}]},{"osd":3,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.70499999999999996,"15min":0.73099999999999998},"min":{"1min":0.38600000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.79200000000000004,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.69699999999999995,"15min":0.71199999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.878,"5min":1.018,"15min":1.018},"last":0.55900000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.80400000000000005,"15min":0.81499999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.78700000000000003,"15min":0.79900000000000004},"min":{"1min":0.58099999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.224,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.57399999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.82199999999999995,"15min":0.84699999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.036,"5min":1.875,"15min":1.875},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.82499999999999996,"15min":0.84799999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.99299999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.89100000000000001,"15min":0.90300000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88500000000000001,"15min":0.89200000000000002},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":0.628}]},{"osd":8,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":0.97299999999999998,"15min":0.95499999999999996},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":7.1070000000000002,"5min":7.1070000000000002,"15min":7.1070000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.075,"5min":0.98599999999999999,"15min":0.97099999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":7.1260000000000003,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.72399999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.88400000000000001,"15min":0.89500000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.96199999999999997,"15min":0.96999999999999997},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.73699999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.7320000000000002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.66100000000000003}]}]},{"osd":8,"up_from":58,"seq":249108103210,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6468,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738684,"statfs":{"total":95995035648,"available":95988412416,"internally_reserved":0,"allocated":978944,"data_stored":487645,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91600000000000004,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":0.996,"15min":0.996},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82599999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.44600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.59799999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.77400000000000002,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0649999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.70199999999999996}]},{"osd":4,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82599999999999996,"5min":1.028,"15min":1.028},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90900000000000003,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.622}]},{"osd":5,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82499999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55900000000000005,"15min":0.54500000000000004},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.89400000000000002}]},{"osd":6,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.179,"15min":1.179},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.499,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.75900000000000001}]},{"osd":7,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.68899999999999995,"15min":0.68100000000000005},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.95299999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.69099999999999995,"15min":0.68300000000000005},"min":{"1min":0.56599999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.99399999999999999,"5min":1.101,"15min":1.101},"last":0.93600000000000005}]},{"osd":9,"last update":"Wed Oct 11 16:30:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.51800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.71999999999999997,"15min":0.71699999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.92000000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.71199999999999997,"15min":0.70099999999999996},"min":{"1min":0.56499999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.69599999999999995,"15min":0.68700000000000006},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.86199999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.876},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.98199999999999998}]}]},{"osd":7,"up_from":51,"seq":219043332142,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6592,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738560,"statfs":{"total":95995035648,"available":95988285440,"internally_reserved":0,"allocated":1040384,"data_stored":1092836,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.438,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68000000000000005,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.439},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.46300000000000002,"15min":0.45300000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.41499999999999998}]},{"osd":2,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59599999999999997,"15min":0.59199999999999997},"min":{"1min":0.42899999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76100000000000001,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83299999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.55100000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.39100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94999999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.875},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.625,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80200000000000005,"5min":1.103,"15min":1.103},"last":0.53700000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.74299999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.502,"15min":0.5},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.40000000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65000000000000002,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90300000000000002,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.66200000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:30:58 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70199999999999996,"15min":0.70799999999999996},"min":{"1min":0.496,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89000000000000001,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.67100000000000004,"15min":0.66100000000000003},"min":{"1min":0.49099999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97799999999999998,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64700000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71199999999999997,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72599999999999998,"15min":0.72799999999999998},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.68300000000000005}]},{"osd":9,"last update":"Wed Oct 11 16:30:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.77100000000000002,"15min":0.77400000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.995,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.76800000000000002,"15min":0.77200000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.99199999999999999,"5min":1.042,"15min":1.042},"last":0.69899999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.70199999999999996,"15min":0.69399999999999995},"min":{"1min":0.436,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.73499999999999999,"15min":0.72999999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0169999999999999,"5min":1.099,"15min":1.099},"last":0.78600000000000003}]},{"osd":11,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.80700000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495667,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6392,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738760,"statfs":{"total":95995035648,"available":95988490240,"internally_reserved":0,"allocated":966656,"data_stored":483254,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.438,"15min":0.43099999999999999},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57499999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.376},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54600000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63900000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.495}]},{"osd":1,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.57899999999999996,"15min":0.56100000000000005},"min":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78400000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.435},{"interface":"front","average":{"1min":0.63,"5min":0.60699999999999998,"15min":0.58499999999999996},"min":{"1min":0.44400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85299999999999998,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.47499999999999998}]},{"osd":2,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.47999999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.75700000000000001,"5min":1.002,"15min":1.002},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.64200000000000002,"15min":0.63400000000000001},"min":{"1min":0.47299999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.83599999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56699999999999995,"15min":0.55500000000000005},"min":{"1min":0.38400000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79700000000000004,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.74099999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.38800000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.70399999999999996,"15min":0.69599999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90400000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66800000000000004,"15min":0.65200000000000002},"min":{"1min":0.52600000000000002,"5min":0.502,"15min":0.502},"max":{"1min":0.89100000000000001,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.52800000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67800000000000005,"15min":0.65900000000000003},"min":{"1min":0.52500000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.871,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.628},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.5,"5min":0.434,"15min":0.434},"max":{"1min":0.877,"5min":1.133,"15min":1.133},"last":0.77700000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:30:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.76700000000000002,"15min":0.755},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72699999999999998,"15min":0.71999999999999997},"min":{"1min":0.58099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.878,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.60899999999999999}]},{"osd":8,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.752},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.875,"15min":0.874},"min":{"1min":0.60099999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.74399999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.85799999999999998,"15min":0.876},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0409999999999999,"5min":1.655,"15min":1.655},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.88800000000000001,"15min":0.90400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.179,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.80200000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:31:07 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.81899999999999995,"15min":0.82899999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0740000000000001,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.873,"15min":0.875},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.2889999999999999,"5min":1.752,"15min":1.752},"last":0.52100000000000002}]},{"osd":11,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.5640000000000001,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":0.73699999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7548,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737604,"statfs":{"total":95995035648,"available":95987306496,"internally_reserved":0,"allocated":1036288,"data_stored":549515,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56200000000000006,"15min":0.61699999999999999},"min":{"1min":0.29499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.84899999999999998,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.872,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.626}]},{"osd":2,"last update":"Wed Oct 11 16:31:05 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.53100000000000003,"15min":0.51500000000000001},"min":{"1min":0.34499999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.71699999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.56399999999999995},"min":{"1min":0.438,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.80700000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.443,"15min":0.41699999999999998},"min":{"1min":0.31,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.82799999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.498,"15min":0.45100000000000001},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.73999999999999999}]},{"osd":5,"last update":"Wed Oct 11 16:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.55200000000000005,"15min":0.498},"min":{"1min":0.38800000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.59999999999999998,"15min":0.53800000000000003},"min":{"1min":0.51900000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.82799999999999996}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.626},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.874}]},{"osd":7,"last update":"Wed Oct 11 16:31:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.53800000000000003,"15min":0.502},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.626,"15min":0.59499999999999997},"min":{"1min":0.40300000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.84199999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.69499999999999995,"15min":0.66600000000000004},"min":{"1min":0.52300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.66200000000000003,"15min":0.63},"min":{"1min":0.59599999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.99199999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:30:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.72199999999999998,"15min":0.71399999999999997},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.76700000000000002,"15min":0.76100000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.93600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.68400000000000005,"15min":0.66800000000000004},"min":{"1min":0.42799999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.72699999999999998},"min":{"1min":0.53700000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.86199999999999999}]},{"osd":11,"last update":"Wed Oct 11 16:31:11 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":1.024},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.90700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7376,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737776,"statfs":{"total":95995035648,"available":95987482624,"internally_reserved":0,"allocated":991232,"data_stored":484642,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:30:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.33100000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57599999999999996,"15min":0.53300000000000003},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.70399999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.62,"15min":0.58399999999999996},"min":{"1min":0.39300000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.64300000000000002,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.78000000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:30:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59499999999999997,"15min":0.55500000000000005},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86599999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.628,"15min":0.58999999999999997},"min":{"1min":0.47099999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90000000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.66200000000000003}]},{"osd":5,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.60699999999999998},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66300000000000003,"15min":0.63900000000000001},"min":{"1min":0.502,"5min":0.436,"15min":0.436},"max":{"1min":0.94699999999999995,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.81000000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:30:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.63500000000000001,"15min":0.61899999999999999},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.755,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.746},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.74299999999999999,"5min":0.999,"15min":0.999},"last":0.72599999999999998}]},{"osd":7,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.64900000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.77300000000000002,"15min":0.753},"min":{"1min":0.63500000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":3.073,"5min":3.073,"15min":3.073},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75800000000000001,"15min":0.76200000000000001},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.68999999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:30:47 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.82599999999999996,"15min":0.84599999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.04,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.81699999999999995,"15min":0.83299999999999996},"min":{"1min":0.47599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93700000000000006,"5min":1.179,"15min":1.179},"last":0.77300000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:31:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.60899999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0649999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.82199999999999995,"15min":0.82599999999999996},"min":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.79200000000000004}]},{"osd":11,"last update":"Wed Oct 11 16:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.64100000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.70399999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.80200000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7440,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737712,"statfs":{"total":95995035648,"available":95987417088,"internally_reserved":0,"allocated":991232,"data_stored":493882,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.41699999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.82899999999999996,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39900000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71099999999999997,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45500000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:31:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56499999999999995,"15min":0.51900000000000002},"min":{"1min":0.36499999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.82499999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56799999999999995,"15min":0.51900000000000002},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.65600000000000003}]},{"osd":3,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57599999999999996,"15min":0.54500000000000004},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.63,"15min":0.57499999999999996},"min":{"1min":0.51100000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Wed Oct 11 16:31:13 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.56599999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:31:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.502,"15min":0.46800000000000003},"min":{"1min":0.35599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.78600000000000003,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.503,"15min":0.48199999999999998},"min":{"1min":0.39600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82399999999999995,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.74299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.64200000000000002,"15min":0.623},"min":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86599999999999999,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.65600000000000003,"15min":0.63900000000000001},"min":{"1min":0.46600000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.69699999999999995}]},{"osd":7,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.89100000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:31:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.64400000000000002,"15min":0.61499999999999999},"min":{"1min":0.48299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.64800000000000002,"15min":0.623},"min":{"1min":0.49199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.78400000000000003}]},{"osd":9,"last update":"Wed Oct 11 16:30:40 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.52600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.73299999999999998,"15min":0.71999999999999997},"min":{"1min":0.60099999999999998,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.77100000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.748,"15min":0.73699999999999999},"min":{"1min":0.56999999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.78500000000000003,"15min":0.77200000000000002},"min":{"1min":0.61299999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.879}]},{"osd":11,"last update":"Wed Oct 11 16:31:13 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.90300000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6720,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738432,"statfs":{"total":95995035648,"available":95988154368,"internally_reserved":0,"allocated":974848,"data_stored":483117,"data_compressed":17551,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41399999999999998,"15min":0.38400000000000001},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.43099999999999999,"15min":0.40899999999999997},"min":{"1min":0.33100000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.56999999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.5}]},{"osd":1,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.62,"15min":0.57499999999999996},"min":{"1min":0.47799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79600000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58999999999999997,"15min":0.55600000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77000000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.55800000000000005}]},{"osd":2,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64700000000000002,"15min":0.60899999999999999},"min":{"1min":0.439,"5min":0.379,"15min":0.379},"max":{"1min":0.91000000000000003,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65000000000000002,"15min":0.59399999999999997},"min":{"1min":0.48999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80100000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.54200000000000004}]},{"osd":4,"last update":"Wed Oct 11 16:30:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.65700000000000003,"15min":0.63800000000000001},"min":{"1min":0.51600000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.83899999999999997,"5min":1.101,"15min":1.101},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.70699999999999996,"15min":0.68400000000000005},"min":{"1min":0.55900000000000005,"5min":0.498,"15min":0.498},"max":{"1min":0.97199999999999998,"5min":1.194,"15min":1.194},"last":0.71999999999999997}]},{"osd":5,"last update":"Wed Oct 11 16:31:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.625,"15min":0.63200000000000001},"min":{"1min":0.41999999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76100000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.68600000000000005,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81899999999999995,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65300000000000002}]},{"osd":6,"last update":"Wed Oct 11 16:30:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.91500000000000004,"5min":1.268,"15min":1.268},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74099999999999999,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.45900000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:31:03 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.68300000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.85499999999999998,"5min":1.119,"15min":1.119},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.72999999999999998,"15min":0.72399999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.84199999999999997,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.623}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:30:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.79900000000000004,"15min":0.80900000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.98699999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.79700000000000004,"15min":0.80600000000000005},"min":{"1min":0.53200000000000003,"5min":0.502,"15min":0.502},"max":{"1min":0.92100000000000004,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.63300000000000001}]},{"osd":10,"last update":"Wed Oct 11 16:31:08 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.81699999999999995,"15min":0.82699999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.94499999999999995,"5min":1.137,"15min":1.137},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.82899999999999996,"15min":0.83499999999999996},"min":{"1min":0.67300000000000004,"5min":0.627,"15min":0.627},"max":{"1min":1.0009999999999999,"5min":1.165,"15min":1.165},"last":0.77100000000000002}]},{"osd":11,"last update":"Wed Oct 11 16:31:14 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.752},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.64400000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6640,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738512,"statfs":{"total":95995035648,"available":95988236288,"internally_reserved":0,"allocated":1024000,"data_stored":553956,"data_compressed":24251,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:30:54 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.375},"min":{"1min":0.28799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.54600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.377},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38500000000000001},"min":{"1min":0.311,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57599999999999996,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40300000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:30:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.57599999999999996,"15min":0.55300000000000005},"min":{"1min":0.442,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.86099999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.81100000000000005,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.495}]},{"osd":3,"last update":"Wed Oct 11 16:30:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.53900000000000003},"min":{"1min":0.439,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68300000000000005,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.52800000000000002},"min":{"1min":0.44,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71599999999999997,"5min":0.875,"15min":0.875},"last":0.51000000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.69599999999999995,"15min":0.74399999999999999},"min":{"1min":0.38200000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86399999999999999,"5min":4.5,"15min":4.5},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.66700000000000004,"15min":0.72299999999999998},"min":{"1min":0.36699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.873,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.65300000000000002}]},{"osd":6,"last update":"Wed Oct 11 16:30:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.55500000000000005}]},{"osd":7,"last update":"Wed Oct 11 16:31:02 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49199999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67500000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.56499999999999995,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83399999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.58599999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.48999999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76800000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.61599999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:30:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.55000000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.91900000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.55400000000000005,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.69299999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.54400000000000004,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.873,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72699999999999998,"15min":0.71899999999999997},"min":{"1min":0.58499999999999996,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.96499999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.80100000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.84999999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6732,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738420,"statfs":{"total":95995035648,"available":95988142080,"internally_reserved":0,"allocated":1052672,"data_stored":1089054,"data_compressed":53301,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53200000000000003,"15min":0.52400000000000002},"min":{"1min":0.375,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.40200000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67800000000000005,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.54200000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.56299999999999994,"15min":0.53400000000000003},"min":{"1min":0.42899999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76400000000000001,"5min":0.874,"15min":0.874},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81999999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.374}]},{"osd":2,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.42999999999999999,"15min":0.40400000000000003},"min":{"1min":0.35499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.63100000000000001,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.42399999999999999},"min":{"1min":0.313,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.67200000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.436}]},{"osd":3,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56899999999999995,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.46700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.60999999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.61399999999999999,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.61399999999999999,"15min":0.58499999999999996},"min":{"1min":0.47799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.499}]},{"osd":6,"last update":"Wed Oct 11 16:30:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.624},"min":{"1min":0.47199999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78100000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.52400000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:31:00 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.71299999999999997,"15min":0.71699999999999997},"min":{"1min":0.496,"5min":0.44,"15min":0.44},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.625},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.70499999999999996,"15min":0.69999999999999996},"min":{"1min":0.502,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57599999999999996}]},{"osd":8,"last update":"Wed Oct 11 16:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68999999999999995},"min":{"1min":0.502,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83699999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58699999999999997}]},{"osd":9,"last update":"Wed Oct 11 16:30:45 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.749,"15min":0.75},"min":{"1min":0.50700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93899999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.76100000000000001,"15min":0.76400000000000001},"min":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.95899999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.65300000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.73099999999999998,"15min":0.72099999999999997},"min":{"1min":0.53800000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.74099999999999999,"15min":0.73199999999999998},"min":{"1min":0.55800000000000005,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.69199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":10718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":66696,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":11426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":11112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:31:34.602 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-10-11T16:31:34.603 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-11T16:31:34.603 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-10-11T16:31:34.603 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph health --format=json 2023-10-11T16:31:35.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:35 smithi111 ceph-mon[98527]: pgmap v316: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:35.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:35 smithi111 ceph-mon[98527]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:31:35.450 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:35.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:35 smithi150 ceph-mon[132217]: pgmap v316: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:35.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:35 smithi150 ceph-mon[132217]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:31:35.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:35 smithi130 ceph-mon[131859]: pgmap v316: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:35.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:35 smithi130 ceph-mon[131859]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:31:36.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 -- 172.21.15.111:0/3286529570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f101055e0 msgr2=0x7f8f1010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:36.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 --2- 172.21.15.111:0/3286529570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f101055e0 0x7f8f1010d160 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800b0d0 tx=0x7f8f0800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:31:36.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 -- 172.21.15.111:0/3286529570 shutdown_connections 2023-10-11T16:31:36.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 --2- 172.21.15.111:0/3286529570 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f101055e0 0x7f8f1010d160 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:36.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 --2- 172.21.15.111:0/3286529570 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f10100540 0x7f8f10100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:36.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 --2- 172.21.15.111:0/3286529570 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f10100ee0 0x7f8f10104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:36.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 -- 172.21.15.111:0/3286529570 >> 172.21.15.111:0/3286529570 conn(0x7f8f100fbdc0 msgr2=0x7f8f100fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:36.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 -- 172.21.15.111:0/3286529570 shutdown_connections 2023-10-11T16:31:36.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.926+0000 7f8f16b41700 1 -- 172.21.15.111:0/3286529570 wait complete. 2023-10-11T16:31:36.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.927+0000 7f8f16b41700 1 Processor -- start 2023-10-11T16:31:36.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.927+0000 7f8f16b41700 1 -- start start 2023-10-11T16:31:36.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f16b41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f10100540 0x7f8f10072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:36.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f16b41700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f10100ee0 0x7f8f10073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:36.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f16b41700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f101055e0 0x7f8f101a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:36.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f16b41700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10073c20 con 0x7f8f10100540 2023-10-11T16:31:36.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f16b41700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101a4c10 con 0x7f8f10100ee0 2023-10-11T16:31:36.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f16b41700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101a4d80 con 0x7f8f101055e0 2023-10-11T16:31:36.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f148dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f10100540 0x7f8f10072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:36.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f148dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f10100540 0x7f8f10072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47190/0 (socket says 172.21.15.111:47190) 2023-10-11T16:31:36.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f148dd700 1 -- 172.21.15.111:0/339363897 learned_addr learned my addr 172.21.15.111:0/339363897 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:36.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f0ffff700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f10100ee0 0x7f8f10073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:36.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.928+0000 7f8f150de700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f101055e0 0x7f8f101a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:36.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f148dd700 1 -- 172.21.15.111:0/339363897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f101055e0 msgr2=0x7f8f101a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:36.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f148dd700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f101055e0 0x7f8f101a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:36.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f148dd700 1 -- 172.21.15.111:0/339363897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f10100ee0 msgr2=0x7f8f10073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:36.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f148dd700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f10100ee0 0x7f8f10073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:36.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f148dd700 1 -- 172.21.15.111:0/339363897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08009d00 con 0x7f8f10100540 2023-10-11T16:31:36.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f148dd700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f10100540 0x7f8f10072df0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400e9a0 tx=0x7f8f0400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:36.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f0dffb700 1 -- 172.21.15.111:0/339363897 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04009790 con 0x7f8f10100540 2023-10-11T16:31:36.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f0dffb700 1 -- 172.21.15.111:0/339363897 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f0400cbd0 con 0x7f8f10100540 2023-10-11T16:31:36.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f10110320 con 0x7f8f10100540 2023-10-11T16:31:36.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f0dffb700 1 -- 172.21.15.111:0/339363897 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04018680 con 0x7f8f10100540 2023-10-11T16:31:36.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.929+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f10110190 con 0x7f8f10100540 2023-10-11T16:31:36.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.931+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f101104b0 con 0x7f8f10100540 2023-10-11T16:31:36.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.932+0000 7f8f0dffb700 1 -- 172.21.15.111:0/339363897 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8f0400cd40 con 0x7f8f10100540 2023-10-11T16:31:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.933+0000 7f8f0dffb700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8f000a1260 0x7f8f000a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.934+0000 7f8f0dffb700 1 -- 172.21.15.111:0/339363897 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8f04014070 con 0x7f8f10100540 2023-10-11T16:31:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.934+0000 7f8f0ffff700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8f000a1260 0x7f8f000a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:36.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.934+0000 7f8f0ffff700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8f000a1260 0x7f8f000a3710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8f100745e0 tx=0x7f8efc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:36.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:36.937+0000 7f8f0dffb700 1 -- 172.21.15.111:0/339363897 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8f04051730 con 0x7f8f10100540 2023-10-11T16:31:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.177+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8f101a57b0 con 0x7f8f10100540 2023-10-11T16:31:37.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.177+0000 7f8f0dffb700 1 -- 172.21.15.111:0/339363897 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f8f040962b0 con 0x7f8f10100540 2023-10-11T16:31:37.179 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:37.179 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-11T16:31:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8f000a1260 msgr2=0x7f8f000a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:37.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8f000a1260 0x7f8f000a3710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8f100745e0 tx=0x7f8efc009260 comp rx=0 tx=0).stop 2023-10-11T16:31:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f10100540 msgr2=0x7f8f10072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f10100540 0x7f8f10072df0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400e9a0 tx=0x7f8f0400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 shutdown_connections 2023-10-11T16:31:37.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8f000a1260 0x7f8f000a3710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f101055e0 0x7f8f101a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f10100540 0x7f8f10072df0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 --2- 172.21.15.111:0/339363897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f10100ee0 0x7f8f10073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 >> 172.21.15.111:0/339363897 conn(0x7f8f100fbdc0 msgr2=0x7f8f100fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:37.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 shutdown_connections 2023-10-11T16:31:37.184 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.180+0000 7f8f16b41700 1 -- 172.21.15.111:0/339363897 wait complete. 2023-10-11T16:31:37.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:37 smithi111 ceph-mon[98527]: pgmap v317: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:37.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:37 smithi150 ceph-mon[132217]: pgmap v317: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:37.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:37 smithi130 ceph-mon[131859]: pgmap v317: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:37.649 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-10-11T16:31:37.650 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:31:37.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/4117900711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e38101e30 msgr2=0x7f6e381086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:37.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/4117900711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e38101e30 0x7f6e381086f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000b0a0 tx=0x7f6e3000b3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/4117900711 shutdown_connections 2023-10-11T16:31:37.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/4117900711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e38101e30 0x7f6e381086f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/4117900711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e381002b0 0x7f6e381006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/4117900711 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e381014a0 0x7f6e381018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/4117900711 >> 172.21.15.111:0/4117900711 conn(0x7f6e380fb890 msgr2=0x7f6e380fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:37.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/4117900711 shutdown_connections 2023-10-11T16:31:37.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.849+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/4117900711 wait complete. 2023-10-11T16:31:37.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.850+0000 7f6e3d9ff700 1 Processor -- start 2023-10-11T16:31:37.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.850+0000 7f6e3d9ff700 1 -- start start 2023-10-11T16:31:37.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e3d9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e381002b0 0x7f6e381982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:37.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e3d9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e381014a0 0x7f6e381987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:37.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e3d9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e38101e30 0x7f6e3819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:37.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e3d9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3819fe20 con 0x7f6e381002b0 2023-10-11T16:31:37.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e3d9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3819ff60 con 0x7f6e38101e30 2023-10-11T16:31:37.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e3d9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e381a00f0 con 0x7f6e381014a0 2023-10-11T16:31:37.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e36ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e381002b0 0x7f6e381982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:37.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e36ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e381002b0 0x7f6e381982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47222/0 (socket says 172.21.15.111:47222) 2023-10-11T16:31:37.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e36ffd700 1 -- 172.21.15.111:0/3081741133 learned_addr learned my addr 172.21.15.111:0/3081741133 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:37.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e377fe700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e38101e30 0x7f6e3819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:37.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e2ffff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e381014a0 0x7f6e381987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:37.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e36ffd700 1 -- 172.21.15.111:0/3081741133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e381014a0 msgr2=0x7f6e381987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:37.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e36ffd700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e381014a0 0x7f6e381987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e36ffd700 1 -- 172.21.15.111:0/3081741133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e38101e30 msgr2=0x7f6e3819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:37.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e36ffd700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e38101e30 0x7f6e3819f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:37.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e36ffd700 1 -- 172.21.15.111:0/3081741133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e30009d00 con 0x7f6e381002b0 2023-10-11T16:31:37.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.851+0000 7f6e377fe700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e38101e30 0x7f6e3819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:37.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.852+0000 7f6e36ffd700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e381002b0 0x7f6e381982a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6e2800e970 tx=0x7f6e2800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:37.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.852+0000 7f6e3c9fd700 1 -- 172.21.15.111:0/3081741133 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28009790 con 0x7f6e381002b0 2023-10-11T16:31:37.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.852+0000 7f6e3c9fd700 1 -- 172.21.15.111:0/3081741133 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e2800c7e0 con 0x7f6e381002b0 2023-10-11T16:31:37.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.852+0000 7f6e3c9fd700 1 -- 172.21.15.111:0/3081741133 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e28003c70 con 0x7f6e381002b0 2023-10-11T16:31:37.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.852+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3810bc80 con 0x7f6e381002b0 2023-10-11T16:31:37.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.852+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3810baf0 con 0x7f6e381002b0 2023-10-11T16:31:37.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.853+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e381a07f0 con 0x7f6e381002b0 2023-10-11T16:31:37.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.855+0000 7f6e3c9fd700 1 -- 172.21.15.111:0/3081741133 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f6e2800cc50 con 0x7f6e381002b0 2023-10-11T16:31:37.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.858+0000 7f6e3c9fd700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6e240a11e0 0x7f6e240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:37.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.858+0000 7f6e3c9fd700 1 -- 172.21.15.111:0/3081741133 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6e28014070 con 0x7f6e381002b0 2023-10-11T16:31:37.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.858+0000 7f6e2ffff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6e240a11e0 0x7f6e240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:37.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.859+0000 7f6e2ffff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6e240a11e0 0x7f6e240a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6e20007be0 tx=0x7f6e200073d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:37.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:37.862+0000 7f6e3c9fd700 1 -- 172.21.15.111:0/3081741133 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6e28096220 con 0x7f6e381002b0 2023-10-11T16:31:38.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.067+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e381a0e90 con 0x7f6e381002b0 2023-10-11T16:31:38.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.069+0000 7f6e3c9fd700 1 -- 172.21.15.111:0/3081741133 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24555 (secure 0 0 0) 0x7f6e28003dd0 con 0x7f6e381002b0 2023-10-11T16:31:38.071 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:38.072 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":89,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:41.177705+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:31:38.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6e240a11e0 msgr2=0x7f6e240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6e240a11e0 0x7f6e240a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6e20007be0 tx=0x7f6e200073d0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e381002b0 msgr2=0x7f6e381982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e381002b0 0x7f6e381982a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6e2800e970 tx=0x7f6e2800ec80 comp rx=0 tx=0).stop 2023-10-11T16:31:38.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 shutdown_connections 2023-10-11T16:31:38.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6e240a11e0 0x7f6e240a3690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e38101e30 0x7f6e3819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e381002b0 0x7f6e381982a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 --2- 172.21.15.111:0/3081741133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e381014a0 0x7f6e381987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 >> 172.21.15.111:0/3081741133 conn(0x7f6e380fb890 msgr2=0x7f6e38106cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 shutdown_connections 2023-10-11T16:31:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.072+0000 7f6e3d9ff700 1 -- 172.21.15.111:0/3081741133 wait complete. 2023-10-11T16:31:38.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:31:38.094 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:31:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 -- 172.21.15.111:0/3220341550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41002c0 msgr2=0x7f90e41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/3220341550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41002c0 0x7f90e41006d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f90d4009990 tx=0x7f90d4009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 -- 172.21.15.111:0/3220341550 shutdown_connections 2023-10-11T16:31:38.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/3220341550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90e4101e40 0x7f90e4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/3220341550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41002c0 0x7f90e41006d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/3220341550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e41014b0 0x7f90e4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 -- 172.21.15.111:0/3220341550 >> 172.21.15.111:0/3220341550 conn(0x7f90e40fb8a0 msgr2=0x7f90e40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:38.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 -- 172.21.15.111:0/3220341550 shutdown_connections 2023-10-11T16:31:38.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.293+0000 7f90ea3ff700 1 -- 172.21.15.111:0/3220341550 wait complete. 2023-10-11T16:31:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90ea3ff700 1 Processor -- start 2023-10-11T16:31:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90ea3ff700 1 -- start start 2023-10-11T16:31:38.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90ea3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e41002c0 0x7f90e4196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90ea3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41014b0 0x7f90e41965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90ea3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90e4101e40 0x7f90e419d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90ea3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e419dbe0 con 0x7f90e41014b0 2023-10-11T16:31:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90ea3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e419dd20 con 0x7f90e4101e40 2023-10-11T16:31:38.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90ea3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e419deb0 con 0x7f90e41002c0 2023-10-11T16:31:38.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.295+0000 7f90e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41014b0 0x7f90e41965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:38.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41014b0 0x7f90e41965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47242/0 (socket says 172.21.15.111:47242) 2023-10-11T16:31:38.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e37fe700 1 -- 172.21.15.111:0/4124284433 learned_addr learned my addr 172.21.15.111:0/4124284433 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:38.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e3fff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e41002c0 0x7f90e4196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:38.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e899c700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90e4101e40 0x7f90e419d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:38.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e37fe700 1 -- 172.21.15.111:0/4124284433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e41002c0 msgr2=0x7f90e4196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e37fe700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e41002c0 0x7f90e4196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e37fe700 1 -- 172.21.15.111:0/4124284433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90e4101e40 msgr2=0x7f90e419d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e37fe700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90e4101e40 0x7f90e419d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e37fe700 1 -- 172.21.15.111:0/4124284433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d40095f0 con 0x7f90e41014b0 2023-10-11T16:31:38.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e37fe700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41014b0 0x7f90e41965a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f90d800e9a0 tx=0x7f90d800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:38.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e17fa700 1 -- 172.21.15.111:0/4124284433 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d8009790 con 0x7f90e41014b0 2023-10-11T16:31:38.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e17fa700 1 -- 172.21.15.111:0/4124284433 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90d8010430 con 0x7f90e41014b0 2023-10-11T16:31:38.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e410bc90 con 0x7f90e41014b0 2023-10-11T16:31:38.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.296+0000 7f90e17fa700 1 -- 172.21.15.111:0/4124284433 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d8018680 con 0x7f90e41014b0 2023-10-11T16:31:38.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.297+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e410bb00 con 0x7f90e41014b0 2023-10-11T16:31:38.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.298+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e419e5a0 con 0x7f90e41014b0 2023-10-11T16:31:38.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.299+0000 7f90e17fa700 1 -- 172.21.15.111:0/4124284433 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f90d80105a0 con 0x7f90e41014b0 2023-10-11T16:31:38.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.301+0000 7f90e17fa700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f90cc0a11e0 0x7f90cc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:38.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.302+0000 7f90e3fff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f90cc0a11e0 0x7f90cc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:38.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.302+0000 7f90e17fa700 1 -- 172.21.15.111:0/4124284433 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f90d8014070 con 0x7f90e41014b0 2023-10-11T16:31:38.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.302+0000 7f90e3fff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f90cc0a11e0 0x7f90cc0a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f90d400b2e0 tx=0x7f90d4005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:38.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.305+0000 7f90e17fa700 1 -- 172.21.15.111:0/4124284433 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f90d8051730 con 0x7f90e41014b0 2023-10-11T16:31:38.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:38 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/339363897' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-11T16:31:38.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:38 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3081741133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:38.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:38 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/339363897' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-11T16:31:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:38 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3081741133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:38.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.508+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f90e4107170 con 0x7f90e41014b0 2023-10-11T16:31:38.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.508+0000 7f90e17fa700 1 -- 172.21.15.111:0/4124284433 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90d80d0030 con 0x7f90e41014b0 2023-10-11T16:31:38.510 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:31:38.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.510+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f90cc0a11e0 msgr2=0x7f90cc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.510+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f90cc0a11e0 0x7f90cc0a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f90d400b2e0 tx=0x7f90d4005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.510+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41014b0 msgr2=0x7f90e41965a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.510+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41014b0 0x7f90e41965a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f90d800e9a0 tx=0x7f90d800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.511+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 shutdown_connections 2023-10-11T16:31:38.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.511+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f90cc0a11e0 0x7f90cc0a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.511+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90e4101e40 0x7f90e419d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.511+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90e41002c0 0x7f90e4196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.511+0000 7f90ea3ff700 1 --2- 172.21.15.111:0/4124284433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e41014b0 0x7f90e41965a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.511+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 >> 172.21.15.111:0/4124284433 conn(0x7f90e40fb8a0 msgr2=0x7f90e40fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:38.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.511+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 shutdown_connections 2023-10-11T16:31:38.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.511+0000 7f90ea3ff700 1 -- 172.21.15.111:0/4124284433 wait complete. 2023-10-11T16:31:38.532 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:31:38.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:38 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/339363897' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-11T16:31:38.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:38 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3081741133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:38.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 -- 172.21.15.111:0/3780411492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8d4101e40 msgr2=0x7ff8d4108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/3780411492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8d4101e40 0x7ff8d4108700 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00b0a0 tx=0x7ff8cc00b3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 -- 172.21.15.111:0/3780411492 shutdown_connections 2023-10-11T16:31:38.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/3780411492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8d4101e40 0x7ff8d4108700 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/3780411492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d41002c0 0x7ff8d41006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/3780411492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d41014b0 0x7ff8d4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 -- 172.21.15.111:0/3780411492 >> 172.21.15.111:0/3780411492 conn(0x7ff8d40fb8a0 msgr2=0x7ff8d40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:38.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 -- 172.21.15.111:0/3780411492 shutdown_connections 2023-10-11T16:31:38.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.732+0000 7ff8da7ff700 1 -- 172.21.15.111:0/3780411492 wait complete. 2023-10-11T16:31:38.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.733+0000 7ff8da7ff700 1 Processor -- start 2023-10-11T16:31:38.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.733+0000 7ff8da7ff700 1 -- start start 2023-10-11T16:31:38.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8da7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d41002c0 0x7ff8d4198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:38.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8da7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8d41014b0 0x7ff8d4198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:38.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8da7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d4101e40 0x7ff8d419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:38.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8da7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d419fec0 con 0x7ff8d4101e40 2023-10-11T16:31:38.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8da7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d41a0000 con 0x7ff8d41014b0 2023-10-11T16:31:38.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8da7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d41a0190 con 0x7ff8d41002c0 2023-10-11T16:31:38.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d41002c0 0x7ff8d4198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:38.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d41002c0 0x7ff8d4198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40220/0 (socket says 172.21.15.111:40220) 2023-10-11T16:31:38.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d3fff700 1 -- 172.21.15.111:0/757591697 learned_addr learned my addr 172.21.15.111:0/757591697 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:38.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d37fe700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8d41014b0 0x7ff8d4198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:38.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d8d9c700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d4101e40 0x7ff8d419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:38.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d3fff700 1 -- 172.21.15.111:0/757591697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8d41014b0 msgr2=0x7ff8d4198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d3fff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8d41014b0 0x7ff8d4198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d3fff700 1 -- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d4101e40 msgr2=0x7ff8d419f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d3fff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d4101e40 0x7ff8d419f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.734+0000 7ff8d3fff700 1 -- 172.21.15.111:0/757591697 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc009d00 con 0x7ff8d41002c0 2023-10-11T16:31:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.735+0000 7ff8d3fff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d41002c0 0x7ff8d4198340 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400d980 tx=0x7ff8c400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.735+0000 7ff8d17fa700 1 -- 172.21.15.111:0/757591697 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400f890 con 0x7ff8d41002c0 2023-10-11T16:31:38.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.735+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d410bc90 con 0x7ff8d41002c0 2023-10-11T16:31:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.735+0000 7ff8d17fa700 1 -- 172.21.15.111:0/757591697 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8c4010460 con 0x7ff8d41002c0 2023-10-11T16:31:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.735+0000 7ff8d17fa700 1 -- 172.21.15.111:0/757591697 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40195c0 con 0x7ff8d41002c0 2023-10-11T16:31:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.735+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d410bb00 con 0x7ff8d41002c0 2023-10-11T16:31:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.736+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d41a0930 con 0x7ff8d41002c0 2023-10-11T16:31:38.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.738+0000 7ff8d17fa700 1 -- 172.21.15.111:0/757591697 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff8c40105d0 con 0x7ff8d41002c0 2023-10-11T16:31:38.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.739+0000 7ff8d17fa700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8bc0a11e0 0x7ff8bc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:38.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.739+0000 7ff8d17fa700 1 -- 172.21.15.111:0/757591697 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff8c40cd0b0 con 0x7ff8d41002c0 2023-10-11T16:31:38.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.740+0000 7ff8d37fe700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8bc0a11e0 0x7ff8bc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:38.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.740+0000 7ff8d37fe700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8bc0a11e0 0x7ff8bc0a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8009be0 tx=0x7ff8c8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:38.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.742+0000 7ff8d17fa700 1 -- 172.21.15.111:0/757591697 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff8c4096330 con 0x7ff8d41002c0 2023-10-11T16:31:38.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.939+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8d41a0ee0 con 0x7ff8d41002c0 2023-10-11T16:31:38.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.940+0000 7ff8d17fa700 1 -- 172.21.15.111:0/757591697 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff8c4016020 con 0x7ff8d41002c0 2023-10-11T16:31:38.942 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:31:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8bc0a11e0 msgr2=0x7ff8bc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8bc0a11e0 0x7ff8bc0a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8009be0 tx=0x7ff8c8009260 comp rx=0 tx=0).stop 2023-10-11T16:31:38.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d41002c0 msgr2=0x7ff8d4198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d41002c0 0x7ff8d4198340 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400d980 tx=0x7ff8c400dc90 comp rx=0 tx=0).stop 2023-10-11T16:31:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 shutdown_connections 2023-10-11T16:31:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff8bc0a11e0 0x7ff8bc0a3690 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8d4101e40 0x7ff8d419f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8d41002c0 0x7ff8d4198340 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 --2- 172.21.15.111:0/757591697 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8d41014b0 0x7ff8d4198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 >> 172.21.15.111:0/757591697 conn(0x7ff8d40fb8a0 msgr2=0x7ff8d4106ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.942+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 shutdown_connections 2023-10-11T16:31:38.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:38.943+0000 7ff8da7ff700 1 -- 172.21.15.111:0/757591697 wait complete. 2023-10-11T16:31:38.963 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:31:39.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.164+0000 7f59015ff700 1 -- 172.21.15.111:0/287696209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc101e50 msgr2=0x7f58fc108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.164+0000 7f59015ff700 1 --2- 172.21.15.111:0/287696209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc101e50 0x7f58fc108710 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58f400b0a0 tx=0x7f58f400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.164+0000 7f59015ff700 1 -- 172.21.15.111:0/287696209 shutdown_connections 2023-10-11T16:31:39.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.164+0000 7f59015ff700 1 --2- 172.21.15.111:0/287696209 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc101e50 0x7f58fc108710 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.164+0000 7f59015ff700 1 --2- 172.21.15.111:0/287696209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58fc1002d0 0x7f58fc1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.164+0000 7f59015ff700 1 --2- 172.21.15.111:0/287696209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58fc1014c0 0x7f58fc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.164+0000 7f59015ff700 1 -- 172.21.15.111:0/287696209 >> 172.21.15.111:0/287696209 conn(0x7f58fc0fb8b0 msgr2=0x7f58fc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:39.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.165+0000 7f59015ff700 1 -- 172.21.15.111:0/287696209 shutdown_connections 2023-10-11T16:31:39.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.165+0000 7f59015ff700 1 -- 172.21.15.111:0/287696209 wait complete. 2023-10-11T16:31:39.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f59015ff700 1 Processor -- start 2023-10-11T16:31:39.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f59015ff700 1 -- start start 2023-10-11T16:31:39.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f59015ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58fc1002d0 0x7f58fc196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:39.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f59015ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc1014c0 0x7f58fc1966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:39.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f59015ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58fc101e50 0x7f58fc19d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:39.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f59015ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc19dcf0 con 0x7f58fc1014c0 2023-10-11T16:31:39.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f59015ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc19de30 con 0x7f58fc101e50 2023-10-11T16:31:39.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f59015ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58fc19dfc0 con 0x7f58fc1002d0 2023-10-11T16:31:39.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f58fa7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc1014c0 0x7f58fc1966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:39.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f58fa7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc1014c0 0x7f58fc1966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47282/0 (socket says 172.21.15.111:47282) 2023-10-11T16:31:39.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f58fa7fc700 1 -- 172.21.15.111:0/4166395940 learned_addr learned my addr 172.21.15.111:0/4166395940 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:39.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f58faffd700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58fc1002d0 0x7f58fc196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:39.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.166+0000 7f58fb7fe700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58fc101e50 0x7f58fc19d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:39.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58fa7fc700 1 -- 172.21.15.111:0/4166395940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58fc1002d0 msgr2=0x7f58fc196170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58fa7fc700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58fc1002d0 0x7f58fc196170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58fa7fc700 1 -- 172.21.15.111:0/4166395940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58fc101e50 msgr2=0x7f58fc19d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58fa7fc700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58fc101e50 0x7f58fc19d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58fa7fc700 1 -- 172.21.15.111:0/4166395940 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f4009d00 con 0x7f58fc1014c0 2023-10-11T16:31:39.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58fa7fc700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc1014c0 0x7f58fc1966b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f58f000e9a0 tx=0x7f58f000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:39.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58e3fff700 1 -- 172.21.15.111:0/4166395940 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0009790 con 0x7f58fc1014c0 2023-10-11T16:31:39.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58e3fff700 1 -- 172.21.15.111:0/4166395940 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58f0010430 con 0x7f58fc1014c0 2023-10-11T16:31:39.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58fc10bca0 con 0x7f58fc1014c0 2023-10-11T16:31:39.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f58e3fff700 1 -- 172.21.15.111:0/4166395940 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0018680 con 0x7f58fc1014c0 2023-10-11T16:31:39.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.167+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58fc10bb10 con 0x7f58fc1014c0 2023-10-11T16:31:39.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.168+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58fc19e700 con 0x7f58fc1014c0 2023-10-11T16:31:39.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.170+0000 7f58e3fff700 1 -- 172.21.15.111:0/4166395940 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f58f00105a0 con 0x7f58fc1014c0 2023-10-11T16:31:39.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.172+0000 7f58e3fff700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f58e40a1260 0x7f58e40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:39.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.172+0000 7f58e3fff700 1 -- 172.21.15.111:0/4166395940 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f58f0014070 con 0x7f58fc1014c0 2023-10-11T16:31:39.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.172+0000 7f58faffd700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f58e40a1260 0x7f58e40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:39.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.172+0000 7f58faffd700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f58e40a1260 0x7f58e40a3710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f58fc101320 tx=0x7f58ec009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:39.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.175+0000 7f58e3fff700 1 -- 172.21.15.111:0/4166395940 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f58f0051730 con 0x7f58fc1014c0 2023-10-11T16:31:39.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.374+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f58fc107160 con 0x7f58fc1014c0 2023-10-11T16:31:39.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.375+0000 7f58e3fff700 1 -- 172.21.15.111:0/4166395940 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58f0096200 con 0x7f58fc1014c0 2023-10-11T16:31:39.377 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:31:39.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f58e40a1260 msgr2=0x7f58e40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f58e40a1260 0x7f58e40a3710 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f58fc101320 tx=0x7f58ec009260 comp rx=0 tx=0).stop 2023-10-11T16:31:39.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc1014c0 msgr2=0x7f58fc1966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc1014c0 0x7f58fc1966b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f58f000e9a0 tx=0x7f58f000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 shutdown_connections 2023-10-11T16:31:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f58e40a1260 0x7f58e40a3710 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58fc101e50 0x7f58fc19d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58fc1002d0 0x7f58fc196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 --2- 172.21.15.111:0/4166395940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58fc1014c0 0x7f58fc1966b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.377+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 >> 172.21.15.111:0/4166395940 conn(0x7f58fc0fb8b0 msgr2=0x7f58fc0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:39.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.378+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 shutdown_connections 2023-10-11T16:31:39.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.378+0000 7f59015ff700 1 -- 172.21.15.111:0/4166395940 wait complete. 2023-10-11T16:31:39.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:39 smithi111 ceph-mon[98527]: pgmap v318: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:39.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:39 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4124284433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:31:39.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:39 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/757591697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:31:39.398 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:31:39.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:39 smithi150 ceph-mon[132217]: pgmap v318: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:39.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:39 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4124284433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:31:39.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:39 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/757591697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:31:39.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:39 smithi130 ceph-mon[131859]: pgmap v318: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:39 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4124284433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:31:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:39 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/757591697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:31:39.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 -- 172.21.15.111:0/3046530867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01014b0 msgr2=0x7fb9d0101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/3046530867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01014b0 0x7fb9d0101900 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009990 tx=0x7fb9c0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 -- 172.21.15.111:0/3046530867 shutdown_connections 2023-10-11T16:31:39.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/3046530867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d0101e40 0x7fb9d0108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/3046530867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d01002c0 0x7fb9d01006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/3046530867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01014b0 0x7fb9d0101900 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 -- 172.21.15.111:0/3046530867 >> 172.21.15.111:0/3046530867 conn(0x7fb9d00fb8a0 msgr2=0x7fb9d00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:39.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 -- 172.21.15.111:0/3046530867 shutdown_connections 2023-10-11T16:31:39.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.602+0000 7fb9d87ff700 1 -- 172.21.15.111:0/3046530867 wait complete. 2023-10-11T16:31:39.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d87ff700 1 Processor -- start 2023-10-11T16:31:39.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d87ff700 1 -- start start 2023-10-11T16:31:39.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d87ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01002c0 0x7fb9d01982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:39.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d87ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d01014b0 0x7fb9d01987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:39.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d87ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d0101e40 0x7fb9d019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:39.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d87ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d019fe30 con 0x7fb9d0101e40 2023-10-11T16:31:39.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d87ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d019ff70 con 0x7fb9d01002c0 2023-10-11T16:31:39.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d87ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d01a0100 con 0x7fb9d01014b0 2023-10-11T16:31:39.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.603+0000 7fb9d6d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d0101e40 0x7fb9d019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:39.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d6d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d0101e40 0x7fb9d019f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47304/0 (socket says 172.21.15.111:47304) 2023-10-11T16:31:39.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d6d9c700 1 -- 172.21.15.111:0/101815778 learned_addr learned my addr 172.21.15.111:0/101815778 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:39.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d659b700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01002c0 0x7fb9d01982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:39.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d5d9a700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d01014b0 0x7fb9d01987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:39.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d6d9c700 1 -- 172.21.15.111:0/101815778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d01014b0 msgr2=0x7fb9d01987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d6d9c700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d01014b0 0x7fb9d01987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d6d9c700 1 -- 172.21.15.111:0/101815778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01002c0 msgr2=0x7fb9d01982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d6d9c700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01002c0 0x7fb9d01982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d6d9c700 1 -- 172.21.15.111:0/101815778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00095f0 con 0x7fb9d0101e40 2023-10-11T16:31:39.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d659b700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01002c0 0x7fb9d01982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:39.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d6d9c700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d0101e40 0x7fb9d019f860 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8007ee0 tx=0x7fb9c800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9c77fe700 1 -- 172.21.15.111:0/101815778 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8017070 con 0x7fb9d0101e40 2023-10-11T16:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d010bc30 con 0x7fb9d0101e40 2023-10-11T16:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9c77fe700 1 -- 172.21.15.111:0/101815778 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9c800f040 con 0x7fb9d0101e40 2023-10-11T16:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.604+0000 7fb9c77fe700 1 -- 172.21.15.111:0/101815778 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8005550 con 0x7fb9d0101e40 2023-10-11T16:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.605+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d010baa0 con 0x7fb9d0101e40 2023-10-11T16:31:39.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.606+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d0192990 con 0x7fb9d0101e40 2023-10-11T16:31:39.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.608+0000 7fb9c77fe700 1 -- 172.21.15.111:0/101815778 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fb9c800db20 con 0x7fb9d0101e40 2023-10-11T16:31:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.610+0000 7fb9c77fe700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb9bc0a11e0 0x7fb9bc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.610+0000 7fb9c77fe700 1 -- 172.21.15.111:0/101815778 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb9c80cc4a0 con 0x7fb9d0101e40 2023-10-11T16:31:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.610+0000 7fb9d659b700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb9bc0a11e0 0x7fb9bc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:39.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.610+0000 7fb9d659b700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb9bc0a11e0 0x7fb9bc0a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0101310 tx=0x7fb9cc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:39.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.614+0000 7fb9c77fe700 1 -- 172.21.15.111:0/101815778 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb9c8095720 con 0x7fb9d0101e40 2023-10-11T16:31:39.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.818+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb9d0061700 con 0x7fb9d0101e40 2023-10-11T16:31:39.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.819+0000 7fb9c77fe700 1 -- 172.21.15.111:0/101815778 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb9c8098d40 con 0x7fb9d0101e40 2023-10-11T16:31:39.821 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:31:39.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb9bc0a11e0 msgr2=0x7fb9bc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb9bc0a11e0 0x7fb9bc0a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0101310 tx=0x7fb9cc009260 comp rx=0 tx=0).stop 2023-10-11T16:31:39.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d0101e40 msgr2=0x7fb9d019f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:39.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d0101e40 0x7fb9d019f860 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8007ee0 tx=0x7fb9c800d3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 shutdown_connections 2023-10-11T16:31:39.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb9bc0a11e0 0x7fb9bc0a3690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9d0101e40 0x7fb9d019f860 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9d01002c0 0x7fb9d01982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 --2- 172.21.15.111:0/101815778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9d01014b0 0x7fb9d01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:39.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 >> 172.21.15.111:0/101815778 conn(0x7fb9d00fb8a0 msgr2=0x7fb9d0106d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:39.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 shutdown_connections 2023-10-11T16:31:39.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:39.821+0000 7fb9d87ff700 1 -- 172.21.15.111:0/101815778 wait complete. 2023-10-11T16:31:39.842 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:31:40.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 -- 172.21.15.111:0/3734084379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c101e30 msgr2=0x7fa20c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 --2- 172.21.15.111:0/3734084379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c101e30 0x7fa20c1086f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa20400b0a0 tx=0x7fa20400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 -- 172.21.15.111:0/3734084379 shutdown_connections 2023-10-11T16:31:40.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 --2- 172.21.15.111:0/3734084379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c101e30 0x7fa20c1086f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 --2- 172.21.15.111:0/3734084379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa20c1002b0 0x7fa20c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 --2- 172.21.15.111:0/3734084379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa20c1014a0 0x7fa20c1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 -- 172.21.15.111:0/3734084379 >> 172.21.15.111:0/3734084379 conn(0x7fa20c0fb890 msgr2=0x7fa20c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:40.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 -- 172.21.15.111:0/3734084379 shutdown_connections 2023-10-11T16:31:40.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.038+0000 7fa2143ff700 1 -- 172.21.15.111:0/3734084379 wait complete. 2023-10-11T16:31:40.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.039+0000 7fa2143ff700 1 Processor -- start 2023-10-11T16:31:40.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.039+0000 7fa2143ff700 1 -- start start 2023-10-11T16:31:40.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa2143ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c1002b0 0x7fa20c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa2143ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa20c1014a0 0x7fa20c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa2143ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa20c101e30 0x7fa20c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa2143ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c073bc0 con 0x7fa20c1002b0 2023-10-11T16:31:40.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa2143ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c073d30 con 0x7fa20c1014a0 2023-10-11T16:31:40.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa2143ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c1a00f0 con 0x7fa20c101e30 2023-10-11T16:31:40.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c1002b0 0x7fa20c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21199a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa20c1014a0 0x7fa20c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21199a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa20c1014a0 0x7fa20c072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:41866/0 (socket says 172.21.15.111:41866) 2023-10-11T16:31:40.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c1002b0 0x7fa20c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47322/0 (socket says 172.21.15.111:47322) 2023-10-11T16:31:40.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21199a700 1 -- 172.21.15.111:0/543083561 learned_addr learned my addr 172.21.15.111:0/543083561 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:40.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21299c700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa20c101e30 0x7fa20c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21219b700 1 -- 172.21.15.111:0/543083561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa20c101e30 msgr2=0x7fa20c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21219b700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa20c101e30 0x7fa20c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21219b700 1 -- 172.21.15.111:0/543083561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa20c1014a0 msgr2=0x7fa20c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21219b700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa20c1014a0 0x7fa20c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.040+0000 7fa21219b700 1 -- 172.21.15.111:0/543083561 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204009d00 con 0x7fa20c1002b0 2023-10-11T16:31:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.041+0000 7fa21219b700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c1002b0 0x7fa20c0748a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa20800e9c0 tx=0x7fa20800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:40.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.041+0000 7fa2037fe700 1 -- 172.21.15.111:0/543083561 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208004220 con 0x7fa20c1002b0 2023-10-11T16:31:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.041+0000 7fa2037fe700 1 -- 172.21.15.111:0/543083561 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa208004380 con 0x7fa20c1002b0 2023-10-11T16:31:40.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.041+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20c10bc80 con 0x7fa20c1002b0 2023-10-11T16:31:40.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.041+0000 7fa2037fe700 1 -- 172.21.15.111:0/543083561 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208003be0 con 0x7fa20c1002b0 2023-10-11T16:31:40.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.041+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20c10baf0 con 0x7fa20c1002b0 2023-10-11T16:31:40.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.042+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20c1077d0 con 0x7fa20c1002b0 2023-10-11T16:31:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.044+0000 7fa2037fe700 1 -- 172.21.15.111:0/543083561 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fa208003d40 con 0x7fa20c1002b0 2023-10-11T16:31:40.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.046+0000 7fa2037fe700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa1f80a1260 0x7fa1f80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.046+0000 7fa2037fe700 1 -- 172.21.15.111:0/543083561 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa208014070 con 0x7fa20c1002b0 2023-10-11T16:31:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.046+0000 7fa21199a700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa1f80a1260 0x7fa1f80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.046+0000 7fa21199a700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa1f80a1260 0x7fa1f80a3710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009d20 tx=0x7fa1fc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:40.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.051+0000 7fa2037fe700 1 -- 172.21.15.111:0/543083561 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa2080963f0 con 0x7fa20c1002b0 2023-10-11T16:31:40.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.297+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa20c061700 con 0x7fa20c1002b0 2023-10-11T16:31:40.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.298+0000 7fa2037fe700 1 -- 172.21.15.111:0/543083561 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10823 (secure 0 0 0) 0x7fa20801f340 con 0x7fa20c1002b0 2023-10-11T16:31:40.299 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:40.300 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:51.227688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:31:40.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa1f80a1260 msgr2=0x7fa1f80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa1f80a1260 0x7fa1f80a3710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009d20 tx=0x7fa1fc009260 comp rx=0 tx=0).stop 2023-10-11T16:31:40.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c1002b0 msgr2=0x7fa20c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c1002b0 0x7fa20c0748a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa20800e9c0 tx=0x7fa20800ecd0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 shutdown_connections 2023-10-11T16:31:40.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa1f80a1260 0x7fa1f80a3710 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa20c101e30 0x7fa20c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa20c1002b0 0x7fa20c0748a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 --2- 172.21.15.111:0/543083561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa20c1014a0 0x7fa20c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 >> 172.21.15.111:0/543083561 conn(0x7fa20c0fb890 msgr2=0x7fa20c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:40.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 shutdown_connections 2023-10-11T16:31:40.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.300+0000 7fa2143ff700 1 -- 172.21.15.111:0/543083561 wait complete. 2023-10-11T16:31:40.306 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-10-11T16:31:40.321 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:31:40.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:40 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4166395940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:31:40.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:40 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/101815778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:31:40.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.517+0000 7f62e7dff700 1 -- 172.21.15.111:0/137918986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e0101e30 msgr2=0x7f62e01086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.517+0000 7f62e7dff700 1 --2- 172.21.15.111:0/137918986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e0101e30 0x7f62e01086f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f62d800b0a0 tx=0x7f62d800b3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.518+0000 7f62e7dff700 1 -- 172.21.15.111:0/137918986 shutdown_connections 2023-10-11T16:31:40.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.518+0000 7f62e7dff700 1 --2- 172.21.15.111:0/137918986 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e0101e30 0x7f62e01086f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.518+0000 7f62e7dff700 1 --2- 172.21.15.111:0/137918986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e01002b0 0x7f62e01006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.518+0000 7f62e7dff700 1 --2- 172.21.15.111:0/137918986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62e01014a0 0x7f62e01018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.518+0000 7f62e7dff700 1 -- 172.21.15.111:0/137918986 >> 172.21.15.111:0/137918986 conn(0x7f62e00fb890 msgr2=0x7f62e00fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:40.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.518+0000 7f62e7dff700 1 -- 172.21.15.111:0/137918986 shutdown_connections 2023-10-11T16:31:40.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.518+0000 7f62e7dff700 1 -- 172.21.15.111:0/137918986 wait complete. 2023-10-11T16:31:40.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e7dff700 1 Processor -- start 2023-10-11T16:31:40.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e7dff700 1 -- start start 2023-10-11T16:31:40.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e7dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e01002b0 0x7f62e0198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e7dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62e01014a0 0x7f62e01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e7dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e0101e30 0x7f62e019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e7dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e019fe10 con 0x7f62e01002b0 2023-10-11T16:31:40.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e7dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e019ff50 con 0x7f62e01014a0 2023-10-11T16:31:40.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e7dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e01a00e0 con 0x7f62e0101e30 2023-10-11T16:31:40.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.519+0000 7f62e5b9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e01002b0 0x7f62e0198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e639c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e0101e30 0x7f62e019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e5b9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e01002b0 0x7f62e0198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47334/0 (socket says 172.21.15.111:47334) 2023-10-11T16:31:40.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e5b9b700 1 -- 172.21.15.111:0/1061010373 learned_addr learned my addr 172.21.15.111:0/1061010373 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:40.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e539a700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62e01014a0 0x7f62e01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e5b9b700 1 -- 172.21.15.111:0/1061010373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e0101e30 msgr2=0x7f62e019f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e5b9b700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e0101e30 0x7f62e019f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e5b9b700 1 -- 172.21.15.111:0/1061010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62e01014a0 msgr2=0x7f62e01987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e5b9b700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62e01014a0 0x7f62e01987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e5b9b700 1 -- 172.21.15.111:0/1061010373 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8009d00 con 0x7f62e01002b0 2023-10-11T16:31:40.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e539a700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62e01014a0 0x7f62e01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:40.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e5b9b700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e01002b0 0x7f62e0198290 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00e970 tx=0x7f62dc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:40.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62d6ffd700 1 -- 172.21.15.111:0/1061010373 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc009790 con 0x7f62e01002b0 2023-10-11T16:31:40.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62d6ffd700 1 -- 172.21.15.111:0/1061010373 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62dc00c7e0 con 0x7f62e01002b0 2023-10-11T16:31:40.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e010bc80 con 0x7f62e01002b0 2023-10-11T16:31:40.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.520+0000 7f62d6ffd700 1 -- 172.21.15.111:0/1061010373 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc003c70 con 0x7f62e01002b0 2023-10-11T16:31:40.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.521+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e010baf0 con 0x7f62e01002b0 2023-10-11T16:31:40.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.521+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e01a0830 con 0x7f62e01002b0 2023-10-11T16:31:40.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.523+0000 7f62d6ffd700 1 -- 172.21.15.111:0/1061010373 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f62dc00cc50 con 0x7f62e01002b0 2023-10-11T16:31:40.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.525+0000 7f62d6ffd700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f62cc0a11e0 0x7f62cc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.525+0000 7f62d6ffd700 1 -- 172.21.15.111:0/1061010373 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62dc014070 con 0x7f62e01002b0 2023-10-11T16:31:40.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.525+0000 7f62e539a700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f62cc0a11e0 0x7f62cc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.525+0000 7f62e539a700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f62cc0a11e0 0x7f62cc0a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f62d0011490 tx=0x7f62d0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:40.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.529+0000 7f62d6ffd700 1 -- 172.21.15.111:0/1061010373 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62dc096240 con 0x7f62e01002b0 2023-10-11T16:31:40.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:40 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4166395940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:31:40.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:40 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/101815778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:31:40.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:40 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4166395940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:31:40.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:40 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/101815778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:31:40.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.728+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62e01a0de0 con 0x7f62e01002b0 2023-10-11T16:31:40.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.730+0000 7f62d6ffd700 1 -- 172.21.15.111:0/1061010373 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24555 (secure 0 0 0) 0x7f62dc003dd0 con 0x7f62e01002b0 2023-10-11T16:31:40.733 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:40.733 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":89,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:30:41.177705+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:31:40.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f62cc0a11e0 msgr2=0x7f62cc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f62cc0a11e0 0x7f62cc0a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f62d0011490 tx=0x7f62d0009260 comp rx=0 tx=0).stop 2023-10-11T16:31:40.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e01002b0 msgr2=0x7f62e0198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e01002b0 0x7f62e0198290 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00e970 tx=0x7f62dc00ec80 comp rx=0 tx=0).stop 2023-10-11T16:31:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 shutdown_connections 2023-10-11T16:31:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f62cc0a11e0 0x7f62cc0a3690 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62e0101e30 0x7f62e019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62e01002b0 0x7f62e0198290 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 --2- 172.21.15.111:0/1061010373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f62e01014a0 0x7f62e01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 >> 172.21.15.111:0/1061010373 conn(0x7f62e00fb890 msgr2=0x7f62e00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 shutdown_connections 2023-10-11T16:31:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.733+0000 7f62e7dff700 1 -- 172.21.15.111:0/1061010373 wait complete. 2023-10-11T16:31:40.754 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:31:40.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:31:40.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2921968068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881014b0 msgr2=0x7f3d88101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2921968068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881014b0 0x7f3d88101900 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009990 tx=0x7f3d74009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2921968068 shutdown_connections 2023-10-11T16:31:40.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2921968068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d88101e40 0x7f3d88108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2921968068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d881002c0 0x7f3d881006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2921968068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881014b0 0x7f3d88101900 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2921968068 >> 172.21.15.111:0/2921968068 conn(0x7f3d880fb8a0 msgr2=0x7f3d880fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:40.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2921968068 shutdown_connections 2023-10-11T16:31:40.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.956+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2921968068 wait complete. 2023-10-11T16:31:40.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.957+0000 7f3d8ebff700 1 Processor -- start 2023-10-11T16:31:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.957+0000 7f3d8ebff700 1 -- start start 2023-10-11T16:31:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.957+0000 7f3d8ebff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881002c0 0x7f3d880748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.957+0000 7f3d8ebff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d881014b0 0x7f3d88072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.957+0000 7f3d8ebff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d88101e40 0x7f3d88073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8ebff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d88073bc0 con 0x7f3d881002c0 2023-10-11T16:31:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8ebff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d88073d30 con 0x7f3d88101e40 2023-10-11T16:31:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8ebff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d881a00a0 con 0x7f3d881014b0 2023-10-11T16:31:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881002c0 0x7f3d880748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881002c0 0x7f3d880748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47350/0 (socket says 172.21.15.111:47350) 2023-10-11T16:31:40.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 -- 172.21.15.111:0/2291925647 learned_addr learned my addr 172.21.15.111:0/2291925647 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d87fff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d881014b0 0x7f3d88072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8d19c700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d88101e40 0x7f3d88073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 -- 172.21.15.111:0/2291925647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d881014b0 msgr2=0x7f3d88072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d881014b0 0x7f3d88072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 -- 172.21.15.111:0/2291925647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d88101e40 msgr2=0x7f3d88073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d88101e40 0x7f3d88073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 -- 172.21.15.111:0/2291925647 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740095f0 con 0x7f3d881002c0 2023-10-11T16:31:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d87fff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d881014b0 0x7f3d88072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:40.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.958+0000 7f3d8c99b700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881002c0 0x7f3d880748a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00e970 tx=0x7f3d7c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:40.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.959+0000 7f3d85ffb700 1 -- 172.21.15.111:0/2291925647 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c009790 con 0x7f3d881002c0 2023-10-11T16:31:40.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.959+0000 7f3d85ffb700 1 -- 172.21.15.111:0/2291925647 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d7c00c7e0 con 0x7f3d881002c0 2023-10-11T16:31:40.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.959+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8810bc70 con 0x7f3d881002c0 2023-10-11T16:31:40.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.959+0000 7f3d85ffb700 1 -- 172.21.15.111:0/2291925647 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c003c70 con 0x7f3d881002c0 2023-10-11T16:31:40.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.959+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8810bae0 con 0x7f3d881002c0 2023-10-11T16:31:40.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.960+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d881071f0 con 0x7f3d881002c0 2023-10-11T16:31:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.962+0000 7f3d85ffb700 1 -- 172.21.15.111:0/2291925647 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f3d7c00cc50 con 0x7f3d881002c0 2023-10-11T16:31:40.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.963+0000 7f3d85ffb700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3d780a1260 0x7f3d780a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:40.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.963+0000 7f3d85ffb700 1 -- 172.21.15.111:0/2291925647 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3d7c014070 con 0x7f3d881002c0 2023-10-11T16:31:40.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.963+0000 7f3d87fff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3d780a1260 0x7f3d780a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:40.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.964+0000 7f3d87fff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3d780a1260 0x7f3d780a3710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400b2e0 tx=0x7f3d74005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:40.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:40.965+0000 7f3d85ffb700 1 -- 172.21.15.111:0/2291925647 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3d7c096310 con 0x7f3d881002c0 2023-10-11T16:31:41.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.177+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d88061700 con 0x7f3d881002c0 2023-10-11T16:31:41.179 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.177+0000 7f3d85ffb700 1 -- 172.21.15.111:0/2291925647 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3d7c099930 con 0x7f3d881002c0 2023-10-11T16:31:41.179 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:31:41.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.179+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3d780a1260 msgr2=0x7f3d780a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.179+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3d780a1260 0x7f3d780a3710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3d7400b2e0 tx=0x7f3d74005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.179+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881002c0 msgr2=0x7f3d880748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.179+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881002c0 0x7f3d880748a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00e970 tx=0x7f3d7c00ec80 comp rx=0 tx=0).stop 2023-10-11T16:31:41.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.180+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 shutdown_connections 2023-10-11T16:31:41.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.180+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3d780a1260 0x7f3d780a3710 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.180+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d88101e40 0x7f3d88073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.180+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d881002c0 0x7f3d880748a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.180+0000 7f3d8ebff700 1 --2- 172.21.15.111:0/2291925647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d881014b0 0x7f3d88072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.180+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 >> 172.21.15.111:0/2291925647 conn(0x7f3d880fb8a0 msgr2=0x7f3d880fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:41.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.180+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 shutdown_connections 2023-10-11T16:31:41.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.180+0000 7f3d8ebff700 1 -- 172.21.15.111:0/2291925647 wait complete. 2023-10-11T16:31:41.200 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:31:41.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.399+0000 7fc565fff700 1 -- 172.21.15.111:0/4036557559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5601002c0 msgr2=0x7fc5601006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.399+0000 7fc565fff700 1 --2- 172.21.15.111:0/4036557559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5601002c0 0x7fc5601006d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc550009990 tx=0x7fc550009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.400+0000 7fc565fff700 1 -- 172.21.15.111:0/4036557559 shutdown_connections 2023-10-11T16:31:41.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.400+0000 7fc565fff700 1 --2- 172.21.15.111:0/4036557559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc560101e40 0x7fc560108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.400+0000 7fc565fff700 1 --2- 172.21.15.111:0/4036557559 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5601002c0 0x7fc5601006d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.400+0000 7fc565fff700 1 --2- 172.21.15.111:0/4036557559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5601014b0 0x7fc560101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.400+0000 7fc565fff700 1 -- 172.21.15.111:0/4036557559 >> 172.21.15.111:0/4036557559 conn(0x7fc5600fb8a0 msgr2=0x7fc5600fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:41.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.400+0000 7fc565fff700 1 -- 172.21.15.111:0/4036557559 shutdown_connections 2023-10-11T16:31:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.400+0000 7fc565fff700 1 -- 172.21.15.111:0/4036557559 wait complete. 2023-10-11T16:31:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc565fff700 1 Processor -- start 2023-10-11T16:31:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc565fff700 1 -- start start 2023-10-11T16:31:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc565fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5601002c0 0x7fc560198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:41.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc565fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5601014b0 0x7fc5601987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc565fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc560101e40 0x7fc56019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc565fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56019fe00 con 0x7fc560101e40 2023-10-11T16:31:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc565fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56019ff40 con 0x7fc5601002c0 2023-10-11T16:31:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc565fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5601a00d0 con 0x7fc5601014b0 2023-10-11T16:31:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc55ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc560101e40 0x7fc56019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:41.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc55ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc560101e40 0x7fc56019f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47360/0 (socket says 172.21.15.111:47360) 2023-10-11T16:31:41.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc55ffff700 1 -- 172.21.15.111:0/3468901861 learned_addr learned my addr 172.21.15.111:0/3468901861 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:41.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.401+0000 7fc55f7fe700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5601002c0 0x7fc560198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:41.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55effd700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5601014b0 0x7fc5601987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:41.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55ffff700 1 -- 172.21.15.111:0/3468901861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5601014b0 msgr2=0x7fc5601987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55ffff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5601014b0 0x7fc5601987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55ffff700 1 -- 172.21.15.111:0/3468901861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5601002c0 msgr2=0x7fc560198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55ffff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5601002c0 0x7fc560198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55ffff700 1 -- 172.21.15.111:0/3468901861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500095f0 con 0x7fc560101e40 2023-10-11T16:31:41.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55effd700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5601014b0 0x7fc5601987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:31:41.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55ffff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc560101e40 0x7fc56019f830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc55800ef30 tx=0x7fc55800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:41.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55cff9700 1 -- 172.21.15.111:0/3468901861 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558018070 con 0x7fc560101e40 2023-10-11T16:31:41.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55cff9700 1 -- 172.21.15.111:0/3468901861 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc55800f040 con 0x7fc560101e40 2023-10-11T16:31:41.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56010bc90 con 0x7fc560101e40 2023-10-11T16:31:41.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc55cff9700 1 -- 172.21.15.111:0/3468901861 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800cdb0 con 0x7fc560101e40 2023-10-11T16:31:41.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.402+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56010bb00 con 0x7fc560101e40 2023-10-11T16:31:41.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.403+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5601929f0 con 0x7fc560101e40 2023-10-11T16:31:41.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.405+0000 7fc55cff9700 1 -- 172.21.15.111:0/3468901861 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fc558003a00 con 0x7fc560101e40 2023-10-11T16:31:41.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.407+0000 7fc55cff9700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5480a1260 0x7fc5480a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:41.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.408+0000 7fc55cff9700 1 -- 172.21.15.111:0/3468901861 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fc5580cca00 con 0x7fc560101e40 2023-10-11T16:31:41.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.408+0000 7fc55f7fe700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5480a1260 0x7fc5480a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:41.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.408+0000 7fc55f7fe700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5480a1260 0x7fc5480a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc55000b2e0 tx=0x7fc550005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:41.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.411+0000 7fc55cff9700 1 -- 172.21.15.111:0/3468901861 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc558095c50 con 0x7fc560101e40 2023-10-11T16:31:41.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: pgmap v319: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:41.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2023-10-11T16:31:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [1, 9]}]: dispatch 2023-10-11T16:31:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6, 5, 9]}]: dispatch 2023-10-11T16:31:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]: dispatch 2023-10-11T16:31:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2023-10-11T16:31:41.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-10-11T16:31:41.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/543083561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:41.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:41 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1061010373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:41.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: pgmap v319: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2023-10-11T16:31:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [1, 9]}]: dispatch 2023-10-11T16:31:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6, 5, 9]}]: dispatch 2023-10-11T16:31:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]: dispatch 2023-10-11T16:31:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2023-10-11T16:31:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-10-11T16:31:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/543083561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:41.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:41 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1061010373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:41.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.611+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc560061700 con 0x7fc560101e40 2023-10-11T16:31:41.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.611+0000 7fc55cff9700 1 -- 172.21.15.111:0/3468901861 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc558099270 con 0x7fc560101e40 2023-10-11T16:31:41.613 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:31:41.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5480a1260 msgr2=0x7fc5480a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5480a1260 0x7fc5480a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc55000b2e0 tx=0x7fc550005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc560101e40 msgr2=0x7fc56019f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc560101e40 0x7fc56019f830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc55800ef30 tx=0x7fc55800c3a0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 shutdown_connections 2023-10-11T16:31:41.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5480a1260 0x7fc5480a3710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc560101e40 0x7fc56019f830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5601002c0 0x7fc560198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 --2- 172.21.15.111:0/3468901861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5601014b0 0x7fc5601987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.613+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 >> 172.21.15.111:0/3468901861 conn(0x7fc5600fb8a0 msgr2=0x7fc560106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:41.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.614+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 shutdown_connections 2023-10-11T16:31:41.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.614+0000 7fc565fff700 1 -- 172.21.15.111:0/3468901861 wait complete. 2023-10-11T16:31:41.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: pgmap v319: 129 pgs: 129 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:41.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2023-10-11T16:31:41.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [1, 9]}]: dispatch 2023-10-11T16:31:41.632 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6, 5, 9]}]: dispatch 2023-10-11T16:31:41.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]: dispatch 2023-10-11T16:31:41.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2023-10-11T16:31:41.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-10-11T16:31:41.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/543083561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:41.633 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:41 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1061010373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:41.634 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:31:41.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 -- 172.21.15.111:0/3083925766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86381002c0 msgr2=0x7f86381006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 --2- 172.21.15.111:0/3083925766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86381002c0 0x7f86381006d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f86280098f0 tx=0x7f8628009c00 comp rx=0 tx=0).stop 2023-10-11T16:31:41.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 -- 172.21.15.111:0/3083925766 shutdown_connections 2023-10-11T16:31:41.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 --2- 172.21.15.111:0/3083925766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8638101e40 0x7f8638108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 --2- 172.21.15.111:0/3083925766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86381002c0 0x7f86381006d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 --2- 172.21.15.111:0/3083925766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86381014b0 0x7f8638101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 -- 172.21.15.111:0/3083925766 >> 172.21.15.111:0/3083925766 conn(0x7f86380fb8a0 msgr2=0x7f86380fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:41.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 -- 172.21.15.111:0/3083925766 shutdown_connections 2023-10-11T16:31:41.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.834+0000 7f863d7ff700 1 -- 172.21.15.111:0/3083925766 wait complete. 2023-10-11T16:31:41.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.835+0000 7f863d7ff700 1 Processor -- start 2023-10-11T16:31:41.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.835+0000 7f863d7ff700 1 -- start start 2023-10-11T16:31:41.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.835+0000 7f863d7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86381002c0 0x7f86380748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:41.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.835+0000 7f863d7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86381014b0 0x7f8638072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:41.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.835+0000 7f863d7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8638101e40 0x7f8638073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:41.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.835+0000 7f863d7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638073bc0 con 0x7f8638101e40 2023-10-11T16:31:41.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.835+0000 7f863d7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638073d30 con 0x7f86381014b0 2023-10-11T16:31:41.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.835+0000 7f863d7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86381a0150 con 0x7f86381002c0 2023-10-11T16:31:41.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8638101e40 0x7f8638073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:41.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f8636ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86381002c0 0x7f86380748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:41.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86367fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86381014b0 0x7f8638072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:41.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8638101e40 0x7f8638073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47376/0 (socket says 172.21.15.111:47376) 2023-10-11T16:31:41.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 -- 172.21.15.111:0/2883686081 learned_addr learned my addr 172.21.15.111:0/2883686081 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:41.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 -- 172.21.15.111:0/2883686081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86381002c0 msgr2=0x7f86380748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86381002c0 0x7f86380748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 -- 172.21.15.111:0/2883686081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86381014b0 msgr2=0x7f8638072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:41.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86381014b0 0x7f8638072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:41.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 -- 172.21.15.111:0/2883686081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8628009520 con 0x7f8638101e40 2023-10-11T16:31:41.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f8636ffd700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86381002c0 0x7f86380748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:41.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f86377fe700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8638101e40 0x7f8638073420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f863000c110 tx=0x7f863000c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:41.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f861ffff700 1 -- 172.21.15.111:0/2883686081 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630017070 con 0x7f8638101e40 2023-10-11T16:31:41.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f861ffff700 1 -- 172.21.15.111:0/2883686081 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f863000e050 con 0x7f8638101e40 2023-10-11T16:31:41.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f861ffff700 1 -- 172.21.15.111:0/2883686081 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86300078f0 con 0x7f8638101e40 2023-10-11T16:31:41.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.836+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863810bc90 con 0x7f8638101e40 2023-10-11T16:31:41.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.837+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863810bb00 con 0x7f8638101e40 2023-10-11T16:31:41.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.838+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86381078c0 con 0x7f8638101e40 2023-10-11T16:31:41.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.839+0000 7f861ffff700 1 -- 172.21.15.111:0/2883686081 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8630007a50 con 0x7f8638101e40 2023-10-11T16:31:41.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.842+0000 7f861ffff700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f86200a11e0 0x7f86200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:41.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.842+0000 7f861ffff700 1 -- 172.21.15.111:0/2883686081 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f86300cd330 con 0x7f8638101e40 2023-10-11T16:31:41.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.842+0000 7f8636ffd700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f86200a11e0 0x7f86200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:41.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.843+0000 7f8636ffd700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f86200a11e0 0x7f86200a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8628009fd0 tx=0x7f8628019040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:41.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:41.844+0000 7f861ffff700 1 -- 172.21.15.111:0/2883686081 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f86300964f0 con 0x7f8638101e40 2023-10-11T16:31:42.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.044+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8638061700 con 0x7f8638101e40 2023-10-11T16:31:42.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.044+0000 7f861ffff700 1 -- 172.21.15.111:0/2883686081 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8630099b10 con 0x7f8638101e40 2023-10-11T16:31:42.046 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:31:42.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f86200a11e0 msgr2=0x7f86200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f86200a11e0 0x7f86200a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8628009fd0 tx=0x7f8628019040 comp rx=0 tx=0).stop 2023-10-11T16:31:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8638101e40 msgr2=0x7f8638073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8638101e40 0x7f8638073420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f863000c110 tx=0x7f863000c420 comp rx=0 tx=0).stop 2023-10-11T16:31:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 shutdown_connections 2023-10-11T16:31:42.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f86200a11e0 0x7f86200a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8638101e40 0x7f8638073420 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86381002c0 0x7f86380748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 --2- 172.21.15.111:0/2883686081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f86381014b0 0x7f8638072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 >> 172.21.15.111:0/2883686081 conn(0x7f86380fb8a0 msgr2=0x7f8638104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:42.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 shutdown_connections 2023-10-11T16:31:42.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.047+0000 7f863d7ff700 1 -- 172.21.15.111:0/2883686081 wait complete. 2023-10-11T16:31:42.067 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:31:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 -- 172.21.15.111:0/1104535088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e0101e50 msgr2=0x7f95e0108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1104535088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e0101e50 0x7f95e0108710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f95d800b0a0 tx=0x7f95d800b3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 -- 172.21.15.111:0/1104535088 shutdown_connections 2023-10-11T16:31:42.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1104535088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e0101e50 0x7f95e0108710 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1104535088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e01002d0 0x7f95e01006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1104535088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95e01014c0 0x7f95e0101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 -- 172.21.15.111:0/1104535088 >> 172.21.15.111:0/1104535088 conn(0x7f95e00fb8b0 msgr2=0x7f95e00fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 -- 172.21.15.111:0/1104535088 shutdown_connections 2023-10-11T16:31:42.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.269+0000 7f95e4fff700 1 -- 172.21.15.111:0/1104535088 wait complete. 2023-10-11T16:31:42.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.270+0000 7f95e4fff700 1 Processor -- start 2023-10-11T16:31:42.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.270+0000 7f95e4fff700 1 -- start start 2023-10-11T16:31:42.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.270+0000 7f95e4fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95e01002d0 0x7f95e0196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:42.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.270+0000 7f95e4fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e01014c0 0x7f95e0196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.270+0000 7f95e4fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e0101e50 0x7f95e019d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.270+0000 7f95e4fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e019dca0 con 0x7f95e01014c0 2023-10-11T16:31:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.270+0000 7f95e4fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e019dde0 con 0x7f95e01002d0 2023-10-11T16:31:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.270+0000 7f95e4fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e019df70 con 0x7f95e0101e50 2023-10-11T16:31:42.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e01014c0 0x7f95e0196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:42.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e01014c0 0x7f95e0196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47384/0 (socket says 172.21.15.111:47384) 2023-10-11T16:31:42.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 -- 172.21.15.111:0/1144069201 learned_addr learned my addr 172.21.15.111:0/1144069201 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:42.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95de59c700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95e01002d0 0x7f95e0196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:42.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95ded9d700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e0101e50 0x7f95e019d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 -- 172.21.15.111:0/1144069201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e0101e50 msgr2=0x7f95e019d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e0101e50 0x7f95e019d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 -- 172.21.15.111:0/1144069201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95e01002d0 msgr2=0x7f95e0196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95e01002d0 0x7f95e0196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 -- 172.21.15.111:0/1144069201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d8009d00 con 0x7f95e01014c0 2023-10-11T16:31:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95de59c700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95e01002d0 0x7f95e0196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:31:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d7fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e01014c0 0x7f95e0196660 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f95c800e9a0 tx=0x7f95c800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d77fe700 1 -- 172.21.15.111:0/1144069201 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c8009790 con 0x7f95e01014c0 2023-10-11T16:31:42.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95d77fe700 1 -- 172.21.15.111:0/1144069201 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95c800cbd0 con 0x7f95e01014c0 2023-10-11T16:31:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.271+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e010bcc0 con 0x7f95e01014c0 2023-10-11T16:31:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.272+0000 7f95d77fe700 1 -- 172.21.15.111:0/1144069201 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c8018680 con 0x7f95e01014c0 2023-10-11T16:31:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.272+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e010bb30 con 0x7f95e01014c0 2023-10-11T16:31:42.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.272+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e010be50 con 0x7f95e01014c0 2023-10-11T16:31:42.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.274+0000 7f95d77fe700 1 -- 172.21.15.111:0/1144069201 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f95c800cd40 con 0x7f95e01014c0 2023-10-11T16:31:42.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.277+0000 7f95d77fe700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95cc0a11e0 0x7f95cc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:42.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.277+0000 7f95d77fe700 1 -- 172.21.15.111:0/1144069201 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f95c8014070 con 0x7f95e01014c0 2023-10-11T16:31:42.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.277+0000 7f95de59c700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95cc0a11e0 0x7f95cc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:42.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.277+0000 7f95de59c700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95cc0a11e0 0x7f95cc0a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f95e0101320 tx=0x7f95d0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:42.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.281+0000 7f95d77fe700 1 -- 172.21.15.111:0/1144069201 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f95c8051730 con 0x7f95e01014c0 2023-10-11T16:31:42.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2023-10-11T16:31:42.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [1, 9]}]': finished 2023-10-11T16:31:42.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6, 5, 9]}]': finished 2023-10-11T16:31:42.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]': finished 2023-10-11T16:31:42.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2023-10-11T16:31:42.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-10-11T16:31:42.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: osdmap e90: 12 total, 12 up, 12 in 2023-10-11T16:31:42.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2291925647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3468901861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:42 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2883686081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2023-10-11T16:31:42.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [1, 9]}]': finished 2023-10-11T16:31:42.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6, 5, 9]}]': finished 2023-10-11T16:31:42.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]': finished 2023-10-11T16:31:42.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2023-10-11T16:31:42.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-10-11T16:31:42.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: osdmap e90: 12 total, 12 up, 12 in 2023-10-11T16:31:42.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2291925647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3468901861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:42 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2883686081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.483+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f95e0107160 con 0x7f95e01014c0 2023-10-11T16:31:42.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.484+0000 7f95d77fe700 1 -- 172.21.15.111:0/1144069201 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f95c8099e40 con 0x7f95e01014c0 2023-10-11T16:31:42.486 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:31:42.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95cc0a11e0 msgr2=0x7f95cc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95cc0a11e0 0x7f95cc0a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f95e0101320 tx=0x7f95d0009260 comp rx=0 tx=0).stop 2023-10-11T16:31:42.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e01014c0 msgr2=0x7f95e0196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e01014c0 0x7f95e0196660 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f95c800e9a0 tx=0x7f95c800ed60 comp rx=0 tx=0).stop 2023-10-11T16:31:42.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 shutdown_connections 2023-10-11T16:31:42.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f95cc0a11e0 0x7f95cc0a3690 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e0101e50 0x7f95e019d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f95e01002d0 0x7f95e0196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 --2- 172.21.15.111:0/1144069201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95e01014c0 0x7f95e0196660 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 >> 172.21.15.111:0/1144069201 conn(0x7f95e00fb8b0 msgr2=0x7f95e00fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:42.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.486+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 shutdown_connections 2023-10-11T16:31:42.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.487+0000 7f95e4fff700 1 -- 172.21.15.111:0/1144069201 wait complete. 2023-10-11T16:31:42.507 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:31:42.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2023-10-11T16:31:42.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [1, 9]}]': finished 2023-10-11T16:31:42.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6, 5, 9]}]': finished 2023-10-11T16:31:42.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8, 5, 6]}]': finished 2023-10-11T16:31:42.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2023-10-11T16:31:42.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='mgr.14148 172.21.15.111:0/3622315281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-10-11T16:31:42.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: osdmap e90: 12 total, 12 up, 12 in 2023-10-11T16:31:42.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2291925647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3468901861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:42 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2883686081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:31:42.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 -- 172.21.15.111:0/4148312523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224100290 msgr2=0x7f72241006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 --2- 172.21.15.111:0/4148312523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224100290 0x7f72241006a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7220009990 tx=0x7f7220009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 -- 172.21.15.111:0/4148312523 shutdown_connections 2023-10-11T16:31:42.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 --2- 172.21.15.111:0/4148312523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7224101e10 0x7f72241086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 --2- 172.21.15.111:0/4148312523 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224100290 0x7f72241006a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 --2- 172.21.15.111:0/4148312523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224101480 0x7f72241018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 -- 172.21.15.111:0/4148312523 >> 172.21.15.111:0/4148312523 conn(0x7f72240fb870 msgr2=0x7f72240fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:42.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 -- 172.21.15.111:0/4148312523 shutdown_connections 2023-10-11T16:31:42.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.705+0000 7f722bfff700 1 -- 172.21.15.111:0/4148312523 wait complete. 2023-10-11T16:31:42.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.706+0000 7f722bfff700 1 Processor -- start 2023-10-11T16:31:42.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722bfff700 1 -- start start 2023-10-11T16:31:42.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224100290 0x7f72241981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:42.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7224101480 0x7f7224198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:42.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224101e10 0x7f722419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:42.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722419fd50 con 0x7f7224101e10 2023-10-11T16:31:42.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722bfff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722419fe90 con 0x7f7224101480 2023-10-11T16:31:42.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72241a0020 con 0x7f7224100290 2023-10-11T16:31:42.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224101e10 0x7f722419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:42.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f7229d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224100290 0x7f72241981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:42.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224101e10 0x7f722419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47400/0 (socket says 172.21.15.111:47400) 2023-10-11T16:31:42.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722a59c700 1 -- 172.21.15.111:0/2663502968 learned_addr learned my addr 172.21.15.111:0/2663502968 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:42.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.707+0000 7f722959a700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7224101480 0x7f7224198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:42.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f722a59c700 1 -- 172.21.15.111:0/2663502968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224100290 msgr2=0x7f72241981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f722a59c700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224100290 0x7f72241981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f722a59c700 1 -- 172.21.15.111:0/2663502968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7224101480 msgr2=0x7f7224198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f722a59c700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7224101480 0x7f7224198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f722a59c700 1 -- 172.21.15.111:0/2663502968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72200095f0 con 0x7f7224101e10 2023-10-11T16:31:42.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f722a59c700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224101e10 0x7f722419f780 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f721c007ee0 tx=0x7f721c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:42.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f721affd700 1 -- 172.21.15.111:0/2663502968 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c00dd20 con 0x7f7224101e10 2023-10-11T16:31:42.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f721affd700 1 -- 172.21.15.111:0/2663502968 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f721c00f040 con 0x7f7224101e10 2023-10-11T16:31:42.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722410bc40 con 0x7f7224101e10 2023-10-11T16:31:42.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f721affd700 1 -- 172.21.15.111:0/2663502968 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c0048a0 con 0x7f7224101e10 2023-10-11T16:31:42.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.708+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722410bab0 con 0x7f7224101e10 2023-10-11T16:31:42.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.709+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72241a0710 con 0x7f7224101e10 2023-10-11T16:31:42.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.712+0000 7f721affd700 1 -- 172.21.15.111:0/2663502968 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f721c003a00 con 0x7f7224101e10 2023-10-11T16:31:42.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.713+0000 7f721affd700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72100a11e0 0x7f72100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:42.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.713+0000 7f721affd700 1 -- 172.21.15.111:0/2663502968 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f721c0cc680 con 0x7f7224101e10 2023-10-11T16:31:42.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.713+0000 7f7229d9b700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72100a11e0 0x7f72100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:42.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.714+0000 7f7229d9b700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72100a11e0 0x7f72100a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f722000b5c0 tx=0x7f7220005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:42.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.717+0000 7f721affd700 1 -- 172.21.15.111:0/2663502968 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f721c04e730 con 0x7f7224101e10 2023-10-11T16:31:42.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.961+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72241a0cc0 con 0x7f7224101e10 2023-10-11T16:31:42.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.962+0000 7f721affd700 1 -- 172.21.15.111:0/2663502968 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f721c095070 con 0x7f7224101e10 2023-10-11T16:31:42.964 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:42.964 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:51.227688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:31:42.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.964+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72100a11e0 msgr2=0x7f72100a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.964+0000 7f722bfff700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72100a11e0 0x7f72100a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f722000b5c0 tx=0x7f7220005fd0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.964+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224101e10 msgr2=0x7f722419f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:42.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.964+0000 7f722bfff700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224101e10 0x7f722419f780 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f721c007ee0 tx=0x7f721c00d3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.965+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 shutdown_connections 2023-10-11T16:31:42.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.965+0000 7f722bfff700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f72100a11e0 0x7f72100a3690 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.965+0000 7f722bfff700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7224101e10 0x7f722419f780 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.965+0000 7f722bfff700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7224100290 0x7f72241981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.965+0000 7f722bfff700 1 --2- 172.21.15.111:0/2663502968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7224101480 0x7f7224198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:42.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.965+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 >> 172.21.15.111:0/2663502968 conn(0x7f72240fb870 msgr2=0x7f72240fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:42.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.965+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 shutdown_connections 2023-10-11T16:31:42.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:42.965+0000 7f722bfff700 1 -- 172.21.15.111:0/2663502968 wait complete. 2023-10-11T16:31:42.969 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-10-11T16:31:42.986 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:31:43.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.184+0000 7f498f7ff700 1 -- 172.21.15.111:0/1824378938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 msgr2=0x7f49881018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.184+0000 7f498f7ff700 1 --2- 172.21.15.111:0/1824378938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 0x7f49881018f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4978009990 tx=0x7f4978009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.185+0000 7f498f7ff700 1 -- 172.21.15.111:0/1824378938 shutdown_connections 2023-10-11T16:31:43.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.185+0000 7f498f7ff700 1 --2- 172.21.15.111:0/1824378938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4988101e30 0x7f49881086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.185+0000 7f498f7ff700 1 --2- 172.21.15.111:0/1824378938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49881002b0 0x7f49881006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.185+0000 7f498f7ff700 1 --2- 172.21.15.111:0/1824378938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 0x7f49881018f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.185+0000 7f498f7ff700 1 -- 172.21.15.111:0/1824378938 >> 172.21.15.111:0/1824378938 conn(0x7f49880fb890 msgr2=0x7f49880fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.185+0000 7f498f7ff700 1 -- 172.21.15.111:0/1824378938 shutdown_connections 2023-10-11T16:31:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.185+0000 7f498f7ff700 1 -- 172.21.15.111:0/1824378938 wait complete. 2023-10-11T16:31:43.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498f7ff700 1 Processor -- start 2023-10-11T16:31:43.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498f7ff700 1 -- start start 2023-10-11T16:31:43.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498f7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49881002b0 0x7f49880748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:43.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498f7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 0x7f4988072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:43.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498f7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4988101e30 0x7f4988073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:43.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498f7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988073bc0 con 0x7f49881014a0 2023-10-11T16:31:43.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498f7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988073d30 con 0x7f4988101e30 2023-10-11T16:31:43.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498f7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49881a0040 con 0x7f49881002b0 2023-10-11T16:31:43.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498cd9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 0x7f4988072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:43.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498cd9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 0x7f4988072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47428/0 (socket says 172.21.15.111:47428) 2023-10-11T16:31:43.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498cd9a700 1 -- 172.21.15.111:0/3596708775 learned_addr learned my addr 172.21.15.111:0/3596708775 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:43.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.186+0000 7f498d59b700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49881002b0 0x7f49880748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:43.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498dd9c700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4988101e30 0x7f4988073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:43.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498cd9a700 1 -- 172.21.15.111:0/3596708775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49881002b0 msgr2=0x7f49880748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498cd9a700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49881002b0 0x7f49880748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498cd9a700 1 -- 172.21.15.111:0/3596708775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4988101e30 msgr2=0x7f4988073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498cd9a700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4988101e30 0x7f4988073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.197 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498cd9a700 1 -- 172.21.15.111:0/3596708775 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780095f0 con 0x7f49881014a0 2023-10-11T16:31:43.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498d59b700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49881002b0 0x7f49880748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:31:43.198 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498dd9c700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4988101e30 0x7f4988073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:31:43.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498cd9a700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 0x7f4988072ee0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f497800b620 tx=0x7f49780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:43.199 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f497e7fc700 1 -- 172.21.15.111:0/3596708775 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497801d070 con 0x7f49881014a0 2023-10-11T16:31:43.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f497e7fc700 1 -- 172.21.15.111:0/3596708775 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f497800fbc0 con 0x7f49881014a0 2023-10-11T16:31:43.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498810bc60 con 0x7f49881014a0 2023-10-11T16:31:43.200 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f497e7fc700 1 -- 172.21.15.111:0/3596708775 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978017650 con 0x7f49881014a0 2023-10-11T16:31:43.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.187+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498810bad0 con 0x7f49881014a0 2023-10-11T16:31:43.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.189+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4988067a60 con 0x7f49881014a0 2023-10-11T16:31:43.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.190+0000 7f497e7fc700 1 -- 172.21.15.111:0/3596708775 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f4978022470 con 0x7f49881014a0 2023-10-11T16:31:43.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.191+0000 7f497e7fc700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49740a11e0 0x7f49740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:43.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.192+0000 7f497e7fc700 1 -- 172.21.15.111:0/3596708775 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f49780ce670 con 0x7f49881014a0 2023-10-11T16:31:43.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.192+0000 7f498d59b700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49740a11e0 0x7f49740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:43.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.192+0000 7f498d59b700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49740a11e0 0x7f49740a3690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4988101300 tx=0x7f4984009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:43.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.195+0000 7f497e7fc700 1 -- 172.21.15.111:0/3596708775 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f49780978c0 con 0x7f49881014a0 2023-10-11T16:31:43.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:43 smithi111 ceph-mon[98527]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:31:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:43 smithi111 ceph-mon[98527]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-10-11T16:31:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:43 smithi111 ceph-mon[98527]: osdmap e91: 12 total, 12 up, 12 in 2023-10-11T16:31:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:43 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1144069201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:31:43.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:43 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2663502968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:43.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.397+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49881071e0 con 0x7f49881014a0 2023-10-11T16:31:43.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.399+0000 7f497e7fc700 1 -- 172.21.15.111:0/3596708775 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24870 (secure 0 0 0) 0x7f497809aee0 con 0x7f49881014a0 2023-10-11T16:31:43.401 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:43.401 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":91,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:31:42.150221+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:31:43.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49740a11e0 msgr2=0x7f49740a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49740a11e0 0x7f49740a3690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4988101300 tx=0x7f4984009260 comp rx=0 tx=0).stop 2023-10-11T16:31:43.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 msgr2=0x7f4988072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 0x7f4988072ee0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f497800b620 tx=0x7f49780057f0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 shutdown_connections 2023-10-11T16:31:43.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49740a11e0 0x7f49740a3690 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4988101e30 0x7f4988073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49881002b0 0x7f49880748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 --2- 172.21.15.111:0/3596708775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49881014a0 0x7f4988072ee0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.401+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 >> 172.21.15.111:0/3596708775 conn(0x7f49880fb890 msgr2=0x7f49880fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:43.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.402+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 shutdown_connections 2023-10-11T16:31:43.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.402+0000 7f498f7ff700 1 -- 172.21.15.111:0/3596708775 wait complete. 2023-10-11T16:31:43.422 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:31:43.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:43 smithi150 ceph-mon[132217]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:31:43.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:43 smithi150 ceph-mon[132217]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-10-11T16:31:43.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:43 smithi150 ceph-mon[132217]: osdmap e91: 12 total, 12 up, 12 in 2023-10-11T16:31:43.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:43 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1144069201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:31:43.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:43 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2663502968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:43.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:43 smithi130 ceph-mon[131859]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:31:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:43 smithi130 ceph-mon[131859]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-10-11T16:31:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:43 smithi130 ceph-mon[131859]: osdmap e91: 12 total, 12 up, 12 in 2023-10-11T16:31:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:43 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1144069201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:31:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:43 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2663502968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:43.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2451624732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ac101e30 msgr2=0x7fc9ac1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2451624732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ac101e30 0x7fc9ac1086f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400b0a0 tx=0x7fc9a400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2451624732 shutdown_connections 2023-10-11T16:31:43.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2451624732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ac101e30 0x7fc9ac1086f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2451624732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ac1002b0 0x7fc9ac1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2451624732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ac1014a0 0x7fc9ac1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2451624732 >> 172.21.15.111:0/2451624732 conn(0x7fc9ac0fb890 msgr2=0x7fc9ac0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:43.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2451624732 shutdown_connections 2023-10-11T16:31:43.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.617+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2451624732 wait complete. 2023-10-11T16:31:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.618+0000 7fc9b1dff700 1 Processor -- start 2023-10-11T16:31:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.618+0000 7fc9b1dff700 1 -- start start 2023-10-11T16:31:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9b1dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ac1002b0 0x7fc9ac0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:43.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9b1dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ac1014a0 0x7fc9ac072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9b1dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ac101e30 0x7fc9ac073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9b1dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac073bc0 con 0x7fc9ac101e30 2023-10-11T16:31:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9b1dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac073d30 con 0x7fc9ac1002b0 2023-10-11T16:31:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9b1dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac1a00d0 con 0x7fc9ac1014a0 2023-10-11T16:31:43.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ac101e30 0x7fc9ac073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:43.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ac101e30 0x7fc9ac073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47444/0 (socket says 172.21.15.111:47444) 2023-10-11T16:31:43.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9abfff700 1 -- 172.21.15.111:0/2234860522 learned_addr learned my addr 172.21.15.111:0/2234860522 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:43.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9aaffd700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ac1014a0 0x7fc9ac072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:43.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9ab7fe700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ac1002b0 0x7fc9ac0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:43.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9abfff700 1 -- 172.21.15.111:0/2234860522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ac1014a0 msgr2=0x7fc9ac072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9abfff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ac1014a0 0x7fc9ac072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9abfff700 1 -- 172.21.15.111:0/2234860522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ac1002b0 msgr2=0x7fc9ac0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9abfff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ac1002b0 0x7fc9ac0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.619+0000 7fc9abfff700 1 -- 172.21.15.111:0/2234860522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4009d00 con 0x7fc9ac101e30 2023-10-11T16:31:43.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.620+0000 7fc9abfff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ac101e30 0x7fc9ac073420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400d010 tx=0x7fc9a400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:43.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.620+0000 7fc9a8ff9700 1 -- 172.21.15.111:0/2234860522 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a400e040 con 0x7fc9ac101e30 2023-10-11T16:31:43.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.620+0000 7fc9a8ff9700 1 -- 172.21.15.111:0/2234860522 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9a400bdb0 con 0x7fc9ac101e30 2023-10-11T16:31:43.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.620+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ac10bc80 con 0x7fc9ac101e30 2023-10-11T16:31:43.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.620+0000 7fc9a8ff9700 1 -- 172.21.15.111:0/2234860522 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4005260 con 0x7fc9ac101e30 2023-10-11T16:31:43.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.620+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac10baf0 con 0x7fc9ac101e30 2023-10-11T16:31:43.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.621+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac067a60 con 0x7fc9ac101e30 2023-10-11T16:31:43.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.623+0000 7fc9a8ff9700 1 -- 172.21.15.111:0/2234860522 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fc9a401d480 con 0x7fc9ac101e30 2023-10-11T16:31:43.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.625+0000 7fc9a8ff9700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc9940a11e0 0x7fc9940a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:43.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.625+0000 7fc9a8ff9700 1 -- 172.21.15.111:0/2234860522 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fc9a40ce1f0 con 0x7fc9ac101e30 2023-10-11T16:31:43.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.625+0000 7fc9ab7fe700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc9940a11e0 0x7fc9940a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:43.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.626+0000 7fc9ab7fe700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc9940a11e0 0x7fc9940a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac101300 tx=0x7fc99c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:43.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.629+0000 7fc9a8ff9700 1 -- 172.21.15.111:0/2234860522 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc9a40974a0 con 0x7fc9ac101e30 2023-10-11T16:31:43.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.874+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9ac107130 con 0x7fc9ac101e30 2023-10-11T16:31:43.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.875+0000 7fc9a8ff9700 1 -- 172.21.15.111:0/2234860522 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10823 (secure 0 0 0) 0x7fc9a409aac0 con 0x7fc9ac101e30 2023-10-11T16:31:43.877 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:31:43.877 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:51.227688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:31:43.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.877+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc9940a11e0 msgr2=0x7fc9940a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.877+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc9940a11e0 0x7fc9940a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac101300 tx=0x7fc99c009260 comp rx=0 tx=0).stop 2023-10-11T16:31:43.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ac101e30 msgr2=0x7fc9ac073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:43.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ac101e30 0x7fc9ac073420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400d010 tx=0x7fc9a400bc80 comp rx=0 tx=0).stop 2023-10-11T16:31:43.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 shutdown_connections 2023-10-11T16:31:43.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc9940a11e0 0x7fc9940a3690 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9ac101e30 0x7fc9ac073420 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9ac1002b0 0x7fc9ac0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 --2- 172.21.15.111:0/2234860522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ac1014a0 0x7fc9ac072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:43.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 >> 172.21.15.111:0/2234860522 conn(0x7fc9ac0fb890 msgr2=0x7fc9ac0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:43.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 shutdown_connections 2023-10-11T16:31:43.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:43.878+0000 7fc9b1dff700 1 -- 172.21.15.111:0/2234860522 wait complete. 2023-10-11T16:31:43.882 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-10-11T16:31:43.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-10-11T16:30:11.633745+0000', 'modified': '2023-10-11T16:30:51.227688+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24569, 'mds_1': 24550, 'mds_2': 24532}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24488': {'gid': 24488, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6835/89303870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 89303870}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 89303870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6835/2357736565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 2357736565}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2357736565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24521': {'gid': 24521, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6837/3927432367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 3927432367}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 3927432367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.150:6837/612139379', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 612139379}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 612139379}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.150:6839/3698951370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 3698951370}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 3698951370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.130:6839/2448702777', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2448702777}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2448702777}]}, 'join_fscid': -1, 'export_targets': [1, 2], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-11T16:31:43.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-11T16:31:43.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-11T16:31:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 -- 172.21.15.111:0/2004448653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101430 msgr2=0x7f97bc101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 --2- 172.21.15.111:0/2004448653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101430 0x7f97bc101880 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009990 tx=0x7f97b0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 -- 172.21.15.111:0/2004448653 shutdown_connections 2023-10-11T16:31:44.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 --2- 172.21.15.111:0/2004448653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97bc101dc0 0x7f97bc108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 --2- 172.21.15.111:0/2004448653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97bc100240 0x7f97bc100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 --2- 172.21.15.111:0/2004448653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101430 0x7f97bc101880 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 -- 172.21.15.111:0/2004448653 >> 172.21.15.111:0/2004448653 conn(0x7f97bc0fb820 msgr2=0x7f97bc0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 -- 172.21.15.111:0/2004448653 shutdown_connections 2023-10-11T16:31:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.097+0000 7f97c1dff700 1 -- 172.21.15.111:0/2004448653 wait complete. 2023-10-11T16:31:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97c1dff700 1 Processor -- start 2023-10-11T16:31:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97c1dff700 1 -- start start 2023-10-11T16:31:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97c1dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97bc100240 0x7f97bc0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97c1dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97bc101430 0x7f97bc072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97c1dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101dc0 0x7f97bc073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97c1dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc073bc0 con 0x7f97bc101dc0 2023-10-11T16:31:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97c1dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc073d30 con 0x7f97bc100240 2023-10-11T16:31:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97c1dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc1a0080 con 0x7f97bc101430 2023-10-11T16:31:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.098+0000 7f97bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101dc0 0x7f97bc073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97bc100240 0x7f97bc0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101dc0 0x7f97bc073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47462/0 (socket says 172.21.15.111:47462) 2023-10-11T16:31:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bbfff700 1 -- 172.21.15.111:0/1540630827 learned_addr learned my addr 172.21.15.111:0/1540630827 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97baffd700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97bc101430 0x7f97bc072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bbfff700 1 -- 172.21.15.111:0/1540630827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97bc101430 msgr2=0x7f97bc072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bbfff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97bc101430 0x7f97bc072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bbfff700 1 -- 172.21.15.111:0/1540630827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97bc100240 msgr2=0x7f97bc0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bbfff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97bc100240 0x7f97bc0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bbfff700 1 -- 172.21.15.111:0/1540630827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b00095f0 con 0x7f97bc101dc0 2023-10-11T16:31:44.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97baffd700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97bc101430 0x7f97bc072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:31:44.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97bbfff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101dc0 0x7f97bc073420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f97b4007ee0 tx=0x7f97b400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97b8ff9700 1 -- 172.21.15.111:0/1540630827 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4017070 con 0x7f97bc101dc0 2023-10-11T16:31:44.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97b8ff9700 1 -- 172.21.15.111:0/1540630827 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97b400f040 con 0x7f97bc101dc0 2023-10-11T16:31:44.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97c1dff700 1 -- 172.21.15.111:0/1540630827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97bc10bc00 con 0x7f97bc101dc0 2023-10-11T16:31:44.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.099+0000 7f97b8ff9700 1 -- 172.21.15.111:0/1540630827 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4005740 con 0x7f97bc101dc0 2023-10-11T16:31:44.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.100+0000 7f97c1dff700 1 -- 172.21.15.111:0/1540630827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc10ba70 con 0x7f97bc101dc0 2023-10-11T16:31:44.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.103+0000 7f97b8ff9700 1 -- 172.21.15.111:0/1540630827 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f97b400dc60 con 0x7f97bc101dc0 2023-10-11T16:31:44.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.104+0000 7f97d564ab80 1 Processor -- start 2023-10-11T16:31:44.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.104+0000 7f97b8ff9700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f97a40a1260 0x7f97a40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.104+0000 7f97b8ff9700 1 -- 172.21.15.111:0/1540630827 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f97b40cc5a0 con 0x7f97bc101dc0 2023-10-11T16:31:44.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.104+0000 7f97d564ab80 1 -- start start 2023-10-11T16:31:44.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.104+0000 7f97baffd700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f97a40a1260 0x7f97a40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.104+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.104+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eb500 0x559ce20ebae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.104+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20ec0c0 0x559ce20ec530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97d564ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1ff0010 con 0x559ce20eabb0 2023-10-11T16:31:44.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97d564ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce2042850 con 0x559ce20ec0c0 2023-10-11T16:31:44.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97d564ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce203bce0 con 0x559ce20eb500 2023-10-11T16:31:44.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20eafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47472/0 (socket says 172.21.15.111:47472) 2023-10-11T16:31:44.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 -- 172.21.15.111:0/1080818918 learned_addr learned my addr 172.21.15.111:0/1080818918 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97baffd700 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eb500 0x559ce20ebae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bbfff700 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20ec0c0 0x559ce20ec530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 -- 172.21.15.111:0/1080818918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eb500 msgr2=0x559ce20ebae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eb500 0x559ce20ebae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 -- 172.21.15.111:0/1080818918 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20ec0c0 msgr2=0x559ce20ec530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20ec0c0 0x559ce20ec530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97baffd700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f97a40a1260 0x7f97a40a3710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f97b000b5c0 tx=0x7f97b0005060 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 -- 172.21.15.111:0/1080818918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ce1ebd5e0 con 0x559ce20eabb0 2023-10-11T16:31:44.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.105+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20eafc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00bd40 tx=0x7f97ac0106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b44bfc47013c04 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97a1ffb700 1 -- 172.21.15.111:0/1080818918 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac010e10 con 0x559ce20eabb0 2023-10-11T16:31:44.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97a1ffb700 1 -- 172.21.15.111:0/1080818918 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97ac015070 con 0x559ce20eabb0 2023-10-11T16:31:44.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97a1ffb700 1 -- 172.21.15.111:0/1080818918 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac005020 con 0x559ce20eabb0 2023-10-11T16:31:44.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97d564ab80 1 -- 172.21.15.111:0/1080818918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 msgr2=0x559ce20eafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20eafc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00bd40 tx=0x7f97ac0106b0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97d564ab80 1 -- 172.21.15.111:0/1080818918 shutdown_connections 2023-10-11T16:31:44.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20ec0c0 0x559ce20ec530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20eafc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.106+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1080818918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eb500 0x559ce20ebae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.107+0000 7f97d564ab80 1 -- 172.21.15.111:0/1080818918 >> 172.21.15.111:0/1080818918 conn(0x559ce20e0310 msgr2=0x559ce20e5620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.107+0000 7f97d564ab80 1 -- 172.21.15.111:0/1080818918 shutdown_connections 2023-10-11T16:31:44.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.107+0000 7f97d564ab80 1 -- 172.21.15.111:0/1080818918 wait complete. 2023-10-11T16:31:44.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.107+0000 7f97d564ab80 1 -- start start 2023-10-11T16:31:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.107+0000 7f97d564ab80 1 Processor -- start 2023-10-11T16:31:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eabb0 0x559ce20f4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20ec0c0 0x559ce20fbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97d564ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1fdd890 con 0x559ce20ec0c0 2023-10-11T16:31:44.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97d564ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1fdd530 con 0x559ce20eb500 2023-10-11T16:31:44.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97d564ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1ebf8d0 con 0x559ce20eabb0 2023-10-11T16:31:44.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97d564ab80 10 client.0 _ll_register_callbacks cb 0x559ce1ffb730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:31:44.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97baffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97baffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20f4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:42060/0 (socket says 172.21.15.111:42060) 2023-10-11T16:31:44.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97baffd700 1 -- 172.21.15.111:0/411034150 learned_addr learned my addr 172.21.15.111:0/411034150 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97bbfff700 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20ec0c0 0x559ce20fbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eabb0 0x559ce20f4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97baffd700 1 -- 172.21.15.111:0/411034150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eabb0 msgr2=0x559ce20f4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97baffd700 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eabb0 0x559ce20f4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97baffd700 1 -- 172.21.15.111:0/411034150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20ec0c0 msgr2=0x559ce20fbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97baffd700 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20ec0c0 0x559ce20fbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.108+0000 7f97baffd700 1 -- 172.21.15.111:0/411034150 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ac00b9a0 con 0x559ce20eb500 2023-10-11T16:31:44.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f97baffd700 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20f4a90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f97b0015000 tx=0x7f97b0009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a2bfffcf4b7c8d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f979affd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:31:44.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f979affd700 1 -- 172.21.15.111:0/411034150 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b00047d0 con 0x559ce20eb500 2023-10-11T16:31:44.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f979affd700 1 -- 172.21.15.111:0/411034150 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f97b0004930 con 0x559ce20eb500 2023-10-11T16:31:44.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f97d564ab80 1 -- 172.21.15.111:0/411034150 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ce20ddad0 con 0x559ce20eb500 2023-10-11T16:31:44.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f979affd700 1 -- 172.21.15.111:0/411034150 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97b0004af0 con 0x559ce20eb500 2023-10-11T16:31:44.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f97d564ab80 10 client.25142 fetch_fsmap learned FSMap version 26 2023-10-11T16:31:44.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f97d564ab80 1 -- 172.21.15.111:0/411034150 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x559ce2002140 con 0x559ce20eb500 2023-10-11T16:31:44.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.109+0000 7f979affd700 1 -- 172.21.15.111:0/411034150 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559ce20ddad0 con 0x559ce20eb500 2023-10-11T16:31:44.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.110+0000 7f979affd700 1 -- 172.21.15.111:0/411034150 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f97b000f460 con 0x559ce20eb500 2023-10-11T16:31:44.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.110+0000 7f97d564ab80 10 client.25142 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:31:44.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.110+0000 7f97d564ab80 10 client.25142 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-11T16:31:44.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.110+0000 7f97d564ab80 20 client.25142 populate_metadata read hostname 'smithi111' 2023-10-11T16:31:44.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.110+0000 7f97d564ab80 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x559ce20ff820 0x559ce20fec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.110+0000 7f97d564ab80 4 client.25142 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:31:44.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.110+0000 7f97d564ab80 1 -- 172.21.15.111:0/411034150 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ce1ffe050 con 0x559ce20ff820 2023-10-11T16:31:44.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.110+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x559ce20ff820 0x559ce20fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.111+0000 7f979affd700 10 client.25142 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:31:44.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.111+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x559ce20ff820 0x559ce20fec20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=79208f4c2cbf561b server_cookie=a8c50eb59f8330c in_seq=0 out_seq=0 2023-10-11T16:31:44.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f979affd700 1 -- 172.21.15.111:0/411034150 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559ce1ffe050 con 0x559ce20ff820 2023-10-11T16:31:44.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f979affd700 10 client.25142 handle_command_reply: tid=0 2023-10-11T16:31:44.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 client.25142 shutdown 2023-10-11T16:31:44.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 -- 172.21.15.111:0/411034150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 msgr2=0x559ce20f4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20f4a90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f97b0015000 tx=0x7f97b0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 -- 172.21.15.111:0/411034150 shutdown_connections 2023-10-11T16:31:44.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x559ce20ff820 0x559ce20fec20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20ec0c0 0x559ce20fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20eabb0 0x559ce20f4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f979affd700 0 client.25142 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:31:44.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 --2- 172.21.15.111:0/411034150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20f4a90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 -- 172.21.15.111:0/411034150 >> 172.21.15.111:0/411034150 conn(0x559ce20e0310 msgr2=0x559ce20e4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 -- 172.21.15.111:0/411034150 shutdown_connections 2023-10-11T16:31:44.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 1 -- 172.21.15.111:0/411034150 wait complete. 2023-10-11T16:31:44.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.112+0000 7f97d564ab80 20 client.25142 trim_cache size 0 max 16384 2023-10-11T16:31:44.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97d564ab80 1 Processor -- start 2023-10-11T16:31:44.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97d564ab80 1 -- start start 2023-10-11T16:31:44.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20fec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20ff160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce20fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97d564ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce20ec9b0 con 0x559ce20eabb0 2023-10-11T16:31:44.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97d564ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce1ffe080 con 0x559ce20eb500 2023-10-11T16:31:44.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97d564ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce20dd9e0 con 0x559ce20ec0c0 2023-10-11T16:31:44.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97bbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce20fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97bb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.131+0000 7f97baffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20ff160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97bb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20fec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47496/0 (socket says 172.21.15.111:47496) 2023-10-11T16:31:44.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97bb7fe700 1 -- 172.21.15.111:0/3839396678 learned_addr learned my addr 172.21.15.111:0/3839396678 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97bb7fe700 1 -- 172.21.15.111:0/3839396678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 msgr2=0x559ce20fb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/3839396678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce20fb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97bb7fe700 1 -- 172.21.15.111:0/3839396678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 msgr2=0x559ce20ff160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/3839396678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20ff160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97bb7fe700 1 -- 172.21.15.111:0/3839396678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ce2001e90 con 0x559ce20eabb0 2023-10-11T16:31:44.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/3839396678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20fec20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f97ac010a00 tx=0x7f97ac00b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13d95599f3db18c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97a27fc700 1 -- 172.21.15.111:0/3839396678 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac020070 con 0x559ce20eabb0 2023-10-11T16:31:44.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97a27fc700 1 -- 172.21.15.111:0/3839396678 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97ac021040 con 0x559ce20eabb0 2023-10-11T16:31:44.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.132+0000 7f97a27fc700 1 -- 172.21.15.111:0/3839396678 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac004340 con 0x559ce20eabb0 2023-10-11T16:31:44.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 -- 172.21.15.111:0/3839396678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 msgr2=0x559ce20fec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 --2- 172.21.15.111:0/3839396678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20fec20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f97ac010a00 tx=0x7f97ac00b430 comp rx=0 tx=0).stop 2023-10-11T16:31:44.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 -- 172.21.15.111:0/3839396678 shutdown_connections 2023-10-11T16:31:44.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 --2- 172.21.15.111:0/3839396678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce20fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 --2- 172.21.15.111:0/3839396678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eabb0 0x559ce20fec20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 --2- 172.21.15.111:0/3839396678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eb500 0x559ce20ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 -- 172.21.15.111:0/3839396678 >> 172.21.15.111:0/3839396678 conn(0x559ce20e0310 msgr2=0x559ce2108610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 -- 172.21.15.111:0/3839396678 shutdown_connections 2023-10-11T16:31:44.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.133+0000 7f97d564ab80 1 -- 172.21.15.111:0/3839396678 wait complete. 2023-10-11T16:31:44.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 1 -- start start 2023-10-11T16:31:44.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 1 Processor -- start 2023-10-11T16:31:44.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eabb0 0x559ce20f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eb500 0x559ce20f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce2153740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce20fba10 con 0x559ce20eb500 2023-10-11T16:31:44.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce20fbb80 con 0x559ce20eabb0 2023-10-11T16:31:44.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ce20fbd40 con 0x559ce20ec0c0 2023-10-11T16:31:44.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97d564ab80 10 client.0 _ll_register_callbacks cb 0x7f97b0005550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:31:44.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.134+0000 7f97baffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eb500 0x559ce20f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97baffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eb500 0x559ce20f69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47504/0 (socket says 172.21.15.111:47504) 2023-10-11T16:31:44.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97bbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce2153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97baffd700 1 -- 172.21.15.111:0/1819920187 learned_addr learned my addr 172.21.15.111:0/1819920187 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97bb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eabb0 0x559ce20f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97bbfff700 1 -- 172.21.15.111:0/1819920187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eabb0 msgr2=0x559ce20f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97bbfff700 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eabb0 0x559ce20f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97bbfff700 1 -- 172.21.15.111:0/1819920187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eb500 msgr2=0x559ce20f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97bbfff700 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eb500 0x559ce20f69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97bbfff700 1 -- 172.21.15.111:0/1819920187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ce1ebd5e0 con 0x559ce20ec0c0 2023-10-11T16:31:44.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f97bbfff700 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce2153740 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f97b40c9b20 tx=0x7f97b400df60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af5bd9a0b0eccaf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.135+0000 7f979affd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:31:44.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.136+0000 7f979affd700 1 -- 172.21.15.111:0/1819920187 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b40cb040 con 0x559ce20ec0c0 2023-10-11T16:31:44.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.136+0000 7f979affd700 1 -- 172.21.15.111:0/1819920187 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f97b40a1870 con 0x559ce20ec0c0 2023-10-11T16:31:44.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.136+0000 7f97d564ab80 1 -- 172.21.15.111:0/1819920187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97ac015070 con 0x559ce20ec0c0 2023-10-11T16:31:44.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.136+0000 7f979affd700 1 -- 172.21.15.111:0/1819920187 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97b409d120 con 0x559ce20ec0c0 2023-10-11T16:31:44.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.136+0000 7f97d564ab80 10 client.25165 fetch_fsmap learned FSMap version 26 2023-10-11T16:31:44.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.136+0000 7f97d564ab80 1 -- 172.21.15.111:0/1819920187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x559ce1e52c10 con 0x559ce20ec0c0 2023-10-11T16:31:44.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.136+0000 7f979affd700 1 -- 172.21.15.111:0/1819920187 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97b40c7070 con 0x559ce20ec0c0 2023-10-11T16:31:44.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.137+0000 7f979affd700 1 -- 172.21.15.111:0/1819920187 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f97b4099680 con 0x559ce20ec0c0 2023-10-11T16:31:44.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.137+0000 7f97d564ab80 10 client.25165 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:31:44.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.137+0000 7f97d564ab80 10 client.25165 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-11T16:31:44.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.137+0000 7f97d564ab80 20 client.25165 populate_metadata read hostname 'smithi111' 2023-10-11T16:31:44.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.137+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x559ce21588a0 0x559ce2155190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.137+0000 7f97d564ab80 4 client.25165 mds_command: new command op to 24532 tid=0[{"prefix": "status"}] 2023-10-11T16:31:44.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.137+0000 7f97d564ab80 1 -- 172.21.15.111:0/1819920187 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559ce200a610 con 0x559ce21588a0 2023-10-11T16:31:44.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.137+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x559ce21588a0 0x559ce2155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97bb7fe700 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x559ce21588a0 0x559ce2155190 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fedb20bd4d1c46d5 server_cookie=97fb879225178f4a in_seq=0 out_seq=0 2023-10-11T16:31:44.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f979affd700 10 client.25165 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:31:44.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f979affd700 1 -- 172.21.15.111:0/1819920187 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x559ce200a610 con 0x559ce21588a0 2023-10-11T16:31:44.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f979affd700 10 client.25165 handle_command_reply: tid=0 2023-10-11T16:31:44.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 client.25165 shutdown 2023-10-11T16:31:44.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 -- 172.21.15.111:0/1819920187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 msgr2=0x559ce2153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce2153740 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f97b40c9b20 tx=0x7f97b400df60 comp rx=0 tx=0).stop 2023-10-11T16:31:44.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 -- 172.21.15.111:0/1819920187 shutdown_connections 2023-10-11T16:31:44.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x559ce21588a0 0x559ce2155190 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559ce20ec0c0 0x559ce2153740 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ce20eabb0 0x559ce20f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f979affd700 0 client.25165 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:31:44.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 --2- 172.21.15.111:0/1819920187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559ce20eb500 0x559ce20f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 -- 172.21.15.111:0/1819920187 >> 172.21.15.111:0/1819920187 conn(0x559ce20e0310 msgr2=0x559ce20fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 -- 172.21.15.111:0/1819920187 shutdown_connections 2023-10-11T16:31:44.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.138+0000 7f97d564ab80 1 -- 172.21.15.111:0/1819920187 wait complete. 2023-10-11T16:31:44.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97d564ab80 20 client.25165 trim_cache size 0 max 16384 2023-10-11T16:31:44.155 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:31:44.156 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:31:44.156 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 2, 2023-10-11T16:31:44.156 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 24532, 2023-10-11T16:31:44.156 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-10-11T16:31:44.156 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-10-11T16:31:44.157 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-10-11T16:31:44.157 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 66.973865849999996, 2023-10-11T16:31:44.157 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 26, 2023-10-11T16:31:44.157 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 91, 2023-10-11T16:31:44.157 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 89, 2023-10-11T16:31:44.158 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 111.486729528 2023-10-11T16:31:44.158 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:31:44.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 -- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f97a40a1260 msgr2=0x7f97a40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f97a40a1260 0x7f97a40a3710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f97b000b5c0 tx=0x7f97b0005060 comp rx=0 tx=0).stop 2023-10-11T16:31:44.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 -- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101dc0 msgr2=0x7f97bc073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101dc0 0x7f97bc073420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f97b4007ee0 tx=0x7f97b400d3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 -- 172.21.15.111:0/1540630827 shutdown_connections 2023-10-11T16:31:44.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f97a40a1260 0x7f97a40a3710 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc101dc0 0x7f97bc073420 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97bc100240 0x7f97bc0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 --2- 172.21.15.111:0/1540630827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97bc101430 0x7f97bc072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 -- 172.21.15.111:0/1540630827 >> 172.21.15.111:0/1540630827 conn(0x7f97bc0fb820 msgr2=0x7f97bc1048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 -- 172.21.15.111:0/1540630827 shutdown_connections 2023-10-11T16:31:44.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.139+0000 7f97c1dff700 1 -- 172.21.15.111:0/1540630827 wait complete. 2023-10-11T16:31:44.162 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-10-11T16:31:44.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.360+0000 7f49c69ff700 1 -- 172.21.15.111:0/1089786841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 msgr2=0x7f49c0100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.360+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1089786841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 0x7f49c0100640 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009990 tx=0x7f49b0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.360+0000 7f49c69ff700 1 -- 172.21.15.111:0/1089786841 shutdown_connections 2023-10-11T16:31:44.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.360+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1089786841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49c0101db0 0x7f49c0108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.360+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1089786841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 0x7f49c0100640 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.360+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1089786841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49c0101420 0x7f49c0101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.360+0000 7f49c69ff700 1 -- 172.21.15.111:0/1089786841 >> 172.21.15.111:0/1089786841 conn(0x7f49c00fb810 msgr2=0x7f49c00fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.360+0000 7f49c69ff700 1 -- 172.21.15.111:0/1089786841 shutdown_connections 2023-10-11T16:31:44.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.361+0000 7f49c69ff700 1 -- 172.21.15.111:0/1089786841 wait complete. 2023-10-11T16:31:44.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.361+0000 7f49c69ff700 1 Processor -- start 2023-10-11T16:31:44.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49c69ff700 1 -- start start 2023-10-11T16:31:44.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49c69ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 0x7f49c01981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49c69ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49c0101420 0x7f49c0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49c69ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49c0101db0 0x7f49c019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49c69ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c019fd60 con 0x7f49c0100230 2023-10-11T16:31:44.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49c69ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c019fea0 con 0x7f49c0101db0 2023-10-11T16:31:44.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49c69ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c01a0030 con 0x7f49c0101420 2023-10-11T16:31:44.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 0x7f49c01981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 0x7f49c01981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47524/0 (socket says 172.21.15.111:47524) 2023-10-11T16:31:44.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49bffff700 1 -- 172.21.15.111:0/1206154005 learned_addr learned my addr 172.21.15.111:0/1206154005 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49c0101420 0x7f49c0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49c4f9c700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49c0101db0 0x7f49c019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49bffff700 1 -- 172.21.15.111:0/1206154005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49c0101420 msgr2=0x7f49c0198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.362+0000 7f49bffff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49c0101420 0x7f49c0198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49bffff700 1 -- 172.21.15.111:0/1206154005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49c0101db0 msgr2=0x7f49c019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49bffff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49c0101db0 0x7f49c019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49bffff700 1 -- 172.21.15.111:0/1206154005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00095f0 con 0x7f49c0100230 2023-10-11T16:31:44.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49bffff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 0x7f49c01981e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f49b0004730 tx=0x7f49b0004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49bd7fa700 1 -- 172.21.15.111:0/1206154005 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b001d070 con 0x7f49c0100230 2023-10-11T16:31:44.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49bd7fa700 1 -- 172.21.15.111:0/1206154005 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49b0021410 con 0x7f49c0100230 2023-10-11T16:31:44.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49c69ff700 1 -- 172.21.15.111:0/1206154005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c010bbe0 con 0x7f49c0100230 2023-10-11T16:31:44.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49bd7fa700 1 -- 172.21.15.111:0/1206154005 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b0017650 con 0x7f49c0100230 2023-10-11T16:31:44.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.363+0000 7f49c69ff700 1 -- 172.21.15.111:0/1206154005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c010ba50 con 0x7f49c0100230 2023-10-11T16:31:44.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.366+0000 7f49bd7fa700 1 -- 172.21.15.111:0/1206154005 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f49b0021810 con 0x7f49c0100230 2023-10-11T16:31:44.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.367+0000 7f49bd7fa700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49a80a11e0 0x7f49a80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.367+0000 7f49da2fdb80 1 Processor -- start 2023-10-11T16:31:44.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49bd7fa700 1 -- 172.21.15.111:0/1206154005 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f49b00ce370 con 0x7f49c0100230 2023-10-11T16:31:44.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49da2fdb80 1 -- start start 2023-10-11T16:31:44.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49a80a11e0 0x7f49a80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1e2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 0x55ef4f1e3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1e4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f0e8010 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f13a850 con 0x55ef4f1e40c0 2023-10-11T16:31:44.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f133ce0 con 0x55ef4f1e3500 2023-10-11T16:31:44.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1e2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1e2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47538/0 (socket says 172.21.15.111:47538) 2023-10-11T16:31:44.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49bffff700 1 -- 172.21.15.111:0/1380976532 learned_addr learned my addr 172.21.15.111:0/1380976532 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.368+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 0x55ef4f1e3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49a80a11e0 0x7f49a80a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009cb0 tx=0x7f49b4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49c4f9c700 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1e4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49bffff700 1 -- 172.21.15.111:0/1380976532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 msgr2=0x55ef4f1e3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49bffff700 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 0x55ef4f1e3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49bffff700 1 -- 172.21.15.111:0/1380976532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e40c0 msgr2=0x55ef4f1e4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49bffff700 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1e4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49bffff700 1 -- 172.21.15.111:0/1380976532 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef4efb55e0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49c4f9c700 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1e4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:31:44.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49bffff700 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1e2fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f49b000b2e0 tx=0x7f49b009aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78070e744eae96e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49a67fc700 1 -- 172.21.15.111:0/1380976532 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b009f2b0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49a67fc700 1 -- 172.21.15.111:0/1380976532 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49b00a3640 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.369+0000 7f49a67fc700 1 -- 172.21.15.111:0/1380976532 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b00ce020 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1380976532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 msgr2=0x55ef4f1e2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1e2fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f49b000b2e0 tx=0x7f49b009aeb0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1380976532 shutdown_connections 2023-10-11T16:31:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1e4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1e2fc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/1380976532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 0x55ef4f1e3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1380976532 >> 172.21.15.111:0/1380976532 conn(0x55ef4f1d8310 msgr2=0x55ef4f1dd620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1380976532 shutdown_connections 2023-10-11T16:31:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.370+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1380976532 wait complete. 2023-10-11T16:31:44.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 1 -- start start 2023-10-11T16:31:44.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 1 Processor -- start 2023-10-11T16:31:44.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1ec230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 0x55ef4f1ec770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f0d5890 con 0x55ef4f1e40c0 2023-10-11T16:31:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f0d5530 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4efb78d0 con 0x55ef4f1e3500 2023-10-11T16:31:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.371+0000 7f49da2fdb80 10 client.0 _ll_register_callbacks cb 0x55ef4f0f3730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:31:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1ec230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1ec230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:42138/0 (socket says 172.21.15.111:42138) 2023-10-11T16:31:44.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 -- 172.21.15.111:0/4020105893 learned_addr learned my addr 172.21.15.111:0/4020105893 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49c4f9c700 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 0x55ef4f1ec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 -- 172.21.15.111:0/4020105893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 msgr2=0x55ef4f1ec770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 0x55ef4f1ec770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 -- 172.21.15.111:0/4020105893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 msgr2=0x55ef4f1f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 -- 172.21.15.111:0/4020105893 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c010ba50 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49bffff700 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1ec230 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f49b009b210 tx=0x7f49b0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=444164e8b18b7151 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f499f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:31:44.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f499f7fe700 1 -- 172.21.15.111:0/4020105893 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b009f2b0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f499f7fe700 1 -- 172.21.15.111:0/4020105893 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f49b00a31b0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.372+0000 7f49da2fdb80 1 -- 172.21.15.111:0/4020105893 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ef4f1d5ad0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f499f7fe700 1 -- 172.21.15.111:0/4020105893 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49b00a3370 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f49da2fdb80 10 client.25157 fetch_fsmap learned FSMap version 26 2023-10-11T16:31:44.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f49da2fdb80 1 -- 172.21.15.111:0/4020105893 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ef4f0fa140 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f499f7fe700 1 -- 172.21.15.111:0/4020105893 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ef4f1d5ad0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f499f7fe700 1 -- 172.21.15.111:0/4020105893 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f49b0084ad0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f49da2fdb80 10 client.25157 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:31:44.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f49da2fdb80 10 client.25157 resolve_mds: resolved name 'i' to daemon mds.i 2023-10-11T16:31:44.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f49da2fdb80 20 client.25157 populate_metadata read hostname 'smithi111' 2023-10-11T16:31:44.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef4f1f75e0 0x55ef4f1f6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f49da2fdb80 4 client.25157 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:31:44.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.373+0000 7f49da2fdb80 1 -- 172.21.15.111:0/4020105893 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef4f0f6050 con 0x55ef4f1f75e0 2023-10-11T16:31:44.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.374+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef4f1f75e0 0x55ef4f1f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.374+0000 7f499f7fe700 10 client.25157 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:31:44.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.374+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef4f1f75e0 0x55ef4f1f6650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5346df0fd6a306a5 server_cookie=7c3ea893fa623e78 in_seq=0 out_seq=0 2023-10-11T16:31:44.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.375+0000 7f499f7fe700 1 -- 172.21.15.111:0/4020105893 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ef4f0f6050 con 0x55ef4f1f75e0 2023-10-11T16:31:44.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.375+0000 7f499f7fe700 10 client.25157 handle_command_reply: tid=0 2023-10-11T16:31:44.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.375+0000 7f49da2fdb80 1 client.25157 shutdown 2023-10-11T16:31:44.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.375+0000 7f49da2fdb80 1 -- 172.21.15.111:0/4020105893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e2bb0 msgr2=0x55ef4f1ec230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.375+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1ec230 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f49b009b210 tx=0x7f49b0009310 comp rx=0 tx=0).stop 2023-10-11T16:31:44.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 1 -- 172.21.15.111:0/4020105893 shutdown_connections 2023-10-11T16:31:44.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef4f1f75e0 0x55ef4f1f6650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1ec230 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f499f7fe700 0 client.25157 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:31:44.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/4020105893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e3500 0x55ef4f1ec770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 1 -- 172.21.15.111:0/4020105893 >> 172.21.15.111:0/4020105893 conn(0x55ef4f1d8310 msgr2=0x55ef4f1dcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 1 -- 172.21.15.111:0/4020105893 shutdown_connections 2023-10-11T16:31:44.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 1 -- 172.21.15.111:0/4020105893 wait complete. 2023-10-11T16:31:44.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.376+0000 7f49da2fdb80 20 client.25157 trim_cache size 0 max 16384 2023-10-11T16:31:44.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.394+0000 7f49da2fdb80 1 Processor -- start 2023-10-11T16:31:44.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.394+0000 7f49da2fdb80 1 -- start start 2023-10-11T16:31:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f13a850 con 0x55ef4f1e40c0 2023-10-11T16:31:44.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f1e38d0 con 0x55ef4f1e3500 2023-10-11T16:31:44.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f0f6050 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49c4f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49bf7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49c4f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1eaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47558/0 (socket says 172.21.15.111:47558) 2023-10-11T16:31:44.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49c4f9c700 1 -- 172.21.15.111:0/1011793161 learned_addr learned my addr 172.21.15.111:0/1011793161 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49bffff700 1 --2- 172.21.15.111:0/1011793161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49bf7fe700 1 -- 172.21.15.111:0/1011793161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e2bb0 msgr2=0x55ef4f1eb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/1011793161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1eb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49bf7fe700 1 -- 172.21.15.111:0/1011793161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 msgr2=0x55ef4f1eaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/1011793161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1eaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.395+0000 7f49bf7fe700 1 -- 172.21.15.111:0/1011793161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef4f0f9e90 con 0x55ef4f1e3500 2023-10-11T16:31:44.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/1011793161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f3740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x55ef4f1ecdc0 tx=0x7f49b4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adfb908005ac04a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49a6ffd700 1 -- 172.21.15.111:0/1011793161 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b401c6b0 con 0x55ef4f1e3500 2023-10-11T16:31:44.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49a6ffd700 1 -- 172.21.15.111:0/1011793161 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49b402d070 con 0x55ef4f1e3500 2023-10-11T16:31:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49a6ffd700 1 -- 172.21.15.111:0/1011793161 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4032c00 con 0x55ef4f1e3500 2023-10-11T16:31:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1011793161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 msgr2=0x55ef4f1f3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/1011793161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f3740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x55ef4f1ecdc0 tx=0x7f49b4014000 comp rx=0 tx=0).stop 2023-10-11T16:31:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1011793161 shutdown_connections 2023-10-11T16:31:44.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/1011793161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/1011793161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1eb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/1011793161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f3740 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.396+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1011793161 >> 172.21.15.111:0/1011793161 conn(0x55ef4f1d8310 msgr2=0x55ef4f1ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.397+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1011793161 shutdown_connections 2023-10-11T16:31:44.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.397+0000 7f49da2fdb80 1 -- 172.21.15.111:0/1011793161 wait complete. 2023-10-11T16:31:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.397+0000 7f49da2fdb80 1 -- start start 2023-10-11T16:31:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.397+0000 7f49da2fdb80 1 Processor -- start 2023-10-11T16:31:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49da2fdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1efa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f1f0010 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f1f0180 con 0x55ef4f1e3500 2023-10-11T16:31:44.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49da2fdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f1f0340 con 0x55ef4f1e40c0 2023-10-11T16:31:44.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49da2fdb80 10 client.0 _ll_register_callbacks cb 0x7f49b4009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:31:44.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1f2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47570/0 (socket says 172.21.15.111:47570) 2023-10-11T16:31:44.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bffff700 1 -- 172.21.15.111:0/3557372830 learned_addr learned my addr 172.21.15.111:0/3557372830 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49c4f9c700 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1efa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bffff700 1 -- 172.21.15.111:0/3557372830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e40c0 msgr2=0x55ef4f1efa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bffff700 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1efa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bffff700 1 -- 172.21.15.111:0/3557372830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 msgr2=0x55ef4f1f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bffff700 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.398+0000 7f49bffff700 1 -- 172.21.15.111:0/3557372830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b4017050 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f49bffff700 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1f2fa0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f49b0084fd0 tx=0x7f49b00976f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e999405ef243cc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f499f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:31:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f499f7fe700 1 -- 172.21.15.111:0/3557372830 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b009f2b0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f499f7fe700 1 -- 172.21.15.111:0/3557372830 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f49b00a31b0 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f49da2fdb80 1 -- 172.21.15.111:0/3557372830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49b00a3640 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f499f7fe700 1 -- 172.21.15.111:0/3557372830 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49b00a3370 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f49da2fdb80 10 client.15120 fetch_fsmap learned FSMap version 26 2023-10-11T16:31:44.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f49da2fdb80 1 -- 172.21.15.111:0/3557372830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ef4ef4ac10 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f499f7fe700 1 -- 172.21.15.111:0/3557372830 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49b0062030 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.399+0000 7f499f7fe700 1 -- 172.21.15.111:0/3557372830 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f49b00cb280 con 0x55ef4f1e2bb0 2023-10-11T16:31:44.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f49da2fdb80 10 client.15120 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:31:44.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f49da2fdb80 10 client.15120 resolve_mds: resolved name 'i' to daemon mds.i 2023-10-11T16:31:44.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f49da2fdb80 20 client.15120 populate_metadata read hostname 'smithi111' 2023-10-11T16:31:44.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef4f24f1f0 0x55ef4f24bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f49da2fdb80 4 client.15120 mds_command: new command op to 24550 tid=0[{"prefix": "status"}] 2023-10-11T16:31:44.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f49da2fdb80 1 -- 172.21.15.111:0/3557372830 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ef4f102610 con 0x55ef4f24f1f0 2023-10-11T16:31:44.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef4f24f1f0 0x55ef4f24bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f499f7fe700 10 client.15120 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:31:44.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.400+0000 7f49bf7fe700 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef4f24f1f0 0x55ef4f24bc60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d12884db31b5df56 server_cookie=7bd8cd5c3dc37985 in_seq=0 out_seq=0 2023-10-11T16:31:44.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f499f7fe700 1 -- 172.21.15.111:0/3557372830 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55ef4f102610 con 0x55ef4f24f1f0 2023-10-11T16:31:44.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f499f7fe700 10 client.15120 handle_command_reply: tid=0 2023-10-11T16:31:44.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 client.15120 shutdown 2023-10-11T16:31:44.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 -- 172.21.15.111:0/3557372830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 msgr2=0x55ef4f1f2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1f2fa0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f49b0084fd0 tx=0x7f49b00976f0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 -- 172.21.15.111:0/3557372830 shutdown_connections 2023-10-11T16:31:44.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef4f24f1f0 0x55ef4f24bc60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef4f1e40c0 0x55ef4f1efa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef4f1e2bb0 0x55ef4f1f2fa0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f499f7fe700 0 client.15120 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:31:44.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 --2- 172.21.15.111:0/3557372830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef4f1e3500 0x55ef4f1f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 -- 172.21.15.111:0/3557372830 >> 172.21.15.111:0/3557372830 conn(0x55ef4f1d8310 msgr2=0x55ef4f1ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 -- 172.21.15.111:0/3557372830 shutdown_connections 2023-10-11T16:31:44.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 1 -- 172.21.15.111:0/3557372830 wait complete. 2023-10-11T16:31:44.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.401+0000 7f49da2fdb80 20 client.15120 trim_cache size 0 max 16384 2023-10-11T16:31:44.417 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:31:44.418 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:31:44.418 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 1, 2023-10-11T16:31:44.418 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 24550, 2023-10-11T16:31:44.418 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-10-11T16:31:44.418 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-10-11T16:31:44.419 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-10-11T16:31:44.419 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 68.245529581, 2023-10-11T16:31:44.419 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 26, 2023-10-11T16:31:44.419 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 91, 2023-10-11T16:31:44.419 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 89, 2023-10-11T16:31:44.420 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 107.262831781 2023-10-11T16:31:44.420 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:31:44.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 -- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49a80a11e0 msgr2=0x7f49a80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49a80a11e0 0x7f49a80a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009cb0 tx=0x7f49b4009260 comp rx=0 tx=0).stop 2023-10-11T16:31:44.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 -- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 msgr2=0x7f49c01981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 0x7f49c01981e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f49b0004730 tx=0x7f49b0004760 comp rx=0 tx=0).stop 2023-10-11T16:31:44.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 -- 172.21.15.111:0/1206154005 shutdown_connections 2023-10-11T16:31:44.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f49a80a11e0 0x7f49a80a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49c0101db0 0x7f49c019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c0100230 0x7f49c01981e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 --2- 172.21.15.111:0/1206154005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49c0101420 0x7f49c0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 -- 172.21.15.111:0/1206154005 >> 172.21.15.111:0/1206154005 conn(0x7f49c00fb810 msgr2=0x7f49c00fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 -- 172.21.15.111:0/1206154005 shutdown_connections 2023-10-11T16:31:44.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.402+0000 7f49c69ff700 1 -- 172.21.15.111:0/1206154005 wait complete. 2023-10-11T16:31:44.423 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:44 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3596708775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:44.423 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:44 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2234860522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:44.424 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-11T16:31:44.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:44 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3596708775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:44.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:44 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2234860522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:44.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:44 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3596708775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:31:44.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:44 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2234860522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:31:44.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 -- 172.21.15.111:0/3623168025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8790101420 msgr2=0x7f8790101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 --2- 172.21.15.111:0/3623168025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8790101420 0x7f8790101870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8784009990 tx=0x7f8784009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 -- 172.21.15.111:0/3623168025 shutdown_connections 2023-10-11T16:31:44.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 --2- 172.21.15.111:0/3623168025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790101db0 0x7f8790108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 --2- 172.21.15.111:0/3623168025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8790100230 0x7f8790100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 --2- 172.21.15.111:0/3623168025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8790101420 0x7f8790101870 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 -- 172.21.15.111:0/3623168025 >> 172.21.15.111:0/3623168025 conn(0x7f87900fb810 msgr2=0x7f87900fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 -- 172.21.15.111:0/3623168025 shutdown_connections 2023-10-11T16:31:44.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.620+0000 7f8795bff700 1 -- 172.21.15.111:0/3623168025 wait complete. 2023-10-11T16:31:44.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.621+0000 7f8795bff700 1 Processor -- start 2023-10-11T16:31:44.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f8795bff700 1 -- start start 2023-10-11T16:31:44.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f8795bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790100230 0x7f87901981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f8795bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8790101420 0x7f8790198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f8795bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8790101db0 0x7f879019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f8795bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879019fd60 con 0x7f8790101db0 2023-10-11T16:31:44.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f8795bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879019fea0 con 0x7f8790101420 2023-10-11T16:31:44.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f8795bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87901a0030 con 0x7f8790100230 2023-10-11T16:31:44.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790100230 0x7f87901981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790100230 0x7f87901981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40472/0 (socket says 172.21.15.111:40472) 2023-10-11T16:31:44.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f878f7fe700 1 -- 172.21.15.111:0/68038498 learned_addr learned my addr 172.21.15.111:0/68038498 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f878ffff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8790101db0 0x7f879019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.622+0000 7f878effd700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8790101420 0x7f8790198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878f7fe700 1 -- 172.21.15.111:0/68038498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8790101420 msgr2=0x7f8790198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878f7fe700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8790101420 0x7f8790198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878f7fe700 1 -- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8790101db0 msgr2=0x7f879019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878f7fe700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8790101db0 0x7f879019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878f7fe700 1 -- 172.21.15.111:0/68038498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8780009520 con 0x7f8790100230 2023-10-11T16:31:44.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878f7fe700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790100230 0x7f87901981e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f878000ea50 tx=0x7f878000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878cff9700 1 -- 172.21.15.111:0/68038498 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878000cbd0 con 0x7f8790100230 2023-10-11T16:31:44.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f8795bff700 1 -- 172.21.15.111:0/68038498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87840095f0 con 0x7f8790100230 2023-10-11T16:31:44.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878cff9700 1 -- 172.21.15.111:0/68038498 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f878000cd30 con 0x7f8790100230 2023-10-11T16:31:44.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f878cff9700 1 -- 172.21.15.111:0/68038498 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780018680 con 0x7f8790100230 2023-10-11T16:31:44.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.623+0000 7f8795bff700 1 -- 172.21.15.111:0/68038498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879010bbf0 con 0x7f8790100230 2023-10-11T16:31:44.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.628+0000 7f878cff9700 1 -- 172.21.15.111:0/68038498 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8780010430 con 0x7f8790100230 2023-10-11T16:31:44.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f87a9539b80 1 Processor -- start 2023-10-11T16:31:44.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f87a9539b80 1 -- start start 2023-10-11T16:31:44.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248bdebb0 0x55a248bdefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248bdfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248be00c0 0x55a248be0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f87a9539b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248ae4010 con 0x55a248be00c0 2023-10-11T16:31:44.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f87a9539b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248b36850 con 0x55a248bdebb0 2023-10-11T16:31:44.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f87a9539b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248b2fce0 con 0x55a248bdf500 2023-10-11T16:31:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f878effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248be00c0 0x55a248be0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f878effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248be00c0 0x55a248be0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47602/0 (socket says 172.21.15.111:47602) 2023-10-11T16:31:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248bdfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f878effd700 1 -- 172.21.15.111:0/1002408278 learned_addr learned my addr 172.21.15.111:0/1002408278 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.629+0000 7f878ffff700 1 --2- 172.21.15.111:0/1002408278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248bdebb0 0x55a248bdefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878effd700 1 -- 172.21.15.111:0/1002408278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 msgr2=0x55a248bdfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878effd700 1 --2- 172.21.15.111:0/1002408278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248bdfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878effd700 1 -- 172.21.15.111:0/1002408278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248bdebb0 msgr2=0x55a248bdefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878effd700 1 --2- 172.21.15.111:0/1002408278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248bdebb0 0x55a248bdefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878effd700 1 -- 172.21.15.111:0/1002408278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2489b15e0 con 0x55a248be00c0 2023-10-11T16:31:44.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878cff9700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87780a11e0 0x7f87780a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878cff9700 1 -- 172.21.15.111:0/68038498 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f8780014070 con 0x7f8790100230 2023-10-11T16:31:44.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878effd700 1 --2- 172.21.15.111:0/1002408278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248be00c0 0x55a248be0530 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8784015000 tx=0x7f8784005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc28339f7616177 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f878ffff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87780a11e0 0x7f87780a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f8775ffb700 1 -- 172.21.15.111:0/1002408278 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878401d070 con 0x55a248be00c0 2023-10-11T16:31:44.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f8775ffb700 1 -- 172.21.15.111:0/1002408278 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f878401e040 con 0x55a248be00c0 2023-10-11T16:31:44.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.630+0000 7f8775ffb700 1 -- 172.21.15.111:0/1002408278 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87840048b0 con 0x55a248be00c0 2023-10-11T16:31:44.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f878ffff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87780a11e0 0x7f87780a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8788009990 tx=0x7f8788009130 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 -- 172.21.15.111:0/1002408278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248be00c0 msgr2=0x55a248be0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1002408278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248be00c0 0x55a248be0530 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8784015000 tx=0x7f8784005bc0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 -- 172.21.15.111:0/1002408278 shutdown_connections 2023-10-11T16:31:44.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1002408278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248be00c0 0x55a248be0530 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1002408278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248bdebb0 0x55a248bdefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1002408278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248bdfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 -- 172.21.15.111:0/1002408278 >> 172.21.15.111:0/1002408278 conn(0x55a248bd4310 msgr2=0x55a248bd9620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 -- 172.21.15.111:0/1002408278 shutdown_connections 2023-10-11T16:31:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.631+0000 7f87a9539b80 1 -- 172.21.15.111:0/1002408278 wait complete. 2023-10-11T16:31:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.632+0000 7f87a9539b80 1 -- start start 2023-10-11T16:31:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.632+0000 7f87a9539b80 1 Processor -- start 2023-10-11T16:31:44.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248be84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248be8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248befa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f87a9539b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248ad1890 con 0x55a248bdf500 2023-10-11T16:31:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f87a9539b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248ad1530 con 0x55a248be00c0 2023-10-11T16:31:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f87a9539b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2489b38d0 con 0x55a248bdebb0 2023-10-11T16:31:44.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f87a9539b80 10 client.0 _ll_register_callbacks cb 0x55a248aef730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:31:44.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248be84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248be8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248be84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40490/0 (socket says 172.21.15.111:40490) 2023-10-11T16:31:44.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878f7fe700 1 -- 172.21.15.111:0/3344705567 learned_addr learned my addr 172.21.15.111:0/3344705567 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878ffff700 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248befa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878f7fe700 1 -- 172.21.15.111:0/3344705567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 msgr2=0x55a248befa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878f7fe700 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248befa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878f7fe700 1 -- 172.21.15.111:0/3344705567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 msgr2=0x55a248be8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878f7fe700 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248be8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.633+0000 7f878f7fe700 1 -- 172.21.15.111:0/3344705567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8790108fa0 con 0x55a248bdebb0 2023-10-11T16:31:44.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f878f7fe700 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248be84d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87800a2130 tx=0x7f87800cae40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd7f0a45490716d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f876effd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:31:44.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f876effd700 1 -- 172.21.15.111:0/3344705567 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780099b70 con 0x55a248bdebb0 2023-10-11T16:31:44.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f87a9539b80 1 -- 172.21.15.111:0/3344705567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a248bd1ad0 con 0x55a248bdebb0 2023-10-11T16:31:44.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f876effd700 1 -- 172.21.15.111:0/3344705567 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f878009a210 con 0x55a248bdebb0 2023-10-11T16:31:44.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f876effd700 1 -- 172.21.15.111:0/3344705567 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f878009e240 con 0x55a248bdebb0 2023-10-11T16:31:44.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f87a9539b80 10 client.25195 fetch_fsmap learned FSMap version 26 2023-10-11T16:31:44.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f87a9539b80 1 -- 172.21.15.111:0/3344705567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a248af6140 con 0x55a248bdebb0 2023-10-11T16:31:44.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f876effd700 1 -- 172.21.15.111:0/3344705567 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87800b1020 con 0x55a248bdebb0 2023-10-11T16:31:44.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.634+0000 7f876effd700 1 -- 172.21.15.111:0/3344705567 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f87800a21d0 con 0x55a248bdebb0 2023-10-11T16:31:44.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f87a9539b80 10 client.25195 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:31:44.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f87a9539b80 10 client.25195 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-11T16:31:44.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f87a9539b80 20 client.25195 populate_metadata read hostname 'smithi111' 2023-10-11T16:31:44.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f87a9539b80 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a248bf3800 0x55a248bf2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f87a9539b80 4 client.25195 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:31:44.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f87a9539b80 1 -- 172.21.15.111:0/3344705567 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a248af2050 con 0x55a248bf3800 2023-10-11T16:31:44.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f878effd700 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a248bf3800 0x55a248bf2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f876effd700 10 client.25195 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:31:44.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.635+0000 7f878effd700 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a248bf3800 0x55a248bf2bb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=490ebae3a7e5c1cc server_cookie=f020b5dc2687ee0a in_seq=0 out_seq=0 2023-10-11T16:31:44.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.636+0000 7f876effd700 1 -- 172.21.15.111:0/3344705567 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a248af2050 con 0x55a248bf3800 2023-10-11T16:31:44.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.636+0000 7f876effd700 10 client.25195 handle_command_reply: tid=0 2023-10-11T16:31:44.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.636+0000 7f87a9539b80 1 client.25195 shutdown 2023-10-11T16:31:44.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.636+0000 7f87a9539b80 1 -- 172.21.15.111:0/3344705567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 msgr2=0x55a248be84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.636+0000 7f87a9539b80 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248be84d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87800a2130 tx=0x7f87800cae40 comp rx=0 tx=0).stop 2023-10-11T16:31:44.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 1 -- 172.21.15.111:0/3344705567 shutdown_connections 2023-10-11T16:31:44.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a248bf3800 0x55a248bf2bb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248befa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248be84d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 1 --2- 172.21.15.111:0/3344705567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248be8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f876effd700 0 client.25195 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:31:44.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 1 -- 172.21.15.111:0/3344705567 >> 172.21.15.111:0/3344705567 conn(0x55a248bd4310 msgr2=0x55a248bd8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 1 -- 172.21.15.111:0/3344705567 shutdown_connections 2023-10-11T16:31:44.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 1 -- 172.21.15.111:0/3344705567 wait complete. 2023-10-11T16:31:44.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.637+0000 7f87a9539b80 20 client.25195 trim_cache size 0 max 16384 2023-10-11T16:31:44.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f87a9539b80 1 Processor -- start 2023-10-11T16:31:44.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f87a9539b80 1 -- start start 2023-10-11T16:31:44.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdebb0 0x55a248be72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248befa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248be2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f87a9539b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248be7ee0 con 0x55a248bdebb0 2023-10-11T16:31:44.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f87a9539b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2489b5a30 con 0x55a248be00c0 2023-10-11T16:31:44.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f87a9539b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248bd3ee0 con 0x55a248bdf500 2023-10-11T16:31:44.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdebb0 0x55a248be72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f878f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdebb0 0x55a248be72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47634/0 (socket says 172.21.15.111:47634) 2023-10-11T16:31:44.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.655+0000 7f878f7fe700 1 -- 172.21.15.111:0/2927916300 learned_addr learned my addr 172.21.15.111:0/2927916300 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878effd700 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248befa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878ffff700 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248be2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878f7fe700 1 -- 172.21.15.111:0/2927916300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 msgr2=0x55a248befa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878f7fe700 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248befa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878f7fe700 1 -- 172.21.15.111:0/2927916300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 msgr2=0x55a248be2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878f7fe700 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248be2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878f7fe700 1 -- 172.21.15.111:0/2927916300 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a248af5e90 con 0x55a248bdebb0 2023-10-11T16:31:44.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878effd700 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248befa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:31:44.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878ffff700 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248be2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:31:44.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f878f7fe700 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdebb0 0x55a248be72a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f878000cf50 tx=0x7f87800987c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f407c6f68665f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f87767fc700 1 -- 172.21.15.111:0/2927916300 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780099b70 con 0x55a248bdebb0 2023-10-11T16:31:44.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f87767fc700 1 -- 172.21.15.111:0/2927916300 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87800b1030 con 0x55a248bdebb0 2023-10-11T16:31:44.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.656+0000 7f87767fc700 1 -- 172.21.15.111:0/2927916300 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780010730 con 0x55a248bdebb0 2023-10-11T16:31:44.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 -- 172.21.15.111:0/2927916300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdebb0 msgr2=0x55a248be72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdebb0 0x55a248be72a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f878000cf50 tx=0x7f87800987c0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 -- 172.21.15.111:0/2927916300 shutdown_connections 2023-10-11T16:31:44.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248be2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdebb0 0x55a248be72a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 --2- 172.21.15.111:0/2927916300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdf500 0x55a248befa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 -- 172.21.15.111:0/2927916300 >> 172.21.15.111:0/2927916300 conn(0x55a248bd4310 msgr2=0x55a248bd94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 -- 172.21.15.111:0/2927916300 shutdown_connections 2023-10-11T16:31:44.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 -- 172.21.15.111:0/2927916300 wait complete. 2023-10-11T16:31:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.657+0000 7f87a9539b80 1 -- start start 2023-10-11T16:31:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f87a9539b80 1 Processor -- start 2023-10-11T16:31:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248bea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248beaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f87a9539b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248c47720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f87a9539b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248befb20 con 0x55a248bdf500 2023-10-11T16:31:44.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f87a9539b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248befc90 con 0x55a248be00c0 2023-10-11T16:31:44.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f87a9539b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a248befe50 con 0x55a248bdebb0 2023-10-11T16:31:44.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f87a9539b80 10 client.0 _ll_register_callbacks cb 0x55a248aef730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:31:44.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f878effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248beaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f878effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248beaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47646/0 (socket says 172.21.15.111:47646) 2023-10-11T16:31:44.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f878effd700 1 -- 172.21.15.111:0/1905912643 learned_addr learned my addr 172.21.15.111:0/1905912643 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:44.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f878ffff700 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248c47720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.658+0000 7f878f7fe700 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248bea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f878effd700 1 -- 172.21.15.111:0/1905912643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 msgr2=0x55a248bea5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f878effd700 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248bea5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f878effd700 1 -- 172.21.15.111:0/1905912643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 msgr2=0x55a248c47720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:31:44.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f878effd700 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248c47720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f878effd700 1 -- 172.21.15.111:0/1905912643 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a248af6140 con 0x55a248bdf500 2023-10-11T16:31:44.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f878effd700 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248beaaf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8784005ff0 tx=0x7f8784005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb13a0d01ef9fadb server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:44.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f876effd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:31:44.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f876effd700 1 -- 172.21.15.111:0/1905912643 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878401d040 con 0x55a248bdf500 2023-10-11T16:31:44.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f87a9539b80 1 -- 172.21.15.111:0/1905912643 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f878401e040 con 0x55a248bdf500 2023-10-11T16:31:44.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f876effd700 1 -- 172.21.15.111:0/1905912643 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f8784033550 con 0x55a248bdf500 2023-10-11T16:31:44.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f876effd700 1 -- 172.21.15.111:0/1905912643 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87840059a0 con 0x55a248bdf500 2023-10-11T16:31:44.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f87a9539b80 10 client.15138 fetch_fsmap learned FSMap version 26 2023-10-11T16:31:44.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.659+0000 7f87a9539b80 1 -- 172.21.15.111:0/1905912643 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a248946c10 con 0x55a248bdf500 2023-10-11T16:31:44.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f876effd700 1 -- 172.21.15.111:0/1905912643 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f878401e040 con 0x55a248bdf500 2023-10-11T16:31:44.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f876effd700 1 -- 172.21.15.111:0/1905912643 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f8784004500 con 0x55a248bdf500 2023-10-11T16:31:44.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f87a9539b80 10 client.15138 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:31:44.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f87a9539b80 10 client.15138 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-11T16:31:44.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f87a9539b80 20 client.15138 populate_metadata read hostname 'smithi111' 2023-10-11T16:31:44.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a248c4c5d0 0x55a248c48ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:44.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f87a9539b80 4 client.15138 mds_command: new command op to 24569 tid=0[{"prefix": "status"}] 2023-10-11T16:31:44.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f87a9539b80 1 -- 172.21.15.111:0/1905912643 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a248afe610 con 0x55a248c4c5d0 2023-10-11T16:31:44.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.660+0000 7f878f7fe700 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a248c4c5d0 0x55a248c48ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:44.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f876effd700 10 client.15138 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:31:44.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f878f7fe700 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a248c4c5d0 0x55a248c48ee0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73e45a9b5a618292 server_cookie=8b8b46b89b7742c in_seq=0 out_seq=0 2023-10-11T16:31:44.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f876effd700 1 -- 172.21.15.111:0/1905912643 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55a248afe610 con 0x55a248c4c5d0 2023-10-11T16:31:44.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f876effd700 10 client.15138 handle_command_reply: tid=0 2023-10-11T16:31:44.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 client.15138 shutdown 2023-10-11T16:31:44.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 -- 172.21.15.111:0/1905912643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 msgr2=0x55a248beaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248beaaf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8784005ff0 tx=0x7f8784005320 comp rx=0 tx=0).stop 2023-10-11T16:31:44.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 -- 172.21.15.111:0/1905912643 shutdown_connections 2023-10-11T16:31:44.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a248c4c5d0 0x55a248c48ee0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a248be00c0 0x55a248c47720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a248bdebb0 0x55a248bea5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f876effd700 0 client.15138 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:31:44.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 --2- 172.21.15.111:0/1905912643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a248bdf500 0x55a248beaaf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 -- 172.21.15.111:0/1905912643 >> 172.21.15.111:0/1905912643 conn(0x55a248bd4310 msgr2=0x55a248be2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 -- 172.21.15.111:0/1905912643 shutdown_connections 2023-10-11T16:31:44.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 1 -- 172.21.15.111:0/1905912643 wait complete. 2023-10-11T16:31:44.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.661+0000 7f87a9539b80 20 client.15138 trim_cache size 0 max 16384 2023-10-11T16:31:44.679 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:31:44.679 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:31:44.679 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-10-11T16:31:44.679 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 24569, 2023-10-11T16:31:44.680 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-10-11T16:31:44.680 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-10-11T16:31:44.680 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-10-11T16:31:44.680 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 93.025552687000001, 2023-10-11T16:31:44.680 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 26, 2023-10-11T16:31:44.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 91, 2023-10-11T16:31:44.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 89, 2023-10-11T16:31:44.681 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 99.104517268999999 2023-10-11T16:31:44.681 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:31:44.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 -- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87780a11e0 msgr2=0x7f87780a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87780a11e0 0x7f87780a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8788009990 tx=0x7f8788009130 comp rx=0 tx=0).stop 2023-10-11T16:31:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 -- 172.21.15.111:0/68038498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790100230 msgr2=0x7f87901981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790100230 0x7f87901981e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f878000ea50 tx=0x7f878000ed60 comp rx=0 tx=0).stop 2023-10-11T16:31:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 -- 172.21.15.111:0/68038498 shutdown_connections 2023-10-11T16:31:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87780a11e0 0x7f87780a3690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8790101db0 0x7f879019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8790100230 0x7f87901981e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 --2- 172.21.15.111:0/68038498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8790101420 0x7f8790198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:44.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 -- 172.21.15.111:0/68038498 >> 172.21.15.111:0/68038498 conn(0x7f87900fb810 msgr2=0x7f8790106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:44.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 -- 172.21.15.111:0/68038498 shutdown_connections 2023-10-11T16:31:44.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:44.662+0000 7f8795bff700 1 -- 172.21.15.111:0/68038498 wait complete. 2023-10-11T16:31:44.685 INFO:tasks.cephadm:Setup complete, yielding 2023-10-11T16:31:44.685 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-10-11T16:31:44.695 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:31:44.696 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch status' 2023-10-11T16:31:45.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:45 smithi150 ceph-mon[132217]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:45.541 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:45.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:45 smithi130 ceph-mon[131859]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:45 smithi111 ceph-mon[98527]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:46.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 -- 172.21.15.111:0/108166981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e681014f0 msgr2=0x7f8e68101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:46.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/108166981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e681014f0 0x7f8e68101940 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8e58009990 tx=0x7f8e58009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:46.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 -- 172.21.15.111:0/108166981 shutdown_connections 2023-10-11T16:31:46.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/108166981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e68101e80 0x7f8e68108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:46.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/108166981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e681002f0 0x7f8e68100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:46.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/108166981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e681014f0 0x7f8e68101940 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:46.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 -- 172.21.15.111:0/108166981 >> 172.21.15.111:0/108166981 conn(0x7f8e680fb900 msgr2=0x7f8e680fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:46.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 -- 172.21.15.111:0/108166981 shutdown_connections 2023-10-11T16:31:46.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.943+0000 7f8e6ff08700 1 -- 172.21.15.111:0/108166981 wait complete. 2023-10-11T16:31:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.944+0000 7f8e6ff08700 1 Processor -- start 2023-10-11T16:31:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.944+0000 7f8e6ff08700 1 -- start start 2023-10-11T16:31:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6ff08700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e681002f0 0x7f8e68196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6ff08700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e681014f0 0x7f8e681965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6ff08700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e68101e80 0x7f8e6819d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:46.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6ff08700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6819dbf0 con 0x7f8e68101e80 2023-10-11T16:31:46.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6ff08700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6819dd30 con 0x7f8e681014f0 2023-10-11T16:31:46.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6ff08700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6819dec0 con 0x7f8e681002f0 2023-10-11T16:31:46.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6d4a3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e681014f0 0x7f8e681965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:46.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6d4a3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e681014f0 0x7f8e681965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54218/0 (socket says 172.21.15.111:54218) 2023-10-11T16:31:46.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6e4a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e68101e80 0x7f8e6819d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:46.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6d4a3700 1 -- 172.21.15.111:0/4237663447 learned_addr learned my addr 172.21.15.111:0/4237663447 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:46.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.945+0000 7f8e6dca4700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e681002f0 0x7f8e68196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:46.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e6d4a3700 1 -- 172.21.15.111:0/4237663447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e681002f0 msgr2=0x7f8e68196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:46.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e6d4a3700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e681002f0 0x7f8e68196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:46.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e6d4a3700 1 -- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e68101e80 msgr2=0x7f8e6819d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:46.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e6d4a3700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e68101e80 0x7f8e6819d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:46.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e6d4a3700 1 -- 172.21.15.111:0/4237663447 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580095f0 con 0x7f8e681014f0 2023-10-11T16:31:46.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e6d4a3700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e681014f0 0x7f8e681965a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e58004730 tx=0x7f8e58004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:46.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e5effd700 1 -- 172.21.15.111:0/4237663447 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5801d070 con 0x7f8e681014f0 2023-10-11T16:31:46.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6810bc90 con 0x7f8e681014f0 2023-10-11T16:31:46.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e5effd700 1 -- 172.21.15.111:0/4237663447 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e58004c90 con 0x7f8e681014f0 2023-10-11T16:31:46.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e5effd700 1 -- 172.21.15.111:0/4237663447 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e580054b0 con 0x7f8e681014f0 2023-10-11T16:31:46.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.946+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6810bb00 con 0x7f8e681014f0 2023-10-11T16:31:46.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.947+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e681078f0 con 0x7f8e681014f0 2023-10-11T16:31:46.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.950+0000 7f8e5effd700 1 -- 172.21.15.111:0/4237663447 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f8e5800fbc0 con 0x7f8e681014f0 2023-10-11T16:31:46.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.951+0000 7f8e5effd700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e540a11e0 0x7f8e540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:46.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.951+0000 7f8e5effd700 1 -- 172.21.15.111:0/4237663447 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f8e580ce5b0 con 0x7f8e681014f0 2023-10-11T16:31:46.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.951+0000 7f8e6dca4700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e540a11e0 0x7f8e540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:46.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.952+0000 7f8e6dca4700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e540a11e0 0x7f8e540a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8e68101350 tx=0x7f8e64009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:46.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:46.953+0000 7f8e5effd700 1 -- 172.21.15.111:0/4237663447 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8e58097800 con 0x7f8e681014f0 2023-10-11T16:31:47.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.169+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e68061bf0 con 0x7f8e540a11e0 2023-10-11T16:31:47.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.170+0000 7f8e5effd700 1 -- 172.21.15.111:0/4237663447 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8e68061bf0 con 0x7f8e540a11e0 2023-10-11T16:31:47.172 INFO:teuthology.orchestra.run.smithi111.stdout:Backend: cephadm 2023-10-11T16:31:47.172 INFO:teuthology.orchestra.run.smithi111.stdout:Available: Yes 2023-10-11T16:31:47.172 INFO:teuthology.orchestra.run.smithi111.stdout:Paused: No 2023-10-11T16:31:47.174 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e540a11e0 msgr2=0x7f8e540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:47.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e540a11e0 0x7f8e540a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8e68101350 tx=0x7f8e64009260 comp rx=0 tx=0).stop 2023-10-11T16:31:47.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e681014f0 msgr2=0x7f8e681965a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:47.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e681014f0 0x7f8e681965a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e58004730 tx=0x7f8e58004760 comp rx=0 tx=0).stop 2023-10-11T16:31:47.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 shutdown_connections 2023-10-11T16:31:47.175 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f8e540a11e0 0x7f8e540a3690 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:47.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e68101e80 0x7f8e6819d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:47.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e681002f0 0x7f8e68196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:47.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 --2- 172.21.15.111:0/4237663447 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e681014f0 0x7f8e681965a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:47.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 >> 172.21.15.111:0/4237663447 conn(0x7f8e680fb900 msgr2=0x7f8e68104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:47.176 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 shutdown_connections 2023-10-11T16:31:47.177 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:47.173+0000 7f8e6ff08700 1 -- 172.21.15.111:0/4237663447 wait complete. 2023-10-11T16:31:47.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:47 smithi150 ceph-mon[132217]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-10-11T16:31:47.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:47 smithi130 ceph-mon[131859]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-10-11T16:31:47.598 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:47 smithi111 ceph-mon[98527]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-10-11T16:31:47.637 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-10-11T16:31:48.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:48 smithi150 ceph-mon[132217]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:48.491 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:48.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:48 smithi130 ceph-mon[131859]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:48.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:48 smithi111 ceph-mon[98527]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:49.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:49 smithi111 ceph-mon[98527]: pgmap v325: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:49.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:49 smithi111 ceph-mon[98527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-10-11T16:31:49.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:49 smithi111 ceph-mon[98527]: Cluster is now healthy 2023-10-11T16:31:49.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:49 smithi150 ceph-mon[132217]: pgmap v325: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:49.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:49 smithi150 ceph-mon[132217]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-10-11T16:31:49.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:49 smithi150 ceph-mon[132217]: Cluster is now healthy 2023-10-11T16:31:49.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:49 smithi130 ceph-mon[131859]: pgmap v325: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:49.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:49 smithi130 ceph-mon[131859]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-10-11T16:31:49.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:49 smithi130 ceph-mon[131859]: Cluster is now healthy 2023-10-11T16:31:50.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 -- 172.21.15.111:0/765371768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c101e80 msgr2=0x7f476c108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:50.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 --2- 172.21.15.111:0/765371768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c101e80 0x7f476c108740 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f476400b0a0 tx=0x7f476400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 -- 172.21.15.111:0/765371768 shutdown_connections 2023-10-11T16:31:50.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 --2- 172.21.15.111:0/765371768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c101e80 0x7f476c108740 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 --2- 172.21.15.111:0/765371768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f476c1002f0 0x7f476c100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 --2- 172.21.15.111:0/765371768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f476c1014f0 0x7f476c101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 -- 172.21.15.111:0/765371768 >> 172.21.15.111:0/765371768 conn(0x7f476c0fb900 msgr2=0x7f476c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:50.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 -- 172.21.15.111:0/765371768 shutdown_connections 2023-10-11T16:31:50.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.001+0000 7f4773815700 1 -- 172.21.15.111:0/765371768 wait complete. 2023-10-11T16:31:50.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4773815700 1 Processor -- start 2023-10-11T16:31:50.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4773815700 1 -- start start 2023-10-11T16:31:50.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4773815700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c1002f0 0x7f476c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:50.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4773815700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f476c1014f0 0x7f476c1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:50.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4773815700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f476c101e80 0x7f476c19f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:50.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4773815700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c19ff20 con 0x7f476c1002f0 2023-10-11T16:31:50.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4773815700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c1a0060 con 0x7f476c101e80 2023-10-11T16:31:50.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4773815700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c1a01f0 con 0x7f476c1014f0 2023-10-11T16:31:50.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f47715b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c1002f0 0x7f476c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:50.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f47715b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c1002f0 0x7f476c198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60384/0 (socket says 172.21.15.111:60384) 2023-10-11T16:31:50.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f47715b1700 1 -- 172.21.15.111:0/197152438 learned_addr learned my addr 172.21.15.111:0/197152438 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:50.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.003+0000 7f4770db0700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f476c1014f0 0x7f476c1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:50.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f4771db2700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f476c101e80 0x7f476c19f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:50.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47715b1700 1 -- 172.21.15.111:0/197152438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f476c1014f0 msgr2=0x7f476c1988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:50.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47715b1700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f476c1014f0 0x7f476c1988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47715b1700 1 -- 172.21.15.111:0/197152438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f476c101e80 msgr2=0x7f476c19f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:50.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47715b1700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f476c101e80 0x7f476c19f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47715b1700 1 -- 172.21.15.111:0/197152438 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4764009d00 con 0x7f476c1002f0 2023-10-11T16:31:50.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47715b1700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c1002f0 0x7f476c198390 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f476800e9a0 tx=0x7f476800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:50.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47627fc700 1 -- 172.21.15.111:0/197152438 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768009790 con 0x7f476c1002f0 2023-10-11T16:31:50.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47627fc700 1 -- 172.21.15.111:0/197152438 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47680044e0 con 0x7f476c1002f0 2023-10-11T16:31:50.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f47627fc700 1 -- 172.21.15.111:0/197152438 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768003d20 con 0x7f476c1002f0 2023-10-11T16:31:50.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476c10bcb0 con 0x7f476c1002f0 2023-10-11T16:31:50.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.004+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476c10bb20 con 0x7f476c1002f0 2023-10-11T16:31:50.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.005+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476c1a08e0 con 0x7f476c1002f0 2023-10-11T16:31:50.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.007+0000 7f47627fc700 1 -- 172.21.15.111:0/197152438 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f4768003e80 con 0x7f476c1002f0 2023-10-11T16:31:50.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.008+0000 7f47627fc700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f47580a11e0 0x7f47580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:50.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.009+0000 7f47627fc700 1 -- 172.21.15.111:0/197152438 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f4768014070 con 0x7f476c1002f0 2023-10-11T16:31:50.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.009+0000 7f4770db0700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f47580a11e0 0x7f47580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:50.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.009+0000 7f4770db0700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f47580a11e0 0x7f47580a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f475c009c60 tx=0x7f475c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:50.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.013+0000 7f47627fc700 1 -- 172.21.15.111:0/197152438 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4768096330 con 0x7f476c1002f0 2023-10-11T16:31:50.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.226+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f476c1a0e90 con 0x7f47580a11e0 2023-10-11T16:31:50.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.239+0000 7f47627fc700 1 -- 172.21.15.111:0/197152438 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f476c1a0e90 con 0x7f47580a11e0 2023-10-11T16:31:50.241 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-10-11T16:31:50.242 INFO:teuthology.orchestra.run.smithi111.stdout:mds.a smithi111 running (107s) 21s ago 106s 13.7M - 18.0.0-3908-g05a43c8f 2d174a053211 4849ccd264a8 2023-10-11T16:31:50.242 INFO:teuthology.orchestra.run.smithi111.stdout:mds.b smithi130 running (105s) 33s ago 104s 30.2M - 18.0.0-3908-g05a43c8f 2d174a053211 f51710e3f499 2023-10-11T16:31:50.242 INFO:teuthology.orchestra.run.smithi111.stdout:mds.c smithi150 running (117s) 33s ago 117s 18.8M - 18.0.0-3908-g05a43c8f 2d174a053211 7dbd832129d2 2023-10-11T16:31:50.242 INFO:teuthology.orchestra.run.smithi111.stdout:mds.d smithi111 running (2m) 21s ago 2m 13.8M - 18.0.0-3908-g05a43c8f 2d174a053211 05dd4edef92d 2023-10-11T16:31:50.242 INFO:teuthology.orchestra.run.smithi111.stdout:mds.e smithi130 running (2m) 33s ago 2m 20.8M - 18.0.0-3908-g05a43c8f 2d174a053211 aad645dd999d 2023-10-11T16:31:50.243 INFO:teuthology.orchestra.run.smithi111.stdout:mds.f smithi150 running (2m) 33s ago 119s 16.9M - 18.0.0-3908-g05a43c8f 2d174a053211 5102df2c7cc7 2023-10-11T16:31:50.243 INFO:teuthology.orchestra.run.smithi111.stdout:mds.g smithi111 running (110s) 21s ago 110s 13.8M - 18.0.0-3908-g05a43c8f 2d174a053211 92ae756e07bb 2023-10-11T16:31:50.243 INFO:teuthology.orchestra.run.smithi111.stdout:mds.h smithi130 running (115s) 33s ago 115s 18.8M - 18.0.0-3908-g05a43c8f 2d174a053211 d5f9d2bcafc9 2023-10-11T16:31:50.243 INFO:teuthology.orchestra.run.smithi111.stdout:mds.i smithi150 running (113s) 33s ago 113s 22.0M - 18.0.0-3908-g05a43c8f 2d174a053211 02434bbfab3d 2023-10-11T16:31:50.243 INFO:teuthology.orchestra.run.smithi111.stdout:mds.j smithi111 running (2m) 21s ago 2m 13.8M - 18.0.0-3908-g05a43c8f 2d174a053211 51ccc626f0d7 2023-10-11T16:31:50.244 INFO:teuthology.orchestra.run.smithi111.stdout:mds.k smithi130 running (2m) 33s ago 2m 16.5M - 18.0.0-3908-g05a43c8f 2d174a053211 906aeca6e4fe 2023-10-11T16:31:50.244 INFO:teuthology.orchestra.run.smithi111.stdout:mds.l smithi150 running (2m) 33s ago 2m 17.6M - 18.0.0-3908-g05a43c8f 2d174a053211 fa5a498d4653 2023-10-11T16:31:50.244 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.x smithi111 *:9283,8765 running (8m) 21s ago 8m 521M - 18.0.0-3908-g05a43c8f 2d174a053211 9e6dab0c08cc 2023-10-11T16:31:50.244 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.y smithi130 *:8443,8765 running (7m) 33s ago 7m 445M - 18.0.0-3908-g05a43c8f 2d174a053211 ba9e7fe172a6 2023-10-11T16:31:50.244 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.z smithi150 *:8443,8765 running (7m) 33s ago 7m 446M - 18.0.0-3908-g05a43c8f 2d174a053211 8e80464a50b8 2023-10-11T16:31:50.245 INFO:teuthology.orchestra.run.smithi111.stdout:mon.a smithi111 running (9m) 21s ago 9m 95.7M 2048M 18.0.0-3908-g05a43c8f 2d174a053211 e020767c462c 2023-10-11T16:31:50.245 INFO:teuthology.orchestra.run.smithi111.stdout:mon.b smithi130 running (7m) 33s ago 7m 66.5M 2048M 18.0.0-3908-g05a43c8f 2d174a053211 4fdf13d1827a 2023-10-11T16:31:50.245 INFO:teuthology.orchestra.run.smithi111.stdout:mon.c smithi150 running (7m) 33s ago 7m 65.6M 2048M 18.0.0-3908-g05a43c8f 2d174a053211 f9148ff0570e 2023-10-11T16:31:50.245 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (6m) 21s ago 6m 71.2M 4317M 18.0.0-3908-g05a43c8f 2d174a053211 4d50a59f725a 2023-10-11T16:31:50.245 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi130 running (6m) 33s ago 6m 76.9M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 66ef91913767 2023-10-11T16:31:50.245 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi150 running (6m) 33s ago 6m 74.9M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 22fb73fb52ba 2023-10-11T16:31:50.246 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi111 running (5m) 21s ago 5m 65.5M 4317M 18.0.0-3908-g05a43c8f 2d174a053211 d36bb4471c32 2023-10-11T16:31:50.246 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi130 running (5m) 33s ago 5m 67.6M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 84cbbd7b3416 2023-10-11T16:31:50.246 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi150 running (4m) 33s ago 4m 72.3M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 1bc923420e91 2023-10-11T16:31:50.246 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6 smithi111 running (4m) 21s ago 4m 60.5M 4317M 18.0.0-3908-g05a43c8f 2d174a053211 076ef3c06d56 2023-10-11T16:31:50.246 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7 smithi130 running (4m) 33s ago 4m 65.0M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 8ce311e3661e 2023-10-11T16:31:50.247 INFO:teuthology.orchestra.run.smithi111.stdout:osd.8 smithi150 running (3m) 33s ago 3m 62.5M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 c56746036999 2023-10-11T16:31:50.247 INFO:teuthology.orchestra.run.smithi111.stdout:osd.9 smithi111 running (3m) 21s ago 3m 59.4M 4317M 18.0.0-3908-g05a43c8f 2d174a053211 c23863335e9d 2023-10-11T16:31:50.247 INFO:teuthology.orchestra.run.smithi111.stdout:osd.10 smithi130 running (2m) 33s ago 2m 62.5M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 7e6638b4db2b 2023-10-11T16:31:50.248 INFO:teuthology.orchestra.run.smithi111.stdout:osd.11 smithi150 running (2m) 33s ago 2m 56.5M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 4165f3d48cae 2023-10-11T16:31:50.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f47580a11e0 msgr2=0x7f47580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:50.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f47580a11e0 0x7f47580a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f475c009c60 tx=0x7f475c009260 comp rx=0 tx=0).stop 2023-10-11T16:31:50.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c1002f0 msgr2=0x7f476c198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:50.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c1002f0 0x7f476c198390 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f476800e9a0 tx=0x7f476800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 shutdown_connections 2023-10-11T16:31:50.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f47580a11e0 0x7f47580a3690 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f476c101e80 0x7f476c19f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f476c1002f0 0x7f476c198390 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 --2- 172.21.15.111:0/197152438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f476c1014f0 0x7f476c1988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:50.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 >> 172.21.15.111:0/197152438 conn(0x7f476c0fb900 msgr2=0x7f476c106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:50.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 shutdown_connections 2023-10-11T16:31:50.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:50.243+0000 7f4773815700 1 -- 172.21.15.111:0/197152438 wait complete. 2023-10-11T16:31:50.682 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-10-11T16:31:51.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:51 smithi130 ceph-mon[131859]: pgmap v326: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-11T16:31:51.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:51 smithi130 ceph-mon[131859]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:51.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:51 smithi111 ceph-mon[98527]: pgmap v326: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-11T16:31:51.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:51 smithi111 ceph-mon[98527]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:51.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:51 smithi150 ceph-mon[132217]: pgmap v326: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-11T16:31:51.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:51 smithi150 ceph-mon[132217]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:51.526 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:52.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 -- 172.21.15.111:0/277413745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8074b20 msgr2=0x7f3ef8072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:52.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 --2- 172.21.15.111:0/277413745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8074b20 0x7f3ef8072f80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80099c0 tx=0x7f3ee8009cd0 comp rx=0 tx=0).stop 2023-10-11T16:31:52.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 -- 172.21.15.111:0/277413745 shutdown_connections 2023-10-11T16:31:52.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 --2- 172.21.15.111:0/277413745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef810c8d0 0x7f3ef810ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:52.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 --2- 172.21.15.111:0/277413745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef8074b20 0x7f3ef8072f80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:52.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 --2- 172.21.15.111:0/277413745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ef8073550 0x7f3ef80739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:52.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 -- 172.21.15.111:0/277413745 >> 172.21.15.111:0/277413745 conn(0x7f3ef80fb900 msgr2=0x7f3ef80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 -- 172.21.15.111:0/277413745 shutdown_connections 2023-10-11T16:31:52.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.926+0000 7f3efe064700 1 -- 172.21.15.111:0/277413745 wait complete. 2023-10-11T16:31:52.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.927+0000 7f3efe064700 1 Processor -- start 2023-10-11T16:31:52.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.927+0000 7f3efe064700 1 -- start start 2023-10-11T16:31:52.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3efe064700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef8073550 0x7f3ef819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:52.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3efe064700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ef8074b20 0x7f3ef819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:52.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3efe064700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef810c8d0 0x7f3ef81a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:52.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3efe064700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef81a41e0 con 0x7f3ef810c8d0 2023-10-11T16:31:52.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3efe064700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef81a4320 con 0x7f3ef8074b20 2023-10-11T16:31:52.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3efe064700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef81a44b0 con 0x7f3ef8073550 2023-10-11T16:31:52.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef8073550 0x7f3ef819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:52.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef8073550 0x7f3ef819c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:54788/0 (socket says 172.21.15.111:54788) 2023-10-11T16:31:52.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef77fe700 1 -- 172.21.15.111:0/3566637802 learned_addr learned my addr 172.21.15.111:0/3566637802 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:52.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef6ffd700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ef8074b20 0x7f3ef819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:52.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef7fff700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef810c8d0 0x7f3ef81a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:52.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef77fe700 1 -- 172.21.15.111:0/3566637802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ef8074b20 msgr2=0x7f3ef819cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:52.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef77fe700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ef8074b20 0x7f3ef819cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:52.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef77fe700 1 -- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef810c8d0 msgr2=0x7f3ef81a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:52.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef77fe700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef810c8d0 0x7f3ef81a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:52.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef77fe700 1 -- 172.21.15.111:0/3566637802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee80095f0 con 0x7f3ef8073550 2023-10-11T16:31:52.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.928+0000 7f3ef7fff700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef810c8d0 0x7f3ef81a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:52.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.929+0000 7f3ef77fe700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef8073550 0x7f3ef819c650 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8000c00 tx=0x7f3ee800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:52.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.929+0000 7f3ef4ff9700 1 -- 172.21.15.111:0/3566637802 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee801c070 con 0x7f3ef8073550 2023-10-11T16:31:52.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.929+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef81122a0 con 0x7f3ef8073550 2023-10-11T16:31:52.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.929+0000 7f3ef4ff9700 1 -- 172.21.15.111:0/3566637802 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ee800fe30 con 0x7f3ef8073550 2023-10-11T16:31:52.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.929+0000 7f3ef4ff9700 1 -- 172.21.15.111:0/3566637802 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8005390 con 0x7f3ef8073550 2023-10-11T16:31:52.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.929+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef8112110 con 0x7f3ef8073550 2023-10-11T16:31:52.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.930+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef8102420 con 0x7f3ef8073550 2023-10-11T16:31:52.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.932+0000 7f3ef4ff9700 1 -- 172.21.15.111:0/3566637802 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f3ee8020070 con 0x7f3ef8073550 2023-10-11T16:31:52.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.934+0000 7f3ef4ff9700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ee00a1260 0x7f3ee00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:52.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.934+0000 7f3ef4ff9700 1 -- 172.21.15.111:0/3566637802 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f3ee80ce2e0 con 0x7f3ef8073550 2023-10-11T16:31:52.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.934+0000 7f3ef6ffd700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ee00a1260 0x7f3ee00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:52.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.934+0000 7f3ef6ffd700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ee00a1260 0x7f3ee00a3710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3eec007c60 tx=0x7f3eec0073d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:52.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:52.936+0000 7f3ef4ff9700 1 -- 172.21.15.111:0/3566637802 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3ee8097530 con 0x7f3ef8073550 2023-10-11T16:31:53.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.142+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ef80619f0 con 0x7f3ee00a1260 2023-10-11T16:31:53.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.148+0000 7f3ef4ff9700 1 -- 172.21.15.111:0/3566637802 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f3ef80619f0 con 0x7f3ee00a1260 2023-10-11T16:31:53.149 INFO:teuthology.orchestra.run.smithi111.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-10-11T16:31:53.149 INFO:teuthology.orchestra.run.smithi111.stdout:mds.all 12/12 36s ago 2m smithi111=a;smithi111=d;smithi111=g;smithi111=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi150=c;smithi150=f;smithi150=i;smithi150=l;count:12 2023-10-11T16:31:53.149 INFO:teuthology.orchestra.run.smithi111.stdout:mgr 3/3 36s ago 7m smithi111=x;smithi130=y;smithi150=z;count:3 2023-10-11T16:31:53.149 INFO:teuthology.orchestra.run.smithi111.stdout:mon 3/3 36s ago 7m smithi111:172.21.15.111=a;smithi130:172.21.15.130=b;smithi150:172.21.15.150=c;count:3 2023-10-11T16:31:53.149 INFO:teuthology.orchestra.run.smithi111.stdout:osd 12 36s ago - 2023-10-11T16:31:53.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.150+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ee00a1260 msgr2=0x7f3ee00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:53.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.150+0000 7f3efe064700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ee00a1260 0x7f3ee00a3710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3eec007c60 tx=0x7f3eec0073d0 comp rx=0 tx=0).stop 2023-10-11T16:31:53.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.150+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef8073550 msgr2=0x7f3ef819c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:53.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.150+0000 7f3efe064700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef8073550 0x7f3ef819c650 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8000c00 tx=0x7f3ee800f690 comp rx=0 tx=0).stop 2023-10-11T16:31:53.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.151+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 shutdown_connections 2023-10-11T16:31:53.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.151+0000 7f3efe064700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f3ee00a1260 0x7f3ee00a3710 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:53.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.151+0000 7f3efe064700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ef810c8d0 0x7f3ef81a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:53.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.151+0000 7f3efe064700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef8073550 0x7f3ef819c650 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:53.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.151+0000 7f3efe064700 1 --2- 172.21.15.111:0/3566637802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ef8074b20 0x7f3ef819cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:53.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.151+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 >> 172.21.15.111:0/3566637802 conn(0x7f3ef80fb900 msgr2=0x7f3ef80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:53.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.151+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 shutdown_connections 2023-10-11T16:31:53.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:53.151+0000 7f3efe064700 1 -- 172.21.15.111:0/3566637802 wait complete. 2023-10-11T16:31:53.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:53 smithi111 ceph-mon[98527]: pgmap v327: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:31:53.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:53 smithi150 ceph-mon[132217]: pgmap v327: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:31:53.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:53 smithi130 ceph-mon[131859]: pgmap v327: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:31:53.623 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-10-11T16:31:54.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:54 smithi111 ceph-mon[98527]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:54.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:54 smithi150 ceph-mon[132217]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:54.469 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:54.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:54 smithi130 ceph-mon[131859]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:55.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:55 smithi111 ceph-mon[98527]: pgmap v328: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:31:55.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:55 smithi150 ceph-mon[132217]: pgmap v328: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:31:55.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:55 smithi130 ceph-mon[131859]: pgmap v328: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:31:56.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.001+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/3718960895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c81014f0 msgr2=0x7fd8c8101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:56.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.001+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/3718960895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c81014f0 0x7fd8c8101940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009990 tx=0x7fd8b8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.001+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/3718960895 shutdown_connections 2023-10-11T16:31:56.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.001+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/3718960895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8c8101e80 0x7fd8c8108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.001+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/3718960895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81002f0 0x7fd8c8100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.001+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/3718960895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c81014f0 0x7fd8c8101940 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.001+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/3718960895 >> 172.21.15.111:0/3718960895 conn(0x7fd8c80fb900 msgr2=0x7fd8c80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:56.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.002+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/3718960895 shutdown_connections 2023-10-11T16:31:56.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.002+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/3718960895 wait complete. 2023-10-11T16:31:56.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cf3e2700 1 Processor -- start 2023-10-11T16:31:56.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cf3e2700 1 -- start start 2023-10-11T16:31:56.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cf3e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81002f0 0x7fd8c8074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:56.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cf3e2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c81014f0 0x7fd8c8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:56.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cf3e2700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8c8101e80 0x7fd8c8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:56.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cf3e2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8073ba0 con 0x7fd8c81002f0 2023-10-11T16:31:56.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cf3e2700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8073d10 con 0x7fd8c8101e80 2023-10-11T16:31:56.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cf3e2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c81a0080 con 0x7fd8c81014f0 2023-10-11T16:31:56.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cd17e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81002f0 0x7fd8c8074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:56.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cd17e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81002f0 0x7fd8c8074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44524/0 (socket says 172.21.15.111:44524) 2023-10-11T16:31:56.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.003+0000 7fd8cd17e700 1 -- 172.21.15.111:0/2212741745 learned_addr learned my addr 172.21.15.111:0/2212741745 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:56.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cd97f700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8c8101e80 0x7fd8c8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:56.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cc97d700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c81014f0 0x7fd8c8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:56.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cd17e700 1 -- 172.21.15.111:0/2212741745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c81014f0 msgr2=0x7fd8c8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:56.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cd17e700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c81014f0 0x7fd8c8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cd17e700 1 -- 172.21.15.111:0/2212741745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8c8101e80 msgr2=0x7fd8c8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:56.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cd17e700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8c8101e80 0x7fd8c8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cd17e700 1 -- 172.21.15.111:0/2212741745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c4009520 con 0x7fd8c81002f0 2023-10-11T16:31:56.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cd17e700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81002f0 0x7fd8c8074890 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400ea50 tx=0x7fd8c400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:56.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8be7fc700 1 -- 172.21.15.111:0/2212741745 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c400cbd0 con 0x7fd8c81002f0 2023-10-11T16:31:56.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8be7fc700 1 -- 172.21.15.111:0/2212741745 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8c400cd30 con 0x7fd8c81002f0 2023-10-11T16:31:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b80095f0 con 0x7fd8c81002f0 2023-10-11T16:31:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.004+0000 7fd8be7fc700 1 -- 172.21.15.111:0/2212741745 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4018680 con 0x7fd8c81002f0 2023-10-11T16:31:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.005+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c810bc50 con 0x7fd8c81002f0 2023-10-11T16:31:56.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.006+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c8107900 con 0x7fd8c81002f0 2023-10-11T16:31:56.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.008+0000 7fd8be7fc700 1 -- 172.21.15.111:0/2212741745 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fd8c4010430 con 0x7fd8c81002f0 2023-10-11T16:31:56.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.009+0000 7fd8be7fc700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd8b40a1260 0x7fd8b40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:56.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.009+0000 7fd8be7fc700 1 -- 172.21.15.111:0/2212741745 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fd8c4014070 con 0x7fd8c81002f0 2023-10-11T16:31:56.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.009+0000 7fd8cc97d700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd8b40a1260 0x7fd8b40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:56.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.010+0000 7fd8cc97d700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd8b40a1260 0x7fd8b40a3710 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800b2e0 tx=0x7fd8b8005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:56.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.013+0000 7fd8be7fc700 1 -- 172.21.15.111:0/2212741745 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd8c40964e0 con 0x7fd8c81002f0 2023-10-11T16:31:56.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.220+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8c8060f30 con 0x7fd8b40a1260 2023-10-11T16:31:56.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.222+0000 7fd8be7fc700 1 -- 172.21.15.111:0/2212741745 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fd8c8060f30 con 0x7fd8b40a1260 2023-10-11T16:31:56.223 INFO:teuthology.orchestra.run.smithi111.stdout:HOST ADDR LABELS STATUS 2023-10-11T16:31:56.223 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111 172.21.15.111 2023-10-11T16:31:56.223 INFO:teuthology.orchestra.run.smithi111.stdout:smithi130 172.21.15.130 2023-10-11T16:31:56.224 INFO:teuthology.orchestra.run.smithi111.stdout:smithi150 172.21.15.150 2023-10-11T16:31:56.224 INFO:teuthology.orchestra.run.smithi111.stdout:3 hosts in cluster 2023-10-11T16:31:56.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.224+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd8b40a1260 msgr2=0x7fd8b40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:56.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.224+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd8b40a1260 0x7fd8b40a3710 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800b2e0 tx=0x7fd8b8005fb0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.224+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81002f0 msgr2=0x7fd8c8074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:56.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.224+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81002f0 0x7fd8c8074890 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400ea50 tx=0x7fd8c400ed60 comp rx=0 tx=0).stop 2023-10-11T16:31:56.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.224+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 shutdown_connections 2023-10-11T16:31:56.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.224+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd8b40a1260 0x7fd8b40a3710 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.225+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8c8101e80 0x7fd8c8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.225+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c81002f0 0x7fd8c8074890 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.225+0000 7fd8cf3e2700 1 --2- 172.21.15.111:0/2212741745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c81014f0 0x7fd8c8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:56.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.225+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 >> 172.21.15.111:0/2212741745 conn(0x7fd8c80fb900 msgr2=0x7fd8c8104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:56.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.225+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 shutdown_connections 2023-10-11T16:31:56.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:56.225+0000 7fd8cf3e2700 1 -- 172.21.15.111:0/2212741745 wait complete. 2023-10-11T16:31:56.765 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-10-11T16:31:57.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:57 smithi111 ceph-mon[98527]: pgmap v329: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:57.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:57 smithi111 ceph-mon[98527]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:57.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:57 smithi150 ceph-mon[132217]: pgmap v329: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:57.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:57 smithi150 ceph-mon[132217]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:57.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:57 smithi130 ceph-mon[131859]: pgmap v329: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:31:57.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:57 smithi130 ceph-mon[131859]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:31:57.607 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:31:59.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 -- 172.21.15.111:0/3916998573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1801002f0 msgr2=0x7ff180100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:59.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 --2- 172.21.15.111:0/3916998573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1801002f0 0x7ff180100700 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0098f0 tx=0x7ff17c009c00 comp rx=0 tx=0).stop 2023-10-11T16:31:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 -- 172.21.15.111:0/3916998573 shutdown_connections 2023-10-11T16:31:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 --2- 172.21.15.111:0/3916998573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff180101e80 0x7ff180108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 --2- 172.21.15.111:0/3916998573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1801002f0 0x7ff180100700 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 --2- 172.21.15.111:0/3916998573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1801014f0 0x7ff180101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 -- 172.21.15.111:0/3916998573 >> 172.21.15.111:0/3916998573 conn(0x7ff1800fb900 msgr2=0x7ff1800fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:59.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 -- 172.21.15.111:0/3916998573 shutdown_connections 2023-10-11T16:31:59.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.136+0000 7ff187af1700 1 -- 172.21.15.111:0/3916998573 wait complete. 2023-10-11T16:31:59.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.137+0000 7ff187af1700 1 Processor -- start 2023-10-11T16:31:59.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.137+0000 7ff187af1700 1 -- start start 2023-10-11T16:31:59.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff187af1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1801002f0 0x7ff180196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:59.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff187af1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1801014f0 0x7ff1801965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:59.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff187af1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180101e80 0x7ff18019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:59.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff187af1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18019dc20 con 0x7ff1801014f0 2023-10-11T16:31:59.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff187af1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18019dd60 con 0x7ff180101e80 2023-10-11T16:31:59.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff187af1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18019def0 con 0x7ff1801002f0 2023-10-11T16:31:59.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff18508c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1801014f0 0x7ff1801965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:59.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff18588d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1801002f0 0x7ff180196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:59.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff18588d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1801002f0 0x7ff180196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:50222/0 (socket says 172.21.15.111:50222) 2023-10-11T16:31:59.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff18588d700 1 -- 172.21.15.111:0/1121704610 learned_addr learned my addr 172.21.15.111:0/1121704610 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:31:59.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff18608e700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180101e80 0x7ff18019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:59.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff18588d700 1 -- 172.21.15.111:0/1121704610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180101e80 msgr2=0x7ff18019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:59.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.138+0000 7ff18588d700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180101e80 0x7ff18019d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff18588d700 1 -- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1801014f0 msgr2=0x7ff1801965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:59.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff18588d700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1801014f0 0x7ff1801965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff18588d700 1 -- 172.21.15.111:0/1121704610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c009520 con 0x7ff1801002f0 2023-10-11T16:31:59.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff18508c700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1801014f0 0x7ff1801965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:31:59.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff18588d700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1801002f0 0x7ff180196090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff17c011040 tx=0x7ff17c00b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:59.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff176ffd700 1 -- 172.21.15.111:0/1121704610 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c01c070 con 0x7ff1801002f0 2023-10-11T16:31:59.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18010bc40 con 0x7ff1801002f0 2023-10-11T16:31:59.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff176ffd700 1 -- 172.21.15.111:0/1121704610 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff17c00f7d0 con 0x7ff1801002f0 2023-10-11T16:31:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff176ffd700 1 -- 172.21.15.111:0/1121704610 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c003b90 con 0x7ff1801002f0 2023-10-11T16:31:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.139+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18010bab0 con 0x7ff1801002f0 2023-10-11T16:31:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.140+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff180066ea0 con 0x7ff1801002f0 2023-10-11T16:31:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.142+0000 7ff176ffd700 1 -- 172.21.15.111:0/1121704610 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff17c003d10 con 0x7ff1801002f0 2023-10-11T16:31:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.143+0000 7ff176ffd700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff16c0a1260 0x7ff16c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:31:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.144+0000 7ff176ffd700 1 -- 172.21.15.111:0/1121704610 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7ff17c0cdba0 con 0x7ff1801002f0 2023-10-11T16:31:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.144+0000 7ff18508c700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff16c0a1260 0x7ff16c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:31:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.144+0000 7ff18508c700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff16c0a1260 0x7ff16c0a3710 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff170011490 tx=0x7ff170009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:31:59.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.146+0000 7ff176ffd700 1 -- 172.21.15.111:0/1121704610 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff17c096df0 con 0x7ff1801002f0 2023-10-11T16:31:59.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.349+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff180060f30 con 0x7ff16c0a1260 2023-10-11T16:31:59.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.351+0000 7ff176ffd700 1 -- 172.21.15.111:0/1121704610 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7ff180060f30 con 0x7ff16c0a1260 2023-10-11T16:31:59.351 INFO:teuthology.orchestra.run.smithi111.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-10-11T16:31:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VZ400BGN 372G No 3m ago LVM detected, locked 2023-10-11T16:31:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:smithi130 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN 372G No 2m ago LVM detected, locked 2023-10-11T16:31:59.352 INFO:teuthology.orchestra.run.smithi111.stdout:smithi150 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GX400BGN 372G No 2m ago LVM detected, locked 2023-10-11T16:31:59.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff16c0a1260 msgr2=0x7ff16c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:59.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff16c0a1260 0x7ff16c0a3710 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff170011490 tx=0x7ff170009260 comp rx=0 tx=0).stop 2023-10-11T16:31:59.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1801002f0 msgr2=0x7ff180196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:31:59.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1801002f0 0x7ff180196090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff17c011040 tx=0x7ff17c00b790 comp rx=0 tx=0).stop 2023-10-11T16:31:59.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 shutdown_connections 2023-10-11T16:31:59.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff16c0a1260 0x7ff16c0a3710 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180101e80 0x7ff18019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1801002f0 0x7ff180196090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 --2- 172.21.15.111:0/1121704610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1801014f0 0x7ff1801965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:31:59.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 >> 172.21.15.111:0/1121704610 conn(0x7ff1800fb900 msgr2=0x7ff1800fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:31:59.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 shutdown_connections 2023-10-11T16:31:59.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:31:59.353+0000 7ff187af1700 1 -- 172.21.15.111:0/1121704610 wait complete. 2023-10-11T16:31:59.375 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:31:59 smithi111 ceph-mon[98527]: pgmap v330: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:31:59 smithi150 ceph-mon[132217]: pgmap v330: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:59.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:31:59 smithi130 ceph-mon[131859]: pgmap v330: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:31:59.804 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-10-11T16:31:59.814 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:31:59.814 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- bash -c 'ceph fs dump' 2023-10-11T16:32:00.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:00 smithi111 ceph-mon[98527]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:00.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:00 smithi150 ceph-mon[132217]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:00.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:00 smithi130 ceph-mon[131859]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:00.664 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:32:01.182 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:01 smithi111 ceph-mon[98527]: pgmap v331: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:01.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:01 smithi150 ceph-mon[132217]: pgmap v331: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:01.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:01 smithi130 ceph-mon[131859]: pgmap v331: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 -- 172.21.15.111:0/3282259819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 msgr2=0x7f3958108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 --2- 172.21.15.111:0/3282259819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 0x7f3958108740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3950009990 tx=0x7f3950009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 -- 172.21.15.111:0/3282259819 shutdown_connections 2023-10-11T16:32:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 --2- 172.21.15.111:0/3282259819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 0x7f3958108740 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 --2- 172.21.15.111:0/3282259819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39581002f0 0x7f3958100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 --2- 172.21.15.111:0/3282259819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39581014f0 0x7f3958101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 -- 172.21.15.111:0/3282259819 >> 172.21.15.111:0/3282259819 conn(0x7f39580fb900 msgr2=0x7f39580fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 -- 172.21.15.111:0/3282259819 shutdown_connections 2023-10-11T16:32:02.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.119+0000 7f396025c700 1 -- 172.21.15.111:0/3282259819 wait complete. 2023-10-11T16:32:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.120+0000 7f396025c700 1 Processor -- start 2023-10-11T16:32:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.120+0000 7f396025c700 1 -- start start 2023-10-11T16:32:02.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f396025c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39581002f0 0x7f3958074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f396025c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39581014f0 0x7f3958072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f396025c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 0x7f3958073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f396025c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3958073ba0 con 0x7f3958101e80 2023-10-11T16:32:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f396025c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3958073d10 con 0x7f39581002f0 2023-10-11T16:32:02.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f396025c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39581a0060 con 0x7f39581014f0 2023-10-11T16:32:02.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f395dff8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39581002f0 0x7f3958074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:02.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f395dff8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39581002f0 0x7f3958074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46448/0 (socket says 172.21.15.111:46448) 2023-10-11T16:32:02.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f395e7f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 0x7f3958073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:02.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f395dff8700 1 -- 172.21.15.111:0/1662300928 learned_addr learned my addr 172.21.15.111:0/1662300928 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:02.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.121+0000 7f395d7f7700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39581014f0 0x7f3958072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:02.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f395e7f9700 1 -- 172.21.15.111:0/1662300928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39581014f0 msgr2=0x7f3958072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:02.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f395e7f9700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39581014f0 0x7f3958072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f395e7f9700 1 -- 172.21.15.111:0/1662300928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39581002f0 msgr2=0x7f3958074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:02.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f395e7f9700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39581002f0 0x7f3958074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f395e7f9700 1 -- 172.21.15.111:0/1662300928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39500095f0 con 0x7f3958101e80 2023-10-11T16:32:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f395d7f7700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39581014f0 0x7f3958072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f395dff8700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39581002f0 0x7f3958074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f395e7f9700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 0x7f3958073420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f395000b680 tx=0x7f3950004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f394effd700 1 -- 172.21.15.111:0/1662300928 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395001d070 con 0x7f3958101e80 2023-10-11T16:32:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f394effd700 1 -- 172.21.15.111:0/1662300928 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3950021410 con 0x7f3958101e80 2023-10-11T16:32:02.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395810bcb0 con 0x7f3958101e80 2023-10-11T16:32:02.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f394effd700 1 -- 172.21.15.111:0/1662300928 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950017650 con 0x7f3958101e80 2023-10-11T16:32:02.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.122+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395810bb20 con 0x7f3958101e80 2023-10-11T16:32:02.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.124+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39581a0650 con 0x7f3958101e80 2023-10-11T16:32:02.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.125+0000 7f394effd700 1 -- 172.21.15.111:0/1662300928 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f3950021580 con 0x7f3958101e80 2023-10-11T16:32:02.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.126+0000 7f394effd700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f39440a1260 0x7f39440a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.127+0000 7f394effd700 1 -- 172.21.15.111:0/1662300928 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f39500ce170 con 0x7f3958101e80 2023-10-11T16:32:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.127+0000 7f395dff8700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f39440a1260 0x7f39440a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.127+0000 7f395dff8700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f39440a1260 0x7f39440a3710 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3954009fd0 tx=0x7f3954009040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:02.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.131+0000 7f394effd700 1 -- 172.21.15.111:0/1662300928 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f39500973c0 con 0x7f3958101e80 2023-10-11T16:32:02.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.369+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f39581071b0 con 0x7f3958101e80 2023-10-11T16:32:02.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.370+0000 7f394effd700 1 -- 172.21.15.111:0/1662300928 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+2928 (secure 0 0 0) 0x7f3950021880 con 0x7f3958101e80 2023-10-11T16:32:02.371 INFO:teuthology.orchestra.run.smithi111.stdout:e26 2023-10-11T16:32:02.371 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-10-11T16:32:02.372 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-11T16:32:02.372 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2023-10-11T16:32:02.372 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:02.372 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2023-10-11T16:32:02.373 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2023-10-11T16:32:02.373 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 26 2023-10-11T16:32:02.373 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-10-11T16:32:02.373 INFO:teuthology.orchestra.run.smithi111.stdout:created 2023-10-11T16:30:11.633745+0000 2023-10-11T16:32:02.373 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2023-10-11T16:30:51.227688+0000 2023-10-11T16:32:02.374 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2023-10-11T16:32:02.374 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2023-10-11T16:32:02.374 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 300 2023-10-11T16:32:02.374 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2023-10-11T16:32:02.375 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2023-10-11T16:32:02.375 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2023-10-11T16:32:02.375 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2023-10-11T16:32:02.376 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2023-10-11T16:32:02.376 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2023-10-11T16:32:02.376 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-11T16:32:02.376 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 3 2023-10-11T16:32:02.376 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1,2 2023-10-11T16:32:02.377 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=24569,1=24550,2=24532} 2023-10-11T16:32:02.377 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2023-10-11T16:32:02.377 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2023-10-11T16:32:02.377 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2023-10-11T16:32:02.378 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3,4] 2023-10-11T16:32:02.378 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2023-10-11T16:32:02.378 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2023-10-11T16:32:02.378 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2023-10-11T16:32:02.378 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2023-10-11T16:32:02.379 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 0 2023-10-11T16:32:02.379 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b{0:24569} state up:active seq 3 export targets 1,2 addr [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.379 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.h{0:24521} state up:standby-replay seq 1 addr [v2:172.21.15.130:6836/3927432367,v1:172.21.15.130:6837/3927432367] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.379 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.i{1:24550} state up:active seq 11 addr [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.380 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.f{1:24520} state up:standby-replay seq 1 addr [v2:172.21.15.150:6834/2357736565,v1:172.21.15.150:6835/2357736565] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.380 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.c{2:24532} state up:active seq 13 addr [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.380 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.e{2:24488} state up:standby-replay seq 1 addr [v2:172.21.15.130:6834/89303870,v1:172.21.15.130:6835/89303870] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.380 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:02.380 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:02.381 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2023-10-11T16:32:02.381 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:02.381 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.j{-1:14424} state up:standby seq 1 addr [v2:172.21.15.111:6834/1910869239,v1:172.21.15.111:6835/1910869239] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.381 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.d{-1:14436} state up:standby seq 1 addr [v2:172.21.15.111:6836/3168828680,v1:172.21.15.111:6837/3168828680] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.381 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.g{-1:14484} state up:standby seq 1 addr [v2:172.21.15.111:6838/2081616561,v1:172.21.15.111:6839/2081616561] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.382 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a{-1:14496} state up:standby seq 1 addr [v2:172.21.15.111:6840/3150990523,v1:172.21.15.111:6841/3150990523] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.382 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.k{-1:24452} state up:standby seq 1 addr [v2:172.21.15.130:6832/447441834,v1:172.21.15.130:6833/447441834] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.382 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.l{-1:24460} state up:standby seq 1 addr [v2:172.21.15.150:6832/2297521930,v1:172.21.15.150:6833/2297521930] compat {c=[1],r=[1],i=[7ff]}] 2023-10-11T16:32:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f39440a1260 msgr2=0x7f39440a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f39440a1260 0x7f39440a3710 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3954009fd0 tx=0x7f3954009040 comp rx=0 tx=0).stop 2023-10-11T16:32:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 msgr2=0x7f3958073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:02.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 0x7f3958073420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f395000b680 tx=0x7f3950004730 comp rx=0 tx=0).stop 2023-10-11T16:32:02.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 shutdown_connections 2023-10-11T16:32:02.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f39440a1260 0x7f39440a3710 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3958101e80 0x7f3958073420 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f39581002f0 0x7f3958074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 --2- 172.21.15.111:0/1662300928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39581014f0 0x7f3958072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 >> 172.21.15.111:0/1662300928 conn(0x7f39580fb900 msgr2=0x7f39580fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:02.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 shutdown_connections 2023-10-11T16:32:02.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:02.372+0000 7f396025c700 1 -- 172.21.15.111:0/1662300928 wait complete. 2023-10-11T16:32:02.386 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-10-11T16:32:02.857 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- bash -c 'ceph osd dump' 2023-10-11T16:32:03.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:03 smithi111 ceph-mon[98527]: pgmap v332: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:03.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:03 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1662300928' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-10-11T16:32:03.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:03 smithi150 ceph-mon[132217]: pgmap v332: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:03.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:03 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1662300928' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-10-11T16:32:03.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:03 smithi130 ceph-mon[131859]: pgmap v332: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:03.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:03 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1662300928' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-10-11T16:32:03.712 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:32:05.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 -- 172.21.15.111:0/3862348319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864100110 msgr2=0x7f1864100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:05.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 --2- 172.21.15.111:0/3862348319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864100110 0x7f1864100520 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1860009990 tx=0x7f1860009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 -- 172.21.15.111:0/3862348319 shutdown_connections 2023-10-11T16:32:05.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 --2- 172.21.15.111:0/3862348319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864101ca0 0x7f1864108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 --2- 172.21.15.111:0/3862348319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864100110 0x7f1864100520 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 --2- 172.21.15.111:0/3862348319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1864101310 0x7f1864101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 -- 172.21.15.111:0/3862348319 >> 172.21.15.111:0/3862348319 conn(0x7f18640fb720 msgr2=0x7f18640fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:05.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 -- 172.21.15.111:0/3862348319 shutdown_connections 2023-10-11T16:32:05.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.333+0000 7f186aba7700 1 -- 172.21.15.111:0/3862348319 wait complete. 2023-10-11T16:32:05.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.334+0000 7f186aba7700 1 Processor -- start 2023-10-11T16:32:05.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186aba7700 1 -- start start 2023-10-11T16:32:05.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186aba7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1864100110 0x7f1864195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:05.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186aba7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864101310 0x7f1864196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:05.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186aba7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864101ca0 0x7f186419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:05.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186aba7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186419da70 con 0x7f1864101ca0 2023-10-11T16:32:05.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186aba7700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186419dbb0 con 0x7f1864100110 2023-10-11T16:32:05.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186aba7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186419dd40 con 0x7f1864101310 2023-10-11T16:32:05.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186a3a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864101ca0 0x7f186419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:05.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186a3a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864101ca0 0x7f186419d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46812/0 (socket says 172.21.15.111:46812) 2023-10-11T16:32:05.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f186a3a6700 1 -- 172.21.15.111:0/1446029901 learned_addr learned my addr 172.21.15.111:0/1446029901 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:05.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f1869ba5700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1864100110 0x7f1864195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:05.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.335+0000 7f18693a4700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864101310 0x7f1864196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:05.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f186a3a6700 1 -- 172.21.15.111:0/1446029901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864101310 msgr2=0x7f1864196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:05.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f186a3a6700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864101310 0x7f1864196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f186a3a6700 1 -- 172.21.15.111:0/1446029901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1864100110 msgr2=0x7f1864195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:05.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f186a3a6700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1864100110 0x7f1864195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f186a3a6700 1 -- 172.21.15.111:0/1446029901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18600095f0 con 0x7f1864101ca0 2023-10-11T16:32:05.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f1869ba5700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1864100110 0x7f1864195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:05.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f186a3a6700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864101ca0 0x7f186419d4a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f185c00ef60 tx=0x7f185c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:05.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f185affd700 1 -- 172.21.15.111:0/1446029901 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c018070 con 0x7f1864101ca0 2023-10-11T16:32:05.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f185affd700 1 -- 172.21.15.111:0/1446029901 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f185c00f040 con 0x7f1864101ca0 2023-10-11T16:32:05.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186410bae0 con 0x7f1864101ca0 2023-10-11T16:32:05.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f185affd700 1 -- 172.21.15.111:0/1446029901 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c00ccc0 con 0x7f1864101ca0 2023-10-11T16:32:05.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.336+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186410b950 con 0x7f1864101ca0 2023-10-11T16:32:05.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.337+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1864107730 con 0x7f1864101ca0 2023-10-11T16:32:05.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.340+0000 7f185affd700 1 -- 172.21.15.111:0/1446029901 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f185c003a00 con 0x7f1864101ca0 2023-10-11T16:32:05.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.341+0000 7f185affd700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f18500a11e0 0x7f18500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:05.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.341+0000 7f185affd700 1 -- 172.21.15.111:0/1446029901 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f185c0cd300 con 0x7f1864101ca0 2023-10-11T16:32:05.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.341+0000 7f1869ba5700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f18500a11e0 0x7f18500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:05.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.342+0000 7f1869ba5700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f18500a11e0 0x7f18500a3690 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f186000b2e0 tx=0x7f1860005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:05.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.344+0000 7f185affd700 1 -- 172.21.15.111:0/1446029901 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f185c096550 con 0x7f1864101ca0 2023-10-11T16:32:05.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:05 smithi111 ceph-mon[98527]: pgmap v333: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:32:05.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:05 smithi150 ceph-mon[132217]: pgmap v333: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:32:05.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.543+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f186402d030 con 0x7f1864101ca0 2023-10-11T16:32:05.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.544+0000 7f185affd700 1 -- 172.21.15.111:0/1446029901 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) v1 ==== 56+0+5790 (secure 0 0 0) 0x7f185c099b70 con 0x7f1864101ca0 2023-10-11T16:32:05.545 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 91 2023-10-11T16:32:05.545 INFO:teuthology.orchestra.run.smithi111.stdout:fsid 495f265e-6852-11ee-8db6-212e2dc638e7 2023-10-11T16:32:05.545 INFO:teuthology.orchestra.run.smithi111.stdout:created 2023-10-11T16:22:44.513312+0000 2023-10-11T16:32:05.545 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2023-10-11T16:31:42.150221+0000 2023-10-11T16:32:05.546 INFO:teuthology.orchestra.run.smithi111.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-10-11T16:32:05.546 INFO:teuthology.orchestra.run.smithi111.stdout:crush_version 27 2023-10-11T16:32:05.546 INFO:teuthology.orchestra.run.smithi111.stdout:full_ratio 0.95 2023-10-11T16:32:05.546 INFO:teuthology.orchestra.run.smithi111.stdout:backfillfull_ratio 0.9 2023-10-11T16:32:05.547 INFO:teuthology.orchestra.run.smithi111.stdout:nearfull_ratio 0.85 2023-10-11T16:32:05.547 INFO:teuthology.orchestra.run.smithi111.stdout:require_min_compat_client luminous 2023-10-11T16:32:05.547 INFO:teuthology.orchestra.run.smithi111.stdout:min_compat_client luminous 2023-10-11T16:32:05.547 INFO:teuthology.orchestra.run.smithi111.stdout:require_osd_release reef 2023-10-11T16:32:05.547 INFO:teuthology.orchestra.run.smithi111.stdout:stretch_mode_enabled false 2023-10-11T16:32:05.548 INFO:teuthology.orchestra.run.smithi111.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-10-11T16:32:05.548 INFO:teuthology.orchestra.run.smithi111.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-10-11T16:32:05.548 INFO:teuthology.orchestra.run.smithi111.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-10-11T16:32:05.548 INFO:teuthology.orchestra.run.smithi111.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 87 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-10-11T16:32:05.548 INFO:teuthology.orchestra.run.smithi111.stdout:max_osd 12 2023-10-11T16:32:05.549 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6802/937472257,v1:172.21.15.111:6803/937472257] [v2:172.21.15.111:6804/937472257,v1:172.21.15.111:6805/937472257] exists,up a4f58cc5-3bfa-4fdb-a127-1926aadd513d 2023-10-11T16:32:05.549 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6800/1526012165,v1:172.21.15.130:6801/1526012165] [v2:172.21.15.130:6802/1526012165,v1:172.21.15.130:6803/1526012165] exists,up 6c29c935-2362-4f82-a918-4fb7c7ce51f8 2023-10-11T16:32:05.549 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6800/1993692554,v1:172.21.15.150:6801/1993692554] [v2:172.21.15.150:6802/1993692554,v1:172.21.15.150:6803/1993692554] exists,up 614135a6-6a84-40f7-8b31-58459c019526 2023-10-11T16:32:05.549 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6810/4229058554,v1:172.21.15.111:6811/4229058554] [v2:172.21.15.111:6812/4229058554,v1:172.21.15.111:6813/4229058554] exists,up 3f2ce623-b8ba-448b-b9b3-f0970bc8be1d 2023-10-11T16:32:05.549 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 up in weight 1 up_from 33 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6808/1737228093,v1:172.21.15.130:6809/1737228093] [v2:172.21.15.130:6810/1737228093,v1:172.21.15.130:6811/1737228093] exists,up 0c35c66a-dcd9-48ed-9b46-d62ea7add46b 2023-10-11T16:32:05.550 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 up in weight 1 up_from 38 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6808/2672847721,v1:172.21.15.150:6809/2672847721] [v2:172.21.15.150:6810/2672847721,v1:172.21.15.150:6811/2672847721] exists,up 9cbf73bf-6335-4f23-946e-b1b50c55d94f 2023-10-11T16:32:05.550 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6 up in weight 1 up_from 46 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6818/2248469058,v1:172.21.15.111:6819/2248469058] [v2:172.21.15.111:6820/2248469058,v1:172.21.15.111:6821/2248469058] exists,up 40e6958d-a7e1-4cd8-abcb-dd770747d086 2023-10-11T16:32:05.550 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6816/2699897389,v1:172.21.15.130:6817/2699897389] [v2:172.21.15.130:6818/2699897389,v1:172.21.15.130:6819/2699897389] exists,up 3e9ae755-b404-4fbd-ad30-08dee80f0a95 2023-10-11T16:32:05.550 INFO:teuthology.orchestra.run.smithi111.stdout:osd.8 up in weight 1 up_from 58 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6816/410436517,v1:172.21.15.150:6817/410436517] [v2:172.21.15.150:6818/410436517,v1:172.21.15.150:6819/410436517] exists,up 1bfb9857-3e28-481a-8a1e-9f3b9b1ea176 2023-10-11T16:32:05.550 INFO:teuthology.orchestra.run.smithi111.stdout:osd.9 up in weight 1 up_from 66 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6826/3788746669,v1:172.21.15.111:6827/3788746669] [v2:172.21.15.111:6828/3788746669,v1:172.21.15.111:6829/3788746669] exists,up 48901f71-e122-4d76-b262-e0f1d8f18ede 2023-10-11T16:32:05.551 INFO:teuthology.orchestra.run.smithi111.stdout:osd.10 up in weight 1 up_from 71 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6824/2844086684,v1:172.21.15.130:6825/2844086684] [v2:172.21.15.130:6826/2844086684,v1:172.21.15.130:6827/2844086684] exists,up 13ec3960-3990-4c4e-a67f-cafbd49e83eb 2023-10-11T16:32:05.551 INFO:teuthology.orchestra.run.smithi111.stdout:osd.11 up in weight 1 up_from 78 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.150:6824/2209861208,v1:172.21.15.150:6825/2209861208] [v2:172.21.15.150:6826/2209861208,v1:172.21.15.150:6827/2209861208] exists,up 7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb 2023-10-11T16:32:05.551 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 2.0 [8,4] 2023-10-11T16:32:05.551 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 2.5 [8,11] 2023-10-11T16:32:05.551 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 2.c [2,1] 2023-10-11T16:32:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 2.14 [2,4] 2023-10-11T16:32:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 2.1f [7,11] 2023-10-11T16:32:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 3.6 [3,7] 2023-10-11T16:32:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 3.a [1,9] 2023-10-11T16:32:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 3.e [4,6,5,9] 2023-10-11T16:32:05.553 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 3.17 [4,8,5,6] 2023-10-11T16:32:05.553 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 3.19 [3,7] 2023-10-11T16:32:05.553 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 3.1d [4,8] 2023-10-11T16:32:05.553 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 3.2c [4,9] 2023-10-11T16:32:05.553 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 4.a [5,10] 2023-10-11T16:32:05.554 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 4.b [9,8] 2023-10-11T16:32:05.554 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 4.e [9,11] 2023-10-11T16:32:05.554 INFO:teuthology.orchestra.run.smithi111.stdout:pg_upmap_items 4.f [9,8] 2023-10-11T16:32:05.554 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:6801/4086392563 expires 2023-10-12T16:23:39.970669+0000 2023-10-11T16:32:05.554 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:6800/4086392563 expires 2023-10-12T16:23:39.970669+0000 2023-10-11T16:32:05.555 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:0/3251673175 expires 2023-10-12T16:23:39.970669+0000 2023-10-11T16:32:05.555 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:0/3319300422 expires 2023-10-12T16:23:39.970669+0000 2023-10-11T16:32:05.555 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:0/3405550474 expires 2023-10-12T16:23:13.996718+0000 2023-10-11T16:32:05.555 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:0/4129094163 expires 2023-10-12T16:23:39.970669+0000 2023-10-11T16:32:05.555 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:0/1272467376 expires 2023-10-12T16:23:13.996718+0000 2023-10-11T16:32:05.556 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:6800/699479694 expires 2023-10-12T16:23:13.996718+0000 2023-10-11T16:32:05.556 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:6801/699479694 expires 2023-10-12T16:23:13.996718+0000 2023-10-11T16:32:05.556 INFO:teuthology.orchestra.run.smithi111.stdout:blocklist 172.21.15.111:0/3715184603 expires 2023-10-12T16:23:13.996718+0000 2023-10-11T16:32:05.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f18500a11e0 msgr2=0x7f18500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:05.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f18500a11e0 0x7f18500a3690 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f186000b2e0 tx=0x7f1860005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864101ca0 msgr2=0x7f186419d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:05.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864101ca0 0x7f186419d4a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f185c00ef60 tx=0x7f185c00ef90 comp rx=0 tx=0).stop 2023-10-11T16:32:05.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 shutdown_connections 2023-10-11T16:32:05.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f18500a11e0 0x7f18500a3690 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1864101ca0 0x7f186419d4a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1864100110 0x7f1864195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 --2- 172.21.15.111:0/1446029901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1864101310 0x7f1864196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 >> 172.21.15.111:0/1446029901 conn(0x7f18640fb720 msgr2=0x7f1864104870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.546+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 shutdown_connections 2023-10-11T16:32:05.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:05.547+0000 7f186aba7700 1 -- 172.21.15.111:0/1446029901 wait complete. 2023-10-11T16:32:05.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:05 smithi130 ceph-mon[131859]: pgmap v333: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:32:06.006 INFO:teuthology.run_tasks:Running task fs.ready... 2023-10-11T16:32:06.021 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:32:06.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 -- 172.21.15.111:0/3098230330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac701014a0 msgr2=0x7fac701018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 --2- 172.21.15.111:0/3098230330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac701014a0 0x7fac701018f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fac58009990 tx=0x7fac58009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 -- 172.21.15.111:0/3098230330 shutdown_connections 2023-10-11T16:32:06.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 --2- 172.21.15.111:0/3098230330 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac70101e30 0x7fac701086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 --2- 172.21.15.111:0/3098230330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac701002b0 0x7fac701006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 --2- 172.21.15.111:0/3098230330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac701014a0 0x7fac701018f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 -- 172.21.15.111:0/3098230330 >> 172.21.15.111:0/3098230330 conn(0x7fac700fb890 msgr2=0x7fac700fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:06.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 -- 172.21.15.111:0/3098230330 shutdown_connections 2023-10-11T16:32:06.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.243+0000 7fac74bff700 1 -- 172.21.15.111:0/3098230330 wait complete. 2023-10-11T16:32:06.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac74bff700 1 Processor -- start 2023-10-11T16:32:06.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac74bff700 1 -- start start 2023-10-11T16:32:06.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac74bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac701002b0 0x7fac70196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:06.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac74bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac701014a0 0x7fac70196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:06.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac74bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac70101e30 0x7fac7019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:06.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac74bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7019dba0 con 0x7fac70101e30 2023-10-11T16:32:06.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac74bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7019dce0 con 0x7fac701002b0 2023-10-11T16:32:06.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac74bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7019de70 con 0x7fac701014a0 2023-10-11T16:32:06.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.245+0000 7fac6ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac70101e30 0x7fac7019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:06.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac70101e30 0x7fac7019d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46836/0 (socket says 172.21.15.111:46836) 2023-10-11T16:32:06.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6ed9d700 1 -- 172.21.15.111:0/1003429257 learned_addr learned my addr 172.21.15.111:0/1003429257 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:06.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac67fff700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac701014a0 0x7fac70196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:06.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6e59c700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac701002b0 0x7fac70196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:06.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6ed9d700 1 -- 172.21.15.111:0/1003429257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac701014a0 msgr2=0x7fac70196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6ed9d700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac701014a0 0x7fac70196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6ed9d700 1 -- 172.21.15.111:0/1003429257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac701002b0 msgr2=0x7fac70196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6ed9d700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac701002b0 0x7fac70196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6ed9d700 1 -- 172.21.15.111:0/1003429257 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580095f0 con 0x7fac70101e30 2023-10-11T16:32:06.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac6ed9d700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac70101e30 0x7fac7019d5d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fac68007ee0 tx=0x7fac6800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:06.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac677fe700 1 -- 172.21.15.111:0/1003429257 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6800dd20 con 0x7fac70101e30 2023-10-11T16:32:06.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac677fe700 1 -- 172.21.15.111:0/1003429257 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac6800f040 con 0x7fac70101e30 2023-10-11T16:32:06.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7010bc60 con 0x7fac70101e30 2023-10-11T16:32:06.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac677fe700 1 -- 172.21.15.111:0/1003429257 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680048a0 con 0x7fac70101e30 2023-10-11T16:32:06.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.246+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7010bad0 con 0x7fac70101e30 2023-10-11T16:32:06.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.248+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac701078c0 con 0x7fac70101e30 2023-10-11T16:32:06.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.250+0000 7fac677fe700 1 -- 172.21.15.111:0/1003429257 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fac68003a00 con 0x7fac70101e30 2023-10-11T16:32:06.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.251+0000 7fac677fe700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac5c0a1260 0x7fac5c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:06.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.251+0000 7fac677fe700 1 -- 172.21.15.111:0/1003429257 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fac680cc5e0 con 0x7fac70101e30 2023-10-11T16:32:06.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.251+0000 7fac6e59c700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac5c0a1260 0x7fac5c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:06.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.252+0000 7fac6e59c700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac5c0a1260 0x7fac5c0a3710 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fac70101300 tx=0x7fac6000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:06.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.254+0000 7fac677fe700 1 -- 172.21.15.111:0/1003429257 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fac6804e730 con 0x7fac70101e30 2023-10-11T16:32:06.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:06 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1446029901' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-10-11T16:32:06.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:06 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1446029901' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-10-11T16:32:06.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.457+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac7002cfe0 con 0x7fac70101e30 2023-10-11T16:32:06.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.458+0000 7fac677fe700 1 -- 172.21.15.111:0/1003429257 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24870 (secure 0 0 0) 0x7fac680957a0 con 0x7fac70101e30 2023-10-11T16:32:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:06.460 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":91,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:31:42.150221+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:32:06.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac5c0a1260 msgr2=0x7fac5c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac5c0a1260 0x7fac5c0a3710 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fac70101300 tx=0x7fac6000b410 comp rx=0 tx=0).stop 2023-10-11T16:32:06.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac70101e30 msgr2=0x7fac7019d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac70101e30 0x7fac7019d5d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fac68007ee0 tx=0x7fac6800d3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 shutdown_connections 2023-10-11T16:32:06.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac5c0a1260 0x7fac5c0a3710 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac70101e30 0x7fac7019d5d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac701002b0 0x7fac70196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 --2- 172.21.15.111:0/1003429257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac701014a0 0x7fac70196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 >> 172.21.15.111:0/1003429257 conn(0x7fac700fb890 msgr2=0x7fac70104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:06.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 shutdown_connections 2023-10-11T16:32:06.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.461+0000 7fac74bff700 1 -- 172.21.15.111:0/1003429257 wait complete. 2023-10-11T16:32:06.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:32:06.481 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:32:06.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:06 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1446029901' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-10-11T16:32:06.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.681+0000 7f60b83ff700 1 -- 172.21.15.111:0/3718096148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0074ad0 msgr2=0x7f60b0072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.681+0000 7f60b83ff700 1 --2- 172.21.15.111:0/3718096148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0074ad0 0x7f60b0072f20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009990 tx=0x7f60ac009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.682+0000 7f60b83ff700 1 -- 172.21.15.111:0/3718096148 shutdown_connections 2023-10-11T16:32:06.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.682+0000 7f60b83ff700 1 --2- 172.21.15.111:0/3718096148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b0108160 0x7f60b010a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.682+0000 7f60b83ff700 1 --2- 172.21.15.111:0/3718096148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0074ad0 0x7f60b0072f20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.682+0000 7f60b83ff700 1 --2- 172.21.15.111:0/3718096148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b00734f0 0x7f60b0073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.682+0000 7f60b83ff700 1 -- 172.21.15.111:0/3718096148 >> 172.21.15.111:0/3718096148 conn(0x7f60b00fb8b0 msgr2=0x7f60b00fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:06.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.682+0000 7f60b83ff700 1 -- 172.21.15.111:0/3718096148 shutdown_connections 2023-10-11T16:32:06.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.682+0000 7f60b83ff700 1 -- 172.21.15.111:0/3718096148 wait complete. 2023-10-11T16:32:06.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.683+0000 7f60b83ff700 1 Processor -- start 2023-10-11T16:32:06.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.683+0000 7f60b83ff700 1 -- start start 2023-10-11T16:32:06.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.683+0000 7f60b83ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b00734f0 0x7f60b0198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:06.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.683+0000 7f60b83ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b0074ad0 0x7f60b01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:06.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.683+0000 7f60b83ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0108160 0x7f60b019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:06.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.683+0000 7f60b83ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019fde0 con 0x7f60b0108160 2023-10-11T16:32:06.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.683+0000 7f60b83ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019ff20 con 0x7f60b00734f0 2023-10-11T16:32:06.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.683+0000 7f60b83ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b01a00b0 con 0x7f60b0074ad0 2023-10-11T16:32:06.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0108160 0x7f60b019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:06.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0108160 0x7f60b019f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46856/0 (socket says 172.21.15.111:46856) 2023-10-11T16:32:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 -- 172.21.15.111:0/1277400549 learned_addr learned my addr 172.21.15.111:0/1277400549 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b599a700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b0074ad0 0x7f60b01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b619b700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b00734f0 0x7f60b0198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 -- 172.21.15.111:0/1277400549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b0074ad0 msgr2=0x7f60b01987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b0074ad0 0x7f60b01987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 -- 172.21.15.111:0/1277400549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b00734f0 msgr2=0x7f60b0198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b00734f0 0x7f60b0198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 -- 172.21.15.111:0/1277400549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac0095f0 con 0x7f60b0108160 2023-10-11T16:32:06.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b699c700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0108160 0x7f60b019f810 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f60a800ef30 tx=0x7f60a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60a77fe700 1 -- 172.21.15.111:0/1277400549 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8018070 con 0x7f60b0108160 2023-10-11T16:32:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60a77fe700 1 -- 172.21.15.111:0/1277400549 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a800f040 con 0x7f60b0108160 2023-10-11T16:32:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.684+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b010de60 con 0x7f60b0108160 2023-10-11T16:32:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.685+0000 7f60a77fe700 1 -- 172.21.15.111:0/1277400549 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a800cd00 con 0x7f60b0108160 2023-10-11T16:32:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.685+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b010dcd0 con 0x7f60b0108160 2023-10-11T16:32:06.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.686+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b01068a0 con 0x7f60b0108160 2023-10-11T16:32:06.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.687+0000 7f60a77fe700 1 -- 172.21.15.111:0/1277400549 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f60a8003a00 con 0x7f60b0108160 2023-10-11T16:32:06.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.689+0000 7f60a77fe700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f609c0a11e0 0x7f609c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.689+0000 7f60a77fe700 1 -- 172.21.15.111:0/1277400549 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f60a80cc9f0 con 0x7f60b0108160 2023-10-11T16:32:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.689+0000 7f60b619b700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f609c0a11e0 0x7f609c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:06.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.690+0000 7f60b619b700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f609c0a11e0 0x7f609c0a3690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00b2e0 tx=0x7f60ac005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:06.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.693+0000 7f60a77fe700 1 -- 172.21.15.111:0/1277400549 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f60a8095c40 con 0x7f60b0108160 2023-10-11T16:32:06.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.891+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60b0067b00 con 0x7f60b0108160 2023-10-11T16:32:06.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.892+0000 7f60a77fe700 1 -- 172.21.15.111:0/1277400549 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f60a8099260 con 0x7f60b0108160 2023-10-11T16:32:06.893 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:32:06.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f609c0a11e0 msgr2=0x7f609c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f609c0a11e0 0x7f609c0a3690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00b2e0 tx=0x7f60ac005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0108160 msgr2=0x7f60b019f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:06.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0108160 0x7f60b019f810 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f60a800ef30 tx=0x7f60a800c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 shutdown_connections 2023-10-11T16:32:06.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f609c0a11e0 0x7f609c0a3690 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b0108160 0x7f60b019f810 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b00734f0 0x7f60b0198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 --2- 172.21.15.111:0/1277400549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b0074ad0 0x7f60b01987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:06.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 >> 172.21.15.111:0/1277400549 conn(0x7f60b00fb8b0 msgr2=0x7f60b00fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:06.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 shutdown_connections 2023-10-11T16:32:06.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:06.894+0000 7f60b83ff700 1 -- 172.21.15.111:0/1277400549 wait complete. 2023-10-11T16:32:06.914 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:32:07.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 -- 172.21.15.111:0/2930844687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1564101e40 msgr2=0x7f1564108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 --2- 172.21.15.111:0/2930844687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1564101e40 0x7f1564108700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f155c00b0a0 tx=0x7f155c00b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 -- 172.21.15.111:0/2930844687 shutdown_connections 2023-10-11T16:32:07.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 --2- 172.21.15.111:0/2930844687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1564101e40 0x7f1564108700 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 --2- 172.21.15.111:0/2930844687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15641002c0 0x7f15641006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 --2- 172.21.15.111:0/2930844687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15641014b0 0x7f1564101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 -- 172.21.15.111:0/2930844687 >> 172.21.15.111:0/2930844687 conn(0x7f15640fb8a0 msgr2=0x7f15640fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:07.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 -- 172.21.15.111:0/2930844687 shutdown_connections 2023-10-11T16:32:07.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.113+0000 7f1568bff700 1 -- 172.21.15.111:0/2930844687 wait complete. 2023-10-11T16:32:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.114+0000 7f1568bff700 1 Processor -- start 2023-10-11T16:32:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.114+0000 7f1568bff700 1 -- start start 2023-10-11T16:32:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f1568bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15641002c0 0x7f1564196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f1568bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15641014b0 0x7f1564196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f1568bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564101e40 0x7f156419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f1568bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156419dcc0 con 0x7f15641014b0 2023-10-11T16:32:07.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f1568bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156419de00 con 0x7f15641002c0 2023-10-11T16:32:07.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f1568bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156419df90 con 0x7f1564101e40 2023-10-11T16:32:07.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f155bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15641014b0 0x7f1564196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f155bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15641014b0 0x7f1564196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46880/0 (socket says 172.21.15.111:46880) 2023-10-11T16:32:07.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f156259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15641002c0 0x7f1564196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f155bfff700 1 -- 172.21.15.111:0/680074107 learned_addr learned my addr 172.21.15.111:0/680074107 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:07.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f1562d9d700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564101e40 0x7f156419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f156259c700 1 -- 172.21.15.111:0/680074107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564101e40 msgr2=0x7f156419d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f156259c700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564101e40 0x7f156419d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f156259c700 1 -- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15641014b0 msgr2=0x7f1564196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.115+0000 7f156259c700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15641014b0 0x7f1564196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.116+0000 7f156259c700 1 -- 172.21.15.111:0/680074107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c009d00 con 0x7f15641002c0 2023-10-11T16:32:07.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.116+0000 7f1562d9d700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564101e40 0x7f156419d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:07.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.116+0000 7f156259c700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15641002c0 0x7f1564196140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1554009ef0 tx=0x7f155400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:07.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.116+0000 7f155b7fe700 1 -- 172.21.15.111:0/680074107 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554013d00 con 0x7f15641002c0 2023-10-11T16:32:07.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.117+0000 7f155b7fe700 1 -- 172.21.15.111:0/680074107 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1554013e60 con 0x7f15641002c0 2023-10-11T16:32:07.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.117+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156410bc90 con 0x7f15641002c0 2023-10-11T16:32:07.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.117+0000 7f155b7fe700 1 -- 172.21.15.111:0/680074107 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155401b440 con 0x7f15641002c0 2023-10-11T16:32:07.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.117+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156410bb00 con 0x7f15641002c0 2023-10-11T16:32:07.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.118+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15641077c0 con 0x7f15641002c0 2023-10-11T16:32:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.120+0000 7f155b7fe700 1 -- 172.21.15.111:0/680074107 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f1554003680 con 0x7f15641002c0 2023-10-11T16:32:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.121+0000 7f155b7fe700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15500a11e0 0x7f15500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.121+0000 7f155b7fe700 1 -- 172.21.15.111:0/680074107 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f15540d1350 con 0x7f15641002c0 2023-10-11T16:32:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.122+0000 7f155bfff700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15500a11e0 0x7f15500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.122+0000 7f155bfff700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15500a11e0 0x7f15500a3690 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f154c009d20 tx=0x7f154c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:07.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.126+0000 7f155b7fe700 1 -- 172.21.15.111:0/680074107 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f155409a680 con 0x7f15641002c0 2023-10-11T16:32:07.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.321+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f156402cfe0 con 0x7f15641002c0 2023-10-11T16:32:07.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.322+0000 7f155b7fe700 1 -- 172.21.15.111:0/680074107 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f155409dca0 con 0x7f15641002c0 2023-10-11T16:32:07.323 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15500a11e0 msgr2=0x7f15500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15500a11e0 0x7f15500a3690 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f154c009d20 tx=0x7f154c009260 comp rx=0 tx=0).stop 2023-10-11T16:32:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15641002c0 msgr2=0x7f1564196140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15641002c0 0x7f1564196140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1554009ef0 tx=0x7f155400eca0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 shutdown_connections 2023-10-11T16:32:07.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f15500a11e0 0x7f15500a3690 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1564101e40 0x7f156419d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f15641002c0 0x7f1564196140 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 --2- 172.21.15.111:0/680074107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15641014b0 0x7f1564196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 >> 172.21.15.111:0/680074107 conn(0x7f15640fb8a0 msgr2=0x7f1564104900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 shutdown_connections 2023-10-11T16:32:07.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.324+0000 7f1568bff700 1 -- 172.21.15.111:0/680074107 wait complete. 2023-10-11T16:32:07.341 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:07 smithi111 ceph-mon[98527]: pgmap v334: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:07.341 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:07 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1003429257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:07.342 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:07 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1277400549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:07.343 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:32:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:07 smithi150 ceph-mon[132217]: pgmap v334: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:07 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1003429257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:07 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1277400549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:07.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 -- 172.21.15.111:0/2069965083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 msgr2=0x7f1d10101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 --2- 172.21.15.111:0/2069965083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 0x7f1d10101900 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1d00009990 tx=0x7f1d00009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 -- 172.21.15.111:0/2069965083 shutdown_connections 2023-10-11T16:32:07.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 --2- 172.21.15.111:0/2069965083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d10101e40 0x7f1d10108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 --2- 172.21.15.111:0/2069965083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d101002c0 0x7f1d101006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 --2- 172.21.15.111:0/2069965083 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 0x7f1d10101900 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 -- 172.21.15.111:0/2069965083 >> 172.21.15.111:0/2069965083 conn(0x7f1d100fb8a0 msgr2=0x7f1d100fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:07.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 -- 172.21.15.111:0/2069965083 shutdown_connections 2023-10-11T16:32:07.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.543+0000 7f1d179ff700 1 -- 172.21.15.111:0/2069965083 wait complete. 2023-10-11T16:32:07.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.544+0000 7f1d179ff700 1 Processor -- start 2023-10-11T16:32:07.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.544+0000 7f1d179ff700 1 -- start start 2023-10-11T16:32:07.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.544+0000 7f1d179ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d101002c0 0x7f1d100748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.544+0000 7f1d179ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 0x7f1d10072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.544+0000 7f1d179ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d10101e40 0x7f1d10073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.544+0000 7f1d179ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10073bc0 con 0x7f1d101014b0 2023-10-11T16:32:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.544+0000 7f1d179ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10073d30 con 0x7f1d101002c0 2023-10-11T16:32:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.544+0000 7f1d179ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d101a00b0 con 0x7f1d10101e40 2023-10-11T16:32:07.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 0x7f1d10072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 0x7f1d10072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46890/0 (socket says 172.21.15.111:46890) 2023-10-11T16:32:07.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 -- 172.21.15.111:0/1920299609 learned_addr learned my addr 172.21.15.111:0/1920299609 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:07.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d1579b700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d101002c0 0x7f1d100748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d15f9c700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d10101e40 0x7f1d10073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 -- 172.21.15.111:0/1920299609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d10101e40 msgr2=0x7f1d10073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d10101e40 0x7f1d10073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 -- 172.21.15.111:0/1920299609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d101002c0 msgr2=0x7f1d100748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d101002c0 0x7f1d100748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 -- 172.21.15.111:0/1920299609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d000095f0 con 0x7f1d101014b0 2023-10-11T16:32:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.545+0000 7f1d14f9a700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 0x7f1d10072ee0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000b620 tx=0x7f1d00005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.546+0000 7f1d067fc700 1 -- 172.21.15.111:0/1920299609 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0001d070 con 0x7f1d101014b0 2023-10-11T16:32:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.546+0000 7f1d067fc700 1 -- 172.21.15.111:0/1920299609 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d00022470 con 0x7f1d101014b0 2023-10-11T16:32:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.546+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1010bc70 con 0x7f1d101014b0 2023-10-11T16:32:07.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.546+0000 7f1d067fc700 1 -- 172.21.15.111:0/1920299609 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00017650 con 0x7f1d101014b0 2023-10-11T16:32:07.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.546+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1010bae0 con 0x7f1d101014b0 2023-10-11T16:32:07.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.547+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d101071f0 con 0x7f1d101014b0 2023-10-11T16:32:07.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.549+0000 7f1d067fc700 1 -- 172.21.15.111:0/1920299609 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f1d000225e0 con 0x7f1d101014b0 2023-10-11T16:32:07.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.551+0000 7f1d067fc700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1cfc0a1260 0x7f1cfc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.551+0000 7f1d067fc700 1 -- 172.21.15.111:0/1920299609 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f1d000cea60 con 0x7f1d101014b0 2023-10-11T16:32:07.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.551+0000 7f1d1579b700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1cfc0a1260 0x7f1cfc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.552+0000 7f1d1579b700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1cfc0a1260 0x7f1cfc0a3710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1d10101310 tx=0x7f1d0c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:07.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.553+0000 7f1d067fc700 1 -- 172.21.15.111:0/1920299609 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1d00097cb0 con 0x7f1d101014b0 2023-10-11T16:32:07.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:07 smithi130 ceph-mon[131859]: pgmap v334: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:07.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:07 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1003429257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:07.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:07 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1277400549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:07.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.753+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1d10061ad0 con 0x7f1d101014b0 2023-10-11T16:32:07.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.754+0000 7f1d067fc700 1 -- 172.21.15.111:0/1920299609 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1d00027020 con 0x7f1d101014b0 2023-10-11T16:32:07.755 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:32:07.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.756+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1cfc0a1260 msgr2=0x7f1cfc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.756+0000 7f1d179ff700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1cfc0a1260 0x7f1cfc0a3710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1d10101310 tx=0x7f1d0c009260 comp rx=0 tx=0).stop 2023-10-11T16:32:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.756+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 msgr2=0x7f1d10072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.756+0000 7f1d179ff700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 0x7f1d10072ee0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000b620 tx=0x7f1d00005740 comp rx=0 tx=0).stop 2023-10-11T16:32:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.756+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 shutdown_connections 2023-10-11T16:32:07.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.757+0000 7f1d179ff700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1cfc0a1260 0x7f1cfc0a3710 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.757+0000 7f1d179ff700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d10101e40 0x7f1d10073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.757+0000 7f1d179ff700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d101002c0 0x7f1d100748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.757+0000 7f1d179ff700 1 --2- 172.21.15.111:0/1920299609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d101014b0 0x7f1d10072ee0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.757+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 >> 172.21.15.111:0/1920299609 conn(0x7f1d100fb8a0 msgr2=0x7f1d100fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:07.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.757+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 shutdown_connections 2023-10-11T16:32:07.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.757+0000 7f1d179ff700 1 -- 172.21.15.111:0/1920299609 wait complete. 2023-10-11T16:32:07.776 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:32:07.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 -- 172.21.15.111:0/2892550258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1014b0 msgr2=0x7f9f9c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/2892550258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1014b0 0x7f9f9c101900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009990 tx=0x7f9f8c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 -- 172.21.15.111:0/2892550258 shutdown_connections 2023-10-11T16:32:07.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/2892550258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f9c101e40 0x7f9f9c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/2892550258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f9c1002c0 0x7f9f9c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/2892550258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1014b0 0x7f9f9c101900 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 -- 172.21.15.111:0/2892550258 >> 172.21.15.111:0/2892550258 conn(0x7f9f9c0fb8a0 msgr2=0x7f9f9c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:07.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 -- 172.21.15.111:0/2892550258 shutdown_connections 2023-10-11T16:32:07.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.975+0000 7f9fa47ff700 1 -- 172.21.15.111:0/2892550258 wait complete. 2023-10-11T16:32:07.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.976+0000 7f9fa47ff700 1 Processor -- start 2023-10-11T16:32:07.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.976+0000 7f9fa47ff700 1 -- start start 2023-10-11T16:32:07.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.976+0000 7f9fa47ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1002c0 0x7f9f9c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.976+0000 7f9fa47ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f9c1014b0 0x7f9f9c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.976+0000 7f9fa47ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f9c101e40 0x7f9f9c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa47ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c073bc0 con 0x7f9f9c1002c0 2023-10-11T16:32:07.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa47ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c073d30 con 0x7f9f9c101e40 2023-10-11T16:32:07.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa47ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c1a0120 con 0x7f9f9c1014b0 2023-10-11T16:32:07.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1002c0 0x7f9f9c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1002c0 0x7f9f9c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46912/0 (socket says 172.21.15.111:46912) 2023-10-11T16:32:07.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 -- 172.21.15.111:0/274415247 learned_addr learned my addr 172.21.15.111:0/274415247 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:07.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa2d9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f9c101e40 0x7f9f9c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa1d9a700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f9c1014b0 0x7f9f9c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 -- 172.21.15.111:0/274415247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f9c1014b0 msgr2=0x7f9f9c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f9c1014b0 0x7f9f9c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 -- 172.21.15.111:0/274415247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f9c101e40 msgr2=0x7f9f9c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:07.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f9c101e40 0x7f9f9c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:07.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 -- 172.21.15.111:0/274415247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c0095f0 con 0x7f9f9c1002c0 2023-10-11T16:32:07.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.977+0000 7f9fa259b700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1002c0 0x7f9f9c0748a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800e970 tx=0x7f9f9800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:07.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.978+0000 7f9f937fe700 1 -- 172.21.15.111:0/274415247 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98009790 con 0x7f9f9c1002c0 2023-10-11T16:32:07.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.978+0000 7f9f937fe700 1 -- 172.21.15.111:0/274415247 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f9800c7e0 con 0x7f9f9c1002c0 2023-10-11T16:32:07.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.978+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c10bc70 con 0x7f9f9c1002c0 2023-10-11T16:32:07.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.978+0000 7f9f937fe700 1 -- 172.21.15.111:0/274415247 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98003c70 con 0x7f9f9c1002c0 2023-10-11T16:32:07.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.978+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c10bae0 con 0x7f9f9c1002c0 2023-10-11T16:32:07.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.979+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c107860 con 0x7f9f9c1002c0 2023-10-11T16:32:07.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.981+0000 7f9f937fe700 1 -- 172.21.15.111:0/274415247 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f9f9800cc50 con 0x7f9f9c1002c0 2023-10-11T16:32:07.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.982+0000 7f9f937fe700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f880a1260 0x7f9f880a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:07.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.982+0000 7f9f937fe700 1 -- 172.21.15.111:0/274415247 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f9f98014070 con 0x7f9f9c1002c0 2023-10-11T16:32:07.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.982+0000 7f9fa1d9a700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f880a1260 0x7f9f880a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:07.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.983+0000 7f9fa1d9a700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f880a1260 0x7f9f880a3710 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c000c00 tx=0x7f9f8c00b260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:07.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:07.984+0000 7f9f937fe700 1 -- 172.21.15.111:0/274415247 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9f980962f0 con 0x7f9f9c1002c0 2023-10-11T16:32:08.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.188+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9f9c061700 con 0x7f9f9c1002c0 2023-10-11T16:32:08.189 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.188+0000 7f9f937fe700 1 -- 172.21.15.111:0/274415247 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9f98099910 con 0x7f9f9c1002c0 2023-10-11T16:32:08.189 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f880a1260 msgr2=0x7f9f880a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f880a1260 0x7f9f880a3710 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c000c00 tx=0x7f9f8c00b260 comp rx=0 tx=0).stop 2023-10-11T16:32:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1002c0 msgr2=0x7f9f9c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1002c0 0x7f9f9c0748a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800e970 tx=0x7f9f9800ec80 comp rx=0 tx=0).stop 2023-10-11T16:32:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 shutdown_connections 2023-10-11T16:32:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f9f880a1260 0x7f9f880a3710 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9f9c101e40 0x7f9f9c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f9c1002c0 0x7f9f9c0748a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 --2- 172.21.15.111:0/274415247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f9c1014b0 0x7f9f9c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 >> 172.21.15.111:0/274415247 conn(0x7f9f9c0fb8a0 msgr2=0x7f9f9c1049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 shutdown_connections 2023-10-11T16:32:08.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.191+0000 7f9fa47ff700 1 -- 172.21.15.111:0/274415247 wait complete. 2023-10-11T16:32:08.210 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:32:08.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:08 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/680074107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:08.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:08 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1920299609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:08.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.407+0000 7f87ad9ff700 1 -- 172.21.15.111:0/2719381720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a81002b0 msgr2=0x7f87a81006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.407+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/2719381720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a81002b0 0x7f87a81006c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8798009990 tx=0x7f8798009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.407+0000 7f87ad9ff700 1 -- 172.21.15.111:0/2719381720 shutdown_connections 2023-10-11T16:32:08.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.407+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/2719381720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87a8101e30 0x7f87a81086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.407+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/2719381720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a81002b0 0x7f87a81006c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.407+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/2719381720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87a81014a0 0x7f87a81018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.407+0000 7f87ad9ff700 1 -- 172.21.15.111:0/2719381720 >> 172.21.15.111:0/2719381720 conn(0x7f87a80fb890 msgr2=0x7f87a80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:08.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.408+0000 7f87ad9ff700 1 -- 172.21.15.111:0/2719381720 shutdown_connections 2023-10-11T16:32:08.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.408+0000 7f87ad9ff700 1 -- 172.21.15.111:0/2719381720 wait complete. 2023-10-11T16:32:08.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87ad9ff700 1 Processor -- start 2023-10-11T16:32:08.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87ad9ff700 1 -- start start 2023-10-11T16:32:08.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87ad9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87a81002b0 0x7f87a80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:08.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87ad9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87a81014a0 0x7f87a8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:08.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87ad9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a8101e30 0x7f87a8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:08.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87ad9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a8073bc0 con 0x7f87a8101e30 2023-10-11T16:32:08.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87ad9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a8073d30 con 0x7f87a81002b0 2023-10-11T16:32:08.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87ad9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a81a00d0 con 0x7f87a81014a0 2023-10-11T16:32:08.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87a77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a8101e30 0x7f87a8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:08.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87a77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a8101e30 0x7f87a8073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46936/0 (socket says 172.21.15.111:46936) 2023-10-11T16:32:08.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87a77fe700 1 -- 172.21.15.111:0/859476842 learned_addr learned my addr 172.21.15.111:0/859476842 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:08.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.409+0000 7f87a6ffd700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87a81002b0 0x7f87a80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:08.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87a67fc700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87a81014a0 0x7f87a8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:08.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87a77fe700 1 -- 172.21.15.111:0/859476842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87a81014a0 msgr2=0x7f87a8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87a77fe700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87a81014a0 0x7f87a8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87a77fe700 1 -- 172.21.15.111:0/859476842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87a81002b0 msgr2=0x7f87a80748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87a77fe700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87a81002b0 0x7f87a80748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87a77fe700 1 -- 172.21.15.111:0/859476842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980095f0 con 0x7f87a8101e30 2023-10-11T16:32:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87a77fe700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a8101e30 0x7f87a8073420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f87a0007ee0 tx=0x7f87a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87ac9fd700 1 -- 172.21.15.111:0/859476842 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a000dd20 con 0x7f87a8101e30 2023-10-11T16:32:08.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87ac9fd700 1 -- 172.21.15.111:0/859476842 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87a000f040 con 0x7f87a8101e30 2023-10-11T16:32:08.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a810bc80 con 0x7f87a8101e30 2023-10-11T16:32:08.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87ac9fd700 1 -- 172.21.15.111:0/859476842 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a00048a0 con 0x7f87a8101e30 2023-10-11T16:32:08.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.410+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a810baf0 con 0x7f87a8101e30 2023-10-11T16:32:08.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.411+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a8107890 con 0x7f87a8101e30 2023-10-11T16:32:08.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.413+0000 7f87ac9fd700 1 -- 172.21.15.111:0/859476842 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f87a0003a00 con 0x7f87a8101e30 2023-10-11T16:32:08.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.414+0000 7f87ac9fd700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87900a1260 0x7f87900a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.415+0000 7f87ac9fd700 1 -- 172.21.15.111:0/859476842 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f87a00cc560 con 0x7f87a8101e30 2023-10-11T16:32:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.415+0000 7f87a6ffd700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87900a1260 0x7f87900a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.415+0000 7f87a6ffd700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87900a1260 0x7f87900a3710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f879800b2e0 tx=0x7f8798005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:08.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.418+0000 7f87ac9fd700 1 -- 172.21.15.111:0/859476842 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f87a004e730 con 0x7f87a8101e30 2023-10-11T16:32:08.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:08 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/680074107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:08.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:08 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1920299609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:08.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:08 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/680074107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:08.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:08 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1920299609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:08.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.659+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87a802cfe0 con 0x7f87a8101e30 2023-10-11T16:32:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.660+0000 7f87ac9fd700 1 -- 172.21.15.111:0/859476842 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f87a0095780 con 0x7f87a8101e30 2023-10-11T16:32:08.661 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:08.661 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:51.227688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:32:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.662+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87900a1260 msgr2=0x7f87900a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.662+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87900a1260 0x7f87900a3710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f879800b2e0 tx=0x7f8798005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.662+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a8101e30 msgr2=0x7f87a8073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.662+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a8101e30 0x7f87a8073420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f87a0007ee0 tx=0x7f87a000d3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.663+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 shutdown_connections 2023-10-11T16:32:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.663+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f87900a1260 0x7f87900a3710 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.663+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87a8101e30 0x7f87a8073420 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.663+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87a81002b0 0x7f87a80748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.663+0000 7f87ad9ff700 1 --2- 172.21.15.111:0/859476842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87a81014a0 0x7f87a8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.663+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 >> 172.21.15.111:0/859476842 conn(0x7f87a80fb890 msgr2=0x7f87a81049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.663+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 shutdown_connections 2023-10-11T16:32:08.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.663+0000 7f87ad9ff700 1 -- 172.21.15.111:0/859476842 wait complete. 2023-10-11T16:32:08.666 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-10-11T16:32:08.684 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:32:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 -- 172.21.15.111:0/737907617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 msgr2=0x7fe2bc103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/737907617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 0x7fe2bc103170 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0009990 tx=0x7fe2b0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 -- 172.21.15.111:0/737907617 shutdown_connections 2023-10-11T16:32:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/737907617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2bc105fd0 0x7fe2bc1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/737907617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 0x7fe2bc103170 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/737907617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2bc1036b0 0x7fe2bc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 -- 172.21.15.111:0/737907617 >> 172.21.15.111:0/737907617 conn(0x7fe2bc0fa750 msgr2=0x7fe2bc0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 -- 172.21.15.111:0/737907617 shutdown_connections 2023-10-11T16:32:08.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.882+0000 7fe2c2bff700 1 -- 172.21.15.111:0/737907617 wait complete. 2023-10-11T16:32:08.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.883+0000 7fe2c2bff700 1 Processor -- start 2023-10-11T16:32:08.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.883+0000 7fe2c2bff700 1 -- start start 2023-10-11T16:32:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.883+0000 7fe2c2bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 0x7fe2bc1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c2bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2bc1036b0 0x7fe2bc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c2bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2bc105fd0 0x7fe2bc19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:08.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c2bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc19fd70 con 0x7fe2bc100d90 2023-10-11T16:32:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c2bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc19feb0 con 0x7fe2bc105fd0 2023-10-11T16:32:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c2bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc1a0040 con 0x7fe2bc1036b0 2023-10-11T16:32:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 0x7fe2bc1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2bbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2bc1036b0 0x7fe2bc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:08.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 0x7fe2bc1981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46948/0 (socket says 172.21.15.111:46948) 2023-10-11T16:32:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2bbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2bc1036b0 0x7fe2bc198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:59874/0 (socket says 172.21.15.111:59874) 2023-10-11T16:32:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 -- 172.21.15.111:0/3063682239 learned_addr learned my addr 172.21.15.111:0/3063682239 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c119c700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2bc105fd0 0x7fe2bc19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 -- 172.21.15.111:0/3063682239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2bc1036b0 msgr2=0x7fe2bc198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2bc1036b0 0x7fe2bc198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 -- 172.21.15.111:0/3063682239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2bc105fd0 msgr2=0x7fe2bc19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:08.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2bc105fd0 0x7fe2bc19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:08.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 -- 172.21.15.111:0/3063682239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b00095f0 con 0x7fe2bc100d90 2023-10-11T16:32:08.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2bbfff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2bc1036b0 0x7fe2bc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:08.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.884+0000 7fe2c099b700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 0x7fe2bc1981f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000b680 tx=0x7fe2b0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:08.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.885+0000 7fe2b9ffb700 1 -- 172.21.15.111:0/3063682239 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b001d070 con 0x7fe2bc100d90 2023-10-11T16:32:08.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.885+0000 7fe2b9ffb700 1 -- 172.21.15.111:0/3063682239 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2b0021410 con 0x7fe2bc100d90 2023-10-11T16:32:08.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.885+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2bc10bc40 con 0x7fe2bc100d90 2023-10-11T16:32:08.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.885+0000 7fe2b9ffb700 1 -- 172.21.15.111:0/3063682239 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0017650 con 0x7fe2bc100d90 2023-10-11T16:32:08.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.885+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2bc10bab0 con 0x7fe2bc100d90 2023-10-11T16:32:08.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.886+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2bc107b40 con 0x7fe2bc100d90 2023-10-11T16:32:08.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.888+0000 7fe2b9ffb700 1 -- 172.21.15.111:0/3063682239 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fe2b0021580 con 0x7fe2bc100d90 2023-10-11T16:32:08.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.889+0000 7fe2b9ffb700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe2ac0a1260 0x7fe2ac0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:08.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.889+0000 7fe2b9ffb700 1 -- 172.21.15.111:0/3063682239 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fe2b00ce2b0 con 0x7fe2bc100d90 2023-10-11T16:32:08.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.890+0000 7fe2bbfff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe2ac0a1260 0x7fe2ac0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:08.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.890+0000 7fe2bbfff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe2ac0a1260 0x7fe2ac0a3710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8010420 tx=0x7fe2a80073d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:08.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:08.893+0000 7fe2b9ffb700 1 -- 172.21.15.111:0/3063682239 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe2b0097500 con 0x7fe2bc100d90 2023-10-11T16:32:09.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.091+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2bc02cfe0 con 0x7fe2bc100d90 2023-10-11T16:32:09.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.092+0000 7fe2b9ffb700 1 -- 172.21.15.111:0/3063682239 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24870 (secure 0 0 0) 0x7fe2b0021860 con 0x7fe2bc100d90 2023-10-11T16:32:09.093 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:09.094 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":91,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:31:42.150221+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:32:09.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe2ac0a1260 msgr2=0x7fe2ac0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe2ac0a1260 0x7fe2ac0a3710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8010420 tx=0x7fe2a80073d0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 msgr2=0x7fe2bc1981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 0x7fe2bc1981f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000b680 tx=0x7fe2b0005740 comp rx=0 tx=0).stop 2023-10-11T16:32:09.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 shutdown_connections 2023-10-11T16:32:09.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe2ac0a1260 0x7fe2ac0a3710 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2bc105fd0 0x7fe2bc19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2bc100d90 0x7fe2bc1981f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 --2- 172.21.15.111:0/3063682239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2bc1036b0 0x7fe2bc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 >> 172.21.15.111:0/3063682239 conn(0x7fe2bc0fa750 msgr2=0x7fe2bc0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 shutdown_connections 2023-10-11T16:32:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.095+0000 7fe2c2bff700 1 -- 172.21.15.111:0/3063682239 wait complete. 2023-10-11T16:32:09.115 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:32:09.116 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:32:09.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 -- 172.21.15.111:0/2459953222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf241002c0 msgr2=0x7fdf241006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 --2- 172.21.15.111:0/2459953222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf241002c0 0x7fdf241006d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009990 tx=0x7fdf18009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 -- 172.21.15.111:0/2459953222 shutdown_connections 2023-10-11T16:32:09.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 --2- 172.21.15.111:0/2459953222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf24101e40 0x7fdf24108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 --2- 172.21.15.111:0/2459953222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf241002c0 0x7fdf241006d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 --2- 172.21.15.111:0/2459953222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf241014b0 0x7fdf24101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 -- 172.21.15.111:0/2459953222 >> 172.21.15.111:0/2459953222 conn(0x7fdf240fb8a0 msgr2=0x7fdf240fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:09.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 -- 172.21.15.111:0/2459953222 shutdown_connections 2023-10-11T16:32:09.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.316+0000 7fdf2adff700 1 -- 172.21.15.111:0/2459953222 wait complete. 2023-10-11T16:32:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.317+0000 7fdf2adff700 1 Processor -- start 2023-10-11T16:32:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.317+0000 7fdf2adff700 1 -- start start 2023-10-11T16:32:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2adff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf241002c0 0x7fdf240748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2adff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf241014b0 0x7fdf24072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:09.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2adff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf24101e40 0x7fdf24073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2adff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24073bc0 con 0x7fdf24101e40 2023-10-11T16:32:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2adff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf24073d30 con 0x7fdf241014b0 2023-10-11T16:32:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2adff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf241a00d0 con 0x7fdf241002c0 2023-10-11T16:32:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf28b9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf241002c0 0x7fdf240748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:09.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2939c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf24101e40 0x7fdf24073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf28b9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf241002c0 0x7fdf240748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:59894/0 (socket says 172.21.15.111:59894) 2023-10-11T16:32:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf28b9b700 1 -- 172.21.15.111:0/3349946220 learned_addr learned my addr 172.21.15.111:0/3349946220 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf23fff700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf241014b0 0x7fdf24072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2939c700 1 -- 172.21.15.111:0/3349946220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf241002c0 msgr2=0x7fdf240748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2939c700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf241002c0 0x7fdf240748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2939c700 1 -- 172.21.15.111:0/3349946220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf241014b0 msgr2=0x7fdf24072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2939c700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf241014b0 0x7fdf24072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2939c700 1 -- 172.21.15.111:0/3349946220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf180095f0 con 0x7fdf24101e40 2023-10-11T16:32:09.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf28b9b700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf241002c0 0x7fdf240748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:09.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.318+0000 7fdf2939c700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf24101e40 0x7fdf24073420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00efc0 tx=0x7fdf1c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:09.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.319+0000 7fdf21ffb700 1 -- 172.21.15.111:0/3349946220 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c018070 con 0x7fdf24101e40 2023-10-11T16:32:09.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.319+0000 7fdf21ffb700 1 -- 172.21.15.111:0/3349946220 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf1c00f040 con 0x7fdf24101e40 2023-10-11T16:32:09.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.319+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2410bc90 con 0x7fdf24101e40 2023-10-11T16:32:09.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.319+0000 7fdf21ffb700 1 -- 172.21.15.111:0/3349946220 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c00cd40 con 0x7fdf24101e40 2023-10-11T16:32:09.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.319+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2410bb00 con 0x7fdf24101e40 2023-10-11T16:32:09.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.320+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf24107860 con 0x7fdf24101e40 2023-10-11T16:32:09.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.322+0000 7fdf21ffb700 1 -- 172.21.15.111:0/3349946220 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fdf1c003a00 con 0x7fdf24101e40 2023-10-11T16:32:09.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.323+0000 7fdf21ffb700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf140a1260 0x7fdf140a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:09.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.323+0000 7fdf21ffb700 1 -- 172.21.15.111:0/3349946220 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fdf1c0cc9a0 con 0x7fdf24101e40 2023-10-11T16:32:09.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.323+0000 7fdf28b9b700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf140a1260 0x7fdf140a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:09.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.324+0000 7fdf28b9b700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf140a1260 0x7fdf140a3710 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800b2e0 tx=0x7fdf18005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:09.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.327+0000 7fdf21ffb700 1 -- 172.21.15.111:0/3349946220 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdf1c095bf0 con 0x7fdf24101e40 2023-10-11T16:32:09.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:09 smithi111 ceph-mon[98527]: pgmap v335: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:32:09.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:09 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/274415247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:09.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:09 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/859476842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:09.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:09 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3063682239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:09.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:09 smithi150 ceph-mon[132217]: pgmap v335: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:32:09.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:09 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/274415247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:09.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:09 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/859476842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:09.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:09 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3063682239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:09.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.523+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf24061700 con 0x7fdf24101e40 2023-10-11T16:32:09.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.524+0000 7fdf21ffb700 1 -- 172.21.15.111:0/3349946220 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf1c099210 con 0x7fdf24101e40 2023-10-11T16:32:09.525 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:32:09.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf140a1260 msgr2=0x7fdf140a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf140a1260 0x7fdf140a3710 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800b2e0 tx=0x7fdf18005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf24101e40 msgr2=0x7fdf24073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf24101e40 0x7fdf24073420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00efc0 tx=0x7fdf1c00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 shutdown_connections 2023-10-11T16:32:09.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf140a1260 0x7fdf140a3710 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf24101e40 0x7fdf24073420 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf241002c0 0x7fdf240748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 --2- 172.21.15.111:0/3349946220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf241014b0 0x7fdf24072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 >> 172.21.15.111:0/3349946220 conn(0x7fdf240fb8a0 msgr2=0x7fdf241049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:09.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 shutdown_connections 2023-10-11T16:32:09.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.526+0000 7fdf2adff700 1 -- 172.21.15.111:0/3349946220 wait complete. 2023-10-11T16:32:09.546 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:32:09.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:09 smithi130 ceph-mon[131859]: pgmap v335: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-11T16:32:09.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:09 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/274415247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:09.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:09 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/859476842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:09.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:09 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3063682239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 -- 172.21.15.111:0/3419852374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa7c8074ab0 msgr2=0x7fa7c8072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/3419852374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa7c8074ab0 0x7fa7c8072f00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40098f0 tx=0x7fa7c4009c00 comp rx=0 tx=0).stop 2023-10-11T16:32:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 -- 172.21.15.111:0/3419852374 shutdown_connections 2023-10-11T16:32:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/3419852374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7c8108180 0x7fa7c810a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/3419852374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa7c8074ab0 0x7fa7c8072f00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/3419852374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c80734d0 0x7fa7c8073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 -- 172.21.15.111:0/3419852374 >> 172.21.15.111:0/3419852374 conn(0x7fa7c80fb8b0 msgr2=0x7fa7c80fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:09.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 -- 172.21.15.111:0/3419852374 shutdown_connections 2023-10-11T16:32:09.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.747+0000 7fa7d07ff700 1 -- 172.21.15.111:0/3419852374 wait complete. 2023-10-11T16:32:09.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.748+0000 7fa7d07ff700 1 Processor -- start 2023-10-11T16:32:09.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.748+0000 7fa7d07ff700 1 -- start start 2023-10-11T16:32:09.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.748+0000 7fa7d07ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7c80734d0 0x7fa7c8198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:09.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.748+0000 7fa7d07ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c8074ab0 0x7fa7c8198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:09.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.748+0000 7fa7d07ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa7c8108180 0x7fa7c819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:09.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.748+0000 7fa7d07ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c819fdc0 con 0x7fa7c80734d0 2023-10-11T16:32:09.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.748+0000 7fa7d07ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c819ff00 con 0x7fa7c8108180 2023-10-11T16:32:09.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.748+0000 7fa7d07ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c81a0090 con 0x7fa7c8074ab0 2023-10-11T16:32:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7c80734d0 0x7fa7c8198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7c80734d0 0x7fa7c8198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46984/0 (socket says 172.21.15.111:46984) 2023-10-11T16:32:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 -- 172.21.15.111:0/4065826471 learned_addr learned my addr 172.21.15.111:0/4065826471 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:09.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ced9c700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa7c8108180 0x7fa7c819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7cdd9a700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c8074ab0 0x7fa7c8198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 -- 172.21.15.111:0/4065826471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c8074ab0 msgr2=0x7fa7c8198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c8074ab0 0x7fa7c8198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 -- 172.21.15.111:0/4065826471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa7c8108180 msgr2=0x7fa7c819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa7c8108180 0x7fa7c819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 -- 172.21.15.111:0/4065826471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c4009520 con 0x7fa7c80734d0 2023-10-11T16:32:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7ce59b700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7c80734d0 0x7fa7c8198240 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009fd0 tx=0x7fa7c400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.749+0000 7fa7bf7fe700 1 -- 172.21.15.111:0/4065826471 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c401c070 con 0x7fa7c80734d0 2023-10-11T16:32:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.750+0000 7fa7bf7fe700 1 -- 172.21.15.111:0/4065826471 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7c400f7d0 con 0x7fa7c80734d0 2023-10-11T16:32:09.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.750+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c810de60 con 0x7fa7c80734d0 2023-10-11T16:32:09.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.750+0000 7fa7bf7fe700 1 -- 172.21.15.111:0/4065826471 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4004ad0 con 0x7fa7c80734d0 2023-10-11T16:32:09.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.750+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c810dcd0 con 0x7fa7c80734d0 2023-10-11T16:32:09.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.751+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c81068c0 con 0x7fa7c80734d0 2023-10-11T16:32:09.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.753+0000 7fa7bf7fe700 1 -- 172.21.15.111:0/4065826471 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7fa7c4020070 con 0x7fa7c80734d0 2023-10-11T16:32:09.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.754+0000 7fa7bf7fe700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa7b40a11e0 0x7fa7b40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:09.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.754+0000 7fa7bf7fe700 1 -- 172.21.15.111:0/4065826471 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fa7c40cdcb0 con 0x7fa7c80734d0 2023-10-11T16:32:09.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.754+0000 7fa7cdd9a700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa7b40a11e0 0x7fa7b40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:09.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.755+0000 7fa7cdd9a700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa7b40a11e0 0x7fa7b40a3690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8009c60 tx=0x7fa7b8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:09.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.756+0000 7fa7bf7fe700 1 -- 172.21.15.111:0/4065826471 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa7c4096f60 con 0x7fa7c80734d0 2023-10-11T16:32:09.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.952+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa7c8067b00 con 0x7fa7c80734d0 2023-10-11T16:32:09.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.953+0000 7fa7bf7fe700 1 -- 172.21.15.111:0/4065826471 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7c409a580 con 0x7fa7c80734d0 2023-10-11T16:32:09.954 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:09.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa7b40a11e0 msgr2=0x7fa7b40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa7b40a11e0 0x7fa7b40a3690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8009c60 tx=0x7fa7b8009260 comp rx=0 tx=0).stop 2023-10-11T16:32:09.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7c80734d0 msgr2=0x7fa7c8198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:09.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7c80734d0 0x7fa7c8198240 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009fd0 tx=0x7fa7c400b790 comp rx=0 tx=0).stop 2023-10-11T16:32:09.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 shutdown_connections 2023-10-11T16:32:09.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa7b40a11e0 0x7fa7b40a3690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa7c8108180 0x7fa7c819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7c80734d0 0x7fa7c8198240 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 --2- 172.21.15.111:0/4065826471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c8074ab0 0x7fa7c8198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:09.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.955+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 >> 172.21.15.111:0/4065826471 conn(0x7fa7c80fb8b0 msgr2=0x7fa7c80fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:09.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.956+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 shutdown_connections 2023-10-11T16:32:09.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:09.956+0000 7fa7d07ff700 1 -- 172.21.15.111:0/4065826471 wait complete. 2023-10-11T16:32:09.975 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:32:10.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/3768414287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881002d0 msgr2=0x7f1d881006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/3768414287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881002d0 0x7f1d881006e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009990 tx=0x7f1d78009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/3768414287 shutdown_connections 2023-10-11T16:32:10.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/3768414287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d88101e50 0x7f1d88108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/3768414287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881002d0 0x7f1d881006e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/3768414287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d881014c0 0x7f1d88101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/3768414287 >> 172.21.15.111:0/3768414287 conn(0x7f1d880fb8b0 msgr2=0x7f1d880fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/3768414287 shutdown_connections 2023-10-11T16:32:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.204+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/3768414287 wait complete. 2023-10-11T16:32:10.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.205+0000 7f1d8d3ff700 1 Processor -- start 2023-10-11T16:32:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.205+0000 7f1d8d3ff700 1 -- start start 2023-10-11T16:32:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.205+0000 7f1d8d3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d881002d0 0x7f1d88198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.205+0000 7f1d8d3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881014c0 0x7f1d88198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.205+0000 7f1d8d3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d88101e50 0x7f1d8819f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.205+0000 7f1d8d3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8819feb0 con 0x7f1d88101e50 2023-10-11T16:32:10.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.205+0000 7f1d8d3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8819fff0 con 0x7f1d881014c0 2023-10-11T16:32:10.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.205+0000 7f1d8d3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d881a0180 con 0x7f1d881002d0 2023-10-11T16:32:10.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d88101e50 0x7f1d8819f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:10.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d86ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d881002d0 0x7f1d88198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:10.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d88101e50 0x7f1d8819f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47006/0 (socket says 172.21.15.111:47006) 2023-10-11T16:32:10.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 -- 172.21.15.111:0/866115071 learned_addr learned my addr 172.21.15.111:0/866115071 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:10.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d7e7fc700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881014c0 0x7f1d88198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:10.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 -- 172.21.15.111:0/866115071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d881002d0 msgr2=0x7f1d88198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d881002d0 0x7f1d88198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 -- 172.21.15.111:0/866115071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881014c0 msgr2=0x7f1d88198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881014c0 0x7f1d88198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 -- 172.21.15.111:0/866115071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d780095f0 con 0x7f1d88101e50 2023-10-11T16:32:10.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d86ffd700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d881002d0 0x7f1d88198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:10.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d7e7fc700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881014c0 0x7f1d88198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:10.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d877fe700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d88101e50 0x7f1d8819f8e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000e940 tx=0x7f1d8000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:10.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d84ff9700 1 -- 172.21.15.111:0/866115071 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d800097b0 con 0x7f1d88101e50 2023-10-11T16:32:10.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.206+0000 7f1d84ff9700 1 -- 172.21.15.111:0/866115071 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d8000ee90 con 0x7f1d88101e50 2023-10-11T16:32:10.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.207+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8810bca0 con 0x7f1d88101e50 2023-10-11T16:32:10.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.207+0000 7f1d84ff9700 1 -- 172.21.15.111:0/866115071 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d80005520 con 0x7f1d88101e50 2023-10-11T16:32:10.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.207+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8810bb10 con 0x7f1d88101e50 2023-10-11T16:32:10.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.208+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d88192a00 con 0x7f1d88101e50 2023-10-11T16:32:10.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.209+0000 7f1d84ff9700 1 -- 172.21.15.111:0/866115071 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f1d80004750 con 0x7f1d88101e50 2023-10-11T16:32:10.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.211+0000 7f1d84ff9700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d740a1260 0x7f1d740a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:10.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.211+0000 7f1d84ff9700 1 -- 172.21.15.111:0/866115071 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f1d80014070 con 0x7f1d88101e50 2023-10-11T16:32:10.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.211+0000 7f1d86ffd700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d740a1260 0x7f1d740a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:10.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.212+0000 7f1d86ffd700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d740a1260 0x7f1d740a3710 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800b2e0 tx=0x7f1d78005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:10.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.215+0000 7f1d84ff9700 1 -- 172.21.15.111:0/866115071 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1d80096180 con 0x7f1d88101e50 2023-10-11T16:32:10.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:10 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3349946220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:10.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:10 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4065826471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:10.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.417+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1d88061700 con 0x7f1d88101e50 2023-10-11T16:32:10.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.417+0000 7f1d84ff9700 1 -- 172.21.15.111:0/866115071 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1d800997a0 con 0x7f1d88101e50 2023-10-11T16:32:10.419 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:32:10.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d740a1260 msgr2=0x7f1d740a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d740a1260 0x7f1d740a3710 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800b2e0 tx=0x7f1d78005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d88101e50 msgr2=0x7f1d8819f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d88101e50 0x7f1d8819f8e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000e940 tx=0x7f1d8000ec50 comp rx=0 tx=0).stop 2023-10-11T16:32:10.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 shutdown_connections 2023-10-11T16:32:10.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1d740a1260 0x7f1d740a3710 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d88101e50 0x7f1d8819f8e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d881002d0 0x7f1d88198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 --2- 172.21.15.111:0/866115071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d881014c0 0x7f1d88198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.419+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 >> 172.21.15.111:0/866115071 conn(0x7f1d880fb8b0 msgr2=0x7f1d88106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:10.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.420+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 shutdown_connections 2023-10-11T16:32:10.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.420+0000 7f1d8d3ff700 1 -- 172.21.15.111:0/866115071 wait complete. 2023-10-11T16:32:10.440 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:32:10.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:10 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3349946220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:10.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:10 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4065826471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:10.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:10 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3349946220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:10.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:10 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4065826471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:10.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 -- 172.21.15.111:0/434483308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 msgr2=0x7f0e8c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 --2- 172.21.15.111:0/434483308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 0x7f0e8c101900 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009990 tx=0x7f0e80009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 -- 172.21.15.111:0/434483308 shutdown_connections 2023-10-11T16:32:10.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 --2- 172.21.15.111:0/434483308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e8c101e40 0x7f0e8c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 --2- 172.21.15.111:0/434483308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e8c1002c0 0x7f0e8c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 --2- 172.21.15.111:0/434483308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 0x7f0e8c101900 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 -- 172.21.15.111:0/434483308 >> 172.21.15.111:0/434483308 conn(0x7f0e8c0fb8a0 msgr2=0x7f0e8c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:10.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 -- 172.21.15.111:0/434483308 shutdown_connections 2023-10-11T16:32:10.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.641+0000 7f0e91bff700 1 -- 172.21.15.111:0/434483308 wait complete. 2023-10-11T16:32:10.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.642+0000 7f0e91bff700 1 Processor -- start 2023-10-11T16:32:10.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.642+0000 7f0e91bff700 1 -- start start 2023-10-11T16:32:10.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.642+0000 7f0e91bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e8c1002c0 0x7f0e8c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:10.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.642+0000 7f0e91bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 0x7f0e8c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:10.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.642+0000 7f0e91bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e8c101e40 0x7f0e8c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:10.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.642+0000 7f0e91bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c19fde0 con 0x7f0e8c1014b0 2023-10-11T16:32:10.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.642+0000 7f0e91bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c19ff20 con 0x7f0e8c1002c0 2023-10-11T16:32:10.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.642+0000 7f0e91bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c1a00b0 con 0x7f0e8c101e40 2023-10-11T16:32:10.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 0x7f0e8c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:10.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 0x7f0e8c1987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47018/0 (socket says 172.21.15.111:47018) 2023-10-11T16:32:10.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 -- 172.21.15.111:0/2968619418 learned_addr learned my addr 172.21.15.111:0/2968619418 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:10.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8b7fe700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e8c1002c0 0x7f0e8c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:10.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8bfff700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e8c101e40 0x7f0e8c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:10.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 -- 172.21.15.111:0/2968619418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e8c101e40 msgr2=0x7f0e8c19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e8c101e40 0x7f0e8c19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 -- 172.21.15.111:0/2968619418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e8c1002c0 msgr2=0x7f0e8c198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e8c1002c0 0x7f0e8c198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 -- 172.21.15.111:0/2968619418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e800095f0 con 0x7f0e8c1014b0 2023-10-11T16:32:10.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.643+0000 7f0e8affd700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 0x7f0e8c1987a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0e80004760 tx=0x7f0e80004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:10.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.644+0000 7f0e88ff9700 1 -- 172.21.15.111:0/2968619418 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8001d070 con 0x7f0e8c1014b0 2023-10-11T16:32:10.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.644+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c10bc70 con 0x7f0e8c1014b0 2023-10-11T16:32:10.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.644+0000 7f0e88ff9700 1 -- 172.21.15.111:0/2968619418 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e80004830 con 0x7f0e8c1014b0 2023-10-11T16:32:10.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.644+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c10bae0 con 0x7f0e8c1014b0 2023-10-11T16:32:10.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.644+0000 7f0e88ff9700 1 -- 172.21.15.111:0/2968619418 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80005370 con 0x7f0e8c1014b0 2023-10-11T16:32:10.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.645+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8c067b00 con 0x7f0e8c1014b0 2023-10-11T16:32:10.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.647+0000 7f0e88ff9700 1 -- 172.21.15.111:0/2968619418 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7f0e800054d0 con 0x7f0e8c1014b0 2023-10-11T16:32:10.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.648+0000 7f0e88ff9700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e740a11e0 0x7f0e740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:10.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.648+0000 7f0e88ff9700 1 -- 172.21.15.111:0/2968619418 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f0e800ce840 con 0x7f0e8c1014b0 2023-10-11T16:32:10.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.648+0000 7f0e8b7fe700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e740a11e0 0x7f0e740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:10.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.649+0000 7f0e8b7fe700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e740a11e0 0x7f0e740a3690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c101310 tx=0x7f0e7c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:10.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.650+0000 7f0e88ff9700 1 -- 172.21.15.111:0/2968619418 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0e80097af0 con 0x7f0e8c1014b0 2023-10-11T16:32:10.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.848+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0e8c1a0c20 con 0x7f0e8c1014b0 2023-10-11T16:32:10.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.849+0000 7f0e88ff9700 1 -- 172.21.15.111:0/2968619418 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0e8009b110 con 0x7f0e8c1014b0 2023-10-11T16:32:10.851 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:10.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e740a11e0 msgr2=0x7f0e740a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e740a11e0 0x7f0e740a3690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c101310 tx=0x7f0e7c009260 comp rx=0 tx=0).stop 2023-10-11T16:32:10.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 msgr2=0x7f0e8c1987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:10.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 0x7f0e8c1987a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0e80004760 tx=0x7f0e80004790 comp rx=0 tx=0).stop 2023-10-11T16:32:10.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 shutdown_connections 2023-10-11T16:32:10.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0e740a11e0 0x7f0e740a3690 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e8c101e40 0x7f0e8c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e8c1002c0 0x7f0e8c198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 --2- 172.21.15.111:0/2968619418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e8c1014b0 0x7f0e8c1987a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:10.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 >> 172.21.15.111:0/2968619418 conn(0x7f0e8c0fb8a0 msgr2=0x7f0e8c0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:10.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 shutdown_connections 2023-10-11T16:32:10.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:10.851+0000 7f0e91bff700 1 -- 172.21.15.111:0/2968619418 wait complete. 2023-10-11T16:32:10.872 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:32:11.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 -- 172.21.15.111:0/3200074728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac101e30 msgr2=0x7ff2ac1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/3200074728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac101e30 0x7ff2ac1086f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400b0a0 tx=0x7ff2a400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 -- 172.21.15.111:0/3200074728 shutdown_connections 2023-10-11T16:32:11.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/3200074728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac101e30 0x7ff2ac1086f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/3200074728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2ac1002b0 0x7ff2ac1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/3200074728 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2ac1014a0 0x7ff2ac1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 -- 172.21.15.111:0/3200074728 >> 172.21.15.111:0/3200074728 conn(0x7ff2ac0fb890 msgr2=0x7ff2ac0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:11.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 -- 172.21.15.111:0/3200074728 shutdown_connections 2023-10-11T16:32:11.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.068+0000 7ff2b39ff700 1 -- 172.21.15.111:0/3200074728 wait complete. 2023-10-11T16:32:11.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.069+0000 7ff2b39ff700 1 Processor -- start 2023-10-11T16:32:11.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b39ff700 1 -- start start 2023-10-11T16:32:11.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b39ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac1002b0 0x7ff2ac0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b39ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2ac1014a0 0x7ff2ac072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b39ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2ac101e30 0x7ff2ac073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b39ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac073bc0 con 0x7ff2ac101e30 2023-10-11T16:32:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b39ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac073d30 con 0x7ff2ac1002b0 2023-10-11T16:32:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b39ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac1a0130 con 0x7ff2ac1014a0 2023-10-11T16:32:11.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b1f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2ac101e30 0x7ff2ac073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:11.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac1002b0 0x7ff2ac0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:11.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b1f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2ac101e30 0x7ff2ac073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47040/0 (socket says 172.21.15.111:47040) 2023-10-11T16:32:11.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac1002b0 0x7ff2ac0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:49962/0 (socket says 172.21.15.111:49962) 2023-10-11T16:32:11.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b0f9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2ac1014a0 0x7ff2ac072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:11.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.070+0000 7ff2b1f9c700 1 -- 172.21.15.111:0/2834997976 learned_addr learned my addr 172.21.15.111:0/2834997976 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:11.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2b1f9c700 1 -- 172.21.15.111:0/2834997976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2ac1014a0 msgr2=0x7ff2ac072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2b1f9c700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2ac1014a0 0x7ff2ac072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2b1f9c700 1 -- 172.21.15.111:0/2834997976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac1002b0 msgr2=0x7ff2ac0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2b1f9c700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac1002b0 0x7ff2ac0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2b1f9c700 1 -- 172.21.15.111:0/2834997976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a4009d00 con 0x7ff2ac101e30 2023-10-11T16:32:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2b1f9c700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2ac101e30 0x7ff2ac073420 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400d010 tx=0x7ff2a400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2a27fc700 1 -- 172.21.15.111:0/2834997976 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a400e040 con 0x7ff2ac101e30 2023-10-11T16:32:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac10bc80 con 0x7ff2ac101e30 2023-10-11T16:32:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2a27fc700 1 -- 172.21.15.111:0/2834997976 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2a4003e10 con 0x7ff2ac101e30 2023-10-11T16:32:11.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2a27fc700 1 -- 172.21.15.111:0/2834997976 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a40052a0 con 0x7ff2ac101e30 2023-10-11T16:32:11.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.071+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ac10baf0 con 0x7ff2ac101e30 2023-10-11T16:32:11.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.073+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac107800 con 0x7ff2ac101e30 2023-10-11T16:32:11.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.074+0000 7ff2a27fc700 1 -- 172.21.15.111:0/2834997976 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 135767+0+0 (secure 0 0 0) 0x7ff2a401d480 con 0x7ff2ac101e30 2023-10-11T16:32:11.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.075+0000 7ff2a27fc700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff2980a11e0 0x7ff2980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:11.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.076+0000 7ff2a27fc700 1 -- 172.21.15.111:0/2834997976 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7ff2a40ce220 con 0x7ff2ac101e30 2023-10-11T16:32:11.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.076+0000 7ff2b179b700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff2980a11e0 0x7ff2980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:11.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.076+0000 7ff2b179b700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff2980a11e0 0x7ff2980a3690 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac101300 tx=0x7ff2a8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:11.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.079+0000 7ff2a27fc700 1 -- 172.21.15.111:0/2834997976 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff2a4097470 con 0x7ff2ac101e30 2023-10-11T16:32:11.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.143+0000 7ff2a27fc700 1 -- 172.21.15.111:0/2834997976 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7ff2a4097840 con 0x7ff2ac101e30 2023-10-11T16:32:11.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.333+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2ac02cfe0 con 0x7ff2ac101e30 2023-10-11T16:32:11.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.334+0000 7ff2a27fc700 1 -- 172.21.15.111:0/2834997976 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10823 (secure 0 0 0) 0x7ff2a40542a0 con 0x7ff2ac101e30 2023-10-11T16:32:11.336 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:11.336 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:51.227688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:32:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.336+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff2980a11e0 msgr2=0x7ff2980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.336+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff2980a11e0 0x7ff2980a3690 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac101300 tx=0x7ff2a8009260 comp rx=0 tx=0).stop 2023-10-11T16:32:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.336+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2ac101e30 msgr2=0x7ff2ac073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.336+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2ac101e30 0x7ff2ac073420 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400d010 tx=0x7ff2a400bc80 comp rx=0 tx=0).stop 2023-10-11T16:32:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.336+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 shutdown_connections 2023-10-11T16:32:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.336+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7ff2980a11e0 0x7ff2980a3690 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.336+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2ac101e30 0x7ff2ac073420 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.336+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2ac1002b0 0x7ff2ac0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.337+0000 7ff2b39ff700 1 --2- 172.21.15.111:0/2834997976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2ac1014a0 0x7ff2ac072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.337+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 >> 172.21.15.111:0/2834997976 conn(0x7ff2ac0fb890 msgr2=0x7ff2ac104940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.337+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 shutdown_connections 2023-10-11T16:32:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.337+0000 7ff2b39ff700 1 -- 172.21.15.111:0/2834997976 wait complete. 2023-10-11T16:32:11.341 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-10-11T16:32:11.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:32:11.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:11 smithi111 ceph-mon[98527]: pgmap v336: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:11.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:11 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/866115071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:11.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:11 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2968619418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:11 smithi150 ceph-mon[132217]: pgmap v336: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:11 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/866115071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:11.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:11 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2968619418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:11.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 -- 172.21.15.111:0/3683477927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 msgr2=0x7f5f9c10a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/3683477927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 0x7f5f9c10a5d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400b0a0 tx=0x7f5f9400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 -- 172.21.15.111:0/3683477927 shutdown_connections 2023-10-11T16:32:11.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/3683477927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 0x7f5f9c10a5d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/3683477927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f9c074b30 0x7f5f9c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/3683477927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f9c073550 0x7f5f9c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 -- 172.21.15.111:0/3683477927 >> 172.21.15.111:0/3683477927 conn(0x7f5f9c0fb870 msgr2=0x7f5f9c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:11.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 -- 172.21.15.111:0/3683477927 shutdown_connections 2023-10-11T16:32:11.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.552+0000 7f5fa47ff700 1 -- 172.21.15.111:0/3683477927 wait complete. 2023-10-11T16:32:11.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.553+0000 7f5fa47ff700 1 Processor -- start 2023-10-11T16:32:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.553+0000 7f5fa47ff700 1 -- start start 2023-10-11T16:32:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.553+0000 7f5fa47ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f9c073550 0x7f5f9c1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.553+0000 7f5fa47ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f9c074b30 0x7f5f9c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.553+0000 7f5fa47ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 0x7f5f9c19f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.553+0000 7f5fa47ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c19fe40 con 0x7f5f9c108150 2023-10-11T16:32:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.553+0000 7f5fa47ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c19ff80 con 0x7f5f9c074b30 2023-10-11T16:32:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.553+0000 7f5fa47ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c1a0110 con 0x7f5f9c073550 2023-10-11T16:32:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 0x7f5f9c19f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa259b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f9c073550 0x7f5f9c1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 0x7f5f9c19f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47054/0 (socket says 172.21.15.111:47054) 2023-10-11T16:32:11.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 -- 172.21.15.111:0/1636090954 learned_addr learned my addr 172.21.15.111:0/1636090954 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:11.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa1d9a700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f9c074b30 0x7f5f9c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:11.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 -- 172.21.15.111:0/1636090954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f9c073550 msgr2=0x7f5f9c1982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f9c073550 0x7f5f9c1982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 -- 172.21.15.111:0/1636090954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f9c074b30 msgr2=0x7f5f9c198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f9c074b30 0x7f5f9c198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 -- 172.21.15.111:0/1636090954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f94009d00 con 0x7f5f9c108150 2023-10-11T16:32:11.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa2d9c700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 0x7f5f9c19f870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009fd0 tx=0x7f5f9400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:11.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5f937fe700 1 -- 172.21.15.111:0/1636090954 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9400e040 con 0x7f5f9c108150 2023-10-11T16:32:11.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5f937fe700 1 -- 172.21.15.111:0/1636090954 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f94003e10 con 0x7f5f9c108150 2023-10-11T16:32:11.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5f937fe700 1 -- 172.21.15.111:0/1636090954 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9401b6a0 con 0x7f5f9c108150 2023-10-11T16:32:11.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.554+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c10de70 con 0x7f5f9c108150 2023-10-11T16:32:11.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.555+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c10dce0 con 0x7f5f9c108150 2023-10-11T16:32:11.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.556+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c106800 con 0x7f5f9c108150 2023-10-11T16:32:11.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.557+0000 7f5f937fe700 1 -- 172.21.15.111:0/1636090954 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f5f9401d470 con 0x7f5f9c108150 2023-10-11T16:32:11.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.559+0000 7f5f937fe700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f880a1150 0x7f5f880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:11.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.559+0000 7f5f937fe700 1 -- 172.21.15.111:0/1636090954 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f5f940ce090 con 0x7f5f9c108150 2023-10-11T16:32:11.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.559+0000 7f5fa259b700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f880a1150 0x7f5f880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:11.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.559+0000 7f5fa259b700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f880a1150 0x7f5f880a3600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c074f00 tx=0x7f5f98009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:11.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.562+0000 7f5f937fe700 1 -- 172.21.15.111:0/1636090954 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5f940974a0 con 0x7f5f9c108150 2023-10-11T16:32:11.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:11 smithi130 ceph-mon[131859]: pgmap v336: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-11T16:32:11.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:11 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/866115071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:11.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:11 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2968619418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:11.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.769+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f9c02cfe0 con 0x7f5f9c108150 2023-10-11T16:32:11.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.771+0000 7f5f937fe700 1 -- 172.21.15.111:0/1636090954 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24870 (secure 0 0 0) 0x7f5f9409aac0 con 0x7f5f9c108150 2023-10-11T16:32:11.773 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:11.774 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":91,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:31:42.150221+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:32:11.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f880a1150 msgr2=0x7f5f880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f880a1150 0x7f5f880a3600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c074f00 tx=0x7f5f98009260 comp rx=0 tx=0).stop 2023-10-11T16:32:11.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 msgr2=0x7f5f9c19f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:11.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 0x7f5f9c19f870 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009fd0 tx=0x7f5f9400bce0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 shutdown_connections 2023-10-11T16:32:11.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f5f880a1150 0x7f5f880a3600 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f9c108150 0x7f5f9c19f870 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f9c073550 0x7f5f9c1982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 --2- 172.21.15.111:0/1636090954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f9c074b30 0x7f5f9c198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 >> 172.21.15.111:0/1636090954 conn(0x7f5f9c0fb870 msgr2=0x7f5f9c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:11.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 shutdown_connections 2023-10-11T16:32:11.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:11.774+0000 7f5fa47ff700 1 -- 172.21.15.111:0/1636090954 wait complete. 2023-10-11T16:32:11.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-10-11T16:30:11.633745+0000', 'modified': '2023-10-11T16:30:51.227688+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24569, 'mds_1': 24550, 'mds_2': 24532}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24488': {'gid': 24488, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6835/89303870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 89303870}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 89303870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.150:6835/2357736565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 2357736565}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2357736565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24521': {'gid': 24521, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6837/3927432367', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 3927432367}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 3927432367}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.150:6837/612139379', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6836', 'nonce': 612139379}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 612139379}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.150:6839/3698951370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6838', 'nonce': 3698951370}, {'type': 'v1', 'addr': '172.21.15.150:6839', 'nonce': 3698951370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.130:6839/2448702777', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2448702777}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2448702777}]}, 'join_fscid': -1, 'export_targets': [1, 2], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-11T16:32:11.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-11T16:32:11.796 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-11T16:32:12.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 -- 172.21.15.111:0/592972007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070100240 msgr2=0x7f6070100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 --2- 172.21.15.111:0/592972007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070100240 0x7f6070100650 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6060009990 tx=0x7f6060009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 -- 172.21.15.111:0/592972007 shutdown_connections 2023-10-11T16:32:12.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 --2- 172.21.15.111:0/592972007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6070101dc0 0x7f6070108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 --2- 172.21.15.111:0/592972007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070100240 0x7f6070100650 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 --2- 172.21.15.111:0/592972007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6070101430 0x7f6070101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 -- 172.21.15.111:0/592972007 >> 172.21.15.111:0/592972007 conn(0x7f60700fb820 msgr2=0x7f60700fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 -- 172.21.15.111:0/592972007 shutdown_connections 2023-10-11T16:32:12.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.031+0000 7f6075fff700 1 -- 172.21.15.111:0/592972007 wait complete. 2023-10-11T16:32:12.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.032+0000 7f6075fff700 1 Processor -- start 2023-10-11T16:32:12.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f6075fff700 1 -- start start 2023-10-11T16:32:12.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f6075fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6070100240 0x7f60701981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f6075fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6070101430 0x7f6070198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f6075fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070101dc0 0x7f607019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f6075fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607019fd50 con 0x7f6070101dc0 2023-10-11T16:32:12.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f6075fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607019fe90 con 0x7f6070101430 2023-10-11T16:32:12.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f6075fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60701a0020 con 0x7f6070100240 2023-10-11T16:32:12.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f606ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070101dc0 0x7f607019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f606ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070101dc0 0x7f607019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47064/0 (socket says 172.21.15.111:47064) 2023-10-11T16:32:12.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f606ffff700 1 -- 172.21.15.111:0/1231363141 learned_addr learned my addr 172.21.15.111:0/1231363141 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6070100240 0x7f60701981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.033+0000 7f606effd700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6070101430 0x7f6070198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606ffff700 1 -- 172.21.15.111:0/1231363141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6070100240 msgr2=0x7f60701981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606ffff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6070100240 0x7f60701981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606ffff700 1 -- 172.21.15.111:0/1231363141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6070101430 msgr2=0x7f6070198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606ffff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6070101430 0x7f6070198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606ffff700 1 -- 172.21.15.111:0/1231363141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60600095f0 con 0x7f6070101dc0 2023-10-11T16:32:12.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606ffff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070101dc0 0x7f607019f780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f606800ef30 tx=0x7f606800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606cff9700 1 -- 172.21.15.111:0/1231363141 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068018070 con 0x7f6070101dc0 2023-10-11T16:32:12.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606cff9700 1 -- 172.21.15.111:0/1231363141 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f606800f040 con 0x7f6070101dc0 2023-10-11T16:32:12.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f6075fff700 1 -- 172.21.15.111:0/1231363141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607010bbf0 con 0x7f6070101dc0 2023-10-11T16:32:12.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f606cff9700 1 -- 172.21.15.111:0/1231363141 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606800cd00 con 0x7f6070101dc0 2023-10-11T16:32:12.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.034+0000 7f6075fff700 1 -- 172.21.15.111:0/1231363141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607010ba60 con 0x7f6070101dc0 2023-10-11T16:32:12.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.037+0000 7f606cff9700 1 -- 172.21.15.111:0/1231363141 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f6068003a00 con 0x7f6070101dc0 2023-10-11T16:32:12.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f606cff9700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f60580a11d0 0x7f60580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f6089919b80 1 Processor -- start 2023-10-11T16:32:12.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f606cff9700 1 -- 172.21.15.111:0/1231363141 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f60680cc7e0 con 0x7f6070101dc0 2023-10-11T16:32:12.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f6089919b80 1 -- start start 2023-10-11T16:32:12.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f60580a11d0 0x7f60580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f6089919b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce339fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f6089919b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce33aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f6089919b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f6089919b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce23f010 con 0x5617ce33a500 2023-10-11T16:32:12.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f6089919b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce291850 con 0x5617ce339bb0 2023-10-11T16:32:12.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.039+0000 7f6089919b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce28ace0 con 0x5617ce33b0c0 2023-10-11T16:32:12.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce33aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce33aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47072/0 (socket says 172.21.15.111:47072) 2023-10-11T16:32:12.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606effd700 1 -- 172.21.15.111:0/775315230 learned_addr learned my addr 172.21.15.111:0/775315230 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f60580a11d0 0x7f60580a3680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f606000b2e0 tx=0x7f6060005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606f7fe700 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce339fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606ffff700 1 -- 172.21.15.111:0/775315230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 msgr2=0x5617ce339fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606ffff700 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce339fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606ffff700 1 -- 172.21.15.111:0/775315230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 msgr2=0x5617ce33aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606ffff700 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce33aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606ffff700 1 -- 172.21.15.111:0/775315230 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617ce10c5e0 con 0x5617ce33b0c0 2023-10-11T16:32:12.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606effd700 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce33aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:12.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.040+0000 7f606ffff700 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33b530 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f60680c9eb0 tx=0x7f60680996f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb2362a133b6faed server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6055ffb700 1 -- 172.21.15.111:0/775315230 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068004ad0 con 0x5617ce33b0c0 2023-10-11T16:32:12.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6055ffb700 1 -- 172.21.15.111:0/775315230 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f606800ce60 con 0x5617ce33b0c0 2023-10-11T16:32:12.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6055ffb700 1 -- 172.21.15.111:0/775315230 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60680cc450 con 0x5617ce33b0c0 2023-10-11T16:32:12.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6089919b80 1 -- 172.21.15.111:0/775315230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 msgr2=0x5617ce33b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6089919b80 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33b530 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f60680c9eb0 tx=0x7f60680996f0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6089919b80 1 -- 172.21.15.111:0/775315230 shutdown_connections 2023-10-11T16:32:12.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6089919b80 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33b530 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6089919b80 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce339fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6089919b80 1 --2- 172.21.15.111:0/775315230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce33aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.041+0000 7f6089919b80 1 -- 172.21.15.111:0/775315230 >> 172.21.15.111:0/775315230 conn(0x5617ce32f310 msgr2=0x5617ce334620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.042+0000 7f6089919b80 1 -- 172.21.15.111:0/775315230 shutdown_connections 2023-10-11T16:32:12.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.042+0000 7f6089919b80 1 -- 172.21.15.111:0/775315230 wait complete. 2023-10-11T16:32:12.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.042+0000 7f6089919b80 1 -- start start 2023-10-11T16:32:12.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.042+0000 7f6089919b80 1 Processor -- start 2023-10-11T16:32:12.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f6089919b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce3434b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f6089919b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce3439f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f6089919b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce34aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f6089919b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce22c890 con 0x5617ce33a500 2023-10-11T16:32:12.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f6089919b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce22c530 con 0x5617ce339bb0 2023-10-11T16:32:12.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f6089919b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce10e8d0 con 0x5617ce33b0c0 2023-10-11T16:32:12.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f6089919b80 10 client.0 _ll_register_callbacks cb 0x5617ce24a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:12.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f606effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce3439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f606effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce3439f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47074/0 (socket says 172.21.15.111:47074) 2023-10-11T16:32:12.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f606effd700 1 -- 172.21.15.111:0/1408009782 learned_addr learned my addr 172.21.15.111:0/1408009782 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.043+0000 7f606ffff700 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce34aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce3434b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f606ffff700 1 -- 172.21.15.111:0/1408009782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 msgr2=0x5617ce3434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f606ffff700 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce3434b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f606ffff700 1 -- 172.21.15.111:0/1408009782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 msgr2=0x5617ce3439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f606ffff700 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce3439f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f606ffff700 1 -- 172.21.15.111:0/1408009782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607010ba60 con 0x5617ce33b0c0 2023-10-11T16:32:12.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f606effd700 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce3439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:12.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f606ffff700 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce34aa30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f606809da40 tx=0x7f606800cfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef73e7dd872b2648 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f604effd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:32:12.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f604effd700 1 -- 172.21.15.111:0/1408009782 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60680b9270 con 0x5617ce33b0c0 2023-10-11T16:32:12.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f6089919b80 1 -- 172.21.15.111:0/1408009782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617ce32cad0 con 0x5617ce33b0c0 2023-10-11T16:32:12.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.044+0000 7f604effd700 1 -- 172.21.15.111:0/1408009782 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f606809d880 con 0x5617ce33b0c0 2023-10-11T16:32:12.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f604effd700 1 -- 172.21.15.111:0/1408009782 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6068099ab0 con 0x5617ce33b0c0 2023-10-11T16:32:12.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f6089919b80 10 client.25333 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:12.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f6089919b80 1 -- 172.21.15.111:0/1408009782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5617ce251140 con 0x5617ce33b0c0 2023-10-11T16:32:12.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f604effd700 1 -- 172.21.15.111:0/1408009782 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5617ce32cad0 con 0x5617ce33b0c0 2023-10-11T16:32:12.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f604effd700 1 -- 172.21.15.111:0/1408009782 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f60680a1860 con 0x5617ce33b0c0 2023-10-11T16:32:12.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f6089919b80 10 client.25333 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:12.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f6089919b80 10 client.25333 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-11T16:32:12.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f6089919b80 20 client.25333 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:12.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f6089919b80 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5617ce34e7d0 0x5617ce34d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f6089919b80 4 client.25333 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:32:12.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f6089919b80 1 -- 172.21.15.111:0/1408009782 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617ce24d050 con 0x5617ce34e7d0 2023-10-11T16:32:12.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.045+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5617ce34e7d0 0x5617ce34d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.046+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5617ce34e7d0 0x5617ce34d960 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75539fb0b4e57cc2 server_cookie=b25f72a96746a5a7 in_seq=0 out_seq=0 2023-10-11T16:32:12.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.046+0000 7f604effd700 10 client.25333 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:32:12.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f604effd700 1 -- 172.21.15.111:0/1408009782 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5617ce24d050 con 0x5617ce34e7d0 2023-10-11T16:32:12.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f604effd700 10 client.25333 handle_command_reply: tid=0 2023-10-11T16:32:12.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 client.25333 shutdown 2023-10-11T16:32:12.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 -- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 msgr2=0x5617ce34aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce34aa30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f606809da40 tx=0x7f606800cfd0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 -- 172.21.15.111:0/1408009782 shutdown_connections 2023-10-11T16:32:12.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5617ce34e7d0 0x5617ce34d960 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce34aa30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce3434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f604effd700 0 client.25333 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:32:12.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 --2- 172.21.15.111:0/1408009782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce3439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 -- 172.21.15.111:0/1408009782 >> 172.21.15.111:0/1408009782 conn(0x5617ce32f310 msgr2=0x5617ce333d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 -- 172.21.15.111:0/1408009782 shutdown_connections 2023-10-11T16:32:12.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 1 -- 172.21.15.111:0/1408009782 wait complete. 2023-10-11T16:32:12.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.047+0000 7f6089919b80 20 client.25333 trim_cache size 0 max 16384 2023-10-11T16:32:12.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.065+0000 7f6089919b80 1 Processor -- start 2023-10-11T16:32:12.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.065+0000 7f6089919b80 1 -- start start 2023-10-11T16:32:12.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f6089919b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce342180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f6089919b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce34a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f6089919b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f6089919b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce234730 con 0x5617ce33a500 2023-10-11T16:32:12.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f6089919b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce196ed0 con 0x5617ce339bb0 2023-10-11T16:32:12.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f6089919b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce226e10 con 0x5617ce33b0c0 2023-10-11T16:32:12.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce34a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606f7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce342180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606f7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce342180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50012/0 (socket says 172.21.15.111:50012) 2023-10-11T16:32:12.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606f7fe700 1 -- 172.21.15.111:0/1010730041 learned_addr learned my addr 172.21.15.111:0/1010730041 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606ffff700 1 -- 172.21.15.111:0/1010730041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 msgr2=0x5617ce342180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606ffff700 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce342180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606ffff700 1 -- 172.21.15.111:0/1010730041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 msgr2=0x5617ce34a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606ffff700 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce34a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606ffff700 1 -- 172.21.15.111:0/1010730041 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617ce250e90 con 0x5617ce33b0c0 2023-10-11T16:32:12.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce342180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:12.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.066+0000 7f606effd700 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce34a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:12.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f606ffff700 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33e000 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6068009f40 tx=0x7f606801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdc4a4c2fb2a3936 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f60567fc700 1 -- 172.21.15.111:0/1010730041 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606801bb40 con 0x5617ce33b0c0 2023-10-11T16:32:12.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f60567fc700 1 -- 172.21.15.111:0/1010730041 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60680c8060 con 0x5617ce33b0c0 2023-10-11T16:32:12.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f60567fc700 1 -- 172.21.15.111:0/1010730041 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068098d40 con 0x5617ce33b0c0 2023-10-11T16:32:12.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f6089919b80 1 -- 172.21.15.111:0/1010730041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 msgr2=0x5617ce33e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f6089919b80 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33e000 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6068009f40 tx=0x7f606801c000 comp rx=0 tx=0).stop 2023-10-11T16:32:12.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f6089919b80 1 -- 172.21.15.111:0/1010730041 shutdown_connections 2023-10-11T16:32:12.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f6089919b80 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33b0c0 0x5617ce33e000 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f6089919b80 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce342180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.067+0000 7f6089919b80 1 --2- 172.21.15.111:0/1010730041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33a500 0x5617ce34a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.068+0000 7f6089919b80 1 -- 172.21.15.111:0/1010730041 >> 172.21.15.111:0/1010730041 conn(0x5617ce32f310 msgr2=0x5617ce356f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.068+0000 7f6089919b80 1 -- 172.21.15.111:0/1010730041 shutdown_connections 2023-10-11T16:32:12.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.068+0000 7f6089919b80 1 -- 172.21.15.111:0/1010730041 wait complete. 2023-10-11T16:32:12.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.068+0000 7f6089919b80 1 -- start start 2023-10-11T16:32:12.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.068+0000 7f6089919b80 1 Processor -- start 2023-10-11T16:32:12.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f6089919b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce345030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f6089919b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33a500 0x5617ce345570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f6089919b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33b0c0 0x5617ce34a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f6089919b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce34a900 con 0x5617ce33b0c0 2023-10-11T16:32:12.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f6089919b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce34aa70 con 0x5617ce339bb0 2023-10-11T16:32:12.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f6089919b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617ce34ac30 con 0x5617ce33a500 2023-10-11T16:32:12.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f6089919b80 10 client.0 _ll_register_callbacks cb 0x5617ce24a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:12.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33a500 0x5617ce345570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606f7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce345030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33b0c0 0x5617ce34a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33a500 0x5617ce345570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:60068/0 (socket says 172.21.15.111:60068) 2023-10-11T16:32:12.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606effd700 1 -- 172.21.15.111:0/1390547333 learned_addr learned my addr 172.21.15.111:0/1390547333 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606ffff700 1 -- 172.21.15.111:0/1390547333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33a500 msgr2=0x5617ce345570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606ffff700 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33a500 0x5617ce345570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606ffff700 1 -- 172.21.15.111:0/1390547333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 msgr2=0x5617ce345030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606ffff700 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce345030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606ffff700 1 -- 172.21.15.111:0/1390547333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6070108fb0 con 0x5617ce33b0c0 2023-10-11T16:32:12.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce345030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:12.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.069+0000 7f606effd700 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33a500 0x5617ce345570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:12.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f606ffff700 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33b0c0 0x5617ce34a3c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f606800c6f0 tx=0x7f606800cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bd2bc24215f348 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f604effd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:12.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f604effd700 1 -- 172.21.15.111:0/1390547333 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606801bb40 con 0x5617ce33b0c0 2023-10-11T16:32:12.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f604effd700 1 -- 172.21.15.111:0/1390547333 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f60680a1c10 con 0x5617ce33b0c0 2023-10-11T16:32:12.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 1 -- 172.21.15.111:0/1390547333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f606800ce60 con 0x5617ce33b0c0 2023-10-11T16:32:12.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f604effd700 1 -- 172.21.15.111:0/1390547333 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f606801bce0 con 0x5617ce33b0c0 2023-10-11T16:32:12.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 10 client.15276 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:12.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 1 -- 172.21.15.111:0/1390547333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f6064012450 con 0x5617ce33b0c0 2023-10-11T16:32:12.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f604effd700 1 -- 172.21.15.111:0/1390547333 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f606800ce60 con 0x5617ce33b0c0 2023-10-11T16:32:12.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f604effd700 1 -- 172.21.15.111:0/1390547333 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f6068095640 con 0x5617ce33b0c0 2023-10-11T16:32:12.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 10 client.15276 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:12.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 10 client.15276 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-11T16:32:12.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 20 client.15276 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:12.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5617ce3a6670 0x5617ce3a3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 4 client.15276 mds_command: new command op to 24532 tid=0[{"prefix": "status"}] 2023-10-11T16:32:12.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.070+0000 7f6089919b80 1 -- 172.21.15.111:0/1390547333 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5617ce259610 con 0x5617ce3a6670 2023-10-11T16:32:12.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.071+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5617ce3a6670 0x5617ce3a3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.071+0000 7f604effd700 10 client.15276 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:32:12.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.071+0000 7f606f7fe700 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5617ce3a6670 0x5617ce3a3020 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=434a57d87e7279b4 server_cookie=64c2d26651fcb315 in_seq=0 out_seq=0 2023-10-11T16:32:12.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f604effd700 1 -- 172.21.15.111:0/1390547333 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5617ce259610 con 0x5617ce3a6670 2023-10-11T16:32:12.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f604effd700 10 client.15276 handle_command_reply: tid=0 2023-10-11T16:32:12.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 client.15276 shutdown 2023-10-11T16:32:12.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 -- 172.21.15.111:0/1390547333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33b0c0 msgr2=0x5617ce34a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33b0c0 0x5617ce34a3c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f606800c6f0 tx=0x7f606800cfd0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 -- 172.21.15.111:0/1390547333 shutdown_connections 2023-10-11T16:32:12.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5617ce3a6670 0x5617ce3a3020 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5617ce33b0c0 0x5617ce34a3c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f604effd700 0 client.15276 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:32:12.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5617ce339bb0 0x5617ce345030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 --2- 172.21.15.111:0/1390547333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5617ce33a500 0x5617ce345570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 -- 172.21.15.111:0/1390547333 >> 172.21.15.111:0/1390547333 conn(0x5617ce32f310 msgr2=0x5617ce3414d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 -- 172.21.15.111:0/1390547333 shutdown_connections 2023-10-11T16:32:12.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 1 -- 172.21.15.111:0/1390547333 wait complete. 2023-10-11T16:32:12.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.072+0000 7f6089919b80 20 client.15276 trim_cache size 0 max 16384 2023-10-11T16:32:12.091 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:32:12.091 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:32:12.091 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 2, 2023-10-11T16:32:12.091 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 24532, 2023-10-11T16:32:12.092 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-10-11T16:32:12.092 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-10-11T16:32:12.092 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-10-11T16:32:12.092 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 94.907374172999994, 2023-10-11T16:32:12.092 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 26, 2023-10-11T16:32:12.093 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 91, 2023-10-11T16:32:12.093 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 89, 2023-10-11T16:32:12.093 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 139.42023725000001 2023-10-11T16:32:12.093 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:32:12.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 -- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f60580a11d0 msgr2=0x7f60580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f60580a11d0 0x7f60580a3680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f606000b2e0 tx=0x7f6060005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 -- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070101dc0 msgr2=0x7f607019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070101dc0 0x7f607019f780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f606800ef30 tx=0x7f606800c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 -- 172.21.15.111:0/1231363141 shutdown_connections 2023-10-11T16:32:12.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f60580a11d0 0x7f60580a3680 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6070101dc0 0x7f607019f780 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6070100240 0x7f60701981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 --2- 172.21.15.111:0/1231363141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6070101430 0x7f6070198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 -- 172.21.15.111:0/1231363141 >> 172.21.15.111:0/1231363141 conn(0x7f60700fb820 msgr2=0x7f6070106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 -- 172.21.15.111:0/1231363141 shutdown_connections 2023-10-11T16:32:12.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.073+0000 7f6075fff700 1 -- 172.21.15.111:0/1231363141 wait complete. 2023-10-11T16:32:12.097 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-10-11T16:32:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 -- 172.21.15.111:0/328296131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb650101420 msgr2=0x7fb650101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 --2- 172.21.15.111:0/328296131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb650101420 0x7fb650101870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb644009990 tx=0x7fb644009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 -- 172.21.15.111:0/328296131 shutdown_connections 2023-10-11T16:32:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 --2- 172.21.15.111:0/328296131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb650101db0 0x7fb650108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 --2- 172.21.15.111:0/328296131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb650100230 0x7fb650100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 --2- 172.21.15.111:0/328296131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb650101420 0x7fb650101870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 -- 172.21.15.111:0/328296131 >> 172.21.15.111:0/328296131 conn(0x7fb6500fb810 msgr2=0x7fb6500fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 -- 172.21.15.111:0/328296131 shutdown_connections 2023-10-11T16:32:12.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.297+0000 7fb6553ff700 1 -- 172.21.15.111:0/328296131 wait complete. 2023-10-11T16:32:12.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.298+0000 7fb6553ff700 1 Processor -- start 2023-10-11T16:32:12.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.298+0000 7fb6553ff700 1 -- start start 2023-10-11T16:32:12.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.298+0000 7fb6553ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb650100230 0x7fb6501981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.298+0000 7fb6553ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb650101420 0x7fb650198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.298+0000 7fb6553ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb650101db0 0x7fb65019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.298+0000 7fb6553ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65019fd50 con 0x7fb650101420 2023-10-11T16:32:12.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.298+0000 7fb6553ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65019fe90 con 0x7fb650100230 2023-10-11T16:32:12.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.298+0000 7fb6553ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6501a0020 con 0x7fb650101db0 2023-10-11T16:32:12.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb650101420 0x7fb650198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb650101420 0x7fb650198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47118/0 (socket says 172.21.15.111:47118) 2023-10-11T16:32:12.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 -- 172.21.15.111:0/3088728388 learned_addr learned my addr 172.21.15.111:0/3088728388 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64effd700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb650100230 0x7fb6501981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb650101db0 0x7fb65019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 -- 172.21.15.111:0/3088728388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb650101db0 msgr2=0x7fb65019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb650101db0 0x7fb65019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 -- 172.21.15.111:0/3088728388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb650100230 msgr2=0x7fb6501981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb650100230 0x7fb6501981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 -- 172.21.15.111:0/3088728388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb640009520 con 0x7fb650101420 2023-10-11T16:32:12.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb650101420 0x7fb650198710 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb64400b680 tx=0x7fb644005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb637fff700 1 -- 172.21.15.111:0/3088728388 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64401d070 con 0x7fb650101420 2023-10-11T16:32:12.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb637fff700 1 -- 172.21.15.111:0/3088728388 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb644022470 con 0x7fb650101420 2023-10-11T16:32:12.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.299+0000 7fb6553ff700 1 -- 172.21.15.111:0/3088728388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6440095f0 con 0x7fb650101420 2023-10-11T16:32:12.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.300+0000 7fb637fff700 1 -- 172.21.15.111:0/3088728388 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644017650 con 0x7fb650101420 2023-10-11T16:32:12.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.300+0000 7fb6553ff700 1 -- 172.21.15.111:0/3088728388 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65010bb80 con 0x7fb650101420 2023-10-11T16:32:12.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.303+0000 7fb637fff700 1 -- 172.21.15.111:0/3088728388 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fb6440225e0 con 0x7fb650101420 2023-10-11T16:32:12.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb637fff700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb6380a11d0 0x7fb6380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb668cf0b80 1 Processor -- start 2023-10-11T16:32:12.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb668cf0b80 1 -- start start 2023-10-11T16:32:12.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb637fff700 1 -- 172.21.15.111:0/3088728388 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fb6440ce150 con 0x7fb650101420 2023-10-11T16:32:12.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb64effd700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb6380a11d0 0x7fb6380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea1ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea20ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea21530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fe925010 con 0x55c4fea210c0 2023-10-11T16:32:12.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fe977850 con 0x55c4fea20500 2023-10-11T16:32:12.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.304+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fe970ce0 con 0x55c4fea1fbb0 2023-10-11T16:32:12.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea20ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea20ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50036/0 (socket says 172.21.15.111:50036) 2023-10-11T16:32:12.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 -- 172.21.15.111:0/4081618226 learned_addr learned my addr 172.21.15.111:0/4081618226 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea21530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64effd700 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea1ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 -- 172.21.15.111:0/4081618226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 msgr2=0x55c4fea1ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea1ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 -- 172.21.15.111:0/4081618226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 msgr2=0x55c4fea21530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea21530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 -- 172.21.15.111:0/4081618226 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4fe7f25e0 con 0x55c4fea20500 2023-10-11T16:32:12.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64effd700 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea1ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:12.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64effd700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb6380a11d0 0x7fb6380a3680 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb650101280 tx=0x7fb640011720 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea20ae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb64409b150 tx=0x7fb644022c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdb93bdd4130c864 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.305+0000 7fb6357fa700 1 -- 172.21.15.111:0/4081618226 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64409b1c0 con 0x55c4fea20500 2023-10-11T16:32:12.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb6357fa700 1 -- 172.21.15.111:0/4081618226 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb64409f1f0 con 0x55c4fea20500 2023-10-11T16:32:12.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb6357fa700 1 -- 172.21.15.111:0/4081618226 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64409b420 con 0x55c4fea20500 2023-10-11T16:32:12.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 -- 172.21.15.111:0/4081618226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 msgr2=0x55c4fea20ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea20ae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb64409b150 tx=0x7fb644022c80 comp rx=0 tx=0).stop 2023-10-11T16:32:12.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 -- 172.21.15.111:0/4081618226 shutdown_connections 2023-10-11T16:32:12.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea21530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea1ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/4081618226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea20ae0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 -- 172.21.15.111:0/4081618226 >> 172.21.15.111:0/4081618226 conn(0x55c4fea15310 msgr2=0x55c4fea1a620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 -- 172.21.15.111:0/4081618226 shutdown_connections 2023-10-11T16:32:12.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.306+0000 7fb668cf0b80 1 -- 172.21.15.111:0/4081618226 wait complete. 2023-10-11T16:32:12.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.307+0000 7fb668cf0b80 1 -- start start 2023-10-11T16:32:12.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.307+0000 7fb668cf0b80 1 Processor -- start 2023-10-11T16:32:12.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea1fbb0 0x55c4fea29180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 0x55c4fea296c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea210c0 0x55c4fea30700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fe912890 con 0x55c4fea1fbb0 2023-10-11T16:32:12.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fe912530 con 0x55c4fea210c0 2023-10-11T16:32:12.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fe7f48d0 con 0x55c4fea20500 2023-10-11T16:32:12.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb668cf0b80 10 client.0 _ll_register_callbacks cb 0x55c4fe930730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:12.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb64effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea1fbb0 0x55c4fea29180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb64effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea1fbb0 0x55c4fea29180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47144/0 (socket says 172.21.15.111:47144) 2023-10-11T16:32:12.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb64effd700 1 -- 172.21.15.111:0/2241913062 learned_addr learned my addr 172.21.15.111:0/2241913062 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea210c0 0x55c4fea30700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.308+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 0x55c4fea296c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb64effd700 1 -- 172.21.15.111:0/2241913062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 msgr2=0x55c4fea296c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb64effd700 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 0x55c4fea296c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb64effd700 1 -- 172.21.15.111:0/2241913062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea210c0 msgr2=0x55c4fea30700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb64effd700 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea210c0 0x55c4fea30700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb64effd700 1 -- 172.21.15.111:0/2241913062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65010bb80 con 0x55c4fea1fbb0 2023-10-11T16:32:12.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb64effd700 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea1fbb0 0x55c4fea29180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb64000fab0 tx=0x7fb6400047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff82a6bbb71a3e92 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb632ff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:12.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb632ff5700 1 -- 172.21.15.111:0/2241913062 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64001eae0 con 0x55c4fea1fbb0 2023-10-11T16:32:12.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb668cf0b80 1 -- 172.21.15.111:0/2241913062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4fea12ad0 con 0x55c4fea1fbb0 2023-10-11T16:32:12.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.309+0000 7fb632ff5700 1 -- 172.21.15.111:0/2241913062 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fb64001ec40 con 0x55c4fea1fbb0 2023-10-11T16:32:12.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb632ff5700 1 -- 172.21.15.111:0/2241913062 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb64001ee00 con 0x55c4fea1fbb0 2023-10-11T16:32:12.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb668cf0b80 10 client.15288 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:12.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb668cf0b80 1 -- 172.21.15.111:0/2241913062 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c4fe937140 con 0x55c4fea1fbb0 2023-10-11T16:32:12.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb632ff5700 1 -- 172.21.15.111:0/2241913062 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c4fea12ad0 con 0x55c4fea1fbb0 2023-10-11T16:32:12.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb632ff5700 1 -- 172.21.15.111:0/2241913062 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fb64000e3b0 con 0x55c4fea1fbb0 2023-10-11T16:32:12.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb668cf0b80 10 client.15288 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:12.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb668cf0b80 10 client.15288 resolve_mds: resolved name 'i' to daemon mds.i 2023-10-11T16:32:12.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb668cf0b80 20 client.15288 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:12.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55c4fea344b0 0x55c4fea33860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb668cf0b80 4 client.15288 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:32:12.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.310+0000 7fb668cf0b80 1 -- 172.21.15.111:0/2241913062 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4fe933050 con 0x55c4fea344b0 2023-10-11T16:32:12.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.311+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55c4fea344b0 0x55c4fea33860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.311+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55c4fea344b0 0x55c4fea33860 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=420cc5f720ffe8bf server_cookie=861f065aa53a7ac7 in_seq=0 out_seq=0 2023-10-11T16:32:12.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.311+0000 7fb632ff5700 10 client.15288 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:32:12.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb632ff5700 1 -- 172.21.15.111:0/2241913062 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c4fe933050 con 0x55c4fea344b0 2023-10-11T16:32:12.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb632ff5700 10 client.15288 handle_command_reply: tid=0 2023-10-11T16:32:12.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 client.15288 shutdown 2023-10-11T16:32:12.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 -- 172.21.15.111:0/2241913062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea1fbb0 msgr2=0x55c4fea29180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea1fbb0 0x55c4fea29180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb64000fab0 tx=0x7fb6400047c0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 -- 172.21.15.111:0/2241913062 shutdown_connections 2023-10-11T16:32:12.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55c4fea344b0 0x55c4fea33860 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea210c0 0x55c4fea30700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea1fbb0 0x55c4fea29180 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/2241913062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 0x55c4fea296c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb632ff5700 0 client.15288 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:32:12.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 -- 172.21.15.111:0/2241913062 >> 172.21.15.111:0/2241913062 conn(0x55c4fea15310 msgr2=0x55c4fea19d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 -- 172.21.15.111:0/2241913062 shutdown_connections 2023-10-11T16:32:12.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.312+0000 7fb668cf0b80 1 -- 172.21.15.111:0/2241913062 wait complete. 2023-10-11T16:32:12.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.313+0000 7fb668cf0b80 20 client.15288 trim_cache size 0 max 16384 2023-10-11T16:32:12.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb668cf0b80 1 Processor -- start 2023-10-11T16:32:12.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb668cf0b80 1 -- start start 2023-10-11T16:32:12.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea1fbb0 0x55c4fea27380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 0x55c4fea278c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea306c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fea219b0 con 0x55c4fea210c0 2023-10-11T16:32:12.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fea2a370 con 0x55c4fea1fbb0 2023-10-11T16:32:12.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fe87ced0 con 0x55c4fea20500 2023-10-11T16:32:12.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.331+0000 7fb64f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea306c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47160/0 (socket says 172.21.15.111:47160) 2023-10-11T16:32:12.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64f7fe700 1 -- 172.21.15.111:0/454587230 learned_addr learned my addr 172.21.15.111:0/454587230 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 0x55c4fea278c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64effd700 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea1fbb0 0x55c4fea27380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64f7fe700 1 -- 172.21.15.111:0/454587230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 msgr2=0x55c4fea278c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 0x55c4fea278c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64f7fe700 1 -- 172.21.15.111:0/454587230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea1fbb0 msgr2=0x55c4fea27380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea1fbb0 0x55c4fea27380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64f7fe700 1 -- 172.21.15.111:0/454587230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4fe936e90 con 0x55c4fea210c0 2023-10-11T16:32:12.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64effd700 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea1fbb0 0x55c4fea27380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:12.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea306c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb648011960 tx=0x7fb648009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2457a7ef5c6d7b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb635ffb700 1 -- 172.21.15.111:0/454587230 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800e040 con 0x55c4fea210c0 2023-10-11T16:32:12.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb635ffb700 1 -- 172.21.15.111:0/454587230 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb64800f040 con 0x55c4fea210c0 2023-10-11T16:32:12.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.332+0000 7fb635ffb700 1 -- 172.21.15.111:0/454587230 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648017c50 con 0x55c4fea210c0 2023-10-11T16:32:12.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 -- 172.21.15.111:0/454587230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 msgr2=0x55c4fea306c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea306c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb648011960 tx=0x7fb648009920 comp rx=0 tx=0).stop 2023-10-11T16:32:12.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 -- 172.21.15.111:0/454587230 shutdown_connections 2023-10-11T16:32:12.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea306c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea1fbb0 0x55c4fea27380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/454587230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea20500 0x55c4fea278c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 -- 172.21.15.111:0/454587230 >> 172.21.15.111:0/454587230 conn(0x55c4fea15310 msgr2=0x55c4fea27f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 -- 172.21.15.111:0/454587230 shutdown_connections 2023-10-11T16:32:12.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.333+0000 7fb668cf0b80 1 -- 172.21.15.111:0/454587230 wait complete. 2023-10-11T16:32:12.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 1 -- start start 2023-10-11T16:32:12.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 1 Processor -- start 2023-10-11T16:32:12.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea2b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea2b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea883b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fea309c0 con 0x55c4fea210c0 2023-10-11T16:32:12.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fea30b30 con 0x55c4fea20500 2023-10-11T16:32:12.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4fea888f0 con 0x55c4fea1fbb0 2023-10-11T16:32:12.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb668cf0b80 10 client.0 _ll_register_callbacks cb 0x7fb64000f880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:12.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb64f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb64f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea883b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47162/0 (socket says 172.21.15.111:47162) 2023-10-11T16:32:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb64f7fe700 1 -- 172.21.15.111:0/3413317406 learned_addr learned my addr 172.21.15.111:0/3413317406 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.334+0000 7fb64effd700 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea2b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb64e7fc700 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea2b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb64f7fe700 1 -- 172.21.15.111:0/3413317406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 msgr2=0x55c4fea2b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea2b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb64f7fe700 1 -- 172.21.15.111:0/3413317406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 msgr2=0x55c4fea2b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea2b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb64f7fe700 1 -- 172.21.15.111:0/3413317406 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb648011610 con 0x55c4fea210c0 2023-10-11T16:32:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb64f7fe700 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea883b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb64800b010 tx=0x7fb648003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3931ba5cfabe33d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb632ff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb632ff5700 1 -- 172.21.15.111:0/3413317406 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800e050 con 0x55c4fea210c0 2023-10-11T16:32:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb632ff5700 1 -- 172.21.15.111:0/3413317406 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fb6480093a0 con 0x55c4fea210c0 2023-10-11T16:32:12.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb668cf0b80 1 -- 172.21.15.111:0/3413317406 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb64409f1f0 con 0x55c4fea210c0 2023-10-11T16:32:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb632ff5700 1 -- 172.21.15.111:0/3413317406 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb648017610 con 0x55c4fea210c0 2023-10-11T16:32:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.335+0000 7fb668cf0b80 10 client.15294 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb668cf0b80 1 -- 172.21.15.111:0/3413317406 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c4fe787c10 con 0x55c4fea210c0 2023-10-11T16:32:12.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb632ff5700 1 -- 172.21.15.111:0/3413317406 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb64409f1f0 con 0x55c4fea210c0 2023-10-11T16:32:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb632ff5700 1 -- 172.21.15.111:0/3413317406 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fb648007500 con 0x55c4fea210c0 2023-10-11T16:32:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb668cf0b80 10 client.15294 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb668cf0b80 10 client.15294 resolve_mds: resolved name 'i' to daemon mds.i 2023-10-11T16:32:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb668cf0b80 20 client.15294 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:12.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55c4fea8d540 0x55c4fea89e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb668cf0b80 4 client.15294 mds_command: new command op to 24550 tid=0[{"prefix": "status"}] 2023-10-11T16:32:12.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb668cf0b80 1 -- 172.21.15.111:0/3413317406 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c4fe93f610 con 0x55c4fea8d540 2023-10-11T16:32:12.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.336+0000 7fb64effd700 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55c4fea8d540 0x55c4fea89e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb64effd700 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55c4fea8d540 0x55c4fea89e60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=232c7eea6fcf1504 server_cookie=7b1d47ea12454e2b in_seq=0 out_seq=0 2023-10-11T16:32:12.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb632ff5700 10 client.15294 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:32:12.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb632ff5700 1 -- 172.21.15.111:0/3413317406 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55c4fe93f610 con 0x55c4fea8d540 2023-10-11T16:32:12.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb632ff5700 10 client.15294 handle_command_reply: tid=0 2023-10-11T16:32:12.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 client.15294 shutdown 2023-10-11T16:32:12.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 -- 172.21.15.111:0/3413317406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 msgr2=0x55c4fea883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea883b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb64800b010 tx=0x7fb648003d60 comp rx=0 tx=0).stop 2023-10-11T16:32:12.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 -- 172.21.15.111:0/3413317406 shutdown_connections 2023-10-11T16:32:12.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55c4fea8d540 0x55c4fea89e60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55c4fea210c0 0x55c4fea883b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c4fea1fbb0 0x55c4fea2b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb632ff5700 0 client.15294 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:32:12.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 --2- 172.21.15.111:0/3413317406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4fea20500 0x55c4fea2b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.337+0000 7fb668cf0b80 1 -- 172.21.15.111:0/3413317406 >> 172.21.15.111:0/3413317406 conn(0x55c4fea15310 msgr2=0x55c4fea33860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb668cf0b80 1 -- 172.21.15.111:0/3413317406 shutdown_connections 2023-10-11T16:32:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb668cf0b80 1 -- 172.21.15.111:0/3413317406 wait complete. 2023-10-11T16:32:12.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb668cf0b80 20 client.15294 trim_cache size 0 max 16384 2023-10-11T16:32:12.360 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:32:12.361 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:32:12.361 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 1, 2023-10-11T16:32:12.361 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 24550, 2023-10-11T16:32:12.361 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-10-11T16:32:12.362 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-10-11T16:32:12.362 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-10-11T16:32:12.362 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 96.182121847000005, 2023-10-11T16:32:12.362 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 26, 2023-10-11T16:32:12.363 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 91, 2023-10-11T16:32:12.363 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 89, 2023-10-11T16:32:12.363 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 135.19943351399999 2023-10-11T16:32:12.363 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:32:12.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 -- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb6380a11d0 msgr2=0x7fb6380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb6380a11d0 0x7fb6380a3680 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb650101280 tx=0x7fb640011720 comp rx=0 tx=0).stop 2023-10-11T16:32:12.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 -- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb650101420 msgr2=0x7fb650198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb650101420 0x7fb650198710 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb64400b680 tx=0x7fb644005740 comp rx=0 tx=0).stop 2023-10-11T16:32:12.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 -- 172.21.15.111:0/3088728388 shutdown_connections 2023-10-11T16:32:12.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fb6380a11d0 0x7fb6380a3680 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb650101db0 0x7fb65019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb650100230 0x7fb6501981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 --2- 172.21.15.111:0/3088728388 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb650101420 0x7fb650198710 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.338+0000 7fb6553ff700 1 -- 172.21.15.111:0/3088728388 >> 172.21.15.111:0/3088728388 conn(0x7fb6500fb810 msgr2=0x7fb650106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.339+0000 7fb6553ff700 1 -- 172.21.15.111:0/3088728388 shutdown_connections 2023-10-11T16:32:12.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.339+0000 7fb6553ff700 1 -- 172.21.15.111:0/3088728388 wait complete. 2023-10-11T16:32:12.367 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:12 smithi111 ceph-mon[98527]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-10-11T16:32:12.367 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:12 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2834997976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:12.367 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:12 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1636090954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:12.368 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-11T16:32:12.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:12 smithi150 ceph-mon[132217]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-10-11T16:32:12.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:12 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2834997976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:12.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:12 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1636090954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:12.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 -- 172.21.15.111:0/3174159435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c101cf0 msgr2=0x7f454c108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 --2- 172.21.15.111:0/3174159435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c101cf0 0x7f454c108590 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4544009f90 tx=0x7f454400b330 comp rx=0 tx=0).stop 2023-10-11T16:32:12.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 -- 172.21.15.111:0/3174159435 shutdown_connections 2023-10-11T16:32:12.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 --2- 172.21.15.111:0/3174159435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c101cf0 0x7f454c108590 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 --2- 172.21.15.111:0/3174159435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f454c1000b0 0x7f454c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 --2- 172.21.15.111:0/3174159435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f454c101340 0x7f454c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 -- 172.21.15.111:0/3174159435 >> 172.21.15.111:0/3174159435 conn(0x7f454c0fb6d0 msgr2=0x7f454c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 -- 172.21.15.111:0/3174159435 shutdown_connections 2023-10-11T16:32:12.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.565+0000 7f454bfff700 1 -- 172.21.15.111:0/3174159435 wait complete. 2023-10-11T16:32:12.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.566+0000 7f454bfff700 1 Processor -- start 2023-10-11T16:32:12.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.566+0000 7f454bfff700 1 -- start start 2023-10-11T16:32:12.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f454c1000b0 0x7f454c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f454c101340 0x7f454c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c101cf0 0x7f454c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c19fbe0 con 0x7f454c1000b0 2023-10-11T16:32:12.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454bfff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c19fd20 con 0x7f454c101cf0 2023-10-11T16:32:12.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c19feb0 con 0x7f454c101340 2023-10-11T16:32:12.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f454c1000b0 0x7f454c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f454c1000b0 0x7f454c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47184/0 (socket says 172.21.15.111:47184) 2023-10-11T16:32:12.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454affd700 1 -- 172.21.15.111:0/102966905 learned_addr learned my addr 172.21.15.111:0/102966905 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454a7fc700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f454c101340 0x7f454c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454b7fe700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c101cf0 0x7f454c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454affd700 1 -- 172.21.15.111:0/102966905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f454c101340 msgr2=0x7f454c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454affd700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f454c101340 0x7f454c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454affd700 1 -- 172.21.15.111:0/102966905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c101cf0 msgr2=0x7f454c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.567+0000 7f454affd700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c101cf0 0x7f454c19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.568+0000 7f454affd700 1 -- 172.21.15.111:0/102966905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544009c40 con 0x7f454c1000b0 2023-10-11T16:32:12.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.568+0000 7f454affd700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f454c1000b0 0x7f454c198060 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f453c00e9a0 tx=0x7f453c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.568+0000 7f4533fff700 1 -- 172.21.15.111:0/102966905 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c009790 con 0x7f454c1000b0 2023-10-11T16:32:12.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.568+0000 7f454bfff700 1 -- 172.21.15.111:0/102966905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c10bab0 con 0x7f454c1000b0 2023-10-11T16:32:12.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.568+0000 7f4533fff700 1 -- 172.21.15.111:0/102966905 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f453c00ce80 con 0x7f454c1000b0 2023-10-11T16:32:12.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.568+0000 7f4533fff700 1 -- 172.21.15.111:0/102966905 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c018680 con 0x7f454c1000b0 2023-10-11T16:32:12.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.568+0000 7f454bfff700 1 -- 172.21.15.111:0/102966905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c10b920 con 0x7f454c1000b0 2023-10-11T16:32:12.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:12 smithi130 ceph-mon[131859]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-10-11T16:32:12.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:12 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2834997976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:12.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:12 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1636090954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:12.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.571+0000 7f4533fff700 1 -- 172.21.15.111:0/102966905 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f453c010430 con 0x7f454c1000b0 2023-10-11T16:32:12.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.572+0000 7f4533fff700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45340a1150 0x7f45340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.572+0000 7f4533fff700 1 -- 172.21.15.111:0/102966905 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f453c014070 con 0x7f454c1000b0 2023-10-11T16:32:12.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f454a7fc700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45340a1150 0x7f45340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f4565094b80 1 Processor -- start 2023-10-11T16:32:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f4565094b80 1 -- start start 2023-10-11T16:32:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f4565094b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a8fe40 0x5595c0a90250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f4565094b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 0x5595c0a90cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f4565094b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0a91700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f454a7fc700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45340a1150 0x7f45340a3600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4540005fd0 tx=0x7f4540009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f4565094b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c0995010 con 0x5595c0a8fe40 2023-10-11T16:32:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f4565094b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c09e7850 con 0x5595c0a90790 2023-10-11T16:32:12.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.573+0000 7f4565094b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c09e0ce0 con 0x5595c0a91260 2023-10-11T16:32:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a8fe40 0x5595c0a90250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a8fe40 0x5595c0a90250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47196/0 (socket says 172.21.15.111:47196) 2023-10-11T16:32:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 -- 172.21.15.111:0/103458268 learned_addr learned my addr 172.21.15.111:0/103458268 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454b7fe700 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0a91700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454a7fc700 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 0x5595c0a90cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 -- 172.21.15.111:0/103458268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 msgr2=0x5595c0a91700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0a91700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 -- 172.21.15.111:0/103458268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 msgr2=0x5595c0a90cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 0x5595c0a90cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 -- 172.21.15.111:0/103458268 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595c08625e0 con 0x5595c0a8fe40 2023-10-11T16:32:12.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f454affd700 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a8fe40 0x5595c0a90250 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f453c099e90 tx=0x7f453c099930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537a10ab97d1f04c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f45317fa700 1 -- 172.21.15.111:0/103458268 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c09df30 con 0x5595c0a8fe40 2023-10-11T16:32:12.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f45317fa700 1 -- 172.21.15.111:0/103458268 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f453c0a2350 con 0x5595c0a8fe40 2023-10-11T16:32:12.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.574+0000 7f45317fa700 1 -- 172.21.15.111:0/103458268 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0a2120 con 0x5595c0a8fe40 2023-10-11T16:32:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 -- 172.21.15.111:0/103458268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a8fe40 msgr2=0x5595c0a90250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a8fe40 0x5595c0a90250 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f453c099e90 tx=0x7f453c099930 comp rx=0 tx=0).stop 2023-10-11T16:32:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 -- 172.21.15.111:0/103458268 shutdown_connections 2023-10-11T16:32:12.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0a91700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a8fe40 0x5595c0a90250 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 --2- 172.21.15.111:0/103458268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 0x5595c0a90cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 -- 172.21.15.111:0/103458268 >> 172.21.15.111:0/103458268 conn(0x5595c0a85560 msgr2=0x5595c0a8a800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 -- 172.21.15.111:0/103458268 shutdown_connections 2023-10-11T16:32:12.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.575+0000 7f4565094b80 1 -- 172.21.15.111:0/103458268 wait complete. 2023-10-11T16:32:12.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.576+0000 7f4565094b80 1 -- start start 2023-10-11T16:32:12.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.576+0000 7f4565094b80 1 Processor -- start 2023-10-11T16:32:12.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.576+0000 7f4565094b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a99730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.576+0000 7f4565094b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a90790 0x5595c0a99c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.576+0000 7f4565094b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.576+0000 7f4565094b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c0982890 con 0x5595c0a91260 2023-10-11T16:32:12.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.576+0000 7f4565094b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c0982530 con 0x5595c0a8fe40 2023-10-11T16:32:12.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.576+0000 7f4565094b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c08648d0 con 0x5595c0a90790 2023-10-11T16:32:12.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f4565094b80 10 client.0 _ll_register_callbacks cb 0x5595c09a0730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:12.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a99730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47198/0 (socket says 172.21.15.111:47198) 2023-10-11T16:32:12.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 -- 172.21.15.111:0/2513145950 learned_addr learned my addr 172.21.15.111:0/2513145950 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454a7fc700 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a90790 0x5595c0a99c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 -- 172.21.15.111:0/2513145950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a90790 msgr2=0x5595c0a99c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a90790 0x5595c0a99c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 -- 172.21.15.111:0/2513145950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 msgr2=0x5595c0a99730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a99730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 -- 172.21.15.111:0/2513145950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c10b920 con 0x5595c0a91260 2023-10-11T16:32:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454affd700 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a99730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f454b7fe700 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa0cb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4544007d00 tx=0x7f4544007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cf71e9934434c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.577+0000 7f452eff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f452eff5700 1 -- 172.21.15.111:0/2513145950 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454400e040 con 0x5595c0a91260 2023-10-11T16:32:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f4565094b80 1 -- 172.21.15.111:0/2513145950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5595c0a82d20 con 0x5595c0a91260 2023-10-11T16:32:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f452eff5700 1 -- 172.21.15.111:0/2513145950 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f4544003c00 con 0x5595c0a91260 2023-10-11T16:32:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f452eff5700 1 -- 172.21.15.111:0/2513145950 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4544009ac0 con 0x5595c0a91260 2023-10-11T16:32:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f4565094b80 10 client.15306 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f4565094b80 1 -- 172.21.15.111:0/2513145950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5595c09a7140 con 0x5595c0a91260 2023-10-11T16:32:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f452eff5700 1 -- 172.21.15.111:0/2513145950 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5595c0a82d20 con 0x5595c0a91260 2023-10-11T16:32:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f452eff5700 1 -- 172.21.15.111:0/2513145950 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f4544019070 con 0x5595c0a91260 2023-10-11T16:32:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.578+0000 7f4565094b80 10 client.15306 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.579+0000 7f4565094b80 10 client.15306 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-11T16:32:12.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.579+0000 7f4565094b80 20 client.15306 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:12.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.579+0000 7f4565094b80 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5595c0aa49e0 0x5595c0aa3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.579+0000 7f4565094b80 4 client.15306 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:32:12.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.579+0000 7f4565094b80 1 -- 172.21.15.111:0/2513145950 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595c09a3050 con 0x5595c0aa49e0 2023-10-11T16:32:12.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.579+0000 7f454affd700 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5595c0aa49e0 0x5595c0aa3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.579+0000 7f452eff5700 10 client.15306 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:12.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.579+0000 7f454affd700 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5595c0aa49e0 0x5595c0aa3db0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59e8f296317a85d4 server_cookie=9c70afa862786c38 in_seq=0 out_seq=0 2023-10-11T16:32:12.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.580+0000 7f452eff5700 1 -- 172.21.15.111:0/2513145950 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5595c09a3050 con 0x5595c0aa49e0 2023-10-11T16:32:12.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.580+0000 7f452eff5700 10 client.15306 handle_command_reply: tid=0 2023-10-11T16:32:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.580+0000 7f4565094b80 1 client.15306 shutdown 2023-10-11T16:32:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.580+0000 7f4565094b80 1 -- 172.21.15.111:0/2513145950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 msgr2=0x5595c0aa0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.580+0000 7f4565094b80 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa0cb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4544007d00 tx=0x7f4544007d30 comp rx=0 tx=0).stop 2023-10-11T16:32:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 1 -- 172.21.15.111:0/2513145950 shutdown_connections 2023-10-11T16:32:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5595c0aa49e0 0x5595c0aa3db0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa0cb0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a99730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 1 --2- 172.21.15.111:0/2513145950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a90790 0x5595c0a99c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 1 -- 172.21.15.111:0/2513145950 >> 172.21.15.111:0/2513145950 conn(0x5595c0a85560 msgr2=0x5595c0a89fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f452eff5700 0 client.15306 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:12.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 1 -- 172.21.15.111:0/2513145950 shutdown_connections 2023-10-11T16:32:12.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 1 -- 172.21.15.111:0/2513145950 wait complete. 2023-10-11T16:32:12.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.581+0000 7f4565094b80 20 client.15306 trim_cache size 0 max 16384 2023-10-11T16:32:12.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f4565094b80 1 Processor -- start 2023-10-11T16:32:12.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f4565094b80 1 -- start start 2023-10-11T16:32:12.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f4565094b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a8fe40 0x5595c0a98410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f4565094b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 0x5595c0aa0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f4565094b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f4565094b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c097ce40 con 0x5595c0a91260 2023-10-11T16:32:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f4565094b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c099efd0 con 0x5595c0a90790 2023-10-11T16:32:12.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f4565094b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c0a850e0 con 0x5595c0a8fe40 2023-10-11T16:32:12.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.599+0000 7f454b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454a7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 0x5595c0aa0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47202/0 (socket says 172.21.15.111:47202) 2023-10-11T16:32:12.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454b7fe700 1 -- 172.21.15.111:0/2064234992 learned_addr learned my addr 172.21.15.111:0/2064234992 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454affd700 1 --2- 172.21.15.111:0/2064234992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a8fe40 0x5595c0a98410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454b7fe700 1 -- 172.21.15.111:0/2064234992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a8fe40 msgr2=0x5595c0a98410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454b7fe700 1 --2- 172.21.15.111:0/2064234992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a8fe40 0x5595c0a98410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454b7fe700 1 -- 172.21.15.111:0/2064234992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 msgr2=0x5595c0aa0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454b7fe700 1 --2- 172.21.15.111:0/2064234992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 0x5595c0aa0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454b7fe700 1 -- 172.21.15.111:0/2064234992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595c09a6e90 con 0x5595c0a91260 2023-10-11T16:32:12.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f454b7fe700 1 --2- 172.21.15.111:0/2064234992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa3db0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f454400b370 tx=0x7f4544012ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71d870ccaa81de7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f4531ffb700 1 -- 172.21.15.111:0/2064234992 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454400e070 con 0x5595c0a91260 2023-10-11T16:32:12.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f4531ffb700 1 -- 172.21.15.111:0/2064234992 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4544021070 con 0x5595c0a91260 2023-10-11T16:32:12.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.600+0000 7f4531ffb700 1 -- 172.21.15.111:0/2064234992 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45440123f0 con 0x5595c0a91260 2023-10-11T16:32:12.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 -- 172.21.15.111:0/2064234992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 msgr2=0x5595c0aa3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 --2- 172.21.15.111:0/2064234992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa3db0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f454400b370 tx=0x7f4544012ec0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 -- 172.21.15.111:0/2064234992 shutdown_connections 2023-10-11T16:32:12.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 --2- 172.21.15.111:0/2064234992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a91260 0x5595c0aa3db0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 --2- 172.21.15.111:0/2064234992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a8fe40 0x5595c0a98410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 --2- 172.21.15.111:0/2064234992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a90790 0x5595c0aa0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 -- 172.21.15.111:0/2064234992 >> 172.21.15.111:0/2064234992 conn(0x5595c0a85560 msgr2=0x5595c0aad790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 -- 172.21.15.111:0/2064234992 shutdown_connections 2023-10-11T16:32:12.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 -- 172.21.15.111:0/2064234992 wait complete. 2023-10-11T16:32:12.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.601+0000 7f4565094b80 1 -- start start 2023-10-11T16:32:12.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f4565094b80 1 Processor -- start 2023-10-11T16:32:12.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f4565094b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a95c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f4565094b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a90790 0x5595c0a96170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f4565094b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0af88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f4565094b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c0aa0c60 con 0x5595c0a90790 2023-10-11T16:32:12.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f4565094b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c0aa0dd0 con 0x5595c0a8fe40 2023-10-11T16:32:12.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f4565094b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595c0aa0f90 con 0x5595c0a91260 2023-10-11T16:32:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f4565094b80 10 client.0 _ll_register_callbacks cb 0x7f4544007730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f454a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a90790 0x5595c0a96170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f454affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a95c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f454a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a90790 0x5595c0a96170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47208/0 (socket says 172.21.15.111:47208) 2023-10-11T16:32:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f454a7fc700 1 -- 172.21.15.111:0/4138597375 learned_addr learned my addr 172.21.15.111:0/4138597375 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.602+0000 7f454b7fe700 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0af88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f454affd700 1 -- 172.21.15.111:0/4138597375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 msgr2=0x5595c0af88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f454affd700 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0af88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f454affd700 1 -- 172.21.15.111:0/4138597375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a90790 msgr2=0x5595c0a96170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:12.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f454affd700 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a90790 0x5595c0a96170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f454affd700 1 -- 172.21.15.111:0/4138597375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595c09a7140 con 0x5595c0a8fe40 2023-10-11T16:32:12.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f454b7fe700 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0af88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:12.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f454affd700 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a95c30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f453c096970 tx=0x7f453c0a21d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dd629cb974c8dc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:12.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f452eff5700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:32:12.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f452eff5700 1 -- 172.21.15.111:0/4138597375 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c010710 con 0x5595c0a8fe40 2023-10-11T16:32:12.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f452eff5700 1 -- 172.21.15.111:0/4138597375 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f453c099460 con 0x5595c0a8fe40 2023-10-11T16:32:12.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f4565094b80 1 -- 172.21.15.111:0/4138597375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f453c0a2350 con 0x5595c0a8fe40 2023-10-11T16:32:12.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f452eff5700 1 -- 172.21.15.111:0/4138597375 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f453c010870 con 0x5595c0a8fe40 2023-10-11T16:32:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f4565094b80 10 client.25355 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.603+0000 7f4565094b80 1 -- 172.21.15.111:0/4138597375 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5595c07f7c10 con 0x5595c0a8fe40 2023-10-11T16:32:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f452eff5700 1 -- 172.21.15.111:0/4138597375 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f453c0ca020 con 0x5595c0a8fe40 2023-10-11T16:32:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f452eff5700 1 -- 172.21.15.111:0/4138597375 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f453c094b10 con 0x5595c0a8fe40 2023-10-11T16:32:12.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f4565094b80 10 client.25355 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f4565094b80 10 client.25355 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-11T16:32:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f4565094b80 20 client.25355 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f4565094b80 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5595c0afd7e0 0x5595c0afa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:12.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f4565094b80 4 client.25355 mds_command: new command op to 24569 tid=0[{"prefix": "status"}] 2023-10-11T16:32:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f4565094b80 1 -- 172.21.15.111:0/4138597375 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5595c09af610 con 0x5595c0afd7e0 2023-10-11T16:32:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f454a7fc700 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5595c0afd7e0 0x5595c0afa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f452eff5700 10 client.25355 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.604+0000 7f454a7fc700 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5595c0afd7e0 0x5595c0afa0b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8a7c087181fe00 server_cookie=e211ca10738e52a6 in_seq=0 out_seq=0 2023-10-11T16:32:12.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f452eff5700 1 -- 172.21.15.111:0/4138597375 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5595c09af610 con 0x5595c0afd7e0 2023-10-11T16:32:12.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f452eff5700 10 client.25355 handle_command_reply: tid=0 2023-10-11T16:32:12.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 client.25355 shutdown 2023-10-11T16:32:12.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 -- 172.21.15.111:0/4138597375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 msgr2=0x5595c0a95c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a95c30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f453c096970 tx=0x7f453c0a21d0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 -- 172.21.15.111:0/4138597375 shutdown_connections 2023-10-11T16:32:12.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5595c0afd7e0 0x5595c0afa0b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5595c0a91260 0x5595c0af88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c0a8fe40 0x5595c0a95c30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 --2- 172.21.15.111:0/4138597375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5595c0a90790 0x5595c0a96170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f452eff5700 0 client.25355 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:12.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 -- 172.21.15.111:0/4138597375 >> 172.21.15.111:0/4138597375 conn(0x5595c0a85560 msgr2=0x5595c0a98410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 -- 172.21.15.111:0/4138597375 shutdown_connections 2023-10-11T16:32:12.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 1 -- 172.21.15.111:0/4138597375 wait complete. 2023-10-11T16:32:12.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.605+0000 7f4565094b80 20 client.25355 trim_cache size 0 max 16384 2023-10-11T16:32:12.632 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:32:12.632 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_fsid": "495f265e-6852-11ee-8db6-212e2dc638e7", 2023-10-11T16:32:12.633 INFO:teuthology.orchestra.run.smithi111.stdout: "whoami": 0, 2023-10-11T16:32:12.633 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 24569, 2023-10-11T16:32:12.633 INFO:teuthology.orchestra.run.smithi111.stdout: "want_state": "up:active", 2023-10-11T16:32:12.633 INFO:teuthology.orchestra.run.smithi111.stdout: "state": "up:active", 2023-10-11T16:32:12.633 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs", 2023-10-11T16:32:12.634 INFO:teuthology.orchestra.run.smithi111.stdout: "rank_uptime": 120.969428061, 2023-10-11T16:32:12.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mdsmap_epoch": 26, 2023-10-11T16:32:12.634 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch": 91, 2023-10-11T16:32:12.634 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_epoch_barrier": 89, 2023-10-11T16:32:12.635 INFO:teuthology.orchestra.run.smithi111.stdout: "uptime": 127.048382147 2023-10-11T16:32:12.635 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:32:12.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 -- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45340a1150 msgr2=0x7f45340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45340a1150 0x7f45340a3600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4540005fd0 tx=0x7f4540009310 comp rx=0 tx=0).stop 2023-10-11T16:32:12.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 -- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f454c1000b0 msgr2=0x7f454c198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:12.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f454c1000b0 0x7f454c198060 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f453c00e9a0 tx=0x7f453c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 -- 172.21.15.111:0/102966905 shutdown_connections 2023-10-11T16:32:12.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f45340a1150 0x7f45340a3600 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f454c101cf0 0x7f454c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f454c1000b0 0x7f454c198060 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 --2- 172.21.15.111:0/102966905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f454c101340 0x7f454c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:12.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 -- 172.21.15.111:0/102966905 >> 172.21.15.111:0/102966905 conn(0x7f454c0fb6d0 msgr2=0x7f454c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:12.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 -- 172.21.15.111:0/102966905 shutdown_connections 2023-10-11T16:32:12.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:12.606+0000 7f454bfff700 1 -- 172.21.15.111:0/102966905 wait complete. 2023-10-11T16:32:12.639 INFO:teuthology.run_tasks:Running task kclient... 2023-10-11T16:32:12.654 INFO:tasks.kclient:Mounting kernel clients... 2023-10-11T16:32:12.654 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-11T16:32:12.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:12.655 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-10-11T16:32:12.699 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-10-11T16:32:12.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:12.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-11T16:32:12.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:12.811 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2023-10-11T16:32:12.938 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2023-10-11T16:32:12.938 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-11T16:32:12.939 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-11T16:32:12.939 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph fs ls 2023-10-11T16:32:13.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:13 smithi150 ceph-mon[132217]: pgmap v337: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:32:13.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:13 smithi130 ceph-mon[131859]: pgmap v337: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:32:13.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:13 smithi111 ceph-mon[98527]: pgmap v337: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:32:13.793 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:32:15.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 -- 172.21.15.111:0/2508673125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a01055e0 msgr2=0x7f13a010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:15.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/2508673125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a01055e0 0x7f13a010d160 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f139800b0d0 tx=0x7f139800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 -- 172.21.15.111:0/2508673125 shutdown_connections 2023-10-11T16:32:15.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/2508673125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a01055e0 0x7f13a010d160 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/2508673125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0100540 0x7f13a0100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/2508673125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13a0100ee0 0x7f13a0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 -- 172.21.15.111:0/2508673125 >> 172.21.15.111:0/2508673125 conn(0x7f13a00fbdc0 msgr2=0x7f13a00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:15.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 -- 172.21.15.111:0/2508673125 shutdown_connections 2023-10-11T16:32:15.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.355+0000 7f13a5f5a700 1 -- 172.21.15.111:0/2508673125 wait complete. 2023-10-11T16:32:15.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.356+0000 7f13a5f5a700 1 Processor -- start 2023-10-11T16:32:15.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.356+0000 7f13a5f5a700 1 -- start start 2023-10-11T16:32:15.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f13a5f5a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0100540 0x7f13a019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:15.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f13a5f5a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0100ee0 0x7f13a019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:15.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f13a5f5a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13a01055e0 0x7f13a01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:15.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f13a5f5a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0198c20 con 0x7f13a0100ee0 2023-10-11T16:32:15.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f13a5f5a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0198d90 con 0x7f13a01055e0 2023-10-11T16:32:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f13a5f5a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0198f50 con 0x7f13a0100540 2023-10-11T16:32:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f139effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0100ee0 0x7f13a019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f139effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0100ee0 0x7f13a019ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59690/0 (socket says 172.21.15.111:59690) 2023-10-11T16:32:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f139effd700 1 -- 172.21.15.111:0/1210248841 learned_addr learned my addr 172.21.15.111:0/1210248841 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f139f7fe700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0100540 0x7f13a019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.357+0000 7f139ffff700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13a01055e0 0x7f13a01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139effd700 1 -- 172.21.15.111:0/1210248841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0100540 msgr2=0x7f13a019e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139effd700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0100540 0x7f13a019e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139effd700 1 -- 172.21.15.111:0/1210248841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13a01055e0 msgr2=0x7f13a01985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139effd700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13a01055e0 0x7f13a01985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139effd700 1 -- 172.21.15.111:0/1210248841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1398009d00 con 0x7f13a0100ee0 2023-10-11T16:32:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139effd700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0100ee0 0x7f13a019ea70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f139400e970 tx=0x7f139400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139cff9700 1 -- 172.21.15.111:0/1210248841 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1394009790 con 0x7f13a0100ee0 2023-10-11T16:32:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139cff9700 1 -- 172.21.15.111:0/1210248841 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f139400c7e0 con 0x7f13a0100ee0 2023-10-11T16:32:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a0110370 con 0x7f13a0100ee0 2023-10-11T16:32:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f139cff9700 1 -- 172.21.15.111:0/1210248841 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13940054e0 con 0x7f13a0100ee0 2023-10-11T16:32:15.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.358+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a01101e0 con 0x7f13a0100ee0 2023-10-11T16:32:15.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.359+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a0110500 con 0x7f13a0100ee0 2023-10-11T16:32:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.361+0000 7f139cff9700 1 -- 172.21.15.111:0/1210248841 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f139400c950 con 0x7f13a0100ee0 2023-10-11T16:32:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.362+0000 7f139cff9700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f13880a11d0 0x7f13880a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.362+0000 7f139cff9700 1 -- 172.21.15.111:0/1210248841 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f1394014070 con 0x7f13a0100ee0 2023-10-11T16:32:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.363+0000 7f139f7fe700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f13880a11d0 0x7f13880a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.363+0000 7f139f7fe700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f13880a11d0 0x7f13880a3680 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1390007be0 tx=0x7f13900073d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.367+0000 7f139cff9700 1 -- 172.21.15.111:0/1210248841 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f13940d3080 con 0x7f13a0100ee0 2023-10-11T16:32:15.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:15 smithi111 ceph-mon[98527]: pgmap v338: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:15.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:15 smithi150 ceph-mon[132217]: pgmap v338: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:15.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:15 smithi130 ceph-mon[131859]: pgmap v338: 129 pgs: 129 active+clean; 655 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:15.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.604+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f13a01a5940 con 0x7f13a0100ee0 2023-10-11T16:32:15.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.605+0000 7f139cff9700 1 -- 172.21.15.111:0/1210248841 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+88 (secure 0 0 0) 0x7f13940961a0 con 0x7f13a0100ee0 2023-10-11T16:32:15.607 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-11T16:32:15.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f13880a11d0 msgr2=0x7f13880a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:15.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f13880a11d0 0x7f13880a3680 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1390007be0 tx=0x7f13900073d0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0100ee0 msgr2=0x7f13a019ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:15.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0100ee0 0x7f13a019ea70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f139400e970 tx=0x7f139400ec80 comp rx=0 tx=0).stop 2023-10-11T16:32:15.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 shutdown_connections 2023-10-11T16:32:15.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f13880a11d0 0x7f13880a3680 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13a01055e0 0x7f13a01985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13a0100540 0x7f13a019e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 --2- 172.21.15.111:0/1210248841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0100ee0 0x7f13a019ea70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:15.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 >> 172.21.15.111:0/1210248841 conn(0x7f13a00fbdc0 msgr2=0x7f13a010b660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:15.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 shutdown_connections 2023-10-11T16:32:15.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:15.607+0000 7f13a5f5a700 1 -- 172.21.15.111:0/1210248841 wait complete. 2023-10-11T16:32:16.073 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-11T16:32:16.073 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-11T16:32:16.074 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2023-10-11T16:32:16.074 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-11T16:32:16.074 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-11T16:32:16.074 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-11T16:32:16.075 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-11T16:32:16.075 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-11T16:32:16.075 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-11T16:32:16.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.076 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2023-10-11T16:32:16.122 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-11T16:32:16.122 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-11T16:32:16.122 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2023-10-11T16:32:16.122 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-10-11T16:32:16.123 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2023-10-11T16:32:16.123 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-10-11T16:32:16.123 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-11T16:32:16.123 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2023-10-11T16:32:16.123 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2023-10-11T16:32:16.124 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-11T16:32:16.124 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2023-10-11T16:32:16.124 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2023-10-11T16:32:16.124 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-11T16:32:16.124 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2023-10-11T16:32:16.125 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-11T16:32:16.125 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2023-10-11T16:32:16.125 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-11T16:32:16.125 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41611sec preferred_lft 41611sec 2023-10-11T16:32:16.125 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::9056:a3ce:d946:5f52/64 scope link noprefixroute 2023-10-11T16:32:16.126 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2023-10-11T16:32:16.126 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-11T16:32:16.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.127 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:32:16.127 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:32:16.127 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add name ceph-brx type bridge 2023-10-11T16:32:16.128 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr flush dev ceph-brx 2023-10-11T16:32:16.128 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set ceph-brx up 2023-10-11T16:32:16.128 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-11T16:32:16.128 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:32:16.276 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:16 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1210248841' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-11T16:32:16.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.372 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-11T16:32:16.441 INFO:teuthology.orchestra.run.smithi111.stdout:1 2023-10-11T16:32:16.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.444 DEBUG:teuthology.orchestra.run.smithi111:> route 2023-10-11T16:32:16.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:16 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1210248841' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-11T16:32:16.491 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2023-10-11T16:32:16.491 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-11T16:32:16.492 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-11T16:32:16.492 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-11T16:32:16.492 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-11T16:32:16.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.493 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:32:16.494 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:32:16.494 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-11T16:32:16.494 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-11T16:32:16.495 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-11T16:32:16.495 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:32:16.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:16 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1210248841' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-11T16:32:16.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.668 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-10-11T16:32:16.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.714 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2023-10-11T16:32:16.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.797 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:32:16.797 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:32:16.798 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-11T16:32:16.798 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-11T16:32:16.798 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:32:16.979 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-11T16:32:16.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:16.980 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:32:16.981 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:32:16.981 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-11T16:32:16.981 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-11T16:32:16.981 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-11T16:32:16.982 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-11T16:32:16.982 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-11T16:32:16.982 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:32:17.194 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:17 smithi111 ceph-mon[98527]: pgmap v339: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-10-11T16:32:17.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:17.230 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2023-10-11T16:32:17.230 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2023-10-11T16:32:17.231 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2023-10-11T16:32:17.231 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.0 master ceph-brx 2023-10-11T16:32:17.232 DEBUG:teuthology.orchestra.run.smithi111:> ') 2023-10-11T16:32:17.405 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-11T16:32:17.405 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-11T16:32:17.449 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-11T16:32:17.450 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-11T16:32:17.450 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-11T16:32:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:17 smithi150 ceph-mon[132217]: pgmap v339: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-10-11T16:32:17.533 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-10-11T16:32:17.533 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-10-11T16:32:17.533 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/_nogroup/sv_1/291f0a93-6079-41ae-a1a4-65679e80ef94 2023-10-11T16:32:17.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:17.534 DEBUG:teuthology.orchestra.run.smithi111:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/_nogroup/sv_1/291f0a93-6079-41ae-a1a4-65679e80ef94 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2023-10-11T16:32:17.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:17 smithi130 ceph-mon[131859]: pgmap v339: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-10-11T16:32:17.677 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2023-10-11T16:32:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2023-10-11T16:32:17.678 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-10-11T16:32:18.138 INFO:teuthology.orchestra.run.smithi111.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2023-10-11T16:32:18.139 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2023-10-11T16:32:18.139 INFO:teuthology.orchestra.run.smithi111.stdout:invalid new device string format 2023-10-11T16:32:18.139 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: resolved to: "172.21.15.111:3300,172.21.15.130:3300,172.21.15.150:3300" 2023-10-11T16:32:18.139 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.111:3300,172.21.15.130:3300,172.21.15.150:3300:/volumes/_nogroup/sv_1/291f0a93-6079-41ae-a1a4-65679e80ef94 2023-10-11T16:32:18.139 INFO:teuthology.orchestra.run.smithi111.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=495f265e-6852-11ee-8db6-212e2dc638e7" will pass to kernel 2023-10-11T16:32:18.146 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-11T16:32:18.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:18.146 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-11T16:32:18.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:18.220 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-11T16:32:18.305 INFO:teuthology.orchestra.run.smithi111.stdout:client25384 2023-10-11T16:32:18.306 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:32:18.306 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-11T16:32:18.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:18.354 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-11T16:32:18.480 INFO:teuthology.orchestra.run.smithi111.stdout:client25384 2023-10-11T16:32:18.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:18.481 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-11T16:32:18.569 INFO:teuthology.orchestra.run.smithi111.stdout:client25384 2023-10-11T16:32:18.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:32:18.570 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/495f265e-6852-11ee-8db6-212e2dc638e7.client25384/status') 2023-10-11T16:32:18.678 INFO:teuthology.orchestra.run.smithi111.stdout:instance: client.25384 (3)172.21.15.111:0/530102895 2023-10-11T16:32:18.679 INFO:teuthology.orchestra.run.smithi111.stdout:blocklisted: false 2023-10-11T16:32:18.679 INFO:teuthology.orchestra.run.smithi111.stderr:0+1 records in 2023-10-11T16:32:18.679 INFO:teuthology.orchestra.run.smithi111.stderr:0+1 records out 2023-10-11T16:32:18.680 INFO:teuthology.orchestra.run.smithi111.stderr:71 bytes copied, 5.3035e-05 s, 1.3 MB/s 2023-10-11T16:32:18.682 INFO:teuthology.run_tasks:Running task check-counter... 2023-10-11T16:32:18.694 INFO:teuthology.run_tasks:Running task fwd_scrub... 2023-10-11T16:32:18.750 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:32:18.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 -- 172.21.15.111:0/2081640424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441002b0 msgr2=0x7fc5441006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:18.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 --2- 172.21.15.111:0/2081640424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441002b0 0x7fc5441006c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc534009990 tx=0x7fc534009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:18.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 -- 172.21.15.111:0/2081640424 shutdown_connections 2023-10-11T16:32:18.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 --2- 172.21.15.111:0/2081640424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc544101e30 0x7fc5441086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:18.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 --2- 172.21.15.111:0/2081640424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441002b0 0x7fc5441006c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:18.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 --2- 172.21.15.111:0/2081640424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5441014a0 0x7fc5441018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:18.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 -- 172.21.15.111:0/2081640424 >> 172.21.15.111:0/2081640424 conn(0x7fc5440fb890 msgr2=0x7fc5440fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:18.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 -- 172.21.15.111:0/2081640424 shutdown_connections 2023-10-11T16:32:18.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.948+0000 7fc548bff700 1 -- 172.21.15.111:0/2081640424 wait complete. 2023-10-11T16:32:18.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.949+0000 7fc548bff700 1 Processor -- start 2023-10-11T16:32:18.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.949+0000 7fc548bff700 1 -- start start 2023-10-11T16:32:18.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc548bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5441002b0 0x7fc5441981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:18.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc548bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441014a0 0x7fc5441986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:18.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc548bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc544101e30 0x7fc54419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:18.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc548bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54419fd20 con 0x7fc5441014a0 2023-10-11T16:32:18.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc548bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54419fe60 con 0x7fc544101e30 2023-10-11T16:32:18.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc548bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54419fff0 con 0x7fc5441002b0 2023-10-11T16:32:18.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc53bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441014a0 0x7fc5441986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:18.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc542d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc544101e30 0x7fc54419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:18.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc53bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441014a0 0x7fc5441986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59710/0 (socket says 172.21.15.111:59710) 2023-10-11T16:32:18.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc53bfff700 1 -- 172.21.15.111:0/1486597856 learned_addr learned my addr 172.21.15.111:0/1486597856 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:18.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc54259c700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5441002b0 0x7fc5441981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:18.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc53bfff700 1 -- 172.21.15.111:0/1486597856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5441002b0 msgr2=0x7fc5441981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:18.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc53bfff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5441002b0 0x7fc5441981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:18.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc53bfff700 1 -- 172.21.15.111:0/1486597856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc544101e30 msgr2=0x7fc54419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:18.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc53bfff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc544101e30 0x7fc54419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:18.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.950+0000 7fc53bfff700 1 -- 172.21.15.111:0/1486597856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5340095f0 con 0x7fc5441014a0 2023-10-11T16:32:18.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.951+0000 7fc53bfff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441014a0 0x7fc5441986e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00d980 tx=0x7fc52c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:18.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.951+0000 7fc53b7fe700 1 -- 172.21.15.111:0/1486597856 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c004220 con 0x7fc5441014a0 2023-10-11T16:32:18.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.951+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54410bc90 con 0x7fc5441014a0 2023-10-11T16:32:18.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.951+0000 7fc53b7fe700 1 -- 172.21.15.111:0/1486597856 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc52c007640 con 0x7fc5441014a0 2023-10-11T16:32:18.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.951+0000 7fc53b7fe700 1 -- 172.21.15.111:0/1486597856 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c005690 con 0x7fc5441014a0 2023-10-11T16:32:18.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.951+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54410bb00 con 0x7fc5441014a0 2023-10-11T16:32:18.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.952+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc544192920 con 0x7fc5441014a0 2023-10-11T16:32:18.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.954+0000 7fc53b7fe700 1 -- 172.21.15.111:0/1486597856 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fc52c004750 con 0x7fc5441014a0 2023-10-11T16:32:18.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.955+0000 7fc53b7fe700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5300a1150 0x7fc5300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:18.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.955+0000 7fc53b7fe700 1 -- 172.21.15.111:0/1486597856 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fc52c0ccb60 con 0x7fc5441014a0 2023-10-11T16:32:18.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.956+0000 7fc54259c700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5300a1150 0x7fc5300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:18.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.956+0000 7fc54259c700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5300a1150 0x7fc5300a3600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc53400b5c0 tx=0x7fc534005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:18.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:18.959+0000 7fc53b7fe700 1 -- 172.21.15.111:0/1486597856 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc52c095e80 con 0x7fc5441014a0 2023-10-11T16:32:19.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.161+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc54402cfe0 con 0x7fc5441014a0 2023-10-11T16:32:19.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.162+0000 7fc53b7fe700 1 -- 172.21.15.111:0/1486597856 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24870 (secure 0 0 0) 0x7fc52c016020 con 0x7fc5441014a0 2023-10-11T16:32:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:19.164 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":91,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:31:42.150221+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:32:19.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.164+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5300a1150 msgr2=0x7fc5300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5300a1150 0x7fc5300a3600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc53400b5c0 tx=0x7fc534005fd0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441014a0 msgr2=0x7fc5441986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441014a0 0x7fc5441986e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00d980 tx=0x7fc52c00dc90 comp rx=0 tx=0).stop 2023-10-11T16:32:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 shutdown_connections 2023-10-11T16:32:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5300a1150 0x7fc5300a3600 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc544101e30 0x7fc54419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5441002b0 0x7fc5441981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 --2- 172.21.15.111:0/1486597856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5441014a0 0x7fc5441986e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 >> 172.21.15.111:0/1486597856 conn(0x7fc5440fb890 msgr2=0x7fc544106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 shutdown_connections 2023-10-11T16:32:19.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.165+0000 7fc548bff700 1 -- 172.21.15.111:0/1486597856 wait complete. 2023-10-11T16:32:19.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:32:19.186 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:32:19.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 -- 172.21.15.111:0/3113337778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c101e40 msgr2=0x7faf7c108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 --2- 172.21.15.111:0/3113337778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c101e40 0x7faf7c108700 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7faf7400b0a0 tx=0x7faf7400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 -- 172.21.15.111:0/3113337778 shutdown_connections 2023-10-11T16:32:19.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 --2- 172.21.15.111:0/3113337778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c101e40 0x7faf7c108700 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 --2- 172.21.15.111:0/3113337778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf7c1002c0 0x7faf7c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 --2- 172.21.15.111:0/3113337778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf7c1014b0 0x7faf7c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 -- 172.21.15.111:0/3113337778 >> 172.21.15.111:0/3113337778 conn(0x7faf7c0fb8a0 msgr2=0x7faf7c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:19.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 -- 172.21.15.111:0/3113337778 shutdown_connections 2023-10-11T16:32:19.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.387+0000 7faf813ff700 1 -- 172.21.15.111:0/3113337778 wait complete. 2023-10-11T16:32:19.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.388+0000 7faf813ff700 1 Processor -- start 2023-10-11T16:32:19.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.388+0000 7faf813ff700 1 -- start start 2023-10-11T16:32:19.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.388+0000 7faf813ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c1002c0 0x7faf7c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:19.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.388+0000 7faf813ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf7c1014b0 0x7faf7c198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:19.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.388+0000 7faf813ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf7c101e40 0x7faf7c19f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:19.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.388+0000 7faf813ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c19fec0 con 0x7faf7c1002c0 2023-10-11T16:32:19.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.388+0000 7faf813ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c1a0000 con 0x7faf7c1014b0 2023-10-11T16:32:19.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.388+0000 7faf813ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c1a0190 con 0x7faf7c101e40 2023-10-11T16:32:19.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c1002c0 0x7faf7c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:19.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c1002c0 0x7faf7c198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59726/0 (socket says 172.21.15.111:59726) 2023-10-11T16:32:19.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 -- 172.21.15.111:0/4121875761 learned_addr learned my addr 172.21.15.111:0/4121875761 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:19.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7b7fe700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf7c101e40 0x7faf7c19f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:19.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf73fff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf7c1014b0 0x7faf7c198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:19.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 -- 172.21.15.111:0/4121875761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf7c101e40 msgr2=0x7faf7c19f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf7c101e40 0x7faf7c19f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 -- 172.21.15.111:0/4121875761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf7c1014b0 msgr2=0x7faf7c198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf7c1014b0 0x7faf7c198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 -- 172.21.15.111:0/4121875761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74009d00 con 0x7faf7c1002c0 2023-10-11T16:32:19.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf7affd700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c1002c0 0x7faf7c198340 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00e970 tx=0x7faf6c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:19.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf78ff9700 1 -- 172.21.15.111:0/4121875761 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c009790 con 0x7faf7c1002c0 2023-10-11T16:32:19.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf78ff9700 1 -- 172.21.15.111:0/4121875761 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf6c00c7e0 con 0x7faf7c1002c0 2023-10-11T16:32:19.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.389+0000 7faf78ff9700 1 -- 172.21.15.111:0/4121875761 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c003c70 con 0x7faf7c1002c0 2023-10-11T16:32:19.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.390+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c10bc90 con 0x7faf7c1002c0 2023-10-11T16:32:19.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.390+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c10bb00 con 0x7faf7c1002c0 2023-10-11T16:32:19.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.391+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c1a08e0 con 0x7faf7c1002c0 2023-10-11T16:32:19.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.392+0000 7faf78ff9700 1 -- 172.21.15.111:0/4121875761 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7faf6c00cc50 con 0x7faf7c1002c0 2023-10-11T16:32:19.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.394+0000 7faf78ff9700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faf680a11d0 0x7faf680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:19.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.394+0000 7faf78ff9700 1 -- 172.21.15.111:0/4121875761 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7faf6c014070 con 0x7faf7c1002c0 2023-10-11T16:32:19.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.394+0000 7faf73fff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faf680a11d0 0x7faf680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:19.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.394+0000 7faf73fff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faf680a11d0 0x7faf680a3680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf64009c60 tx=0x7faf64009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:19.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.397+0000 7faf78ff9700 1 -- 172.21.15.111:0/4121875761 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faf6c096170 con 0x7faf7c1002c0 2023-10-11T16:32:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:19 smithi150 ceph-mon[132217]: pgmap v340: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:19.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:19 smithi130 ceph-mon[131859]: pgmap v340: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:19.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.593+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faf7c1a0e90 con 0x7faf7c1002c0 2023-10-11T16:32:19.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.593+0000 7faf78ff9700 1 -- 172.21.15.111:0/4121875761 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7faf6c003dd0 con 0x7faf7c1002c0 2023-10-11T16:32:19.595 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:32:19.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faf680a11d0 msgr2=0x7faf680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faf680a11d0 0x7faf680a3680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf64009c60 tx=0x7faf64009260 comp rx=0 tx=0).stop 2023-10-11T16:32:19.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c1002c0 msgr2=0x7faf7c198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c1002c0 0x7faf7c198340 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00e970 tx=0x7faf6c00ec80 comp rx=0 tx=0).stop 2023-10-11T16:32:19.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 shutdown_connections 2023-10-11T16:32:19.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7faf680a11d0 0x7faf680a3680 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf7c101e40 0x7faf7c19f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf7c1002c0 0x7faf7c198340 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 --2- 172.21.15.111:0/4121875761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf7c1014b0 0x7faf7c198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 >> 172.21.15.111:0/4121875761 conn(0x7faf7c0fb8a0 msgr2=0x7faf7c106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:19.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 shutdown_connections 2023-10-11T16:32:19.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.596+0000 7faf813ff700 1 -- 172.21.15.111:0/4121875761 wait complete. 2023-10-11T16:32:19.614 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:19 smithi111 ceph-mon[98527]: pgmap v340: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:19.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:32:19.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.818+0000 7fe5dcdff700 1 -- 172.21.15.111:0/3230767407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 msgr2=0x7fe5d81006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.818+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/3230767407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 0x7fe5d81006e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009990 tx=0x7fe5c8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.819+0000 7fe5dcdff700 1 -- 172.21.15.111:0/3230767407 shutdown_connections 2023-10-11T16:32:19.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.819+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/3230767407 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5d8101e50 0x7fe5d8108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.819+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/3230767407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 0x7fe5d81006e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.819+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/3230767407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5d81014c0 0x7fe5d8101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.819+0000 7fe5dcdff700 1 -- 172.21.15.111:0/3230767407 >> 172.21.15.111:0/3230767407 conn(0x7fe5d80fb8b0 msgr2=0x7fe5d80fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.819+0000 7fe5dcdff700 1 -- 172.21.15.111:0/3230767407 shutdown_connections 2023-10-11T16:32:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.819+0000 7fe5dcdff700 1 -- 172.21.15.111:0/3230767407 wait complete. 2023-10-11T16:32:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5dcdff700 1 Processor -- start 2023-10-11T16:32:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5dcdff700 1 -- start start 2023-10-11T16:32:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5dcdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 0x7fe5d80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5dcdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5d81014c0 0x7fe5d8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5dcdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5d8101e50 0x7fe5d8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5dcdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8073bc0 con 0x7fe5d81002d0 2023-10-11T16:32:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5dcdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d8073d30 con 0x7fe5d8101e50 2023-10-11T16:32:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5dcdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d81a0100 con 0x7fe5d81014c0 2023-10-11T16:32:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5d659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 0x7fe5d80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5d659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 0x7fe5d80748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59744/0 (socket says 172.21.15.111:59744) 2023-10-11T16:32:19.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.820+0000 7fe5d659c700 1 -- 172.21.15.111:0/2484979572 learned_addr learned my addr 172.21.15.111:0/2484979572 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:19.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5d6d9d700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5d8101e50 0x7fe5d8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:19.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5cffff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5d81014c0 0x7fe5d8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:19.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5d659c700 1 -- 172.21.15.111:0/2484979572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5d81014c0 msgr2=0x7fe5d8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5d659c700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5d81014c0 0x7fe5d8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5d659c700 1 -- 172.21.15.111:0/2484979572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5d8101e50 msgr2=0x7fe5d8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5d659c700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5d8101e50 0x7fe5d8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5d659c700 1 -- 172.21.15.111:0/2484979572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c80095f0 con 0x7fe5d81002d0 2023-10-11T16:32:19.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5d659c700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 0x7fe5d80748a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8005740 tx=0x7fe5c8005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:19.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5cf7fe700 1 -- 172.21.15.111:0/2484979572 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c801d070 con 0x7fe5d81002d0 2023-10-11T16:32:19.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5cf7fe700 1 -- 172.21.15.111:0/2484979572 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5c8021410 con 0x7fe5d81002d0 2023-10-11T16:32:19.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d810bca0 con 0x7fe5d81002d0 2023-10-11T16:32:19.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5cf7fe700 1 -- 172.21.15.111:0/2484979572 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8017650 con 0x7fe5d81002d0 2023-10-11T16:32:19.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.821+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d810bb10 con 0x7fe5d81002d0 2023-10-11T16:32:19.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.822+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d8107860 con 0x7fe5d81002d0 2023-10-11T16:32:19.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.824+0000 7fe5cf7fe700 1 -- 172.21.15.111:0/2484979572 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fe5c8021580 con 0x7fe5d81002d0 2023-10-11T16:32:19.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.825+0000 7fe5cf7fe700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe5c40a1150 0x7fe5c40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:19.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.826+0000 7fe5cf7fe700 1 -- 172.21.15.111:0/2484979572 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fe5c80ce100 con 0x7fe5d81002d0 2023-10-11T16:32:19.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.826+0000 7fe5cffff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe5c40a1150 0x7fe5c40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:19.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.826+0000 7fe5cffff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe5c40a1150 0x7fe5c40a3600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009bf0 tx=0x7fe5c0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:19.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:19.829+0000 7fe5cf7fe700 1 -- 172.21.15.111:0/2484979572 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe5c8097540 con 0x7fe5d81002d0 2023-10-11T16:32:20.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.028+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe5d8067b00 con 0x7fe5d81002d0 2023-10-11T16:32:20.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.028+0000 7fe5cf7fe700 1 -- 172.21.15.111:0/2484979572 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe5c809ab60 con 0x7fe5d81002d0 2023-10-11T16:32:20.030 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:20.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe5c40a1150 msgr2=0x7fe5c40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe5c40a1150 0x7fe5c40a3600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009bf0 tx=0x7fe5c0009260 comp rx=0 tx=0).stop 2023-10-11T16:32:20.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 msgr2=0x7fe5d80748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 0x7fe5d80748a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8005740 tx=0x7fe5c8005770 comp rx=0 tx=0).stop 2023-10-11T16:32:20.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 shutdown_connections 2023-10-11T16:32:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fe5c40a1150 0x7fe5c40a3600 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5d8101e50 0x7fe5d8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d81002d0 0x7fe5d80748a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 --2- 172.21.15.111:0/2484979572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5d81014c0 0x7fe5d8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 >> 172.21.15.111:0/2484979572 conn(0x7fe5d80fb8b0 msgr2=0x7fe5d81049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:20.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 shutdown_connections 2023-10-11T16:32:20.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.031+0000 7fe5dcdff700 1 -- 172.21.15.111:0/2484979572 wait complete. 2023-10-11T16:32:20.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:32:20.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 -- 172.21.15.111:0/1176350274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 msgr2=0x7fcc0c10a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 --2- 172.21.15.111:0/1176350274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 0x7fcc0c10a610 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400b0a0 tx=0x7fcc0400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 -- 172.21.15.111:0/1176350274 shutdown_connections 2023-10-11T16:32:20.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 --2- 172.21.15.111:0/1176350274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 0x7fcc0c10a610 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 --2- 172.21.15.111:0/1176350274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc0c074b30 0x7fcc0c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 --2- 172.21.15.111:0/1176350274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc0c073550 0x7fcc0c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 -- 172.21.15.111:0/1176350274 >> 172.21.15.111:0/1176350274 conn(0x7fcc0c0fb8b0 msgr2=0x7fcc0c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:20.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 -- 172.21.15.111:0/1176350274 shutdown_connections 2023-10-11T16:32:20.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.254+0000 7fcc133ff700 1 -- 172.21.15.111:0/1176350274 wait complete. 2023-10-11T16:32:20.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.255+0000 7fcc133ff700 1 Processor -- start 2023-10-11T16:32:20.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.255+0000 7fcc133ff700 1 -- start start 2023-10-11T16:32:20.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.255+0000 7fcc133ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc0c073550 0x7fcc0c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:20.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.255+0000 7fcc133ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc0c074b30 0x7fcc0c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:20.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.255+0000 7fcc133ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 0x7fcc0c19f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:20.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.255+0000 7fcc133ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c19ff50 con 0x7fcc0c108190 2023-10-11T16:32:20.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.255+0000 7fcc133ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c1a0090 con 0x7fcc0c073550 2023-10-11T16:32:20.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.255+0000 7fcc133ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c1a0220 con 0x7fcc0c074b30 2023-10-11T16:32:20.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 0x7fcc0c19f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:20.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1099a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc0c074b30 0x7fcc0c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:20.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 0x7fcc0c19f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59762/0 (socket says 172.21.15.111:59762) 2023-10-11T16:32:20.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 -- 172.21.15.111:0/2242789613 learned_addr learned my addr 172.21.15.111:0/2242789613 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:20.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1119b700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc0c073550 0x7fcc0c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:20.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 -- 172.21.15.111:0/2242789613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc0c074b30 msgr2=0x7fcc0c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc0c074b30 0x7fcc0c198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 -- 172.21.15.111:0/2242789613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc0c073550 msgr2=0x7fcc0c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc0c073550 0x7fcc0c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 -- 172.21.15.111:0/2242789613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc04009d00 con 0x7fcc0c108190 2023-10-11T16:32:20.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.256+0000 7fcc1199c700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 0x7fcc0c19f980 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009fd0 tx=0x7fcc0400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:20.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.257+0000 7fcc027fc700 1 -- 172.21.15.111:0/2242789613 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400e040 con 0x7fcc0c108190 2023-10-11T16:32:20.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.257+0000 7fcc027fc700 1 -- 172.21.15.111:0/2242789613 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc04003e10 con 0x7fcc0c108190 2023-10-11T16:32:20.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.257+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c10deb0 con 0x7fcc0c108190 2023-10-11T16:32:20.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.257+0000 7fcc027fc700 1 -- 172.21.15.111:0/2242789613 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0401b6a0 con 0x7fcc0c108190 2023-10-11T16:32:20.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.257+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c10dd20 con 0x7fcc0c108190 2023-10-11T16:32:20.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.258+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c106840 con 0x7fcc0c108190 2023-10-11T16:32:20.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.260+0000 7fcc027fc700 1 -- 172.21.15.111:0/2242789613 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fcc0401b800 con 0x7fcc0c108190 2023-10-11T16:32:20.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.261+0000 7fcc027fc700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcbf80a1150 0x7fcbf80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:20.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.261+0000 7fcc027fc700 1 -- 172.21.15.111:0/2242789613 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fcc040ce730 con 0x7fcc0c108190 2023-10-11T16:32:20.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.261+0000 7fcc1119b700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcbf80a1150 0x7fcbf80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:20.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.262+0000 7fcc1119b700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcbf80a1150 0x7fcbf80a3600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c074f00 tx=0x7fcc08009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:20.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.263+0000 7fcc027fc700 1 -- 172.21.15.111:0/2242789613 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcc04097ae0 con 0x7fcc0c108190 2023-10-11T16:32:20.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:20 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1486597856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:20.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:20 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4121875761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:20.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:20 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2484979572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:20 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1486597856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:20 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4121875761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:20.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:20 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2484979572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.461+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc0c061700 con 0x7fcc0c108190 2023-10-11T16:32:20.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.462+0000 7fcc027fc700 1 -- 172.21.15.111:0/2242789613 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc0409b100 con 0x7fcc0c108190 2023-10-11T16:32:20.464 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:32:20.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.464+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcbf80a1150 msgr2=0x7fcbf80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.464+0000 7fcc133ff700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcbf80a1150 0x7fcbf80a3600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c074f00 tx=0x7fcc08009260 comp rx=0 tx=0).stop 2023-10-11T16:32:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.464+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 msgr2=0x7fcc0c19f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.464+0000 7fcc133ff700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 0x7fcc0c19f980 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009fd0 tx=0x7fcc0400bce0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.464+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 shutdown_connections 2023-10-11T16:32:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.464+0000 7fcc133ff700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fcbf80a1150 0x7fcbf80a3600 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.464+0000 7fcc133ff700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc0c108190 0x7fcc0c19f980 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.464+0000 7fcc133ff700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc0c073550 0x7fcc0c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.465+0000 7fcc133ff700 1 --2- 172.21.15.111:0/2242789613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc0c074b30 0x7fcc0c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.465+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 >> 172.21.15.111:0/2242789613 conn(0x7fcc0c0fb8b0 msgr2=0x7fcc0c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.465+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 shutdown_connections 2023-10-11T16:32:20.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.465+0000 7fcc133ff700 1 -- 172.21.15.111:0/2242789613 wait complete. 2023-10-11T16:32:20.485 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:32:20.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:20 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1486597856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:20.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:20 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4121875761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:20.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:20 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2484979572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:20.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 -- 172.21.15.111:0/666004922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c101e50 msgr2=0x7fba3c108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 --2- 172.21.15.111:0/666004922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c101e50 0x7fba3c108710 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fba3400b0a0 tx=0x7fba3400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 -- 172.21.15.111:0/666004922 shutdown_connections 2023-10-11T16:32:20.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 --2- 172.21.15.111:0/666004922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c101e50 0x7fba3c108710 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 --2- 172.21.15.111:0/666004922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba3c1002d0 0x7fba3c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 --2- 172.21.15.111:0/666004922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba3c1014c0 0x7fba3c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 -- 172.21.15.111:0/666004922 >> 172.21.15.111:0/666004922 conn(0x7fba3c0fb8b0 msgr2=0x7fba3c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:20.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 -- 172.21.15.111:0/666004922 shutdown_connections 2023-10-11T16:32:20.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.686+0000 7fba431ff700 1 -- 172.21.15.111:0/666004922 wait complete. 2023-10-11T16:32:20.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.687+0000 7fba431ff700 1 Processor -- start 2023-10-11T16:32:20.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.687+0000 7fba431ff700 1 -- start start 2023-10-11T16:32:20.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.687+0000 7fba431ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba3c1002d0 0x7fba3c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:20.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.687+0000 7fba431ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba3c1014c0 0x7fba3c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:20.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.687+0000 7fba431ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c101e50 0x7fba3c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:20.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.687+0000 7fba431ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c073bc0 con 0x7fba3c1014c0 2023-10-11T16:32:20.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba431ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c073d30 con 0x7fba3c1002d0 2023-10-11T16:32:20.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba431ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c19ffa0 con 0x7fba3c101e50 2023-10-11T16:32:20.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba3c1014c0 0x7fba3c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:20.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba3c1014c0 0x7fba3c072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59786/0 (socket says 172.21.15.111:59786) 2023-10-11T16:32:20.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 -- 172.21.15.111:0/1136120263 learned_addr learned my addr 172.21.15.111:0/1136120263 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:20.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba40f9b700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba3c1002d0 0x7fba3c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:20.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba4179c700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c101e50 0x7fba3c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:20.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 -- 172.21.15.111:0/1136120263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c101e50 msgr2=0x7fba3c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c101e50 0x7fba3c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 -- 172.21.15.111:0/1136120263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba3c1002d0 msgr2=0x7fba3c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba3c1002d0 0x7fba3c0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 -- 172.21.15.111:0/1136120263 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34009d00 con 0x7fba3c1014c0 2023-10-11T16:32:20.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.688+0000 7fba3bfff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba3c1014c0 0x7fba3c072ee0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fba2800e9a0 tx=0x7fba2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:20.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.689+0000 7fba39ffb700 1 -- 172.21.15.111:0/1136120263 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28009790 con 0x7fba3c1014c0 2023-10-11T16:32:20.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.689+0000 7fba39ffb700 1 -- 172.21.15.111:0/1136120263 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba2800cbd0 con 0x7fba3c1014c0 2023-10-11T16:32:20.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.689+0000 7fba39ffb700 1 -- 172.21.15.111:0/1136120263 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28018680 con 0x7fba3c1014c0 2023-10-11T16:32:20.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.689+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c10bbe0 con 0x7fba3c1014c0 2023-10-11T16:32:20.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.689+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c10ba50 con 0x7fba3c1014c0 2023-10-11T16:32:20.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.690+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c107160 con 0x7fba3c1014c0 2023-10-11T16:32:20.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.692+0000 7fba39ffb700 1 -- 172.21.15.111:0/1136120263 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fba2800cd40 con 0x7fba3c1014c0 2023-10-11T16:32:20.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.693+0000 7fba39ffb700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fba2c0a1150 0x7fba2c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:20.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.694+0000 7fba39ffb700 1 -- 172.21.15.111:0/1136120263 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fba28014070 con 0x7fba3c1014c0 2023-10-11T16:32:20.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.694+0000 7fba40f9b700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fba2c0a1150 0x7fba2c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:20.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.694+0000 7fba40f9b700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fba2c0a1150 0x7fba2c0a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fba3c101320 tx=0x7fba30009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:20.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.697+0000 7fba39ffb700 1 -- 172.21.15.111:0/1136120263 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fba28051730 con 0x7fba3c1014c0 2023-10-11T16:32:20.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.896+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fba3c061700 con 0x7fba3c1014c0 2023-10-11T16:32:20.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.896+0000 7fba39ffb700 1 -- 172.21.15.111:0/1136120263 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fba280962b0 con 0x7fba3c1014c0 2023-10-11T16:32:20.898 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:20.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.898+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fba2c0a1150 msgr2=0x7fba2c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.898+0000 7fba431ff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fba2c0a1150 0x7fba2c0a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fba3c101320 tx=0x7fba30009260 comp rx=0 tx=0).stop 2023-10-11T16:32:20.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.898+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba3c1014c0 msgr2=0x7fba3c072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:20.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.898+0000 7fba431ff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba3c1014c0 0x7fba3c072ee0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fba2800e9a0 tx=0x7fba2800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.899+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 shutdown_connections 2023-10-11T16:32:20.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.899+0000 7fba431ff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fba2c0a1150 0x7fba2c0a3600 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.899+0000 7fba431ff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c101e50 0x7fba3c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.899+0000 7fba431ff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba3c1002d0 0x7fba3c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.899+0000 7fba431ff700 1 --2- 172.21.15.111:0/1136120263 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba3c1014c0 0x7fba3c072ee0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:20.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.899+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 >> 172.21.15.111:0/1136120263 conn(0x7fba3c0fb8b0 msgr2=0x7fba3c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:20.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.899+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 shutdown_connections 2023-10-11T16:32:20.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:20.899+0000 7fba431ff700 1 -- 172.21.15.111:0/1136120263 wait complete. 2023-10-11T16:32:20.919 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:32:21.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.117+0000 7f671dfff700 1 -- 172.21.15.111:0/2175539272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101480 msgr2=0x7f67181018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.117+0000 7f671dfff700 1 --2- 172.21.15.111:0/2175539272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101480 0x7f67181018d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f670c0098f0 tx=0x7f670c009c00 comp rx=0 tx=0).stop 2023-10-11T16:32:21.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.118+0000 7f671dfff700 1 -- 172.21.15.111:0/2175539272 shutdown_connections 2023-10-11T16:32:21.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.118+0000 7f671dfff700 1 --2- 172.21.15.111:0/2175539272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718101e10 0x7f67181086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.118+0000 7f671dfff700 1 --2- 172.21.15.111:0/2175539272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6718100290 0x7f67181006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.118+0000 7f671dfff700 1 --2- 172.21.15.111:0/2175539272 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101480 0x7f67181018d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.118+0000 7f671dfff700 1 -- 172.21.15.111:0/2175539272 >> 172.21.15.111:0/2175539272 conn(0x7f67180fb870 msgr2=0x7f67180fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:21.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.118+0000 7f671dfff700 1 -- 172.21.15.111:0/2175539272 shutdown_connections 2023-10-11T16:32:21.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.118+0000 7f671dfff700 1 -- 172.21.15.111:0/2175539272 wait complete. 2023-10-11T16:32:21.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f671dfff700 1 Processor -- start 2023-10-11T16:32:21.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f671dfff700 1 -- start start 2023-10-11T16:32:21.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f671dfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6718100290 0x7f6718195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f671dfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718101480 0x7f6718196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f671dfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101e10 0x7f671819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f671dfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671819db60 con 0x7f6718101e10 2023-10-11T16:32:21.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f671dfff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671819dca0 con 0x7f6718101480 2023-10-11T16:32:21.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f671dfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671819de30 con 0x7f6718100290 2023-10-11T16:32:21.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f6717fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101e10 0x7f671819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f6717fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101e10 0x7f671819d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59806/0 (socket says 172.21.15.111:59806) 2023-10-11T16:32:21.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.119+0000 7f6717fff700 1 -- 172.21.15.111:0/2220232019 learned_addr learned my addr 172.21.15.111:0/2220232019 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:21.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f67177fe700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6718100290 0x7f6718195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6716ffd700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718101480 0x7f6718196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6717fff700 1 -- 172.21.15.111:0/2220232019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6718100290 msgr2=0x7f6718195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6717fff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6718100290 0x7f6718195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6717fff700 1 -- 172.21.15.111:0/2220232019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718101480 msgr2=0x7f6718196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6717fff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718101480 0x7f6718196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6717fff700 1 -- 172.21.15.111:0/2220232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c009520 con 0x7f6718101e10 2023-10-11T16:32:21.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6717fff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101e10 0x7f671819d590 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f671000ef30 tx=0x7f671000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:21.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6714ff9700 1 -- 172.21.15.111:0/2220232019 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710018070 con 0x7f6718101e10 2023-10-11T16:32:21.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6714ff9700 1 -- 172.21.15.111:0/2220232019 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f671000f040 con 0x7f6718101e10 2023-10-11T16:32:21.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671810bc50 con 0x7f6718101e10 2023-10-11T16:32:21.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.120+0000 7f6714ff9700 1 -- 172.21.15.111:0/2220232019 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671000ccf0 con 0x7f6718101e10 2023-10-11T16:32:21.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.121+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671810bac0 con 0x7f6718101e10 2023-10-11T16:32:21.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.122+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671819e510 con 0x7f6718101e10 2023-10-11T16:32:21.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.123+0000 7f6714ff9700 1 -- 172.21.15.111:0/2220232019 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f6710003a00 con 0x7f6718101e10 2023-10-11T16:32:21.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.126+0000 7f6714ff9700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67000a11d0 0x7f67000a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.126+0000 7f6714ff9700 1 -- 172.21.15.111:0/2220232019 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f67100cc7b0 con 0x7f6718101e10 2023-10-11T16:32:21.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.126+0000 7f67177fe700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67000a11d0 0x7f67000a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.127+0000 7f67177fe700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67000a11d0 0x7f67000a3680 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f67181012e0 tx=0x7f6708009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:21.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.130+0000 7f6714ff9700 1 -- 172.21.15.111:0/2220232019 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6710095b60 con 0x7f6718101e10 2023-10-11T16:32:21.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.331+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f671819ebb0 con 0x7f6718101e10 2023-10-11T16:32:21.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.332+0000 7f6714ff9700 1 -- 172.21.15.111:0/2220232019 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24870 (secure 0 0 0) 0x7f6710099180 con 0x7f6718101e10 2023-10-11T16:32:21.334 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:21.335 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":91,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:31:42.150221+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:32:21.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67000a11d0 msgr2=0x7f67000a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67000a11d0 0x7f67000a3680 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f67181012e0 tx=0x7f6708009310 comp rx=0 tx=0).stop 2023-10-11T16:32:21.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101e10 msgr2=0x7f671819d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101e10 0x7f671819d590 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f671000ef30 tx=0x7f671000c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 shutdown_connections 2023-10-11T16:32:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f67000a11d0 0x7f67000a3680 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6718101e10 0x7f671819d590 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6718100290 0x7f6718195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 --2- 172.21.15.111:0/2220232019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6718101480 0x7f6718196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 >> 172.21.15.111:0/2220232019 conn(0x7f67180fb870 msgr2=0x7f67180fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 shutdown_connections 2023-10-11T16:32:21.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.335+0000 7f671dfff700 1 -- 172.21.15.111:0/2220232019 wait complete. 2023-10-11T16:32:21.353 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:21 smithi111 ceph-mon[98527]: pgmap v341: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-10-11T16:32:21.354 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:21 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2242789613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:21.354 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:21 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1136120263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:21.356 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:32:21.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:32:21.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:21 smithi150 ceph-mon[132217]: pgmap v341: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-10-11T16:32:21.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:21 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2242789613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:21.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:21 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1136120263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:21.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 -- 172.21.15.111:0/2984736841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 msgr2=0x7f02d40739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2984736841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 0x7f02d40739c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f02c8009990 tx=0x7f02c8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 -- 172.21.15.111:0/2984736841 shutdown_connections 2023-10-11T16:32:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2984736841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02d410c8a0 0x7f02d410ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2984736841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02d4074b30 0x7f02d4072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2984736841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 0x7f02d40739c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 -- 172.21.15.111:0/2984736841 >> 172.21.15.111:0/2984736841 conn(0x7f02d40fb8b0 msgr2=0x7f02d40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:21.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 -- 172.21.15.111:0/2984736841 shutdown_connections 2023-10-11T16:32:21.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.556+0000 7f02da3ff700 1 -- 172.21.15.111:0/2984736841 wait complete. 2023-10-11T16:32:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.557+0000 7f02da3ff700 1 Processor -- start 2023-10-11T16:32:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.557+0000 7f02da3ff700 1 -- start start 2023-10-11T16:32:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02da3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 0x7f02d419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02da3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02d4074b30 0x7f02d419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02da3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02d410c8a0 0x7f02d41a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02da3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d41a4210 con 0x7f02d4073550 2023-10-11T16:32:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02da3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d41a4350 con 0x7f02d410c8a0 2023-10-11T16:32:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02da3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d41a44e0 con 0x7f02d4074b30 2023-10-11T16:32:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 0x7f02d419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 0x7f02d419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59818/0 (socket says 172.21.15.111:59818) 2023-10-11T16:32:21.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d3fff700 1 -- 172.21.15.111:0/2313077903 learned_addr learned my addr 172.21.15.111:0/2313077903 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:21.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d37fe700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02d4074b30 0x7f02d419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d899c700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02d410c8a0 0x7f02d41a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d3fff700 1 -- 172.21.15.111:0/2313077903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02d4074b30 msgr2=0x7f02d419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d3fff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02d4074b30 0x7f02d419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d3fff700 1 -- 172.21.15.111:0/2313077903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02d410c8a0 msgr2=0x7f02d41a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d3fff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02d410c8a0 0x7f02d41a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d3fff700 1 -- 172.21.15.111:0/2313077903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c80095f0 con 0x7f02d4073550 2023-10-11T16:32:21.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.558+0000 7f02d899c700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02d410c8a0 0x7f02d41a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:21.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.559+0000 7f02d3fff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 0x7f02d419c690 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f02c400e970 tx=0x7f02c400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:21.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.559+0000 7f02d17fa700 1 -- 172.21.15.111:0/2313077903 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4009790 con 0x7f02d4073550 2023-10-11T16:32:21.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.559+0000 7f02d17fa700 1 -- 172.21.15.111:0/2313077903 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02c400eec0 con 0x7f02d4073550 2023-10-11T16:32:21.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.559+0000 7f02d17fa700 1 -- 172.21.15.111:0/2313077903 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4005590 con 0x7f02d4073550 2023-10-11T16:32:21.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.559+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d41122b0 con 0x7f02d4073550 2023-10-11T16:32:21.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.559+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d4112120 con 0x7f02d4073550 2023-10-11T16:32:21.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.560+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d41023f0 con 0x7f02d4073550 2023-10-11T16:32:21.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.562+0000 7f02d17fa700 1 -- 172.21.15.111:0/2313077903 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f02c400cbd0 con 0x7f02d4073550 2023-10-11T16:32:21.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.563+0000 7f02d17fa700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f02bc0a1250 0x7f02bc0a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.563+0000 7f02d17fa700 1 -- 172.21.15.111:0/2313077903 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f02c4014070 con 0x7f02d4073550 2023-10-11T16:32:21.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.563+0000 7f02d37fe700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f02bc0a1250 0x7f02bc0a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.564+0000 7f02d37fe700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f02bc0a1250 0x7f02bc0a3700 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f02c800b2e0 tx=0x7f02c8005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:21.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.567+0000 7f02d17fa700 1 -- 172.21.15.111:0/2313077903 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f02c4096250 con 0x7f02d4073550 2023-10-11T16:32:21.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:21 smithi130 ceph-mon[131859]: pgmap v341: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-10-11T16:32:21.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:21 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2242789613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:21.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:21 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1136120263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:21.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.766+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02d4061700 con 0x7f02d4073550 2023-10-11T16:32:21.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.767+0000 7f02d17fa700 1 -- 172.21.15.111:0/2313077903 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02c4004ef0 con 0x7f02d4073550 2023-10-11T16:32:21.769 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:32:21.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f02bc0a1250 msgr2=0x7f02bc0a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f02bc0a1250 0x7f02bc0a3700 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f02c800b2e0 tx=0x7f02c8005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 msgr2=0x7f02d419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 0x7f02d419c690 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f02c400e970 tx=0x7f02c400ec80 comp rx=0 tx=0).stop 2023-10-11T16:32:21.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 shutdown_connections 2023-10-11T16:32:21.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f02bc0a1250 0x7f02bc0a3700 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02d410c8a0 0x7f02d41a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02d4073550 0x7f02d419c690 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 --2- 172.21.15.111:0/2313077903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02d4074b30 0x7f02d419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.769+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 >> 172.21.15.111:0/2313077903 conn(0x7f02d40fb8b0 msgr2=0x7f02d40fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:21.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.770+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 shutdown_connections 2023-10-11T16:32:21.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.770+0000 7f02da3ff700 1 -- 172.21.15.111:0/2313077903 wait complete. 2023-10-11T16:32:21.790 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:32:21.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 -- 172.21.15.111:0/1898628148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01002c0 msgr2=0x7fa9b01006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/1898628148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01002c0 0x7fa9b01006d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009990 tx=0x7fa9ac009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 -- 172.21.15.111:0/1898628148 shutdown_connections 2023-10-11T16:32:21.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/1898628148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9b0101e40 0x7fa9b0108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/1898628148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01002c0 0x7fa9b01006d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/1898628148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9b01014b0 0x7fa9b0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 -- 172.21.15.111:0/1898628148 >> 172.21.15.111:0/1898628148 conn(0x7fa9b00fb8a0 msgr2=0x7fa9b00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:21.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 -- 172.21.15.111:0/1898628148 shutdown_connections 2023-10-11T16:32:21.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.989+0000 7fa9b81ff700 1 -- 172.21.15.111:0/1898628148 wait complete. 2023-10-11T16:32:21.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b81ff700 1 Processor -- start 2023-10-11T16:32:21.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b81ff700 1 -- start start 2023-10-11T16:32:21.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b81ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9b01002c0 0x7fa9b0196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b81ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01014b0 0x7fa9b01965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b81ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9b0101e40 0x7fa9b019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:21.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b81ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b019dc10 con 0x7fa9b01014b0 2023-10-11T16:32:21.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b81ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b019dd50 con 0x7fa9b01002c0 2023-10-11T16:32:21.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b81ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b019dee0 con 0x7fa9b0101e40 2023-10-11T16:32:21.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.990+0000 7fa9b579a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01014b0 0x7fa9b01965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b579a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01014b0 0x7fa9b01965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59842/0 (socket says 172.21.15.111:59842) 2023-10-11T16:32:21.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b579a700 1 -- 172.21.15.111:0/2499712624 learned_addr learned my addr 172.21.15.111:0/2499712624 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:21.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b5f9b700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9b01002c0 0x7fa9b0196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b679c700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9b0101e40 0x7fa9b019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:21.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b579a700 1 -- 172.21.15.111:0/2499712624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9b0101e40 msgr2=0x7fa9b019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b579a700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9b0101e40 0x7fa9b019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b579a700 1 -- 172.21.15.111:0/2499712624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9b01002c0 msgr2=0x7fa9b0196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:21.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b579a700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9b01002c0 0x7fa9b0196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:21.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b579a700 1 -- 172.21.15.111:0/2499712624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac0095f0 con 0x7fa9b01014b0 2023-10-11T16:32:21.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b5f9b700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9b01002c0 0x7fa9b0196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:21.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b579a700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01014b0 0x7fa9b01965d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000e9a0 tx=0x7fa9a000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:21.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9a6ffd700 1 -- 172.21.15.111:0/2499712624 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0009790 con 0x7fa9b01014b0 2023-10-11T16:32:21.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b010bc90 con 0x7fa9b01014b0 2023-10-11T16:32:21.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.991+0000 7fa9a6ffd700 1 -- 172.21.15.111:0/2499712624 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9a0010430 con 0x7fa9b01014b0 2023-10-11T16:32:21.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.992+0000 7fa9a6ffd700 1 -- 172.21.15.111:0/2499712624 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0018680 con 0x7fa9b01014b0 2023-10-11T16:32:22.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.992+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b010bb00 con 0x7fa9b01014b0 2023-10-11T16:32:22.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.993+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b0107850 con 0x7fa9b01014b0 2023-10-11T16:32:22.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.995+0000 7fa9a6ffd700 1 -- 172.21.15.111:0/2499712624 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fa9a00105a0 con 0x7fa9b01014b0 2023-10-11T16:32:22.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.996+0000 7fa9a6ffd700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa99c0a1150 0x7fa99c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.996+0000 7fa9a6ffd700 1 -- 172.21.15.111:0/2499712624 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fa9a0014070 con 0x7fa9b01014b0 2023-10-11T16:32:22.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.996+0000 7fa9b5f9b700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa99c0a1150 0x7fa99c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.997+0000 7fa9b5f9b700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa99c0a1150 0x7fa99c0a3600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00b2e0 tx=0x7fa9ac005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:22.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:21.998+0000 7fa9a6ffd700 1 -- 172.21.15.111:0/2499712624 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa9a0096420 con 0x7fa9b01014b0 2023-10-11T16:32:22.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.201+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa9b0067b00 con 0x7fa9b01014b0 2023-10-11T16:32:22.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.202+0000 7fa9a6ffd700 1 -- 172.21.15.111:0/2499712624 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa9a00260a0 con 0x7fa9b01014b0 2023-10-11T16:32:22.204 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:22.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa99c0a1150 msgr2=0x7fa99c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa99c0a1150 0x7fa99c0a3600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00b2e0 tx=0x7fa9ac005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01014b0 msgr2=0x7fa9b01965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01014b0 0x7fa9b01965d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000e9a0 tx=0x7fa9a000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 shutdown_connections 2023-10-11T16:32:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fa99c0a1150 0x7fa99c0a3600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9b0101e40 0x7fa9b019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9b01002c0 0x7fa9b0196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 --2- 172.21.15.111:0/2499712624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b01014b0 0x7fa9b01965d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 >> 172.21.15.111:0/2499712624 conn(0x7fa9b00fb8a0 msgr2=0x7fa9b0104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 shutdown_connections 2023-10-11T16:32:22.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.204+0000 7fa9b81ff700 1 -- 172.21.15.111:0/2499712624 wait complete. 2023-10-11T16:32:22.225 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:32:22.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 -- 172.21.15.111:0/3756155280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6810c8a0 msgr2=0x7fde6810ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3756155280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6810c8a0 0x7fde6810ec80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fde6000b0a0 tx=0x7fde6000b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 -- 172.21.15.111:0/3756155280 shutdown_connections 2023-10-11T16:32:22.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3756155280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde6810c8a0 0x7fde6810ec80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3756155280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde68074b30 0x7fde68072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3756155280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde68073550 0x7fde680739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 -- 172.21.15.111:0/3756155280 >> 172.21.15.111:0/3756155280 conn(0x7fde680fb8b0 msgr2=0x7fde680fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:22.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 -- 172.21.15.111:0/3756155280 shutdown_connections 2023-10-11T16:32:22.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.424+0000 7fde6fbff700 1 -- 172.21.15.111:0/3756155280 wait complete. 2023-10-11T16:32:22.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.425+0000 7fde6fbff700 1 Processor -- start 2023-10-11T16:32:22.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.425+0000 7fde6fbff700 1 -- start start 2023-10-11T16:32:22.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6fbff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde68073550 0x7fde6819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6fbff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde68074b30 0x7fde6819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6fbff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6810c8a0 0x7fde681a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6fbff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde681a4300 con 0x7fde68074b30 2023-10-11T16:32:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6fbff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde681a4440 con 0x7fde6810c8a0 2023-10-11T16:32:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6fbff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde681a45d0 con 0x7fde68073550 2023-10-11T16:32:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d99b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde68073550 0x7fde6819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d19a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde68074b30 0x7fde6819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d99b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde68073550 0x7fde6819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:44614/0 (socket says 172.21.15.111:44614) 2023-10-11T16:32:22.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d99b700 1 -- 172.21.15.111:0/3919639538 learned_addr learned my addr 172.21.15.111:0/3919639538 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:22.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6e19c700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6810c8a0 0x7fde681a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d19a700 1 -- 172.21.15.111:0/3919639538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde68073550 msgr2=0x7fde6819c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d19a700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde68073550 0x7fde6819c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d19a700 1 -- 172.21.15.111:0/3919639538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6810c8a0 msgr2=0x7fde681a3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d19a700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6810c8a0 0x7fde681a3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d19a700 1 -- 172.21.15.111:0/3919639538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde60009d00 con 0x7fde68074b30 2023-10-11T16:32:22.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.426+0000 7fde6d99b700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde68073550 0x7fde6819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:22.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.427+0000 7fde6d19a700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde68074b30 0x7fde6819ccc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fde5800e9c0 tx=0x7fde5800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:22.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.427+0000 7fde5effd700 1 -- 172.21.15.111:0/3919639538 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58004220 con 0x7fde68074b30 2023-10-11T16:32:22.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.427+0000 7fde5effd700 1 -- 172.21.15.111:0/3919639538 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde58004380 con 0x7fde68074b30 2023-10-11T16:32:22.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.427+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde681122b0 con 0x7fde68074b30 2023-10-11T16:32:22.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.427+0000 7fde5effd700 1 -- 172.21.15.111:0/3919639538 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58003be0 con 0x7fde68074b30 2023-10-11T16:32:22.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.427+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68112120 con 0x7fde68074b30 2023-10-11T16:32:22.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.428+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde681023f0 con 0x7fde68074b30 2023-10-11T16:32:22.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.430+0000 7fde5effd700 1 -- 172.21.15.111:0/3919639538 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fde58003d40 con 0x7fde68074b30 2023-10-11T16:32:22.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.431+0000 7fde5effd700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fde540a1150 0x7fde540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.432+0000 7fde5effd700 1 -- 172.21.15.111:0/3919639538 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fde58014070 con 0x7fde68074b30 2023-10-11T16:32:22.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.432+0000 7fde6d99b700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fde540a1150 0x7fde540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.432+0000 7fde6d99b700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fde540a1150 0x7fde540a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fde64005950 tx=0x7fde64009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:22.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.435+0000 7fde5effd700 1 -- 172.21.15.111:0/3919639538 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fde580d0030 con 0x7fde68074b30 2023-10-11T16:32:22.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:22 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2220232019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:22.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:22 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2313077903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:22.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:22 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2220232019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:22.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:22 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2313077903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:22.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:22 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2220232019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:22.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:22 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2313077903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:22.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.638+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde68061bb0 con 0x7fde68074b30 2023-10-11T16:32:22.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.638+0000 7fde5effd700 1 -- 172.21.15.111:0/3919639538 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde580962f0 con 0x7fde68074b30 2023-10-11T16:32:22.640 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:32:22.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fde540a1150 msgr2=0x7fde540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fde540a1150 0x7fde540a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fde64005950 tx=0x7fde64009260 comp rx=0 tx=0).stop 2023-10-11T16:32:22.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde68074b30 msgr2=0x7fde6819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde68074b30 0x7fde6819ccc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fde5800e9c0 tx=0x7fde5800ecd0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 shutdown_connections 2023-10-11T16:32:22.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fde540a1150 0x7fde540a3600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde6810c8a0 0x7fde681a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde68073550 0x7fde6819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 --2- 172.21.15.111:0/3919639538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde68074b30 0x7fde6819ccc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 >> 172.21.15.111:0/3919639538 conn(0x7fde680fb8b0 msgr2=0x7fde680fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:22.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 shutdown_connections 2023-10-11T16:32:22.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.641+0000 7fde6fbff700 1 -- 172.21.15.111:0/3919639538 wait complete. 2023-10-11T16:32:22.662 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:32:22.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 -- 172.21.15.111:0/1934379496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1002d0 msgr2=0x7f1edc1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/1934379496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1002d0 0x7f1edc1006e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009990 tx=0x7f1ecc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 -- 172.21.15.111:0/1934379496 shutdown_connections 2023-10-11T16:32:22.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/1934379496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1edc101e50 0x7f1edc108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/1934379496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1002d0 0x7f1edc1006e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/1934379496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1edc1014c0 0x7f1edc101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 -- 172.21.15.111:0/1934379496 >> 172.21.15.111:0/1934379496 conn(0x7f1edc0fb8b0 msgr2=0x7f1edc0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:22.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 -- 172.21.15.111:0/1934379496 shutdown_connections 2023-10-11T16:32:22.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.862+0000 7f1ee29ff700 1 -- 172.21.15.111:0/1934379496 wait complete. 2023-10-11T16:32:22.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1ee29ff700 1 Processor -- start 2023-10-11T16:32:22.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1ee29ff700 1 -- start start 2023-10-11T16:32:22.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1ee29ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1edc1002d0 0x7f1edc198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1ee29ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1014c0 0x7f1edc198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1ee29ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1edc101e50 0x7f1edc19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1ee29ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc19feb0 con 0x7f1edc1002d0 2023-10-11T16:32:22.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1ee29ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc19fff0 con 0x7f1edc1014c0 2023-10-11T16:32:22.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1ee29ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc1a0180 con 0x7f1edc101e50 2023-10-11T16:32:22.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.863+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1edc1002d0 0x7f1edc198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1014c0 0x7f1edc198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1edc1002d0 0x7f1edc198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59874/0 (socket says 172.21.15.111:59874) 2023-10-11T16:32:22.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1014c0 0x7f1edc198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:38150/0 (socket says 172.21.15.111:38150) 2023-10-11T16:32:22.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edbfff700 1 -- 172.21.15.111:0/3889813044 learned_addr learned my addr 172.21.15.111:0/3889813044 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:22.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1ee0f9c700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1edc101e50 0x7f1edc19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edbfff700 1 -- 172.21.15.111:0/3889813044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1edc101e50 msgr2=0x7f1edc19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edbfff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1edc101e50 0x7f1edc19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edbfff700 1 -- 172.21.15.111:0/3889813044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1014c0 msgr2=0x7f1edc198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:22.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edbfff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1014c0 0x7f1edc198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:22.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edbfff700 1 -- 172.21.15.111:0/3889813044 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc0095f0 con 0x7f1edc1002d0 2023-10-11T16:32:22.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edb7fe700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1014c0 0x7f1edc198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:22.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1ee0f9c700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1edc101e50 0x7f1edc19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:22.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1edbfff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1edc1002d0 0x7f1edc198330 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc004730 tx=0x7f1ecc004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:22.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1ed97fa700 1 -- 172.21.15.111:0/3889813044 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc01d070 con 0x7f1edc1002d0 2023-10-11T16:32:22.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc10bca0 con 0x7f1edc1002d0 2023-10-11T16:32:22.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1ed97fa700 1 -- 172.21.15.111:0/3889813044 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ecc021410 con 0x7f1edc1002d0 2023-10-11T16:32:22.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.864+0000 7f1ed97fa700 1 -- 172.21.15.111:0/3889813044 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc017650 con 0x7f1edc1002d0 2023-10-11T16:32:22.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.865+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc10bb10 con 0x7f1edc1002d0 2023-10-11T16:32:22.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.866+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc192a00 con 0x7f1edc1002d0 2023-10-11T16:32:22.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.867+0000 7f1ed97fa700 1 -- 172.21.15.111:0/3889813044 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f1ecc021810 con 0x7f1edc1002d0 2023-10-11T16:32:22.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.869+0000 7f1ed97fa700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ec40a1150 0x7f1ec40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:22.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.869+0000 7f1ed97fa700 1 -- 172.21.15.111:0/3889813044 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f1ecc0ce1e0 con 0x7f1edc1002d0 2023-10-11T16:32:22.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.869+0000 7f1edb7fe700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ec40a1150 0x7f1ec40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:22.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.870+0000 7f1edb7fe700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ec40a1150 0x7f1ec40a3600 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0011490 tx=0x7f1ed0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:22.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:22.872+0000 7f1ed97fa700 1 -- 172.21.15.111:0/3889813044 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1ecc097590 con 0x7f1edc1002d0 2023-10-11T16:32:23.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.081+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1edc061c40 con 0x7f1edc1002d0 2023-10-11T16:32:23.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.082+0000 7f1ed97fa700 1 -- 172.21.15.111:0/3889813044 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1ecc095e60 con 0x7f1edc1002d0 2023-10-11T16:32:23.084 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:23.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ec40a1150 msgr2=0x7f1ec40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ec40a1150 0x7f1ec40a3600 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0011490 tx=0x7f1ed0009260 comp rx=0 tx=0).stop 2023-10-11T16:32:23.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1edc1002d0 msgr2=0x7f1edc198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1edc1002d0 0x7f1edc198330 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc004730 tx=0x7f1ecc004760 comp rx=0 tx=0).stop 2023-10-11T16:32:23.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 shutdown_connections 2023-10-11T16:32:23.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1ec40a1150 0x7f1ec40a3600 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1edc101e50 0x7f1edc19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1edc1002d0 0x7f1edc198330 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 --2- 172.21.15.111:0/3889813044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc1014c0 0x7f1edc198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 >> 172.21.15.111:0/3889813044 conn(0x7f1edc0fb8b0 msgr2=0x7f1edc106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:23.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 shutdown_connections 2023-10-11T16:32:23.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.084+0000 7f1ee29ff700 1 -- 172.21.15.111:0/3889813044 wait complete. 2023-10-11T16:32:23.104 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2023-10-11T16:32:23.104 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:32:23.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 -- 172.21.15.111:0/3911526119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 msgr2=0x7f6f601006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 --2- 172.21.15.111:0/3911526119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 0x7f6f601006c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009990 tx=0x7f6f50009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 -- 172.21.15.111:0/3911526119 shutdown_connections 2023-10-11T16:32:23.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 --2- 172.21.15.111:0/3911526119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f60101e30 0x7f6f601086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 --2- 172.21.15.111:0/3911526119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 0x7f6f601006c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 --2- 172.21.15.111:0/3911526119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f601014a0 0x7f6f601018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 -- 172.21.15.111:0/3911526119 >> 172.21.15.111:0/3911526119 conn(0x7f6f600fb890 msgr2=0x7f6f600fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:23.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 -- 172.21.15.111:0/3911526119 shutdown_connections 2023-10-11T16:32:23.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.302+0000 7f6f65dff700 1 -- 172.21.15.111:0/3911526119 wait complete. 2023-10-11T16:32:23.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.303+0000 7f6f65dff700 1 Processor -- start 2023-10-11T16:32:23.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.303+0000 7f6f65dff700 1 -- start start 2023-10-11T16:32:23.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.303+0000 7f6f65dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 0x7f6f60196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:23.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.303+0000 7f6f65dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f601014a0 0x7f6f60196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:23.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.303+0000 7f6f65dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f60101e30 0x7f6f6019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:23.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.303+0000 7f6f65dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019dbc0 con 0x7f6f601002b0 2023-10-11T16:32:23.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.303+0000 7f6f65dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019dd00 con 0x7f6f60101e30 2023-10-11T16:32:23.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.303+0000 7f6f65dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019de90 con 0x7f6f601014a0 2023-10-11T16:32:23.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 0x7f6f60196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:23.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 0x7f6f60196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59892/0 (socket says 172.21.15.111:59892) 2023-10-11T16:32:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 -- 172.21.15.111:0/856548750 learned_addr learned my addr 172.21.15.111:0/856548750 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5ffff700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f60101e30 0x7f6f6019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5effd700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f601014a0 0x7f6f60196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:23.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 -- 172.21.15.111:0/856548750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f601014a0 msgr2=0x7f6f60196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f601014a0 0x7f6f60196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 -- 172.21.15.111:0/856548750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f60101e30 msgr2=0x7f6f6019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f60101e30 0x7f6f6019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 -- 172.21.15.111:0/856548750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f54009520 con 0x7f6f601002b0 2023-10-11T16:32:23.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.304+0000 7f6f5f7fe700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 0x7f6f60196040 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6f50005740 tx=0x7f6f50005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:23.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.305+0000 7f6f5cff9700 1 -- 172.21.15.111:0/856548750 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5001d070 con 0x7f6f601002b0 2023-10-11T16:32:23.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.305+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f500095f0 con 0x7f6f601002b0 2023-10-11T16:32:23.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.305+0000 7f6f5cff9700 1 -- 172.21.15.111:0/856548750 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f50021410 con 0x7f6f601002b0 2023-10-11T16:32:23.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.305+0000 7f6f5cff9700 1 -- 172.21.15.111:0/856548750 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50017650 con 0x7f6f601002b0 2023-10-11T16:32:23.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.305+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6010bc80 con 0x7f6f601002b0 2023-10-11T16:32:23.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.306+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6010be10 con 0x7f6f601002b0 2023-10-11T16:32:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.309+0000 7f6f5cff9700 1 -- 172.21.15.111:0/856548750 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f6f50021580 con 0x7f6f601002b0 2023-10-11T16:32:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.310+0000 7f6f5cff9700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6f480a1150 0x7f6f480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.310+0000 7f6f5cff9700 1 -- 172.21.15.111:0/856548750 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f6f500ce2e0 con 0x7f6f601002b0 2023-10-11T16:32:23.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.310+0000 7f6f5effd700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6f480a1150 0x7f6f480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:23.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.311+0000 7f6f5effd700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6f480a1150 0x7f6f480a3600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009df0 tx=0x7f6f54009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:23.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.312+0000 7f6f5cff9700 1 -- 172.21.15.111:0/856548750 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6f50097690 con 0x7f6f601002b0 2023-10-11T16:32:23.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:23 smithi111 ceph-mon[98527]: pgmap v342: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-10-11T16:32:23.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:23 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2499712624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:23 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3919639538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:23 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3889813044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:23 smithi150 ceph-mon[132217]: pgmap v342: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-10-11T16:32:23.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:23 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2499712624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:23 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3919639538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:23 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3889813044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.556+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f6002cfe0 con 0x7f6f601002b0 2023-10-11T16:32:23.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.557+0000 7f6f5cff9700 1 -- 172.21.15.111:0/856548750 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f6f5009acb0 con 0x7f6f601002b0 2023-10-11T16:32:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:23.560 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:51.227688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:32:23.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.559+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6f480a1150 msgr2=0x7f6f480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.559+0000 7f6f65dff700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6f480a1150 0x7f6f480a3600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009df0 tx=0x7f6f54009260 comp rx=0 tx=0).stop 2023-10-11T16:32:23.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.559+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 msgr2=0x7f6f60196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.559+0000 7f6f65dff700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 0x7f6f60196040 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6f50005740 tx=0x7f6f50005770 comp rx=0 tx=0).stop 2023-10-11T16:32:23.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.560+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 shutdown_connections 2023-10-11T16:32:23.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.560+0000 7f6f65dff700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f6f480a1150 0x7f6f480a3600 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.560+0000 7f6f65dff700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6f60101e30 0x7f6f6019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.560+0000 7f6f65dff700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f601002b0 0x7f6f60196040 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.560+0000 7f6f65dff700 1 --2- 172.21.15.111:0/856548750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f601014a0 0x7f6f60196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.560+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 >> 172.21.15.111:0/856548750 conn(0x7f6f600fb890 msgr2=0x7f6f60104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:23.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.560+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 shutdown_connections 2023-10-11T16:32:23.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.560+0000 7f6f65dff700 1 -- 172.21.15.111:0/856548750 wait complete. 2023-10-11T16:32:23.564 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-10-11T16:32:23.580 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2023-10-11T16:32:23.580 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2023-10-11T16:32:23.581 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T16:32:23.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:23 smithi130 ceph-mon[131859]: pgmap v342: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-10-11T16:32:23.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:23 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2499712624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:23 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3919639538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:23 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3889813044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.778+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1819796065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 msgr2=0x7f20a40739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.778+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1819796065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 0x7f20a40739c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f20980098f0 tx=0x7f2098009c00 comp rx=0 tx=0).stop 2023-10-11T16:32:23.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.778+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1819796065 shutdown_connections 2023-10-11T16:32:23.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.778+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1819796065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20a410c860 0x7f20a410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.778+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1819796065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20a4074b30 0x7f20a4072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.778+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1819796065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 0x7f20a40739c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.778+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1819796065 >> 172.21.15.111:0/1819796065 conn(0x7f20a40fb870 msgr2=0x7f20a40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:23.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.779+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1819796065 shutdown_connections 2023-10-11T16:32:23.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.779+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1819796065 wait complete. 2023-10-11T16:32:23.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20aa9ff700 1 Processor -- start 2023-10-11T16:32:23.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20aa9ff700 1 -- start start 2023-10-11T16:32:23.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20aa9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 0x7f20a419c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:23.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20aa9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20a4074b30 0x7f20a419cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:23.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20aa9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20a410c860 0x7f20a41a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:23.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20aa9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a41a4190 con 0x7f20a4074b30 2023-10-11T16:32:23.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20aa9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a41a42d0 con 0x7f20a4073550 2023-10-11T16:32:23.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20aa9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a41a4460 con 0x7f20a410c860 2023-10-11T16:32:23.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20a37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20a4074b30 0x7f20a419cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:23.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20a37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20a4074b30 0x7f20a419cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59916/0 (socket says 172.21.15.111:59916) 2023-10-11T16:32:23.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20a37fe700 1 -- 172.21.15.111:0/1052152521 learned_addr learned my addr 172.21.15.111:0/1052152521 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:23.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20a3fff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 0x7f20a419c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:23.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.780+0000 7f20a8f9c700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20a410c860 0x7f20a41a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:23.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a37fe700 1 -- 172.21.15.111:0/1052152521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20a410c860 msgr2=0x7f20a41a3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a37fe700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20a410c860 0x7f20a41a3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a37fe700 1 -- 172.21.15.111:0/1052152521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 msgr2=0x7f20a419c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:23.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a37fe700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 0x7f20a419c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:23.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a37fe700 1 -- 172.21.15.111:0/1052152521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20940095f0 con 0x7f20a4074b30 2023-10-11T16:32:23.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a3fff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 0x7f20a419c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:23.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a37fe700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20a4074b30 0x7f20a419cb50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2098000c00 tx=0x7f209800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:23.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a17fa700 1 -- 172.21.15.111:0/1052152521 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209801c070 con 0x7f20a4074b30 2023-10-11T16:32:23.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a17fa700 1 -- 172.21.15.111:0/1052152521 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f209800f7d0 con 0x7f20a4074b30 2023-10-11T16:32:23.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2098009520 con 0x7f20a4074b30 2023-10-11T16:32:23.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20a17fa700 1 -- 172.21.15.111:0/1052152521 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098003ba0 con 0x7f20a4074b30 2023-10-11T16:32:23.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.781+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a4112220 con 0x7f20a4074b30 2023-10-11T16:32:23.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.782+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a41023b0 con 0x7f20a4074b30 2023-10-11T16:32:23.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.784+0000 7f20a17fa700 1 -- 172.21.15.111:0/1052152521 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f2098020030 con 0x7f20a4074b30 2023-10-11T16:32:23.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.786+0000 7f20a17fa700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f208c0a11d0 0x7f208c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:23.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.786+0000 7f20a17fa700 1 -- 172.21.15.111:0/1052152521 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f20980cdd80 con 0x7f20a4074b30 2023-10-11T16:32:23.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.786+0000 7f20a3fff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f208c0a11d0 0x7f208c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:23.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.786+0000 7f20a3fff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f208c0a11d0 0x7f208c0a3680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f20a4074f00 tx=0x7f2094009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:23.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.789+0000 7f20a17fa700 1 -- 172.21.15.111:0/1052152521 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2098097130 con 0x7f20a4074b30 2023-10-11T16:32:23.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.984+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20a4061900 con 0x7f208c0a11d0 2023-10-11T16:32:23.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:23.998+0000 7f20a17fa700 1 -- 172.21.15.111:0/1052152521 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427540 (secure 0 0 0) 0x7f20a4061900 con 0x7f208c0a11d0 2023-10-11T16:32:24.005 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:24.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f208c0a11d0 msgr2=0x7f208c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f208c0a11d0 0x7f208c0a3680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f20a4074f00 tx=0x7f2094009310 comp rx=0 tx=0).stop 2023-10-11T16:32:24.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20a4074b30 msgr2=0x7f20a419cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20a4074b30 0x7f20a419cb50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2098000c00 tx=0x7f209800b840 comp rx=0 tx=0).stop 2023-10-11T16:32:24.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 shutdown_connections 2023-10-11T16:32:24.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f208c0a11d0 0x7f208c0a3680 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20a410c860 0x7f20a41a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20a4073550 0x7f20a419c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 --2- 172.21.15.111:0/1052152521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20a4074b30 0x7f20a419cb50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 >> 172.21.15.111:0/1052152521 conn(0x7f20a40fb870 msgr2=0x7f20a40fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 shutdown_connections 2023-10-11T16:32:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.003+0000 7f20aa9ff700 1 -- 172.21.15.111:0/1052152521 wait complete. 2023-10-11T16:32:24.013 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:32:24.023 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2023-10-11T16:32:22.086984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670932,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":766,"num_read_kb":515,"num_write":357,"num_write_kb":3077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1220437,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82464,"kb_used_data":12160,"kb_used_omap":0,"kb_used_meta":70208,"kb_avail":1124859360,"statfs":{"total":1151940427776,"available":1151855984640,"internally_reserved":0,"allocated":12451840,"data_stored":8079545,"data_compressed":395806,"data_compressed_allocated":536576,"data_compressed_original":2592768,"omap_allocated":0,"internal_metadata":71892992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":706,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":33,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003564"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100200+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:32:10.100200+0000","last_peered":"2023-10-11T16:32:10.100200+0000","last_clean":"2023-10-11T16:32:10.100200+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:32:10.100200+0000","last_undegraded":"2023-10-11T16:32:10.100200+0000","last_fullsized":"2023-10-11T16:32:10.100200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:26:00.751881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100937+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:32:10.100937+0000","last_peered":"2023-10-11T16:32:10.100937+0000","last_clean":"2023-10-11T16:32:10.100937+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:32:10.100937+0000","last_undegraded":"2023-10-11T16:32:10.100937+0000","last_fullsized":"2023-10-11T16:32:10.100937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:51:20.593620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101715+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:32:10.101715+0000","last_peered":"2023-10-11T16:32:10.101715+0000","last_clean":"2023-10-11T16:32:10.101715+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:32:10.101715+0000","last_undegraded":"2023-10-11T16:32:10.101715+0000","last_fullsized":"2023-10-11T16:32:10.101715+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:20:45.562554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083360+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:32:10.083360+0000","last_peered":"2023-10-11T16:32:10.083360+0000","last_clean":"2023-10-11T16:32:10.083360+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:32:10.083360+0000","last_undegraded":"2023-10-11T16:32:10.083360+0000","last_fullsized":"2023-10-11T16:32:10.083360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:03:22.258325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.071382+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:32:10.071382+0000","last_peered":"2023-10-11T16:32:10.071382+0000","last_clean":"2023-10-11T16:32:10.071382+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:32:10.071382+0000","last_undegraded":"2023-10-11T16:32:10.071382+0000","last_fullsized":"2023-10-11T16:32:10.071382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:59:15.949889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.086763+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:32:10.086763+0000","last_peered":"2023-10-11T16:32:10.086763+0000","last_clean":"2023-10-11T16:32:10.086763+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:32:10.086763+0000","last_undegraded":"2023-10-11T16:32:10.086763+0000","last_fullsized":"2023-10-11T16:32:10.086763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:17:51.767698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.087839+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:32:10.087839+0000","last_peered":"2023-10-11T16:32:10.087839+0000","last_clean":"2023-10-11T16:32:10.087839+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:32:10.087839+0000","last_undegraded":"2023-10-11T16:32:10.087839+0000","last_fullsized":"2023-10-11T16:32:10.087839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:31:16.375555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.099303+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:32:10.099303+0000","last_peered":"2023-10-11T16:32:10.099303+0000","last_clean":"2023-10-11T16:32:10.099303+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:32:10.099303+0000","last_undegraded":"2023-10-11T16:32:10.099303+0000","last_fullsized":"2023-10-11T16:32:10.099303+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:14:00.865418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092992+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:32:10.092992+0000","last_peered":"2023-10-11T16:32:10.092992+0000","last_clean":"2023-10-11T16:32:10.092992+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:32:10.092992+0000","last_undegraded":"2023-10-11T16:32:10.092992+0000","last_fullsized":"2023-10-11T16:32:10.092992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:48:35.792817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.091540+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:32:10.091540+0000","last_peered":"2023-10-11T16:32:10.091540+0000","last_clean":"2023-10-11T16:32:10.091540+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:32:10.091540+0000","last_undegraded":"2023-10-11T16:32:10.091540+0000","last_fullsized":"2023-10-11T16:32:10.091540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:35.706510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.094525+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:32:10.094525+0000","last_peered":"2023-10-11T16:32:10.094525+0000","last_clean":"2023-10-11T16:32:10.094525+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:32:10.094525+0000","last_undegraded":"2023-10-11T16:32:10.094525+0000","last_fullsized":"2023-10-11T16:32:10.094525+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:39:30.482685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089635+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:32:10.089635+0000","last_peered":"2023-10-11T16:32:10.089635+0000","last_clean":"2023-10-11T16:32:10.089635+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:32:10.089635+0000","last_undegraded":"2023-10-11T16:32:10.089635+0000","last_fullsized":"2023-10-11T16:32:10.089635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:34:42.597400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089114+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:32:10.089114+0000","last_peered":"2023-10-11T16:32:10.089114+0000","last_clean":"2023-10-11T16:32:10.089114+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:32:10.089114+0000","last_undegraded":"2023-10-11T16:32:10.089114+0000","last_fullsized":"2023-10-11T16:32:10.089114+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:46:38.920873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100645+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:32:10.100645+0000","last_peered":"2023-10-11T16:32:10.100645+0000","last_clean":"2023-10-11T16:32:10.100645+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:32:10.100645+0000","last_undegraded":"2023-10-11T16:32:10.100645+0000","last_fullsized":"2023-10-11T16:32:10.100645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:53:32.036585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100988+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:32:10.100988+0000","last_peered":"2023-10-11T16:32:10.100988+0000","last_clean":"2023-10-11T16:32:10.100988+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:32:10.100988+0000","last_undegraded":"2023-10-11T16:32:10.100988+0000","last_fullsized":"2023-10-11T16:32:10.100988+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:32.595236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100326+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:32:10.100326+0000","last_peered":"2023-10-11T16:32:10.100326+0000","last_clean":"2023-10-11T16:32:10.100326+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:32:10.100326+0000","last_undegraded":"2023-10-11T16:32:10.100326+0000","last_fullsized":"2023-10-11T16:32:10.100326+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:51.221018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100337+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:32:10.100337+0000","last_peered":"2023-10-11T16:32:10.100337+0000","last_clean":"2023-10-11T16:32:10.100337+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:32:10.100337+0000","last_undegraded":"2023-10-11T16:32:10.100337+0000","last_fullsized":"2023-10-11T16:32:10.100337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:40:38.294389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083579+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:32:10.083579+0000","last_peered":"2023-10-11T16:32:10.083579+0000","last_clean":"2023-10-11T16:32:10.083579+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:32:10.083579+0000","last_undegraded":"2023-10-11T16:32:10.083579+0000","last_fullsized":"2023-10-11T16:32:10.083579+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:56:56.119945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102789+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:32:10.102789+0000","last_peered":"2023-10-11T16:32:10.102789+0000","last_clean":"2023-10-11T16:32:10.102789+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:32:10.102789+0000","last_undegraded":"2023-10-11T16:32:10.102789+0000","last_fullsized":"2023-10-11T16:32:10.102789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:24:13.338895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159916+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:31:42.159916+0000","last_peered":"2023-10-11T16:31:42.159916+0000","last_clean":"2023-10-11T16:31:42.159916+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:31:42.159916+0000","last_undegraded":"2023-10-11T16:31:42.159916+0000","last_fullsized":"2023-10-11T16:31:42.159916+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:50:38.133970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159320+0000","last_change":"2023-10-11T16:30:41.198854+0000","last_active":"2023-10-11T16:31:42.159320+0000","last_peered":"2023-10-11T16:31:42.159320+0000","last_clean":"2023-10-11T16:31:42.159320+0000","last_became_active":"2023-10-11T16:30:41.189677+0000","last_became_peered":"2023-10-11T16:30:41.189677+0000","last_unstale":"2023-10-11T16:31:42.159320+0000","last_undegraded":"2023-10-11T16:31:42.159320+0000","last_fullsized":"2023-10-11T16:31:42.159320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:02:16.154156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097708+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:32:10.097708+0000","last_peered":"2023-10-11T16:32:10.097708+0000","last_clean":"2023-10-11T16:32:10.097708+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:32:10.097708+0000","last_undegraded":"2023-10-11T16:32:10.097708+0000","last_fullsized":"2023-10-11T16:32:10.097708+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:10.365459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288525+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:31:42.288525+0000","last_peered":"2023-10-11T16:31:42.288525+0000","last_clean":"2023-10-11T16:31:42.288525+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:31:42.288525+0000","last_undegraded":"2023-10-11T16:31:42.288525+0000","last_fullsized":"2023-10-11T16:31:42.288525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:37:11.027756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534803+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:31:42.534803+0000","last_peered":"2023-10-11T16:31:42.534803+0000","last_clean":"2023-10-11T16:31:42.534803+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:31:42.534803+0000","last_undegraded":"2023-10-11T16:31:42.534803+0000","last_fullsized":"2023-10-11T16:31:42.534803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:29:20.287428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102278+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:32:10.102278+0000","last_peered":"2023-10-11T16:32:10.102278+0000","last_clean":"2023-10-11T16:32:10.102278+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:32:10.102278+0000","last_undegraded":"2023-10-11T16:32:10.102278+0000","last_fullsized":"2023-10-11T16:32:10.102278+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:23:33.919634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288642+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:31:42.288642+0000","last_peered":"2023-10-11T16:31:42.288642+0000","last_clean":"2023-10-11T16:31:42.288642+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:31:42.288642+0000","last_undegraded":"2023-10-11T16:31:42.288642+0000","last_fullsized":"2023-10-11T16:31:42.288642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:42:02.132395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462596+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:31:42.462596+0000","last_peered":"2023-10-11T16:31:42.462596+0000","last_clean":"2023-10-11T16:31:42.462596+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:31:42.462596+0000","last_undegraded":"2023-10-11T16:31:42.462596+0000","last_fullsized":"2023-10-11T16:31:42.462596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:38:07.419529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092994+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:32:10.092994+0000","last_peered":"2023-10-11T16:32:10.092994+0000","last_clean":"2023-10-11T16:32:10.092994+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:32:10.092994+0000","last_undegraded":"2023-10-11T16:32:10.092994+0000","last_fullsized":"2023-10-11T16:32:10.092994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:44:22.080004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288644+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:31:42.288644+0000","last_peered":"2023-10-11T16:31:42.288644+0000","last_clean":"2023-10-11T16:31:42.288644+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:31:42.288644+0000","last_undegraded":"2023-10-11T16:31:42.288644+0000","last_fullsized":"2023-10-11T16:31:42.288644+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:40.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462545+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:31:42.462545+0000","last_peered":"2023-10-11T16:31:42.462545+0000","last_clean":"2023-10-11T16:31:42.462545+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:31:42.462545+0000","last_undegraded":"2023-10-11T16:31:42.462545+0000","last_fullsized":"2023-10-11T16:31:42.462545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:32:50.524951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083771+0000","last_change":"2023-10-11T16:30:41.195736+0000","last_active":"2023-10-11T16:32:10.083771+0000","last_peered":"2023-10-11T16:32:10.083771+0000","last_clean":"2023-10-11T16:32:10.083771+0000","last_became_active":"2023-10-11T16:30:41.195352+0000","last_became_peered":"2023-10-11T16:30:41.195352+0000","last_unstale":"2023-10-11T16:32:10.083771+0000","last_undegraded":"2023-10-11T16:32:10.083771+0000","last_fullsized":"2023-10-11T16:32:10.083771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:29:40.859051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160178+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:31:42.160178+0000","last_peered":"2023-10-11T16:31:42.160178+0000","last_clean":"2023-10-11T16:31:42.160178+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:31:42.160178+0000","last_undegraded":"2023-10-11T16:31:42.160178+0000","last_fullsized":"2023-10-11T16:31:42.160178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:55:21.244206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'10","reported_seq":148,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:19.340286+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:32:19.340286+0000","last_peered":"2023-10-11T16:32:19.340286+0000","last_clean":"2023-10-11T16:32:19.340286+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:32:19.340286+0000","last_undegraded":"2023-10-11T16:32:19.340286+0000","last_fullsized":"2023-10-11T16:32:19.340286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:37:47.462085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11029,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":14,"num_write":7,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.078744+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:32:10.078744+0000","last_peered":"2023-10-11T16:32:10.078744+0000","last_clean":"2023-10-11T16:32:10.078744+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:32:10.078744+0000","last_undegraded":"2023-10-11T16:32:10.078744+0000","last_fullsized":"2023-10-11T16:32:10.078744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:49:22.705791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160227+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:31:42.160227+0000","last_peered":"2023-10-11T16:31:42.160227+0000","last_clean":"2023-10-11T16:31:42.160227+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:31:42.160227+0000","last_undegraded":"2023-10-11T16:31:42.160227+0000","last_fullsized":"2023-10-11T16:31:42.160227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:18:51.568546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285229+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:31:42.285229+0000","last_peered":"2023-10-11T16:31:42.285229+0000","last_clean":"2023-10-11T16:31:42.285229+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:31:42.285229+0000","last_undegraded":"2023-10-11T16:31:42.285229+0000","last_fullsized":"2023-10-11T16:31:42.285229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:13:18.888864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.085765+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:32:10.085765+0000","last_peered":"2023-10-11T16:32:10.085765+0000","last_clean":"2023-10-11T16:32:10.085765+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:32:10.085765+0000","last_undegraded":"2023-10-11T16:32:10.085765+0000","last_fullsized":"2023-10-11T16:32:10.085765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:46:58.254928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.164266+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:31:42.164266+0000","last_peered":"2023-10-11T16:31:42.164266+0000","last_clean":"2023-10-11T16:31:42.164266+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:31:42.164266+0000","last_undegraded":"2023-10-11T16:31:42.164266+0000","last_fullsized":"2023-10-11T16:31:42.164266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:20:42.850191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285323+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:31:42.285323+0000","last_peered":"2023-10-11T16:31:42.285323+0000","last_clean":"2023-10-11T16:31:42.285323+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:31:42.285323+0000","last_undegraded":"2023-10-11T16:31:42.285323+0000","last_fullsized":"2023-10-11T16:31:42.285323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:11:41.038914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101540+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:32:10.101540+0000","last_peered":"2023-10-11T16:32:10.101540+0000","last_clean":"2023-10-11T16:32:10.101540+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:32:10.101540+0000","last_undegraded":"2023-10-11T16:32:10.101540+0000","last_fullsized":"2023-10-11T16:32:10.101540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:04:56.860908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161225+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:31:42.161225+0000","last_peered":"2023-10-11T16:31:42.161225+0000","last_clean":"2023-10-11T16:31:42.161225+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:31:42.161225+0000","last_undegraded":"2023-10-11T16:31:42.161225+0000","last_fullsized":"2023-10-11T16:31:42.161225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:40.600507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288380+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:31:42.288380+0000","last_peered":"2023-10-11T16:31:42.288380+0000","last_clean":"2023-10-11T16:31:42.288380+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:31:42.288380+0000","last_undegraded":"2023-10-11T16:31:42.288380+0000","last_fullsized":"2023-10-11T16:31:42.288380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:37:32.836177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088886+0000","last_change":"2023-10-11T16:31:42.286928+0000","last_active":"2023-10-11T16:32:10.088886+0000","last_peered":"2023-10-11T16:32:10.088886+0000","last_clean":"2023-10-11T16:32:10.088886+0000","last_became_active":"2023-10-11T16:31:42.286409+0000","last_became_peered":"2023-10-11T16:31:42.286409+0000","last_unstale":"2023-10-11T16:32:10.088886+0000","last_undegraded":"2023-10-11T16:32:10.088886+0000","last_fullsized":"2023-10-11T16:32:10.088886+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:16:50.559256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533799+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:31:42.533799+0000","last_peered":"2023-10-11T16:31:42.533799+0000","last_clean":"2023-10-11T16:31:42.533799+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:31:42.533799+0000","last_undegraded":"2023-10-11T16:31:42.533799+0000","last_fullsized":"2023-10-11T16:31:42.533799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:53:33.531388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":134,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:16.335262+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:32:16.335262+0000","last_peered":"2023-10-11T16:32:16.335262+0000","last_clean":"2023-10-11T16:32:16.335262+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:32:16.335262+0000","last_undegraded":"2023-10-11T16:32:16.335262+0000","last_fullsized":"2023-10-11T16:32:16.335262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:22.346902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092896+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:32:10.092896+0000","last_peered":"2023-10-11T16:32:10.092896+0000","last_clean":"2023-10-11T16:32:10.092896+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:32:10.092896+0000","last_undegraded":"2023-10-11T16:32:10.092896+0000","last_fullsized":"2023-10-11T16:32:10.092896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:36:02.019758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.157442+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:31:42.157442+0000","last_peered":"2023-10-11T16:31:42.157442+0000","last_clean":"2023-10-11T16:31:42.157442+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:31:42.157442+0000","last_undegraded":"2023-10-11T16:31:42.157442+0000","last_fullsized":"2023-10-11T16:31:42.157442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:20:41.440369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161173+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:31:42.161173+0000","last_peered":"2023-10-11T16:31:42.161173+0000","last_clean":"2023-10-11T16:31:42.161173+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:31:42.161173+0000","last_undegraded":"2023-10-11T16:31:42.161173+0000","last_fullsized":"2023-10-11T16:31:42.161173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:48.986542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101928+0000","last_change":"2023-10-11T16:31:42.464453+0000","last_active":"2023-10-11T16:32:10.101928+0000","last_peered":"2023-10-11T16:32:10.101928+0000","last_clean":"2023-10-11T16:32:10.101928+0000","last_became_active":"2023-10-11T16:31:42.463997+0000","last_became_peered":"2023-10-11T16:31:42.463997+0000","last_unstale":"2023-10-11T16:32:10.101928+0000","last_undegraded":"2023-10-11T16:32:10.101928+0000","last_fullsized":"2023-10-11T16:32:10.101928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:44:47.706246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288834+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:31:42.288834+0000","last_peered":"2023-10-11T16:31:42.288834+0000","last_clean":"2023-10-11T16:31:42.288834+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:31:42.288834+0000","last_undegraded":"2023-10-11T16:31:42.288834+0000","last_fullsized":"2023-10-11T16:31:42.288834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:13:12.138603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289000+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:31:42.289000+0000","last_peered":"2023-10-11T16:31:42.289000+0000","last_clean":"2023-10-11T16:31:42.289000+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:31:42.289000+0000","last_undegraded":"2023-10-11T16:31:42.289000+0000","last_fullsized":"2023-10-11T16:31:42.289000+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:15:55.186995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092874+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:32:10.092874+0000","last_peered":"2023-10-11T16:32:10.092874+0000","last_clean":"2023-10-11T16:32:10.092874+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:32:10.092874+0000","last_undegraded":"2023-10-11T16:32:10.092874+0000","last_fullsized":"2023-10-11T16:32:10.092874+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:23:25.760798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158398+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:31:42.158398+0000","last_peered":"2023-10-11T16:31:42.158398+0000","last_clean":"2023-10-11T16:31:42.158398+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:31:42.158398+0000","last_undegraded":"2023-10-11T16:31:42.158398+0000","last_fullsized":"2023-10-11T16:31:42.158398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:04:51.440115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":293,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:17.336262+0000","last_change":"2023-10-11T16:30:41.788994+0000","last_active":"2023-10-11T16:32:17.336262+0000","last_peered":"2023-10-11T16:32:17.336262+0000","last_clean":"2023-10-11T16:32:17.336262+0000","last_became_active":"2023-10-11T16:30:41.189949+0000","last_became_peered":"2023-10-11T16:30:41.189949+0000","last_unstale":"2023-10-11T16:32:17.336262+0000","last_undegraded":"2023-10-11T16:32:17.336262+0000","last_fullsized":"2023-10-11T16:32:17.336262+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:58:15.728137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66349,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":207,"num_read_kb":169,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38515,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.082731+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:32:10.082731+0000","last_peered":"2023-10-11T16:32:10.082731+0000","last_clean":"2023-10-11T16:32:10.082731+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:32:10.082731+0000","last_undegraded":"2023-10-11T16:32:10.082731+0000","last_fullsized":"2023-10-11T16:32:10.082731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:04:39.079541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289353+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:31:42.289353+0000","last_peered":"2023-10-11T16:31:42.289353+0000","last_clean":"2023-10-11T16:31:42.289353+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:31:42.289353+0000","last_undegraded":"2023-10-11T16:31:42.289353+0000","last_fullsized":"2023-10-11T16:31:42.289353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:31:29.246174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160086+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:31:42.160086+0000","last_peered":"2023-10-11T16:31:42.160086+0000","last_clean":"2023-10-11T16:31:42.160086+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:31:42.160086+0000","last_undegraded":"2023-10-11T16:31:42.160086+0000","last_fullsized":"2023-10-11T16:31:42.160086+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:39:49.561867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.104692+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:32:10.104692+0000","last_peered":"2023-10-11T16:32:10.104692+0000","last_clean":"2023-10-11T16:32:10.104692+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:32:10.104692+0000","last_undegraded":"2023-10-11T16:32:10.104692+0000","last_fullsized":"2023-10-11T16:32:10.104692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:17:46.544148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.163839+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:31:42.163839+0000","last_peered":"2023-10-11T16:31:42.163839+0000","last_clean":"2023-10-11T16:31:42.163839+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:31:42.163839+0000","last_undegraded":"2023-10-11T16:31:42.163839+0000","last_fullsized":"2023-10-11T16:31:42.163839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:36:11.677512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288800+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:31:42.288800+0000","last_peered":"2023-10-11T16:31:42.288800+0000","last_clean":"2023-10-11T16:31:42.288800+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:31:42.288800+0000","last_undegraded":"2023-10-11T16:31:42.288800+0000","last_fullsized":"2023-10-11T16:31:42.288800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:55:39.104456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.091980+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:32:10.091980+0000","last_peered":"2023-10-11T16:32:10.091980+0000","last_clean":"2023-10-11T16:32:10.091980+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:32:10.091980+0000","last_undegraded":"2023-10-11T16:32:10.091980+0000","last_fullsized":"2023-10-11T16:32:10.091980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:26:20.985716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158790+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:31:42.158790+0000","last_peered":"2023-10-11T16:31:42.158790+0000","last_clean":"2023-10-11T16:31:42.158790+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:31:42.158790+0000","last_undegraded":"2023-10-11T16:31:42.158790+0000","last_fullsized":"2023-10-11T16:31:42.158790+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:05:47.475689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":135,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:18.337663+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:32:18.337663+0000","last_peered":"2023-10-11T16:32:18.337663+0000","last_clean":"2023-10-11T16:32:18.337663+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:32:18.337663+0000","last_undegraded":"2023-10-11T16:32:18.337663+0000","last_fullsized":"2023-10-11T16:32:18.337663+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:37:12.900675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093444+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:32:10.093444+0000","last_peered":"2023-10-11T16:32:10.093444+0000","last_clean":"2023-10-11T16:32:10.093444+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:32:10.093444+0000","last_undegraded":"2023-10-11T16:32:10.093444+0000","last_fullsized":"2023-10-11T16:32:10.093444+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:27:01.920127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285928+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:31:42.285928+0000","last_peered":"2023-10-11T16:31:42.285928+0000","last_clean":"2023-10-11T16:31:42.285928+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:31:42.285928+0000","last_undegraded":"2023-10-11T16:31:42.285928+0000","last_fullsized":"2023-10-11T16:31:42.285928+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:08:54.198092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462035+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:31:42.462035+0000","last_peered":"2023-10-11T16:31:42.462035+0000","last_clean":"2023-10-11T16:31:42.462035+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:31:42.462035+0000","last_undegraded":"2023-10-11T16:31:42.462035+0000","last_fullsized":"2023-10-11T16:31:42.462035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:55:08.925417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092273+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:32:10.092273+0000","last_peered":"2023-10-11T16:32:10.092273+0000","last_clean":"2023-10-11T16:32:10.092273+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:32:10.092273+0000","last_undegraded":"2023-10-11T16:32:10.092273+0000","last_fullsized":"2023-10-11T16:32:10.092273+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:15:28.155402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288455+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:31:42.288455+0000","last_peered":"2023-10-11T16:31:42.288455+0000","last_clean":"2023-10-11T16:31:42.288455+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:31:42.288455+0000","last_undegraded":"2023-10-11T16:31:42.288455+0000","last_fullsized":"2023-10-11T16:31:42.288455+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:28:12.958080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462997+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:31:42.462997+0000","last_peered":"2023-10-11T16:31:42.462997+0000","last_clean":"2023-10-11T16:31:42.462997+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:31:42.462997+0000","last_undegraded":"2023-10-11T16:31:42.462997+0000","last_fullsized":"2023-10-11T16:31:42.462997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:03:09.115056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.080956+0000","last_change":"2023-10-11T16:31:42.287141+0000","last_active":"2023-10-11T16:32:10.080956+0000","last_peered":"2023-10-11T16:32:10.080956+0000","last_clean":"2023-10-11T16:32:10.080956+0000","last_became_active":"2023-10-11T16:31:42.286758+0000","last_became_peered":"2023-10-11T16:31:42.286758+0000","last_unstale":"2023-10-11T16:32:10.080956+0000","last_undegraded":"2023-10-11T16:32:10.080956+0000","last_fullsized":"2023-10-11T16:32:10.080956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:40:33.920330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462187+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:31:42.462187+0000","last_peered":"2023-10-11T16:31:42.462187+0000","last_clean":"2023-10-11T16:31:42.462187+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:31:42.462187+0000","last_undegraded":"2023-10-11T16:31:42.462187+0000","last_fullsized":"2023-10-11T16:31:42.462187+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:43.818267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158964+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:31:42.158964+0000","last_peered":"2023-10-11T16:31:42.158964+0000","last_clean":"2023-10-11T16:31:42.158964+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:31:42.158964+0000","last_undegraded":"2023-10-11T16:31:42.158964+0000","last_fullsized":"2023-10-11T16:31:42.158964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:16:18.715348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093724+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:32:10.093724+0000","last_peered":"2023-10-11T16:32:10.093724+0000","last_clean":"2023-10-11T16:32:10.093724+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:32:10.093724+0000","last_undegraded":"2023-10-11T16:32:10.093724+0000","last_fullsized":"2023-10-11T16:32:10.093724+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:03:22.304558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285696+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:31:42.285696+0000","last_peered":"2023-10-11T16:31:42.285696+0000","last_clean":"2023-10-11T16:31:42.285696+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:31:42.285696+0000","last_undegraded":"2023-10-11T16:31:42.285696+0000","last_fullsized":"2023-10-11T16:31:42.285696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:45:09.560952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160923+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:31:42.160923+0000","last_peered":"2023-10-11T16:31:42.160923+0000","last_clean":"2023-10-11T16:31:42.160923+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:31:42.160923+0000","last_undegraded":"2023-10-11T16:31:42.160923+0000","last_fullsized":"2023-10-11T16:31:42.160923+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:11:25.625028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.095113+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:32:10.095113+0000","last_peered":"2023-10-11T16:32:10.095113+0000","last_clean":"2023-10-11T16:32:10.095113+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:32:10.095113+0000","last_undegraded":"2023-10-11T16:32:10.095113+0000","last_fullsized":"2023-10-11T16:32:10.095113+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:33:24.458841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.286057+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:31:42.286057+0000","last_peered":"2023-10-11T16:31:42.286057+0000","last_clean":"2023-10-11T16:31:42.286057+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:31:42.286057+0000","last_undegraded":"2023-10-11T16:31:42.286057+0000","last_fullsized":"2023-10-11T16:31:42.286057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:23:48.290443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083891+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:32:10.083891+0000","last_peered":"2023-10-11T16:32:10.083891+0000","last_clean":"2023-10-11T16:32:10.083891+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:32:10.083891+0000","last_undegraded":"2023-10-11T16:32:10.083891+0000","last_fullsized":"2023-10-11T16:32:10.083891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:18:07.940669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":135,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:17.335778+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:32:17.335778+0000","last_peered":"2023-10-11T16:32:17.335778+0000","last_clean":"2023-10-11T16:32:17.335778+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:32:17.335778+0000","last_undegraded":"2023-10-11T16:32:17.335778+0000","last_fullsized":"2023-10-11T16:32:17.335778+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:13:34.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161339+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:31:42.161339+0000","last_peered":"2023-10-11T16:31:42.161339+0000","last_clean":"2023-10-11T16:31:42.161339+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:31:42.161339+0000","last_undegraded":"2023-10-11T16:31:42.161339+0000","last_fullsized":"2023-10-11T16:31:42.161339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:05:48.188359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101162+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:32:10.101162+0000","last_peered":"2023-10-11T16:32:10.101162+0000","last_clean":"2023-10-11T16:32:10.101162+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:32:10.101162+0000","last_undegraded":"2023-10-11T16:32:10.101162+0000","last_fullsized":"2023-10-11T16:32:10.101162+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:12:34.720396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288780+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:31:42.288780+0000","last_peered":"2023-10-11T16:31:42.288780+0000","last_clean":"2023-10-11T16:31:42.288780+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:31:42.288780+0000","last_undegraded":"2023-10-11T16:31:42.288780+0000","last_fullsized":"2023-10-11T16:31:42.288780+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:17:09.996158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160483+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:31:42.160483+0000","last_peered":"2023-10-11T16:31:42.160483+0000","last_clean":"2023-10-11T16:31:42.160483+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:31:42.160483+0000","last_undegraded":"2023-10-11T16:31:42.160483+0000","last_fullsized":"2023-10-11T16:31:42.160483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:36.114850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289135+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:31:42.289135+0000","last_peered":"2023-10-11T16:31:42.289135+0000","last_clean":"2023-10-11T16:31:42.289135+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:31:42.289135+0000","last_undegraded":"2023-10-11T16:31:42.289135+0000","last_fullsized":"2023-10-11T16:31:42.289135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:30:07.695611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.074567+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:32:10.074567+0000","last_peered":"2023-10-11T16:32:10.074567+0000","last_clean":"2023-10-11T16:32:10.074567+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:32:10.074567+0000","last_undegraded":"2023-10-11T16:32:10.074567+0000","last_fullsized":"2023-10-11T16:32:10.074567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:33:14.082898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160623+0000","last_change":"2023-10-11T16:30:41.938949+0000","last_active":"2023-10-11T16:31:42.160623+0000","last_peered":"2023-10-11T16:31:42.160623+0000","last_clean":"2023-10-11T16:31:42.160623+0000","last_became_active":"2023-10-11T16:30:41.192902+0000","last_became_peered":"2023-10-11T16:30:41.192902+0000","last_unstale":"2023-10-11T16:31:42.160623+0000","last_undegraded":"2023-10-11T16:31:42.160623+0000","last_fullsized":"2023-10-11T16:31:42.160623+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:57:28.224909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.163836+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:31:42.163836+0000","last_peered":"2023-10-11T16:31:42.163836+0000","last_clean":"2023-10-11T16:31:42.163836+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:31:42.163836+0000","last_undegraded":"2023-10-11T16:31:42.163836+0000","last_fullsized":"2023-10-11T16:31:42.163836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:42:22.315441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.164029+0000","last_change":"2023-10-11T16:30:41.205550+0000","last_active":"2023-10-11T16:31:42.164029+0000","last_peered":"2023-10-11T16:31:42.164029+0000","last_clean":"2023-10-11T16:31:42.164029+0000","last_became_active":"2023-10-11T16:30:41.190345+0000","last_became_peered":"2023-10-11T16:30:41.190345+0000","last_unstale":"2023-10-11T16:31:42.164029+0000","last_undegraded":"2023-10-11T16:31:42.164029+0000","last_fullsized":"2023-10-11T16:31:42.164029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:32:00.140298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.074906+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:32:10.074906+0000","last_peered":"2023-10-11T16:32:10.074906+0000","last_clean":"2023-10-11T16:32:10.074906+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:32:10.074906+0000","last_undegraded":"2023-10-11T16:32:10.074906+0000","last_fullsized":"2023-10-11T16:32:10.074906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:57:43.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534261+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:31:42.534261+0000","last_peered":"2023-10-11T16:31:42.534261+0000","last_clean":"2023-10-11T16:31:42.534261+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:31:42.534261+0000","last_undegraded":"2023-10-11T16:31:42.534261+0000","last_fullsized":"2023-10-11T16:31:42.534261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:13:32.227680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.564394+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:32:10.564394+0000","last_peered":"2023-10-11T16:32:10.564394+0000","last_clean":"2023-10-11T16:32:10.564394+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:32:10.564394+0000","last_undegraded":"2023-10-11T16:32:10.564394+0000","last_fullsized":"2023-10-11T16:32:10.564394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:02:28.868871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101284+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:32:10.101284+0000","last_peered":"2023-10-11T16:32:10.101284+0000","last_clean":"2023-10-11T16:32:10.101284+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:32:10.101284+0000","last_undegraded":"2023-10-11T16:32:10.101284+0000","last_fullsized":"2023-10-11T16:32:10.101284+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:50:14.165387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159765+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:31:42.159765+0000","last_peered":"2023-10-11T16:31:42.159765+0000","last_clean":"2023-10-11T16:31:42.159765+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:31:42.159765+0000","last_undegraded":"2023-10-11T16:31:42.159765+0000","last_fullsized":"2023-10-11T16:31:42.159765+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:04:09.873784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289422+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:31:42.289422+0000","last_peered":"2023-10-11T16:31:42.289422+0000","last_clean":"2023-10-11T16:31:42.289422+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:31:42.289422+0000","last_undegraded":"2023-10-11T16:31:42.289422+0000","last_fullsized":"2023-10-11T16:31:42.289422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:03.781734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100102+0000","last_change":"2023-10-11T16:31:42.290019+0000","last_active":"2023-10-11T16:32:10.100102+0000","last_peered":"2023-10-11T16:32:10.100102+0000","last_clean":"2023-10-11T16:32:10.100102+0000","last_became_active":"2023-10-11T16:31:42.289721+0000","last_became_peered":"2023-10-11T16:31:42.289721+0000","last_unstale":"2023-10-11T16:32:10.100102+0000","last_undegraded":"2023-10-11T16:32:10.100102+0000","last_fullsized":"2023-10-11T16:32:10.100102+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:04:10.132863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289138+0000","last_change":"2023-10-11T16:30:41.193489+0000","last_active":"2023-10-11T16:31:42.289138+0000","last_peered":"2023-10-11T16:31:42.289138+0000","last_clean":"2023-10-11T16:31:42.289138+0000","last_became_active":"2023-10-11T16:30:41.193060+0000","last_became_peered":"2023-10-11T16:30:41.193060+0000","last_unstale":"2023-10-11T16:31:42.289138+0000","last_undegraded":"2023-10-11T16:31:42.289138+0000","last_fullsized":"2023-10-11T16:31:42.289138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:51:45.983921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533728+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:31:42.533728+0000","last_peered":"2023-10-11T16:31:42.533728+0000","last_clean":"2023-10-11T16:31:42.533728+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:31:42.533728+0000","last_undegraded":"2023-10-11T16:31:42.533728+0000","last_fullsized":"2023-10-11T16:31:42.533728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:43:17.566817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.057040+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:32:10.057040+0000","last_peered":"2023-10-11T16:32:10.057040+0000","last_clean":"2023-10-11T16:32:10.057040+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:32:10.057040+0000","last_undegraded":"2023-10-11T16:32:10.057040+0000","last_fullsized":"2023-10-11T16:32:10.057040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:20:18.016311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462851+0000","last_change":"2023-10-11T16:30:41.404891+0000","last_active":"2023-10-11T16:31:42.462851+0000","last_peered":"2023-10-11T16:31:42.462851+0000","last_clean":"2023-10-11T16:31:42.462851+0000","last_became_active":"2023-10-11T16:30:41.404396+0000","last_became_peered":"2023-10-11T16:30:41.404396+0000","last_unstale":"2023-10-11T16:31:42.462851+0000","last_undegraded":"2023-10-11T16:31:42.462851+0000","last_fullsized":"2023-10-11T16:31:42.462851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:41:36.636884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533832+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:31:42.533832+0000","last_peered":"2023-10-11T16:31:42.533832+0000","last_clean":"2023-10-11T16:31:42.533832+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:31:42.533832+0000","last_undegraded":"2023-10-11T16:31:42.533832+0000","last_fullsized":"2023-10-11T16:31:42.533832+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:11:17.333111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097932+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:32:10.097932+0000","last_peered":"2023-10-11T16:32:10.097932+0000","last_clean":"2023-10-11T16:32:10.097932+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:32:10.097932+0000","last_undegraded":"2023-10-11T16:32:10.097932+0000","last_fullsized":"2023-10-11T16:32:10.097932+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:14:58.964194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289262+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:31:42.289262+0000","last_peered":"2023-10-11T16:31:42.289262+0000","last_clean":"2023-10-11T16:31:42.289262+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:31:42.289262+0000","last_undegraded":"2023-10-11T16:31:42.289262+0000","last_fullsized":"2023-10-11T16:31:42.289262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:43:35.192572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:12.146327+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:32:12.146327+0000","last_peered":"2023-10-11T16:32:12.146327+0000","last_clean":"2023-10-11T16:32:12.146327+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:32:12.146327+0000","last_undegraded":"2023-10-11T16:32:12.146327+0000","last_fullsized":"2023-10-11T16:32:12.146327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:52:56.087178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102288+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:32:10.102288+0000","last_peered":"2023-10-11T16:32:10.102288+0000","last_clean":"2023-10-11T16:32:10.102288+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:32:10.102288+0000","last_undegraded":"2023-10-11T16:32:10.102288+0000","last_fullsized":"2023-10-11T16:32:10.102288+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:38.182167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288761+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:31:42.288761+0000","last_peered":"2023-10-11T16:31:42.288761+0000","last_clean":"2023-10-11T16:31:42.288761+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:31:42.288761+0000","last_undegraded":"2023-10-11T16:31:42.288761+0000","last_fullsized":"2023-10-11T16:31:42.288761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:29:21.139277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462324+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:31:42.462324+0000","last_peered":"2023-10-11T16:31:42.462324+0000","last_clean":"2023-10-11T16:31:42.462324+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:31:42.462324+0000","last_undegraded":"2023-10-11T16:31:42.462324+0000","last_fullsized":"2023-10-11T16:31:42.462324+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:50:35.582480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088149+0000","last_change":"2023-10-11T16:31:42.286822+0000","last_active":"2023-10-11T16:32:10.088149+0000","last_peered":"2023-10-11T16:32:10.088149+0000","last_clean":"2023-10-11T16:32:10.088149+0000","last_became_active":"2023-10-11T16:31:42.286327+0000","last_became_peered":"2023-10-11T16:31:42.286327+0000","last_unstale":"2023-10-11T16:32:10.088149+0000","last_undegraded":"2023-10-11T16:32:10.088149+0000","last_fullsized":"2023-10-11T16:32:10.088149+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:21:36.527593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533754+0000","last_change":"2023-10-11T16:30:41.200104+0000","last_active":"2023-10-11T16:31:42.533754+0000","last_peered":"2023-10-11T16:31:42.533754+0000","last_clean":"2023-10-11T16:31:42.533754+0000","last_became_active":"2023-10-11T16:30:41.199637+0000","last_became_peered":"2023-10-11T16:30:41.199637+0000","last_unstale":"2023-10-11T16:31:42.533754+0000","last_undegraded":"2023-10-11T16:31:42.533754+0000","last_fullsized":"2023-10-11T16:31:42.533754+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:07:59.931450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534011+0000","last_change":"2023-10-11T16:30:41.202718+0000","last_active":"2023-10-11T16:31:42.534011+0000","last_peered":"2023-10-11T16:31:42.534011+0000","last_clean":"2023-10-11T16:31:42.534011+0000","last_became_active":"2023-10-11T16:30:41.190741+0000","last_became_peered":"2023-10-11T16:30:41.190741+0000","last_unstale":"2023-10-11T16:31:42.534011+0000","last_undegraded":"2023-10-11T16:31:42.534011+0000","last_fullsized":"2023-10-11T16:31:42.534011+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:13:39.958791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.094846+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:32:10.094846+0000","last_peered":"2023-10-11T16:32:10.094846+0000","last_clean":"2023-10-11T16:32:10.094846+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:32:10.094846+0000","last_undegraded":"2023-10-11T16:32:10.094846+0000","last_fullsized":"2023-10-11T16:32:10.094846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:44:05.420593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161952+0000","last_change":"2023-10-11T16:30:41.190477+0000","last_active":"2023-10-11T16:31:42.161952+0000","last_peered":"2023-10-11T16:31:42.161952+0000","last_clean":"2023-10-11T16:31:42.161952+0000","last_became_active":"2023-10-11T16:30:41.189929+0000","last_became_peered":"2023-10-11T16:30:41.189929+0000","last_unstale":"2023-10-11T16:31:42.161952+0000","last_undegraded":"2023-10-11T16:31:42.161952+0000","last_fullsized":"2023-10-11T16:31:42.161952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:14:01.200276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289168+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:31:42.289168+0000","last_peered":"2023-10-11T16:31:42.289168+0000","last_clean":"2023-10-11T16:31:42.289168+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:31:42.289168+0000","last_undegraded":"2023-10-11T16:31:42.289168+0000","last_fullsized":"2023-10-11T16:31:42.289168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:13.844885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093645+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:32:10.093645+0000","last_peered":"2023-10-11T16:32:10.093645+0000","last_clean":"2023-10-11T16:32:10.093645+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:32:10.093645+0000","last_undegraded":"2023-10-11T16:32:10.093645+0000","last_fullsized":"2023-10-11T16:32:10.093645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:07:18.219144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288217+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:31:42.288217+0000","last_peered":"2023-10-11T16:31:42.288217+0000","last_clean":"2023-10-11T16:31:42.288217+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:31:42.288217+0000","last_undegraded":"2023-10-11T16:31:42.288217+0000","last_fullsized":"2023-10-11T16:31:42.288217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:07:28.047642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285779+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:31:42.285779+0000","last_peered":"2023-10-11T16:31:42.285779+0000","last_clean":"2023-10-11T16:31:42.285779+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:31:42.285779+0000","last_undegraded":"2023-10-11T16:31:42.285779+0000","last_fullsized":"2023-10-11T16:31:42.285779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:43:48.783409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088126+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:32:10.088126+0000","last_peered":"2023-10-11T16:32:10.088126+0000","last_clean":"2023-10-11T16:32:10.088126+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:32:10.088126+0000","last_undegraded":"2023-10-11T16:32:10.088126+0000","last_fullsized":"2023-10-11T16:32:10.088126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:29:09.209568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.085185+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:32:10.085185+0000","last_peered":"2023-10-11T16:32:10.085185+0000","last_clean":"2023-10-11T16:32:10.085185+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:32:10.085185+0000","last_undegraded":"2023-10-11T16:32:10.085185+0000","last_fullsized":"2023-10-11T16:32:10.085185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:00:29.421748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089406+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:32:10.089406+0000","last_peered":"2023-10-11T16:32:10.089406+0000","last_clean":"2023-10-11T16:32:10.089406+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:32:10.089406+0000","last_undegraded":"2023-10-11T16:32:10.089406+0000","last_fullsized":"2023-10-11T16:32:10.089406+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:24.461888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.104714+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:32:10.104714+0000","last_peered":"2023-10-11T16:32:10.104714+0000","last_clean":"2023-10-11T16:32:10.104714+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:32:10.104714+0000","last_undegraded":"2023-10-11T16:32:10.104714+0000","last_fullsized":"2023-10-11T16:32:10.104714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:25:45.955319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100277+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:32:10.100277+0000","last_peered":"2023-10-11T16:32:10.100277+0000","last_clean":"2023-10-11T16:32:10.100277+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:32:10.100277+0000","last_undegraded":"2023-10-11T16:32:10.100277+0000","last_fullsized":"2023-10-11T16:32:10.100277+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:50:42.892510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.086621+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:32:10.086621+0000","last_peered":"2023-10-11T16:32:10.086621+0000","last_clean":"2023-10-11T16:32:10.086621+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:32:10.086621+0000","last_undegraded":"2023-10-11T16:32:10.086621+0000","last_fullsized":"2023-10-11T16:32:10.086621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:21:58.187591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.081932+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:32:10.081932+0000","last_peered":"2023-10-11T16:32:10.081932+0000","last_clean":"2023-10-11T16:32:10.081932+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:32:10.081932+0000","last_undegraded":"2023-10-11T16:32:10.081932+0000","last_fullsized":"2023-10-11T16:32:10.081932+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:38:42.257752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.082937+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:32:10.082937+0000","last_peered":"2023-10-11T16:32:10.082937+0000","last_clean":"2023-10-11T16:32:10.082937+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:32:10.082937+0000","last_undegraded":"2023-10-11T16:32:10.082937+0000","last_fullsized":"2023-10-11T16:32:10.082937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:53:03.240456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.098026+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:32:10.098026+0000","last_peered":"2023-10-11T16:32:10.098026+0000","last_clean":"2023-10-11T16:32:10.098026+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:32:10.098026+0000","last_undegraded":"2023-10-11T16:32:10.098026+0000","last_fullsized":"2023-10-11T16:32:10.098026+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:14:19.917814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083854+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:32:10.083854+0000","last_peered":"2023-10-11T16:32:10.083854+0000","last_clean":"2023-10-11T16:32:10.083854+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:32:10.083854+0000","last_undegraded":"2023-10-11T16:32:10.083854+0000","last_fullsized":"2023-10-11T16:32:10.083854+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:39.214839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097729+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:32:10.097729+0000","last_peered":"2023-10-11T16:32:10.097729+0000","last_clean":"2023-10-11T16:32:10.097729+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:32:10.097729+0000","last_undegraded":"2023-10-11T16:32:10.097729+0000","last_fullsized":"2023-10-11T16:32:10.097729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:58.913211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.105194+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:32:10.105194+0000","last_peered":"2023-10-11T16:32:10.105194+0000","last_clean":"2023-10-11T16:32:10.105194+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:32:10.105194+0000","last_undegraded":"2023-10-11T16:32:10.105194+0000","last_fullsized":"2023-10-11T16:32:10.105194+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:54:53.313375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.095040+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:32:10.095040+0000","last_peered":"2023-10-11T16:32:10.095040+0000","last_clean":"2023-10-11T16:32:10.095040+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:32:10.095040+0000","last_undegraded":"2023-10-11T16:32:10.095040+0000","last_fullsized":"2023-10-11T16:32:10.095040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:18:45.218633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.080249+0000","last_change":"2023-10-11T16:31:42.463155+0000","last_active":"2023-10-11T16:32:10.080249+0000","last_peered":"2023-10-11T16:32:10.080249+0000","last_clean":"2023-10-11T16:32:10.080249+0000","last_became_active":"2023-10-11T16:31:42.462678+0000","last_became_peered":"2023-10-11T16:31:42.462678+0000","last_unstale":"2023-10-11T16:32:10.080249+0000","last_undegraded":"2023-10-11T16:32:10.080249+0000","last_fullsized":"2023-10-11T16:32:10.080249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:50:11.499350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80318,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":630,"num_read_kb":396,"num_write":174,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":39701,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":240954,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449126,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":1003520,"data_stored":508258,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.443},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":1.204,"15min":1.204},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73699999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57099999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.39100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.46999999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79200000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70399999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60499999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.39400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48499999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.50600000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76500000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.70799999999999996},"min":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.006,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":3.75}]},{"osd":7,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83199999999999996,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.621},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73599999999999999,"15min":0.74299999999999999},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.86099999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.64000000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:32:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68799999999999994,"15min":0.68899999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.871,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.753},"min":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96799999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.69499999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78700000000000003,"15min":0.76900000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.96,"5min":2.96,"15min":2.96},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.79700000000000004,"15min":0.80100000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.86499999999999999,"5min":1.286,"15min":1.286},"last":0.71999999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.86599999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76500000000000001,"15min":0.77100000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92100000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.65600000000000003}]}]},{"osd":10,"up_from":71,"seq":304942678058,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6696,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738456,"statfs":{"total":95995035648,"available":95988178944,"internally_reserved":0,"allocated":1146880,"data_stored":1161223,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.499}]},{"osd":1,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.33200000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.59799999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.49299999999999999,"15min":0.51100000000000001},"min":{"1min":0.318,"5min":0.314,"15min":0.314},"max":{"1min":0.56000000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57899999999999996}]},{"osd":2,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60999999999999999,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64900000000000002,"15min":0.66300000000000003},"min":{"1min":0.41299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71499999999999997,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.624}]},{"osd":3,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70099999999999996,"5min":0.995,"15min":0.995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64300000000000002,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.745,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.67800000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.38300000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69299999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.36899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76000000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.69199999999999995}]},{"osd":5,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.443,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.71099999999999997,"15min":0.72099999999999997},"min":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.76200000000000001,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.66000000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72099999999999997,"15min":0.74099999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.84499999999999997,"5min":1.085,"15min":1.085},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82499999999999996,"5min":1.03,"15min":1.03},"last":0.72899999999999998}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.45700000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90200000000000002,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.755},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.83999999999999997,"15min":0.877},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.84799999999999998,"5min":2.46,"15min":2.46},"last":0.70699999999999996}]},{"osd":9,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.79200000000000004,"15min":0.81699999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.85099999999999998,"5min":1.286,"15min":1.286},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.79900000000000004,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92600000000000005,"5min":1.347,"15min":1.347},"last":0.83499999999999996}]},{"osd":11,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.876,"5min":1.073,"15min":1.073},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90100000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.78200000000000003}]}]},{"osd":9,"up_from":66,"seq":283467841582,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":1015808,"data_stored":511986,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.496}]},{"osd":1,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80000000000000004,"15min":0.76700000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.198,"5min":7.056,"15min":7.056},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.80300000000000005,"15min":0.78200000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.2250000000000001,"5min":7.1529999999999996,"15min":7.1529999999999996},"last":0.44800000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69499999999999995,"15min":0.72799999999999998},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.379,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.758,"5min":2.758,"15min":2.758},"last":0.63100000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.286,"5min":2.286,"15min":2.286},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77800000000000002,"15min":0.79600000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.58299999999999996}]},{"osd":5,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.81699999999999995,"15min":0.84499999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.83099999999999996,"15min":0.84999999999999998},"min":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.6480000000000001,"5min":2.6480000000000001,"15min":2.6480000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86699999999999999,"15min":0.89500000000000002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88200000000000001,"15min":0.89100000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6789999999999998,"5min":2.6789999999999998,"15min":2.6789999999999998},"last":0.51000000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.96599999999999997,"15min":0.95299999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":3.387,"5min":7.1070000000000002,"15min":7.1070000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.96699999999999997,"15min":0.96499999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.4860000000000002,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.77400000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.84899999999999998,"15min":0.88300000000000001},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1699999999999999,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.92900000000000005,"15min":0.95899999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.3300000000000001,"5min":2.706,"15min":2.706},"last":0.75600000000000001}]},{"osd":11,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.88300000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.995,"5min":2.46,"15min":2.46},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.95099999999999996,"15min":0.98499999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.79600000000000004}]}]},{"osd":8,"up_from":58,"seq":249108103220,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6556,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738596,"statfs":{"total":95995035648,"available":95988322304,"internally_reserved":0,"allocated":1003520,"data_stored":507457,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91600000000000004,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":0.996,"15min":0.996},"last":0.45200000000000001}]},{"osd":1,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82599999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.63}]},{"osd":2,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.44600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.77400000000000002,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0649999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.57699999999999996}]},{"osd":4,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82599999999999996,"5min":1.028,"15min":1.028},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90900000000000003,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82499999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55900000000000005,"15min":0.54500000000000004},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.179,"15min":1.179},"last":0.874},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.499,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.751}]},{"osd":7,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.68899999999999995,"15min":0.68100000000000005},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.95299999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.69099999999999995,"15min":0.68300000000000005},"min":{"1min":0.56599999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.99399999999999999,"5min":1.101,"15min":1.101},"last":0.73499999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.73299999999999998,"15min":0.69899999999999995},"min":{"1min":0.69799999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.74299999999999999,"15min":0.72499999999999998},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.83899999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.74399999999999999,"15min":0.71199999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0449999999999999,"5min":1.069,"15min":1.069},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.72199999999999998,"15min":0.69599999999999995},"min":{"1min":0.57699999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0549999999999999,"5min":1.083,"15min":1.083},"last":0.80300000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:32:14 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.012,"5min":1.048,"15min":1.048},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.78800000000000003,"15min":0.77800000000000002},"min":{"1min":0.59599999999999997,"5min":0.436,"15min":0.436},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.81699999999999995}]}]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6744,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738408,"statfs":{"total":95995035648,"available":95988129792,"internally_reserved":0,"allocated":1064960,"data_stored":1113105,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.66100000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.45400000000000001},"min":{"1min":0.29199999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63900000000000001,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.374}]},{"osd":2,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77700000000000002,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.57599999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.628}]},{"osd":3,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58099999999999996,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.621,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80900000000000005,"5min":1.103,"15min":1.103},"last":0.51500000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.71099999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.501,"15min":0.5},"min":{"1min":0.29399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45000000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.45700000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91800000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66100000000000003,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86399999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68700000000000006}]},{"osd":6,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.53900000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.083,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.67900000000000005,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.71899999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71199999999999997,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72599999999999998,"15min":0.72799999999999998},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77800000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.77700000000000002,"15min":0.77600000000000002},"min":{"1min":0.63500000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.079,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.77000000000000002,"15min":0.77300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.60499999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70699999999999996,"15min":0.69499999999999995},"min":{"1min":0.48499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.054,"5min":1.071,"15min":1.071},"last":0.751},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.47699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.99199999999999999,"5min":1.099,"15min":1.099},"last":0.79500000000000004}]},{"osd":11,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.82899999999999996}]}]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6544,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738608,"statfs":{"total":95995035648,"available":95988334592,"internally_reserved":0,"allocated":991232,"data_stored":503066,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.47899999999999998,"15min":0.44500000000000001},"min":{"1min":0.32700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.59399999999999997,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.73099999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.60399999999999998,"15min":0.56999999999999995},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.63500000000000001,"15min":0.59499999999999997},"min":{"1min":0.44,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.71799999999999997,"15min":0.67700000000000005},"min":{"1min":0.48699999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.48699999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.47799999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.39100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.374,"5min":0.317,"15min":0.317},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55900000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.73699999999999999,"15min":0.70699999999999996},"min":{"1min":0.52000000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.69299999999999995,"15min":0.66000000000000003},"min":{"1min":0.52800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.64200000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70899999999999996,"15min":0.66900000000000004},"min":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.79800000000000004,"15min":0.75800000000000001},"min":{"1min":0.65000000000000002,"5min":0.434,"15min":0.434},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.79000000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:31:59 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.79700000000000004,"15min":0.76500000000000001},"min":{"1min":0.53700000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.73999999999999999,"15min":0.72399999999999998},"min":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.67600000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.752},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.875,"15min":0.874},"min":{"1min":0.60099999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.77500000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:31:46 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.872},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.89500000000000002,"15min":0.90700000000000003},"min":{"1min":0.57899999999999996,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.3719999999999999,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":0.80600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.66200000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.871,"15min":0.875},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.71499999999999997}]},{"osd":11,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.5640000000000001,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":0.73899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":569576,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55100000000000005,"15min":0.61299999999999999},"min":{"1min":0.39200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.8520000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.52400000000000002,"15min":0.53800000000000003},"min":{"1min":0.41499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83099999999999996,"5min":0.872,"15min":0.89800000000000002},"last":0.51000000000000001}]},{"osd":2,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.57299999999999995,"15min":0.52300000000000002},"min":{"1min":0.51500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.56599999999999995,"15min":0.52600000000000002},"min":{"1min":0.495,"5min":0.32900000000000001,"15min":0.307},"max":{"1min":0.89300000000000002,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.70999999999999996}]},{"osd":3,"last update":"Wed Oct 11 16:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61299999999999999,"15min":0.57999999999999996},"min":{"1min":0.48899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.88800000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.60799999999999998,"15min":0.57199999999999995},"min":{"1min":0.53000000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.65300000000000002}]},{"osd":4,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.47399999999999998,"15min":0.42699999999999999},"min":{"1min":0.34300000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.752,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.54300000000000004,"15min":0.46600000000000003},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.91000000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.73099999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.59599999999999997,"15min":0.51200000000000001},"min":{"1min":0.505,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.64900000000000002,"15min":0.55400000000000005},"min":{"1min":0.58199999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85999999999999999,"5min":0.997,"15min":0.997},"last":0.77600000000000002}]},{"osd":6,"last update":"Wed Oct 11 16:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.67900000000000005,"15min":0.627},"min":{"1min":0.60199999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":2.3580000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.65900000000000003,"15min":0.63500000000000001},"min":{"1min":0.59499999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.79200000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.57999999999999996,"15min":0.51600000000000001},"min":{"1min":0.47699999999999998,"5min":0.253,"15min":0.253},"max":{"1min":1.004,"5min":1.127,"15min":1.127},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.66900000000000004,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90700000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.80500000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.73199999999999998,"15min":0.67800000000000005},"min":{"1min":0.65500000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.70499999999999996,"15min":0.64500000000000002},"min":{"1min":0.67700000000000005,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.94699999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.746,"15min":0.72199999999999998},"min":{"1min":0.70699999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97099999999999997,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.79200000000000004,"15min":0.77000000000000002},"min":{"1min":0.67000000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.98699999999999999,"5min":1.179,"15min":1.179},"last":0.89700000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.71299999999999997,"15min":0.67800000000000005},"min":{"1min":0.48999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96099999999999997,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.871},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.76000000000000001,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93799999999999994,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.82199999999999995}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.84199999999999997,"15min":0.83699999999999997},"min":{"1min":0.629,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.024,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.876,"5min":0.86399999999999999,"15min":0.86199999999999999},"min":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.97599999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.88800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7464,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737688,"statfs":{"total":95995035648,"available":95987392512,"internally_reserved":0,"allocated":1015808,"data_stored":504454,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.41599999999999998,"5min":0.33100000000000002,"15min":0.309},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.60999999999999999,"15min":0.54400000000000004},"min":{"1min":0.54500000000000004,"5min":0.38500000000000001,"15min":0.33800000000000002},"max":{"1min":0.82499999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47199999999999998}]},{"osd":2,"last update":"Wed Oct 11 16:32:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.65300000000000002,"15min":0.59499999999999997},"min":{"1min":0.46200000000000002,"5min":0.38500000000000001,"15min":0.35899999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66700000000000004,"15min":0.61299999999999999},"min":{"1min":0.54300000000000004,"5min":0.42599999999999999,"15min":0.40500000000000003},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.59999999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.42299999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61599999999999999,"15min":0.56200000000000006},"min":{"1min":0.41499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64800000000000002,"15min":0.59699999999999998},"min":{"1min":0.47399999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81899999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.52900000000000003}]},{"osd":5,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.60699999999999998},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66300000000000003,"15min":0.63900000000000001},"min":{"1min":0.502,"5min":0.436,"15min":0.436},"max":{"1min":0.94699999999999995,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.73899999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.626},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.63,"15min":0.60399999999999998},"min":{"1min":0.374,"5min":0.37,"15min":0.37},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:31:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.56299999999999994}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.77300000000000002,"15min":0.753},"min":{"1min":0.63500000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":3.073,"5min":3.073,"15min":3.073},"last":0.622},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75800000000000001,"15min":0.76200000000000001},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.63500000000000001}]},{"osd":9,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.80600000000000005,"15min":0.83899999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.80400000000000005,"15min":0.82899999999999996},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.72699999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81399999999999995,"15min":0.81100000000000005},"min":{"1min":0.69499999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.82299999999999995,"15min":0.82599999999999996},"min":{"1min":0.624,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.76900000000000002}]},{"osd":11,"last update":"Wed Oct 11 16:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.64100000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.751},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.70399999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.71099999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7592,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737560,"statfs":{"total":95995035648,"available":95987261440,"internally_reserved":0,"allocated":1015808,"data_stored":514151,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55100000000000005,"15min":0.52700000000000002},"min":{"1min":0.39500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.96099999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.42899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":0.91800000000000004,"15min":1.0549999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59399999999999997,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.29199999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.91200000000000003},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.59299999999999997,"15min":0.52700000000000002},"min":{"1min":0.40799999999999997,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57599999999999996,"15min":0.54500000000000004},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.63,"15min":0.57499999999999996},"min":{"1min":0.51100000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.68500000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85699999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69899999999999995,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.89400000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.70399999999999996}]},{"osd":5,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51900000000000002,"15min":0.47399999999999998},"min":{"1min":0.39000000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.88800000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52000000000000002,"15min":0.48799999999999999},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85099999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.54000000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.67300000000000004,"15min":0.63300000000000001},"min":{"1min":0.51900000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.66900000000000004,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82999999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.64300000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.498,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93899999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52400000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94799999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.77400000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.64400000000000002,"15min":0.61499999999999999},"min":{"1min":0.48299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.64800000000000002,"15min":0.623},"min":{"1min":0.49199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.747}]},{"osd":9,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.74099999999999999,"15min":0.71599999999999997},"min":{"1min":0.66300000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.746,"15min":0.72399999999999998},"min":{"1min":0.61699999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.93100000000000005,"5min":1.024,"15min":1.024},"last":0.80000000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.76300000000000001,"15min":0.74199999999999999},"min":{"1min":0.626,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94499999999999995,"5min":0.997,"15min":0.997},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.80300000000000005,"15min":0.77800000000000002},"min":{"1min":0.626,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0029999999999999,"5min":1.038,"15min":1.038},"last":0.879}]},{"osd":11,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82999999999999996,"15min":0.83199999999999996},"min":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.85499999999999998},"min":{"1min":0.66400000000000003,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.90800000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6808,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738344,"statfs":{"total":95995035648,"available":95988064256,"internally_reserved":0,"allocated":999424,"data_stored":502929,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41399999999999998,"15min":0.38400000000000001},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.43099999999999999,"15min":0.40899999999999997},"min":{"1min":0.33100000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.56999999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.62,"15min":0.57499999999999996},"min":{"1min":0.47799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79600000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58999999999999997,"15min":0.55600000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77000000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.68999999999999995}]},{"osd":2,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64700000000000002,"15min":0.60899999999999999},"min":{"1min":0.439,"5min":0.379,"15min":0.379},"max":{"1min":0.91000000000000003,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65000000000000002,"15min":0.59399999999999997},"min":{"1min":0.48999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80100000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Wed Oct 11 16:31:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.64000000000000001},"min":{"1min":0.54200000000000004,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82099999999999995,"5min":1.101,"15min":1.101},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71499999999999997,"15min":0.68700000000000006},"min":{"1min":0.624,"5min":0.498,"15min":0.498},"max":{"1min":0.83799999999999997,"5min":1.194,"15min":1.194},"last":0.54300000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.621,"15min":0.63100000000000001},"min":{"1min":0.46300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88600000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92900000000000005,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.78800000000000003,"5min":1.268,"15min":1.268},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.749,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.626}]},{"osd":7,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.57299999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.85699999999999998,"5min":1.119,"15min":1.119},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.623,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95799999999999996,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.745}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:31:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.77600000000000002,"15min":0.80200000000000005},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.90800000000000003,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.77600000000000002,"15min":0.79900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.876,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69799999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.81200000000000006,"15min":0.82499999999999996},"min":{"1min":0.66700000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.028,"5min":1.137,"15min":1.137},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.82199999999999995,"15min":0.83199999999999996},"min":{"1min":0.68400000000000005,"5min":0.627,"15min":0.627},"max":{"1min":0.876,"5min":1.165,"15min":1.165},"last":0.64400000000000002}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78700000000000003,"15min":0.78800000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.90900000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72099999999999997,"15min":0.71799999999999997},"min":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.71199999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6796,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738356,"statfs":{"total":95995035648,"available":95988076544,"internally_reserved":0,"allocated":1052672,"data_stored":574017,"data_compressed":30188,"data_compressed_allocated":40960,"data_compressed_original":102400,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51500000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39000000000000001,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48999999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59799999999999998,"15min":0.56000000000000005},"min":{"1min":0.41899999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81599999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52700000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75700000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.69099999999999995,"15min":0.58399999999999996},"min":{"1min":0.433,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.234,"5min":0.69699999999999995,"15min":0.57599999999999996},"min":{"1min":0.46800000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.0129999999999999,"5min":7.0129999999999999,"15min":7.0129999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67600000000000005,"15min":0.73799999999999999},"min":{"1min":0.52100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78500000000000003,"5min":4.5,"15min":4.5},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.71599999999999997},"min":{"1min":0.51500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76700000000000002,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.57699999999999996}]},{"osd":6,"last update":"Wed Oct 11 16:31:39 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.70799999999999996,"15min":0.65500000000000003},"min":{"1min":0.52300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.9420000000000002,"5min":6.9420000000000002,"15min":6.9420000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.58199999999999996},"min":{"1min":0.52400000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79200000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56200000000000006}]},{"osd":7,"last update":"Wed Oct 11 16:32:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.373,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55400000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58099999999999996,"15min":0.54600000000000004},"min":{"1min":0.42799999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79200000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.40100000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.48999999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76800000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.622}]},{"osd":9,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.64800000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.84899999999999998,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.454,"5min":0.87,"15min":0.76800000000000002},"min":{"1min":0.63800000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":6.968,"5min":6.968,"15min":6.968},"last":0.71399999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.46200000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89200000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.52900000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.94099999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.59999999999999998}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.68300000000000005,"15min":0.68799999999999994},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73899999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1077248,"data_stored":1109323,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.53100000000000003},"min":{"1min":0.434,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.309,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57199999999999995,"15min":0.53700000000000003},"min":{"1min":0.375,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.874,"15min":0.874},"last":0.62},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57999999999999996,"15min":0.55800000000000005},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77100000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55000000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.439,"15min":0.40699999999999997},"min":{"1min":0.34300000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.59799999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.46600000000000003,"15min":0.42599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65100000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.309}]},{"osd":3,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72599999999999998,"5min":1.089,"15min":1.089},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60299999999999998,"15min":0.56599999999999995},"min":{"1min":0.47599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.79700000000000004}]},{"osd":4,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.621,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82399999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.62,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.754,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65600000000000003,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.629},"min":{"1min":0.52400000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.753}]},{"osd":7,"last update":"Wed Oct 11 16:32:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88800000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70499999999999996,"15min":0.69999999999999996},"min":{"1min":0.45200000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57699999999999996}]},{"osd":8,"last update":"Wed Oct 11 16:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68999999999999995},"min":{"1min":0.502,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83699999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.42199999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:31:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.751,"15min":0.751},"min":{"1min":0.59699999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90900000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77300000000000002,"15min":0.76800000000000002},"min":{"1min":0.57699999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.83499999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72899999999999998,"15min":0.71999999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89700000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.745,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":11175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":66945,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":11883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":11569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":67879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:32:24.034 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T16:32:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/344775192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481002b0 msgr2=0x7fdf481006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/344775192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481002b0 0x7fdf481006c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009990 tx=0x7fdf38009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/344775192 shutdown_connections 2023-10-11T16:32:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/344775192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf48101e30 0x7fdf481086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/344775192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481002b0 0x7fdf481006c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/344775192 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf481014a0 0x7fdf481018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/344775192 >> 172.21.15.111:0/344775192 conn(0x7fdf480fb890 msgr2=0x7fdf480fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:24.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/344775192 shutdown_connections 2023-10-11T16:32:24.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.264+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/344775192 wait complete. 2023-10-11T16:32:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.265+0000 7fdf4c9ff700 1 Processor -- start 2023-10-11T16:32:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.265+0000 7fdf4c9ff700 1 -- start start 2023-10-11T16:32:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf4c9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf481002b0 0x7fdf48198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf4c9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481014a0 0x7fdf48198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:24.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf4c9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf48101e30 0x7fdf4819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf4c9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4819fdb0 con 0x7fdf481014a0 2023-10-11T16:32:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf4c9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4819fef0 con 0x7fdf481002b0 2023-10-11T16:32:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf4c9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf481a0080 con 0x7fdf48101e30 2023-10-11T16:32:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf45d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481014a0 0x7fdf48198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:24.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf4659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf481002b0 0x7fdf48198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf45d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481014a0 0x7fdf48198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59928/0 (socket says 172.21.15.111:59928) 2023-10-11T16:32:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf45d9b700 1 -- 172.21.15.111:0/2605209597 learned_addr learned my addr 172.21.15.111:0/2605209597 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf46d9d700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf48101e30 0x7fdf4819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf45d9b700 1 -- 172.21.15.111:0/2605209597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf48101e30 msgr2=0x7fdf4819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf45d9b700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf48101e30 0x7fdf4819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf45d9b700 1 -- 172.21.15.111:0/2605209597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf481002b0 msgr2=0x7fdf48198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf45d9b700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf481002b0 0x7fdf48198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.266+0000 7fdf45d9b700 1 -- 172.21.15.111:0/2605209597 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380095f0 con 0x7fdf481014a0 2023-10-11T16:32:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.267+0000 7fdf45d9b700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481014a0 0x7fdf48198770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00e9a0 tx=0x7fdf3c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:24.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.267+0000 7fdf377fe700 1 -- 172.21.15.111:0/2605209597 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c009790 con 0x7fdf481014a0 2023-10-11T16:32:24.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.267+0000 7fdf377fe700 1 -- 172.21.15.111:0/2605209597 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf3c010430 con 0x7fdf481014a0 2023-10-11T16:32:24.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.267+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4810bc80 con 0x7fdf481014a0 2023-10-11T16:32:24.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.267+0000 7fdf377fe700 1 -- 172.21.15.111:0/2605209597 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c018680 con 0x7fdf481014a0 2023-10-11T16:32:24.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.267+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4810baf0 con 0x7fdf481014a0 2023-10-11T16:32:24.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.268+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf48192910 con 0x7fdf481014a0 2023-10-11T16:32:24.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.270+0000 7fdf377fe700 1 -- 172.21.15.111:0/2605209597 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fdf3c0105a0 con 0x7fdf481014a0 2023-10-11T16:32:24.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.272+0000 7fdf377fe700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf300a11d0 0x7fdf300a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:24.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.272+0000 7fdf377fe700 1 -- 172.21.15.111:0/2605209597 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fdf3c014070 con 0x7fdf481014a0 2023-10-11T16:32:24.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.272+0000 7fdf4659c700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf300a11d0 0x7fdf300a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:24.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.273+0000 7fdf4659c700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf300a11d0 0x7fdf300a3680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800b2e0 tx=0x7fdf38005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:24.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.276+0000 7fdf377fe700 1 -- 172.21.15.111:0/2605209597 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdf3c051730 con 0x7fdf481014a0 2023-10-11T16:32:24.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:24 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/856548750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:24.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:24 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/856548750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:24.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.469+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf48061c60 con 0x7fdf300a11d0 2023-10-11T16:32:24.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.482+0000 7fdf377fe700 1 -- 172.21.15.111:0/2605209597 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427546 (secure 0 0 0) 0x7fdf48061c60 con 0x7fdf300a11d0 2023-10-11T16:32:24.490 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:24.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf300a11d0 msgr2=0x7fdf300a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf300a11d0 0x7fdf300a3680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800b2e0 tx=0x7fdf38005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481014a0 msgr2=0x7fdf48198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481014a0 0x7fdf48198770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00e9a0 tx=0x7fdf3c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 shutdown_connections 2023-10-11T16:32:24.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdf300a11d0 0x7fdf300a3680 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf48101e30 0x7fdf4819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf481002b0 0x7fdf48198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 --2- 172.21.15.111:0/2605209597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf481014a0 0x7fdf48198770 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 >> 172.21.15.111:0/2605209597 conn(0x7fdf480fb890 msgr2=0x7fdf48106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:24.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 shutdown_connections 2023-10-11T16:32:24.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.487+0000 7fdf4c9ff700 1 -- 172.21.15.111:0/2605209597 wait complete. 2023-10-11T16:32:24.496 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:32:24.507 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2023-10-11T16:32:24.087409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670932,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":786,"num_read_kb":530,"num_write":357,"num_write_kb":3077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1220437,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82464,"kb_used_data":12160,"kb_used_omap":0,"kb_used_meta":70208,"kb_avail":1124859360,"statfs":{"total":1151940427776,"available":1151855984640,"internally_reserved":0,"allocated":12451840,"data_stored":8079545,"data_compressed":395806,"data_compressed_allocated":536576,"data_compressed_original":2592768,"omap_allocated":0,"internal_metadata":71892992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":565,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":47,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100200+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:32:10.100200+0000","last_peered":"2023-10-11T16:32:10.100200+0000","last_clean":"2023-10-11T16:32:10.100200+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:32:10.100200+0000","last_undegraded":"2023-10-11T16:32:10.100200+0000","last_fullsized":"2023-10-11T16:32:10.100200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:26:00.751881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100937+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:32:10.100937+0000","last_peered":"2023-10-11T16:32:10.100937+0000","last_clean":"2023-10-11T16:32:10.100937+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:32:10.100937+0000","last_undegraded":"2023-10-11T16:32:10.100937+0000","last_fullsized":"2023-10-11T16:32:10.100937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:51:20.593620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101715+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:32:10.101715+0000","last_peered":"2023-10-11T16:32:10.101715+0000","last_clean":"2023-10-11T16:32:10.101715+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:32:10.101715+0000","last_undegraded":"2023-10-11T16:32:10.101715+0000","last_fullsized":"2023-10-11T16:32:10.101715+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:20:45.562554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083360+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:32:10.083360+0000","last_peered":"2023-10-11T16:32:10.083360+0000","last_clean":"2023-10-11T16:32:10.083360+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:32:10.083360+0000","last_undegraded":"2023-10-11T16:32:10.083360+0000","last_fullsized":"2023-10-11T16:32:10.083360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:03:22.258325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.071382+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:32:10.071382+0000","last_peered":"2023-10-11T16:32:10.071382+0000","last_clean":"2023-10-11T16:32:10.071382+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:32:10.071382+0000","last_undegraded":"2023-10-11T16:32:10.071382+0000","last_fullsized":"2023-10-11T16:32:10.071382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:59:15.949889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.086763+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:32:10.086763+0000","last_peered":"2023-10-11T16:32:10.086763+0000","last_clean":"2023-10-11T16:32:10.086763+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:32:10.086763+0000","last_undegraded":"2023-10-11T16:32:10.086763+0000","last_fullsized":"2023-10-11T16:32:10.086763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:17:51.767698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.087839+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:32:10.087839+0000","last_peered":"2023-10-11T16:32:10.087839+0000","last_clean":"2023-10-11T16:32:10.087839+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:32:10.087839+0000","last_undegraded":"2023-10-11T16:32:10.087839+0000","last_fullsized":"2023-10-11T16:32:10.087839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:31:16.375555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.099303+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:32:10.099303+0000","last_peered":"2023-10-11T16:32:10.099303+0000","last_clean":"2023-10-11T16:32:10.099303+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:32:10.099303+0000","last_undegraded":"2023-10-11T16:32:10.099303+0000","last_fullsized":"2023-10-11T16:32:10.099303+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:14:00.865418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092992+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:32:10.092992+0000","last_peered":"2023-10-11T16:32:10.092992+0000","last_clean":"2023-10-11T16:32:10.092992+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:32:10.092992+0000","last_undegraded":"2023-10-11T16:32:10.092992+0000","last_fullsized":"2023-10-11T16:32:10.092992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:48:35.792817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.091540+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:32:10.091540+0000","last_peered":"2023-10-11T16:32:10.091540+0000","last_clean":"2023-10-11T16:32:10.091540+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:32:10.091540+0000","last_undegraded":"2023-10-11T16:32:10.091540+0000","last_fullsized":"2023-10-11T16:32:10.091540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:35.706510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.094525+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:32:10.094525+0000","last_peered":"2023-10-11T16:32:10.094525+0000","last_clean":"2023-10-11T16:32:10.094525+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:32:10.094525+0000","last_undegraded":"2023-10-11T16:32:10.094525+0000","last_fullsized":"2023-10-11T16:32:10.094525+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:39:30.482685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089635+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:32:10.089635+0000","last_peered":"2023-10-11T16:32:10.089635+0000","last_clean":"2023-10-11T16:32:10.089635+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:32:10.089635+0000","last_undegraded":"2023-10-11T16:32:10.089635+0000","last_fullsized":"2023-10-11T16:32:10.089635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:34:42.597400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089114+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:32:10.089114+0000","last_peered":"2023-10-11T16:32:10.089114+0000","last_clean":"2023-10-11T16:32:10.089114+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:32:10.089114+0000","last_undegraded":"2023-10-11T16:32:10.089114+0000","last_fullsized":"2023-10-11T16:32:10.089114+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:46:38.920873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100645+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:32:10.100645+0000","last_peered":"2023-10-11T16:32:10.100645+0000","last_clean":"2023-10-11T16:32:10.100645+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:32:10.100645+0000","last_undegraded":"2023-10-11T16:32:10.100645+0000","last_fullsized":"2023-10-11T16:32:10.100645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:53:32.036585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100988+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:32:10.100988+0000","last_peered":"2023-10-11T16:32:10.100988+0000","last_clean":"2023-10-11T16:32:10.100988+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:32:10.100988+0000","last_undegraded":"2023-10-11T16:32:10.100988+0000","last_fullsized":"2023-10-11T16:32:10.100988+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:32.595236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100326+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:32:10.100326+0000","last_peered":"2023-10-11T16:32:10.100326+0000","last_clean":"2023-10-11T16:32:10.100326+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:32:10.100326+0000","last_undegraded":"2023-10-11T16:32:10.100326+0000","last_fullsized":"2023-10-11T16:32:10.100326+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:51.221018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100337+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:32:10.100337+0000","last_peered":"2023-10-11T16:32:10.100337+0000","last_clean":"2023-10-11T16:32:10.100337+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:32:10.100337+0000","last_undegraded":"2023-10-11T16:32:10.100337+0000","last_fullsized":"2023-10-11T16:32:10.100337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:40:38.294389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083579+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:32:10.083579+0000","last_peered":"2023-10-11T16:32:10.083579+0000","last_clean":"2023-10-11T16:32:10.083579+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:32:10.083579+0000","last_undegraded":"2023-10-11T16:32:10.083579+0000","last_fullsized":"2023-10-11T16:32:10.083579+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:56:56.119945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102789+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:32:10.102789+0000","last_peered":"2023-10-11T16:32:10.102789+0000","last_clean":"2023-10-11T16:32:10.102789+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:32:10.102789+0000","last_undegraded":"2023-10-11T16:32:10.102789+0000","last_fullsized":"2023-10-11T16:32:10.102789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:24:13.338895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159916+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:31:42.159916+0000","last_peered":"2023-10-11T16:31:42.159916+0000","last_clean":"2023-10-11T16:31:42.159916+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:31:42.159916+0000","last_undegraded":"2023-10-11T16:31:42.159916+0000","last_fullsized":"2023-10-11T16:31:42.159916+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:50:38.133970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159320+0000","last_change":"2023-10-11T16:30:41.198854+0000","last_active":"2023-10-11T16:31:42.159320+0000","last_peered":"2023-10-11T16:31:42.159320+0000","last_clean":"2023-10-11T16:31:42.159320+0000","last_became_active":"2023-10-11T16:30:41.189677+0000","last_became_peered":"2023-10-11T16:30:41.189677+0000","last_unstale":"2023-10-11T16:31:42.159320+0000","last_undegraded":"2023-10-11T16:31:42.159320+0000","last_fullsized":"2023-10-11T16:31:42.159320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:02:16.154156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097708+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:32:10.097708+0000","last_peered":"2023-10-11T16:32:10.097708+0000","last_clean":"2023-10-11T16:32:10.097708+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:32:10.097708+0000","last_undegraded":"2023-10-11T16:32:10.097708+0000","last_fullsized":"2023-10-11T16:32:10.097708+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:10.365459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288525+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:31:42.288525+0000","last_peered":"2023-10-11T16:31:42.288525+0000","last_clean":"2023-10-11T16:31:42.288525+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:31:42.288525+0000","last_undegraded":"2023-10-11T16:31:42.288525+0000","last_fullsized":"2023-10-11T16:31:42.288525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:37:11.027756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534803+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:31:42.534803+0000","last_peered":"2023-10-11T16:31:42.534803+0000","last_clean":"2023-10-11T16:31:42.534803+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:31:42.534803+0000","last_undegraded":"2023-10-11T16:31:42.534803+0000","last_fullsized":"2023-10-11T16:31:42.534803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:29:20.287428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102278+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:32:10.102278+0000","last_peered":"2023-10-11T16:32:10.102278+0000","last_clean":"2023-10-11T16:32:10.102278+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:32:10.102278+0000","last_undegraded":"2023-10-11T16:32:10.102278+0000","last_fullsized":"2023-10-11T16:32:10.102278+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:23:33.919634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288642+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:31:42.288642+0000","last_peered":"2023-10-11T16:31:42.288642+0000","last_clean":"2023-10-11T16:31:42.288642+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:31:42.288642+0000","last_undegraded":"2023-10-11T16:31:42.288642+0000","last_fullsized":"2023-10-11T16:31:42.288642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:42:02.132395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462596+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:31:42.462596+0000","last_peered":"2023-10-11T16:31:42.462596+0000","last_clean":"2023-10-11T16:31:42.462596+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:31:42.462596+0000","last_undegraded":"2023-10-11T16:31:42.462596+0000","last_fullsized":"2023-10-11T16:31:42.462596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:38:07.419529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092994+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:32:10.092994+0000","last_peered":"2023-10-11T16:32:10.092994+0000","last_clean":"2023-10-11T16:32:10.092994+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:32:10.092994+0000","last_undegraded":"2023-10-11T16:32:10.092994+0000","last_fullsized":"2023-10-11T16:32:10.092994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:44:22.080004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288644+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:31:42.288644+0000","last_peered":"2023-10-11T16:31:42.288644+0000","last_clean":"2023-10-11T16:31:42.288644+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:31:42.288644+0000","last_undegraded":"2023-10-11T16:31:42.288644+0000","last_fullsized":"2023-10-11T16:31:42.288644+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:40.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462545+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:31:42.462545+0000","last_peered":"2023-10-11T16:31:42.462545+0000","last_clean":"2023-10-11T16:31:42.462545+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:31:42.462545+0000","last_undegraded":"2023-10-11T16:31:42.462545+0000","last_fullsized":"2023-10-11T16:31:42.462545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:32:50.524951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083771+0000","last_change":"2023-10-11T16:30:41.195736+0000","last_active":"2023-10-11T16:32:10.083771+0000","last_peered":"2023-10-11T16:32:10.083771+0000","last_clean":"2023-10-11T16:32:10.083771+0000","last_became_active":"2023-10-11T16:30:41.195352+0000","last_became_peered":"2023-10-11T16:30:41.195352+0000","last_unstale":"2023-10-11T16:32:10.083771+0000","last_undegraded":"2023-10-11T16:32:10.083771+0000","last_fullsized":"2023-10-11T16:32:10.083771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:29:40.859051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160178+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:31:42.160178+0000","last_peered":"2023-10-11T16:31:42.160178+0000","last_clean":"2023-10-11T16:31:42.160178+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:31:42.160178+0000","last_undegraded":"2023-10-11T16:31:42.160178+0000","last_fullsized":"2023-10-11T16:31:42.160178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:55:21.244206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'10","reported_seq":148,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:19.340286+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:32:19.340286+0000","last_peered":"2023-10-11T16:32:19.340286+0000","last_clean":"2023-10-11T16:32:19.340286+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:32:19.340286+0000","last_undegraded":"2023-10-11T16:32:19.340286+0000","last_fullsized":"2023-10-11T16:32:19.340286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:37:47.462085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11029,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":14,"num_write":7,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.078744+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:32:10.078744+0000","last_peered":"2023-10-11T16:32:10.078744+0000","last_clean":"2023-10-11T16:32:10.078744+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:32:10.078744+0000","last_undegraded":"2023-10-11T16:32:10.078744+0000","last_fullsized":"2023-10-11T16:32:10.078744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:49:22.705791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160227+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:31:42.160227+0000","last_peered":"2023-10-11T16:31:42.160227+0000","last_clean":"2023-10-11T16:31:42.160227+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:31:42.160227+0000","last_undegraded":"2023-10-11T16:31:42.160227+0000","last_fullsized":"2023-10-11T16:31:42.160227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:18:51.568546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285229+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:31:42.285229+0000","last_peered":"2023-10-11T16:31:42.285229+0000","last_clean":"2023-10-11T16:31:42.285229+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:31:42.285229+0000","last_undegraded":"2023-10-11T16:31:42.285229+0000","last_fullsized":"2023-10-11T16:31:42.285229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:13:18.888864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.085765+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:32:10.085765+0000","last_peered":"2023-10-11T16:32:10.085765+0000","last_clean":"2023-10-11T16:32:10.085765+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:32:10.085765+0000","last_undegraded":"2023-10-11T16:32:10.085765+0000","last_fullsized":"2023-10-11T16:32:10.085765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:46:58.254928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.164266+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:31:42.164266+0000","last_peered":"2023-10-11T16:31:42.164266+0000","last_clean":"2023-10-11T16:31:42.164266+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:31:42.164266+0000","last_undegraded":"2023-10-11T16:31:42.164266+0000","last_fullsized":"2023-10-11T16:31:42.164266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:20:42.850191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285323+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:31:42.285323+0000","last_peered":"2023-10-11T16:31:42.285323+0000","last_clean":"2023-10-11T16:31:42.285323+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:31:42.285323+0000","last_undegraded":"2023-10-11T16:31:42.285323+0000","last_fullsized":"2023-10-11T16:31:42.285323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:11:41.038914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101540+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:32:10.101540+0000","last_peered":"2023-10-11T16:32:10.101540+0000","last_clean":"2023-10-11T16:32:10.101540+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:32:10.101540+0000","last_undegraded":"2023-10-11T16:32:10.101540+0000","last_fullsized":"2023-10-11T16:32:10.101540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:04:56.860908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161225+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:31:42.161225+0000","last_peered":"2023-10-11T16:31:42.161225+0000","last_clean":"2023-10-11T16:31:42.161225+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:31:42.161225+0000","last_undegraded":"2023-10-11T16:31:42.161225+0000","last_fullsized":"2023-10-11T16:31:42.161225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:40.600507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288380+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:31:42.288380+0000","last_peered":"2023-10-11T16:31:42.288380+0000","last_clean":"2023-10-11T16:31:42.288380+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:31:42.288380+0000","last_undegraded":"2023-10-11T16:31:42.288380+0000","last_fullsized":"2023-10-11T16:31:42.288380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:37:32.836177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088886+0000","last_change":"2023-10-11T16:31:42.286928+0000","last_active":"2023-10-11T16:32:10.088886+0000","last_peered":"2023-10-11T16:32:10.088886+0000","last_clean":"2023-10-11T16:32:10.088886+0000","last_became_active":"2023-10-11T16:31:42.286409+0000","last_became_peered":"2023-10-11T16:31:42.286409+0000","last_unstale":"2023-10-11T16:32:10.088886+0000","last_undegraded":"2023-10-11T16:32:10.088886+0000","last_fullsized":"2023-10-11T16:32:10.088886+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:16:50.559256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533799+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:31:42.533799+0000","last_peered":"2023-10-11T16:31:42.533799+0000","last_clean":"2023-10-11T16:31:42.533799+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:31:42.533799+0000","last_undegraded":"2023-10-11T16:31:42.533799+0000","last_fullsized":"2023-10-11T16:31:42.533799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:53:33.531388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":139,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:21.342444+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:32:21.342444+0000","last_peered":"2023-10-11T16:32:21.342444+0000","last_clean":"2023-10-11T16:32:21.342444+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:32:21.342444+0000","last_undegraded":"2023-10-11T16:32:21.342444+0000","last_fullsized":"2023-10-11T16:32:21.342444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:22.346902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092896+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:32:10.092896+0000","last_peered":"2023-10-11T16:32:10.092896+0000","last_clean":"2023-10-11T16:32:10.092896+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:32:10.092896+0000","last_undegraded":"2023-10-11T16:32:10.092896+0000","last_fullsized":"2023-10-11T16:32:10.092896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:36:02.019758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.157442+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:31:42.157442+0000","last_peered":"2023-10-11T16:31:42.157442+0000","last_clean":"2023-10-11T16:31:42.157442+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:31:42.157442+0000","last_undegraded":"2023-10-11T16:31:42.157442+0000","last_fullsized":"2023-10-11T16:31:42.157442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:20:41.440369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161173+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:31:42.161173+0000","last_peered":"2023-10-11T16:31:42.161173+0000","last_clean":"2023-10-11T16:31:42.161173+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:31:42.161173+0000","last_undegraded":"2023-10-11T16:31:42.161173+0000","last_fullsized":"2023-10-11T16:31:42.161173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:48.986542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101928+0000","last_change":"2023-10-11T16:31:42.464453+0000","last_active":"2023-10-11T16:32:10.101928+0000","last_peered":"2023-10-11T16:32:10.101928+0000","last_clean":"2023-10-11T16:32:10.101928+0000","last_became_active":"2023-10-11T16:31:42.463997+0000","last_became_peered":"2023-10-11T16:31:42.463997+0000","last_unstale":"2023-10-11T16:32:10.101928+0000","last_undegraded":"2023-10-11T16:32:10.101928+0000","last_fullsized":"2023-10-11T16:32:10.101928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:44:47.706246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288834+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:31:42.288834+0000","last_peered":"2023-10-11T16:31:42.288834+0000","last_clean":"2023-10-11T16:31:42.288834+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:31:42.288834+0000","last_undegraded":"2023-10-11T16:31:42.288834+0000","last_fullsized":"2023-10-11T16:31:42.288834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:13:12.138603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289000+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:31:42.289000+0000","last_peered":"2023-10-11T16:31:42.289000+0000","last_clean":"2023-10-11T16:31:42.289000+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:31:42.289000+0000","last_undegraded":"2023-10-11T16:31:42.289000+0000","last_fullsized":"2023-10-11T16:31:42.289000+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:15:55.186995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092874+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:32:10.092874+0000","last_peered":"2023-10-11T16:32:10.092874+0000","last_clean":"2023-10-11T16:32:10.092874+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:32:10.092874+0000","last_undegraded":"2023-10-11T16:32:10.092874+0000","last_fullsized":"2023-10-11T16:32:10.092874+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:23:25.760798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158398+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:31:42.158398+0000","last_peered":"2023-10-11T16:31:42.158398+0000","last_clean":"2023-10-11T16:31:42.158398+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:31:42.158398+0000","last_undegraded":"2023-10-11T16:31:42.158398+0000","last_fullsized":"2023-10-11T16:31:42.158398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:04:51.440115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":303,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:22.343291+0000","last_change":"2023-10-11T16:30:41.788994+0000","last_active":"2023-10-11T16:32:22.343291+0000","last_peered":"2023-10-11T16:32:22.343291+0000","last_clean":"2023-10-11T16:32:22.343291+0000","last_became_active":"2023-10-11T16:30:41.189949+0000","last_became_peered":"2023-10-11T16:30:41.189949+0000","last_unstale":"2023-10-11T16:32:22.343291+0000","last_undegraded":"2023-10-11T16:32:22.343291+0000","last_fullsized":"2023-10-11T16:32:22.343291+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:58:15.728137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66349,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":217,"num_read_kb":174,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38515,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.082731+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:32:10.082731+0000","last_peered":"2023-10-11T16:32:10.082731+0000","last_clean":"2023-10-11T16:32:10.082731+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:32:10.082731+0000","last_undegraded":"2023-10-11T16:32:10.082731+0000","last_fullsized":"2023-10-11T16:32:10.082731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:04:39.079541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289353+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:31:42.289353+0000","last_peered":"2023-10-11T16:31:42.289353+0000","last_clean":"2023-10-11T16:31:42.289353+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:31:42.289353+0000","last_undegraded":"2023-10-11T16:31:42.289353+0000","last_fullsized":"2023-10-11T16:31:42.289353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:31:29.246174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160086+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:31:42.160086+0000","last_peered":"2023-10-11T16:31:42.160086+0000","last_clean":"2023-10-11T16:31:42.160086+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:31:42.160086+0000","last_undegraded":"2023-10-11T16:31:42.160086+0000","last_fullsized":"2023-10-11T16:31:42.160086+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:39:49.561867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.104692+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:32:10.104692+0000","last_peered":"2023-10-11T16:32:10.104692+0000","last_clean":"2023-10-11T16:32:10.104692+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:32:10.104692+0000","last_undegraded":"2023-10-11T16:32:10.104692+0000","last_fullsized":"2023-10-11T16:32:10.104692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:17:46.544148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.163839+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:31:42.163839+0000","last_peered":"2023-10-11T16:31:42.163839+0000","last_clean":"2023-10-11T16:31:42.163839+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:31:42.163839+0000","last_undegraded":"2023-10-11T16:31:42.163839+0000","last_fullsized":"2023-10-11T16:31:42.163839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:36:11.677512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288800+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:31:42.288800+0000","last_peered":"2023-10-11T16:31:42.288800+0000","last_clean":"2023-10-11T16:31:42.288800+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:31:42.288800+0000","last_undegraded":"2023-10-11T16:31:42.288800+0000","last_fullsized":"2023-10-11T16:31:42.288800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:55:39.104456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.091980+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:32:10.091980+0000","last_peered":"2023-10-11T16:32:10.091980+0000","last_clean":"2023-10-11T16:32:10.091980+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:32:10.091980+0000","last_undegraded":"2023-10-11T16:32:10.091980+0000","last_fullsized":"2023-10-11T16:32:10.091980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:26:20.985716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158790+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:31:42.158790+0000","last_peered":"2023-10-11T16:31:42.158790+0000","last_clean":"2023-10-11T16:31:42.158790+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:31:42.158790+0000","last_undegraded":"2023-10-11T16:31:42.158790+0000","last_fullsized":"2023-10-11T16:31:42.158790+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:05:47.475689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":135,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:18.337663+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:32:18.337663+0000","last_peered":"2023-10-11T16:32:18.337663+0000","last_clean":"2023-10-11T16:32:18.337663+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:32:18.337663+0000","last_undegraded":"2023-10-11T16:32:18.337663+0000","last_fullsized":"2023-10-11T16:32:18.337663+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:37:12.900675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093444+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:32:10.093444+0000","last_peered":"2023-10-11T16:32:10.093444+0000","last_clean":"2023-10-11T16:32:10.093444+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:32:10.093444+0000","last_undegraded":"2023-10-11T16:32:10.093444+0000","last_fullsized":"2023-10-11T16:32:10.093444+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:27:01.920127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285928+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:31:42.285928+0000","last_peered":"2023-10-11T16:31:42.285928+0000","last_clean":"2023-10-11T16:31:42.285928+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:31:42.285928+0000","last_undegraded":"2023-10-11T16:31:42.285928+0000","last_fullsized":"2023-10-11T16:31:42.285928+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:08:54.198092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462035+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:31:42.462035+0000","last_peered":"2023-10-11T16:31:42.462035+0000","last_clean":"2023-10-11T16:31:42.462035+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:31:42.462035+0000","last_undegraded":"2023-10-11T16:31:42.462035+0000","last_fullsized":"2023-10-11T16:31:42.462035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:55:08.925417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092273+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:32:10.092273+0000","last_peered":"2023-10-11T16:32:10.092273+0000","last_clean":"2023-10-11T16:32:10.092273+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:32:10.092273+0000","last_undegraded":"2023-10-11T16:32:10.092273+0000","last_fullsized":"2023-10-11T16:32:10.092273+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:15:28.155402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288455+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:31:42.288455+0000","last_peered":"2023-10-11T16:31:42.288455+0000","last_clean":"2023-10-11T16:31:42.288455+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:31:42.288455+0000","last_undegraded":"2023-10-11T16:31:42.288455+0000","last_fullsized":"2023-10-11T16:31:42.288455+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:28:12.958080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462997+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:31:42.462997+0000","last_peered":"2023-10-11T16:31:42.462997+0000","last_clean":"2023-10-11T16:31:42.462997+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:31:42.462997+0000","last_undegraded":"2023-10-11T16:31:42.462997+0000","last_fullsized":"2023-10-11T16:31:42.462997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:03:09.115056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.080956+0000","last_change":"2023-10-11T16:31:42.287141+0000","last_active":"2023-10-11T16:32:10.080956+0000","last_peered":"2023-10-11T16:32:10.080956+0000","last_clean":"2023-10-11T16:32:10.080956+0000","last_became_active":"2023-10-11T16:31:42.286758+0000","last_became_peered":"2023-10-11T16:31:42.286758+0000","last_unstale":"2023-10-11T16:32:10.080956+0000","last_undegraded":"2023-10-11T16:32:10.080956+0000","last_fullsized":"2023-10-11T16:32:10.080956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:40:33.920330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462187+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:31:42.462187+0000","last_peered":"2023-10-11T16:31:42.462187+0000","last_clean":"2023-10-11T16:31:42.462187+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:31:42.462187+0000","last_undegraded":"2023-10-11T16:31:42.462187+0000","last_fullsized":"2023-10-11T16:31:42.462187+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:43.818267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158964+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:31:42.158964+0000","last_peered":"2023-10-11T16:31:42.158964+0000","last_clean":"2023-10-11T16:31:42.158964+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:31:42.158964+0000","last_undegraded":"2023-10-11T16:31:42.158964+0000","last_fullsized":"2023-10-11T16:31:42.158964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:16:18.715348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093724+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:32:10.093724+0000","last_peered":"2023-10-11T16:32:10.093724+0000","last_clean":"2023-10-11T16:32:10.093724+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:32:10.093724+0000","last_undegraded":"2023-10-11T16:32:10.093724+0000","last_fullsized":"2023-10-11T16:32:10.093724+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:03:22.304558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285744+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:31:42.285744+0000","last_peered":"2023-10-11T16:31:42.285744+0000","last_clean":"2023-10-11T16:31:42.285744+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:31:42.285744+0000","last_undegraded":"2023-10-11T16:31:42.285744+0000","last_fullsized":"2023-10-11T16:31:42.285744+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:45:09.560952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160923+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:31:42.160923+0000","last_peered":"2023-10-11T16:31:42.160923+0000","last_clean":"2023-10-11T16:31:42.160923+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:31:42.160923+0000","last_undegraded":"2023-10-11T16:31:42.160923+0000","last_fullsized":"2023-10-11T16:31:42.160923+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:11:25.625028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.095113+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:32:10.095113+0000","last_peered":"2023-10-11T16:32:10.095113+0000","last_clean":"2023-10-11T16:32:10.095113+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:32:10.095113+0000","last_undegraded":"2023-10-11T16:32:10.095113+0000","last_fullsized":"2023-10-11T16:32:10.095113+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:33:24.458841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.286057+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:31:42.286057+0000","last_peered":"2023-10-11T16:31:42.286057+0000","last_clean":"2023-10-11T16:31:42.286057+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:31:42.286057+0000","last_undegraded":"2023-10-11T16:31:42.286057+0000","last_fullsized":"2023-10-11T16:31:42.286057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:23:48.290443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083891+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:32:10.083891+0000","last_peered":"2023-10-11T16:32:10.083891+0000","last_clean":"2023-10-11T16:32:10.083891+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:32:10.083891+0000","last_undegraded":"2023-10-11T16:32:10.083891+0000","last_fullsized":"2023-10-11T16:32:10.083891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:18:07.940669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":140,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:21.341422+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:32:21.341422+0000","last_peered":"2023-10-11T16:32:21.341422+0000","last_clean":"2023-10-11T16:32:21.341422+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:32:21.341422+0000","last_undegraded":"2023-10-11T16:32:21.341422+0000","last_fullsized":"2023-10-11T16:32:21.341422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:13:34.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161339+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:31:42.161339+0000","last_peered":"2023-10-11T16:31:42.161339+0000","last_clean":"2023-10-11T16:31:42.161339+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:31:42.161339+0000","last_undegraded":"2023-10-11T16:31:42.161339+0000","last_fullsized":"2023-10-11T16:31:42.161339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:05:48.188359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101162+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:32:10.101162+0000","last_peered":"2023-10-11T16:32:10.101162+0000","last_clean":"2023-10-11T16:32:10.101162+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:32:10.101162+0000","last_undegraded":"2023-10-11T16:32:10.101162+0000","last_fullsized":"2023-10-11T16:32:10.101162+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:12:34.720396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288780+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:31:42.288780+0000","last_peered":"2023-10-11T16:31:42.288780+0000","last_clean":"2023-10-11T16:31:42.288780+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:31:42.288780+0000","last_undegraded":"2023-10-11T16:31:42.288780+0000","last_fullsized":"2023-10-11T16:31:42.288780+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:17:09.996158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160483+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:31:42.160483+0000","last_peered":"2023-10-11T16:31:42.160483+0000","last_clean":"2023-10-11T16:31:42.160483+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:31:42.160483+0000","last_undegraded":"2023-10-11T16:31:42.160483+0000","last_fullsized":"2023-10-11T16:31:42.160483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:36.114850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289135+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:31:42.289135+0000","last_peered":"2023-10-11T16:31:42.289135+0000","last_clean":"2023-10-11T16:31:42.289135+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:31:42.289135+0000","last_undegraded":"2023-10-11T16:31:42.289135+0000","last_fullsized":"2023-10-11T16:31:42.289135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:30:07.695611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.074567+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:32:10.074567+0000","last_peered":"2023-10-11T16:32:10.074567+0000","last_clean":"2023-10-11T16:32:10.074567+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:32:10.074567+0000","last_undegraded":"2023-10-11T16:32:10.074567+0000","last_fullsized":"2023-10-11T16:32:10.074567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:33:14.082898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160623+0000","last_change":"2023-10-11T16:30:41.938949+0000","last_active":"2023-10-11T16:31:42.160623+0000","last_peered":"2023-10-11T16:31:42.160623+0000","last_clean":"2023-10-11T16:31:42.160623+0000","last_became_active":"2023-10-11T16:30:41.192902+0000","last_became_peered":"2023-10-11T16:30:41.192902+0000","last_unstale":"2023-10-11T16:31:42.160623+0000","last_undegraded":"2023-10-11T16:31:42.160623+0000","last_fullsized":"2023-10-11T16:31:42.160623+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:57:28.224909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.163836+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:31:42.163836+0000","last_peered":"2023-10-11T16:31:42.163836+0000","last_clean":"2023-10-11T16:31:42.163836+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:31:42.163836+0000","last_undegraded":"2023-10-11T16:31:42.163836+0000","last_fullsized":"2023-10-11T16:31:42.163836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:42:22.315441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.164029+0000","last_change":"2023-10-11T16:30:41.205550+0000","last_active":"2023-10-11T16:31:42.164029+0000","last_peered":"2023-10-11T16:31:42.164029+0000","last_clean":"2023-10-11T16:31:42.164029+0000","last_became_active":"2023-10-11T16:30:41.190345+0000","last_became_peered":"2023-10-11T16:30:41.190345+0000","last_unstale":"2023-10-11T16:31:42.164029+0000","last_undegraded":"2023-10-11T16:31:42.164029+0000","last_fullsized":"2023-10-11T16:31:42.164029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:32:00.140298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.074906+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:32:10.074906+0000","last_peered":"2023-10-11T16:32:10.074906+0000","last_clean":"2023-10-11T16:32:10.074906+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:32:10.074906+0000","last_undegraded":"2023-10-11T16:32:10.074906+0000","last_fullsized":"2023-10-11T16:32:10.074906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:57:43.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534261+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:31:42.534261+0000","last_peered":"2023-10-11T16:31:42.534261+0000","last_clean":"2023-10-11T16:31:42.534261+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:31:42.534261+0000","last_undegraded":"2023-10-11T16:31:42.534261+0000","last_fullsized":"2023-10-11T16:31:42.534261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:13:32.227680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.564394+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:32:10.564394+0000","last_peered":"2023-10-11T16:32:10.564394+0000","last_clean":"2023-10-11T16:32:10.564394+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:32:10.564394+0000","last_undegraded":"2023-10-11T16:32:10.564394+0000","last_fullsized":"2023-10-11T16:32:10.564394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:02:28.868871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101410+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:32:10.101410+0000","last_peered":"2023-10-11T16:32:10.101410+0000","last_clean":"2023-10-11T16:32:10.101410+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:32:10.101410+0000","last_undegraded":"2023-10-11T16:32:10.101410+0000","last_fullsized":"2023-10-11T16:32:10.101410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:50:14.165387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159765+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:31:42.159765+0000","last_peered":"2023-10-11T16:31:42.159765+0000","last_clean":"2023-10-11T16:31:42.159765+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:31:42.159765+0000","last_undegraded":"2023-10-11T16:31:42.159765+0000","last_fullsized":"2023-10-11T16:31:42.159765+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:04:09.873784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289422+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:31:42.289422+0000","last_peered":"2023-10-11T16:31:42.289422+0000","last_clean":"2023-10-11T16:31:42.289422+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:31:42.289422+0000","last_undegraded":"2023-10-11T16:31:42.289422+0000","last_fullsized":"2023-10-11T16:31:42.289422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:03.781734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100197+0000","last_change":"2023-10-11T16:31:42.290019+0000","last_active":"2023-10-11T16:32:10.100197+0000","last_peered":"2023-10-11T16:32:10.100197+0000","last_clean":"2023-10-11T16:32:10.100197+0000","last_became_active":"2023-10-11T16:31:42.289721+0000","last_became_peered":"2023-10-11T16:31:42.289721+0000","last_unstale":"2023-10-11T16:32:10.100197+0000","last_undegraded":"2023-10-11T16:32:10.100197+0000","last_fullsized":"2023-10-11T16:32:10.100197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:04:10.132863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289138+0000","last_change":"2023-10-11T16:30:41.193489+0000","last_active":"2023-10-11T16:31:42.289138+0000","last_peered":"2023-10-11T16:31:42.289138+0000","last_clean":"2023-10-11T16:31:42.289138+0000","last_became_active":"2023-10-11T16:30:41.193060+0000","last_became_peered":"2023-10-11T16:30:41.193060+0000","last_unstale":"2023-10-11T16:31:42.289138+0000","last_undegraded":"2023-10-11T16:31:42.289138+0000","last_fullsized":"2023-10-11T16:31:42.289138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:51:45.983921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533782+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:31:42.533782+0000","last_peered":"2023-10-11T16:31:42.533782+0000","last_clean":"2023-10-11T16:31:42.533782+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:31:42.533782+0000","last_undegraded":"2023-10-11T16:31:42.533782+0000","last_fullsized":"2023-10-11T16:31:42.533782+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:43:17.566817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.057040+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:32:10.057040+0000","last_peered":"2023-10-11T16:32:10.057040+0000","last_clean":"2023-10-11T16:32:10.057040+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:32:10.057040+0000","last_undegraded":"2023-10-11T16:32:10.057040+0000","last_fullsized":"2023-10-11T16:32:10.057040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:20:18.016311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462851+0000","last_change":"2023-10-11T16:30:41.404891+0000","last_active":"2023-10-11T16:31:42.462851+0000","last_peered":"2023-10-11T16:31:42.462851+0000","last_clean":"2023-10-11T16:31:42.462851+0000","last_became_active":"2023-10-11T16:30:41.404396+0000","last_became_peered":"2023-10-11T16:30:41.404396+0000","last_unstale":"2023-10-11T16:31:42.462851+0000","last_undegraded":"2023-10-11T16:31:42.462851+0000","last_fullsized":"2023-10-11T16:31:42.462851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:41:36.636884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533885+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:31:42.533885+0000","last_peered":"2023-10-11T16:31:42.533885+0000","last_clean":"2023-10-11T16:31:42.533885+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:31:42.533885+0000","last_undegraded":"2023-10-11T16:31:42.533885+0000","last_fullsized":"2023-10-11T16:31:42.533885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:11:17.333111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097932+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:32:10.097932+0000","last_peered":"2023-10-11T16:32:10.097932+0000","last_clean":"2023-10-11T16:32:10.097932+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:32:10.097932+0000","last_undegraded":"2023-10-11T16:32:10.097932+0000","last_fullsized":"2023-10-11T16:32:10.097932+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:14:58.964194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289262+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:31:42.289262+0000","last_peered":"2023-10-11T16:31:42.289262+0000","last_clean":"2023-10-11T16:31:42.289262+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:31:42.289262+0000","last_undegraded":"2023-10-11T16:31:42.289262+0000","last_fullsized":"2023-10-11T16:31:42.289262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:43:35.192572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:12.146327+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:32:12.146327+0000","last_peered":"2023-10-11T16:32:12.146327+0000","last_clean":"2023-10-11T16:32:12.146327+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:32:12.146327+0000","last_undegraded":"2023-10-11T16:32:12.146327+0000","last_fullsized":"2023-10-11T16:32:12.146327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:52:56.087178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102288+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:32:10.102288+0000","last_peered":"2023-10-11T16:32:10.102288+0000","last_clean":"2023-10-11T16:32:10.102288+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:32:10.102288+0000","last_undegraded":"2023-10-11T16:32:10.102288+0000","last_fullsized":"2023-10-11T16:32:10.102288+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:38.182167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288761+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:31:42.288761+0000","last_peered":"2023-10-11T16:31:42.288761+0000","last_clean":"2023-10-11T16:31:42.288761+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:31:42.288761+0000","last_undegraded":"2023-10-11T16:31:42.288761+0000","last_fullsized":"2023-10-11T16:31:42.288761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:29:21.139277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462324+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:31:42.462324+0000","last_peered":"2023-10-11T16:31:42.462324+0000","last_clean":"2023-10-11T16:31:42.462324+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:31:42.462324+0000","last_undegraded":"2023-10-11T16:31:42.462324+0000","last_fullsized":"2023-10-11T16:31:42.462324+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:50:35.582480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088149+0000","last_change":"2023-10-11T16:31:42.286822+0000","last_active":"2023-10-11T16:32:10.088149+0000","last_peered":"2023-10-11T16:32:10.088149+0000","last_clean":"2023-10-11T16:32:10.088149+0000","last_became_active":"2023-10-11T16:31:42.286327+0000","last_became_peered":"2023-10-11T16:31:42.286327+0000","last_unstale":"2023-10-11T16:32:10.088149+0000","last_undegraded":"2023-10-11T16:32:10.088149+0000","last_fullsized":"2023-10-11T16:32:10.088149+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:21:36.527593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533754+0000","last_change":"2023-10-11T16:30:41.200104+0000","last_active":"2023-10-11T16:31:42.533754+0000","last_peered":"2023-10-11T16:31:42.533754+0000","last_clean":"2023-10-11T16:31:42.533754+0000","last_became_active":"2023-10-11T16:30:41.199637+0000","last_became_peered":"2023-10-11T16:30:41.199637+0000","last_unstale":"2023-10-11T16:31:42.533754+0000","last_undegraded":"2023-10-11T16:31:42.533754+0000","last_fullsized":"2023-10-11T16:31:42.533754+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:07:59.931450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534011+0000","last_change":"2023-10-11T16:30:41.202718+0000","last_active":"2023-10-11T16:31:42.534011+0000","last_peered":"2023-10-11T16:31:42.534011+0000","last_clean":"2023-10-11T16:31:42.534011+0000","last_became_active":"2023-10-11T16:30:41.190741+0000","last_became_peered":"2023-10-11T16:30:41.190741+0000","last_unstale":"2023-10-11T16:31:42.534011+0000","last_undegraded":"2023-10-11T16:31:42.534011+0000","last_fullsized":"2023-10-11T16:31:42.534011+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:13:39.958791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.094846+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:32:10.094846+0000","last_peered":"2023-10-11T16:32:10.094846+0000","last_clean":"2023-10-11T16:32:10.094846+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:32:10.094846+0000","last_undegraded":"2023-10-11T16:32:10.094846+0000","last_fullsized":"2023-10-11T16:32:10.094846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:44:05.420593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161952+0000","last_change":"2023-10-11T16:30:41.190477+0000","last_active":"2023-10-11T16:31:42.161952+0000","last_peered":"2023-10-11T16:31:42.161952+0000","last_clean":"2023-10-11T16:31:42.161952+0000","last_became_active":"2023-10-11T16:30:41.189929+0000","last_became_peered":"2023-10-11T16:30:41.189929+0000","last_unstale":"2023-10-11T16:31:42.161952+0000","last_undegraded":"2023-10-11T16:31:42.161952+0000","last_fullsized":"2023-10-11T16:31:42.161952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:14:01.200276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289168+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:31:42.289168+0000","last_peered":"2023-10-11T16:31:42.289168+0000","last_clean":"2023-10-11T16:31:42.289168+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:31:42.289168+0000","last_undegraded":"2023-10-11T16:31:42.289168+0000","last_fullsized":"2023-10-11T16:31:42.289168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:13.844885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093645+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:32:10.093645+0000","last_peered":"2023-10-11T16:32:10.093645+0000","last_clean":"2023-10-11T16:32:10.093645+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:32:10.093645+0000","last_undegraded":"2023-10-11T16:32:10.093645+0000","last_fullsized":"2023-10-11T16:32:10.093645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:07:18.219144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288217+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:31:42.288217+0000","last_peered":"2023-10-11T16:31:42.288217+0000","last_clean":"2023-10-11T16:31:42.288217+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:31:42.288217+0000","last_undegraded":"2023-10-11T16:31:42.288217+0000","last_fullsized":"2023-10-11T16:31:42.288217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:07:28.047642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285779+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:31:42.285779+0000","last_peered":"2023-10-11T16:31:42.285779+0000","last_clean":"2023-10-11T16:31:42.285779+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:31:42.285779+0000","last_undegraded":"2023-10-11T16:31:42.285779+0000","last_fullsized":"2023-10-11T16:31:42.285779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:43:48.783409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088126+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:32:10.088126+0000","last_peered":"2023-10-11T16:32:10.088126+0000","last_clean":"2023-10-11T16:32:10.088126+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:32:10.088126+0000","last_undegraded":"2023-10-11T16:32:10.088126+0000","last_fullsized":"2023-10-11T16:32:10.088126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:29:09.209568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.085185+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:32:10.085185+0000","last_peered":"2023-10-11T16:32:10.085185+0000","last_clean":"2023-10-11T16:32:10.085185+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:32:10.085185+0000","last_undegraded":"2023-10-11T16:32:10.085185+0000","last_fullsized":"2023-10-11T16:32:10.085185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:00:29.421748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089406+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:32:10.089406+0000","last_peered":"2023-10-11T16:32:10.089406+0000","last_clean":"2023-10-11T16:32:10.089406+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:32:10.089406+0000","last_undegraded":"2023-10-11T16:32:10.089406+0000","last_fullsized":"2023-10-11T16:32:10.089406+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:24.461888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.104714+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:32:10.104714+0000","last_peered":"2023-10-11T16:32:10.104714+0000","last_clean":"2023-10-11T16:32:10.104714+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:32:10.104714+0000","last_undegraded":"2023-10-11T16:32:10.104714+0000","last_fullsized":"2023-10-11T16:32:10.104714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:25:45.955319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100277+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:32:10.100277+0000","last_peered":"2023-10-11T16:32:10.100277+0000","last_clean":"2023-10-11T16:32:10.100277+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:32:10.100277+0000","last_undegraded":"2023-10-11T16:32:10.100277+0000","last_fullsized":"2023-10-11T16:32:10.100277+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:50:42.892510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.087129+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:32:10.087129+0000","last_peered":"2023-10-11T16:32:10.087129+0000","last_clean":"2023-10-11T16:32:10.087129+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:32:10.087129+0000","last_undegraded":"2023-10-11T16:32:10.087129+0000","last_fullsized":"2023-10-11T16:32:10.087129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:21:58.187591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.081932+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:32:10.081932+0000","last_peered":"2023-10-11T16:32:10.081932+0000","last_clean":"2023-10-11T16:32:10.081932+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:32:10.081932+0000","last_undegraded":"2023-10-11T16:32:10.081932+0000","last_fullsized":"2023-10-11T16:32:10.081932+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:38:42.257752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.082937+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:32:10.082937+0000","last_peered":"2023-10-11T16:32:10.082937+0000","last_clean":"2023-10-11T16:32:10.082937+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:32:10.082937+0000","last_undegraded":"2023-10-11T16:32:10.082937+0000","last_fullsized":"2023-10-11T16:32:10.082937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:53:03.240456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.098142+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:32:10.098142+0000","last_peered":"2023-10-11T16:32:10.098142+0000","last_clean":"2023-10-11T16:32:10.098142+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:32:10.098142+0000","last_undegraded":"2023-10-11T16:32:10.098142+0000","last_fullsized":"2023-10-11T16:32:10.098142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:14:19.917814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083854+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:32:10.083854+0000","last_peered":"2023-10-11T16:32:10.083854+0000","last_clean":"2023-10-11T16:32:10.083854+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:32:10.083854+0000","last_undegraded":"2023-10-11T16:32:10.083854+0000","last_fullsized":"2023-10-11T16:32:10.083854+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:39.214839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097729+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:32:10.097729+0000","last_peered":"2023-10-11T16:32:10.097729+0000","last_clean":"2023-10-11T16:32:10.097729+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:32:10.097729+0000","last_undegraded":"2023-10-11T16:32:10.097729+0000","last_fullsized":"2023-10-11T16:32:10.097729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:58.913211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.105194+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:32:10.105194+0000","last_peered":"2023-10-11T16:32:10.105194+0000","last_clean":"2023-10-11T16:32:10.105194+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:32:10.105194+0000","last_undegraded":"2023-10-11T16:32:10.105194+0000","last_fullsized":"2023-10-11T16:32:10.105194+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:54:53.313375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.095040+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:32:10.095040+0000","last_peered":"2023-10-11T16:32:10.095040+0000","last_clean":"2023-10-11T16:32:10.095040+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:32:10.095040+0000","last_undegraded":"2023-10-11T16:32:10.095040+0000","last_fullsized":"2023-10-11T16:32:10.095040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:18:45.218633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.080249+0000","last_change":"2023-10-11T16:31:42.463155+0000","last_active":"2023-10-11T16:32:10.080249+0000","last_peered":"2023-10-11T16:32:10.080249+0000","last_clean":"2023-10-11T16:32:10.080249+0000","last_became_active":"2023-10-11T16:31:42.462678+0000","last_became_peered":"2023-10-11T16:31:42.462678+0000","last_unstale":"2023-10-11T16:32:10.080249+0000","last_undegraded":"2023-10-11T16:32:10.080249+0000","last_fullsized":"2023-10-11T16:32:10.080249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:50:11.499350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80318,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":650,"num_read_kb":411,"num_write":174,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":39701,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":240954,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449126,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":1003520,"data_stored":508258,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.443},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":1.204,"15min":1.204},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73699999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57099999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.39100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.46999999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79200000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70399999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60499999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.39400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48499999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.50600000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76500000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.70799999999999996},"min":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.006,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":3.75}]},{"osd":7,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83199999999999996,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.621},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73599999999999999,"15min":0.74299999999999999},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.86099999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.64000000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:32:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68799999999999994,"15min":0.68899999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.871,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.753},"min":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96799999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.69499999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78700000000000003,"15min":0.76900000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.96,"5min":2.96,"15min":2.96},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.79700000000000004,"15min":0.80100000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.86499999999999999,"5min":1.286,"15min":1.286},"last":0.71999999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.86599999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76500000000000001,"15min":0.77100000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92100000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.65600000000000003}]}]},{"osd":10,"up_from":71,"seq":304942678059,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6696,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738456,"statfs":{"total":95995035648,"available":95988178944,"internally_reserved":0,"allocated":1146880,"data_stored":1161223,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.496},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.33200000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.59799999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.49299999999999999,"15min":0.51100000000000001},"min":{"1min":0.318,"5min":0.314,"15min":0.314},"max":{"1min":0.56000000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60999999999999999,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64900000000000002,"15min":0.66300000000000003},"min":{"1min":0.41299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71499999999999997,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70099999999999996,"5min":0.995,"15min":0.995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64300000000000002,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.745,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.62}]},{"osd":4,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.38300000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69299999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.438},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.36899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76000000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.47899999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.443,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.71099999999999997,"15min":0.72099999999999997},"min":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.76200000000000001,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72099999999999997,"15min":0.74099999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.84499999999999997,"5min":1.085,"15min":1.085},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82499999999999996,"5min":1.03,"15min":1.03},"last":0.53900000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.45700000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90200000000000002,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.83999999999999997,"15min":0.877},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.84799999999999998,"5min":2.46,"15min":2.46},"last":0.751}]},{"osd":9,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.79200000000000004,"15min":0.81699999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.85099999999999998,"5min":1.286,"15min":1.286},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.79900000000000004,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92600000000000005,"5min":1.347,"15min":1.347},"last":0.66000000000000003}]},{"osd":11,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.876,"5min":1.073,"15min":1.073},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90100000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.83399999999999996}]}]},{"osd":9,"up_from":66,"seq":283467841582,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":1015808,"data_stored":511986,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.496}]},{"osd":1,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80000000000000004,"15min":0.76700000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.198,"5min":7.056,"15min":7.056},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.80300000000000005,"15min":0.78200000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.2250000000000001,"5min":7.1529999999999996,"15min":7.1529999999999996},"last":0.44800000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69499999999999995,"15min":0.72799999999999998},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.379,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.758,"5min":2.758,"15min":2.758},"last":0.63100000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.286,"5min":2.286,"15min":2.286},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77800000000000002,"15min":0.79600000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.58299999999999996}]},{"osd":5,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.81699999999999995,"15min":0.84499999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.83099999999999996,"15min":0.84999999999999998},"min":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.6480000000000001,"5min":2.6480000000000001,"15min":2.6480000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86699999999999999,"15min":0.89500000000000002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88200000000000001,"15min":0.89100000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6789999999999998,"5min":2.6789999999999998,"15min":2.6789999999999998},"last":0.51000000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.96599999999999997,"15min":0.95299999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":3.387,"5min":7.1070000000000002,"15min":7.1070000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.96699999999999997,"15min":0.96499999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.4860000000000002,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.77400000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.84899999999999998,"15min":0.88300000000000001},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1699999999999999,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.92900000000000005,"15min":0.95899999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.3300000000000001,"5min":2.706,"15min":2.706},"last":0.75600000000000001}]},{"osd":11,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.88300000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.995,"5min":2.46,"15min":2.46},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.95099999999999996,"15min":0.98499999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.79600000000000004}]}]},{"osd":8,"up_from":58,"seq":249108103220,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6556,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738596,"statfs":{"total":95995035648,"available":95988322304,"internally_reserved":0,"allocated":1003520,"data_stored":507457,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91600000000000004,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":0.996,"15min":0.996},"last":0.45200000000000001}]},{"osd":1,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82599999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.63}]},{"osd":2,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.44600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.77400000000000002,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0649999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.57699999999999996}]},{"osd":4,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82599999999999996,"5min":1.028,"15min":1.028},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90900000000000003,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82499999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55900000000000005,"15min":0.54500000000000004},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.179,"15min":1.179},"last":0.874},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.499,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.751}]},{"osd":7,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.68899999999999995,"15min":0.68100000000000005},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.95299999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.69099999999999995,"15min":0.68300000000000005},"min":{"1min":0.56599999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.99399999999999999,"5min":1.101,"15min":1.101},"last":0.73499999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.73299999999999998,"15min":0.69899999999999995},"min":{"1min":0.69799999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.74299999999999999,"15min":0.72499999999999998},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.83899999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.74399999999999999,"15min":0.71199999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0449999999999999,"5min":1.069,"15min":1.069},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.72199999999999998,"15min":0.69599999999999995},"min":{"1min":0.57699999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0549999999999999,"5min":1.083,"15min":1.083},"last":0.80300000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:32:14 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.012,"5min":1.048,"15min":1.048},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.78800000000000003,"15min":0.77800000000000002},"min":{"1min":0.59599999999999997,"5min":0.436,"15min":0.436},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.81699999999999995}]}]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6744,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738408,"statfs":{"total":95995035648,"available":95988129792,"internally_reserved":0,"allocated":1064960,"data_stored":1113105,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.66100000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.45400000000000001},"min":{"1min":0.29199999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63900000000000001,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.374}]},{"osd":2,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77700000000000002,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.57599999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.628}]},{"osd":3,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58099999999999996,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.621,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80900000000000005,"5min":1.103,"15min":1.103},"last":0.51500000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.71099999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.501,"15min":0.5},"min":{"1min":0.29399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45000000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.45700000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91800000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66100000000000003,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86399999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68700000000000006}]},{"osd":6,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.53900000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.083,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.67900000000000005,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.71899999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71199999999999997,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72599999999999998,"15min":0.72799999999999998},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77800000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.77700000000000002,"15min":0.77600000000000002},"min":{"1min":0.63500000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.079,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.77000000000000002,"15min":0.77300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.60499999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70699999999999996,"15min":0.69499999999999995},"min":{"1min":0.48499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.054,"5min":1.071,"15min":1.071},"last":0.751},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.47699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.99199999999999999,"5min":1.099,"15min":1.099},"last":0.79500000000000004}]},{"osd":11,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.82899999999999996}]}]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6544,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738608,"statfs":{"total":95995035648,"available":95988334592,"internally_reserved":0,"allocated":991232,"data_stored":503066,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.47899999999999998,"15min":0.44500000000000001},"min":{"1min":0.32700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.59399999999999997,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.73099999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.60399999999999998,"15min":0.56999999999999995},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.63500000000000001,"15min":0.59499999999999997},"min":{"1min":0.44,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.71799999999999997,"15min":0.67700000000000005},"min":{"1min":0.48699999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.48699999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.47799999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.39100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.374,"5min":0.317,"15min":0.317},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55900000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.73699999999999999,"15min":0.70699999999999996},"min":{"1min":0.52000000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.69299999999999995,"15min":0.66000000000000003},"min":{"1min":0.52800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.64200000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70899999999999996,"15min":0.66900000000000004},"min":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.79800000000000004,"15min":0.75800000000000001},"min":{"1min":0.65000000000000002,"5min":0.434,"15min":0.434},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.79000000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:31:59 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.79700000000000004,"15min":0.76500000000000001},"min":{"1min":0.53700000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.73999999999999999,"15min":0.72399999999999998},"min":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.67600000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.752},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.875,"15min":0.874},"min":{"1min":0.60099999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.77500000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:31:46 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.872},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.89500000000000002,"15min":0.90700000000000003},"min":{"1min":0.57899999999999996,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.3719999999999999,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":0.80600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.66200000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.871,"15min":0.875},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.71499999999999997}]},{"osd":11,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.5640000000000001,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":0.73899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":569576,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55100000000000005,"15min":0.61299999999999999},"min":{"1min":0.39200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.8520000000000001},"last":0.433},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.52400000000000002,"15min":0.53800000000000003},"min":{"1min":0.41499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83099999999999996,"5min":0.872,"15min":0.89800000000000002},"last":0.40999999999999998}]},{"osd":2,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.57299999999999995,"15min":0.52300000000000002},"min":{"1min":0.51500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.56599999999999995,"15min":0.52600000000000002},"min":{"1min":0.495,"5min":0.32900000000000001,"15min":0.307},"max":{"1min":0.89300000000000002,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61299999999999999,"15min":0.57999999999999996},"min":{"1min":0.48899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.88800000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.60799999999999998,"15min":0.57199999999999995},"min":{"1min":0.53000000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53700000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.47399999999999998,"15min":0.42699999999999999},"min":{"1min":0.34300000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.752,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.54300000000000004,"15min":0.46600000000000003},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.91000000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.51100000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.59599999999999997,"15min":0.51200000000000001},"min":{"1min":0.505,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.64900000000000002,"15min":0.55400000000000005},"min":{"1min":0.58199999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85999999999999999,"5min":0.997,"15min":0.997},"last":0.80800000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.67900000000000005,"15min":0.627},"min":{"1min":0.60199999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":2.3580000000000001},"last":0.747},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.65900000000000003,"15min":0.63500000000000001},"min":{"1min":0.59499999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.58299999999999996}]},{"osd":7,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.57999999999999996,"15min":0.51600000000000001},"min":{"1min":0.47699999999999998,"5min":0.253,"15min":0.253},"max":{"1min":1.004,"5min":1.127,"15min":1.127},"last":0.621},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.66900000000000004,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90700000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.69799999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.73199999999999998,"15min":0.67800000000000005},"min":{"1min":0.65500000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.70499999999999996,"15min":0.64500000000000002},"min":{"1min":0.67700000000000005,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.68100000000000005}]},{"osd":9,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.746,"15min":0.72199999999999998},"min":{"1min":0.70699999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97099999999999997,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.79200000000000004,"15min":0.77000000000000002},"min":{"1min":0.67000000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.98699999999999999,"5min":1.179,"15min":1.179},"last":0.82799999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.71299999999999997,"15min":0.67800000000000005},"min":{"1min":0.48999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96099999999999997,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.76000000000000001,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93799999999999994,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.76600000000000001}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.84199999999999997,"15min":0.83699999999999997},"min":{"1min":0.629,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.024,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.86399999999999999,"15min":0.86199999999999999},"min":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.97599999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.84299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7464,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737688,"statfs":{"total":95995035648,"available":95987392512,"internally_reserved":0,"allocated":1015808,"data_stored":504454,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.41599999999999998,"5min":0.33100000000000002,"15min":0.309},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.60999999999999999,"15min":0.54400000000000004},"min":{"1min":0.54500000000000004,"5min":0.38500000000000001,"15min":0.33800000000000002},"max":{"1min":0.82499999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Oct 11 16:32:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.65300000000000002,"15min":0.59499999999999997},"min":{"1min":0.46200000000000002,"5min":0.38500000000000001,"15min":0.35899999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66700000000000004,"15min":0.61299999999999999},"min":{"1min":0.54300000000000004,"5min":0.42599999999999999,"15min":0.40500000000000003},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.69399999999999995}]},{"osd":3,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.42299999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61599999999999999,"15min":0.56200000000000006},"min":{"1min":0.41499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64800000000000002,"15min":0.59699999999999998},"min":{"1min":0.47399999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81899999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76400000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:32:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64300000000000002,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.377,"15min":0.377},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.68200000000000005,"15min":0.64600000000000002},"min":{"1min":0.5,"5min":0.436,"15min":0.436},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.80000000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.626},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.63,"15min":0.60399999999999998},"min":{"1min":0.374,"5min":0.37,"15min":0.37},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Wed Oct 11 16:32:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.76800000000000002,"15min":0.77300000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.56299999999999994,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0840000000000001,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.77300000000000002,"15min":0.753},"min":{"1min":0.63500000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":3.073,"5min":3.073,"15min":3.073},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75800000000000001,"15min":0.76200000000000001},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.73199999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.80600000000000005,"15min":0.83899999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.80400000000000005,"15min":0.82899999999999996},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.52100000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81399999999999995,"15min":0.81100000000000005},"min":{"1min":0.69499999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.82299999999999995,"15min":0.82599999999999996},"min":{"1min":0.624,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63500000000000001}]},{"osd":11,"last update":"Wed Oct 11 16:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.64100000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.70399999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.78700000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7592,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737560,"statfs":{"total":95995035648,"available":95987261440,"internally_reserved":0,"allocated":1015808,"data_stored":514151,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55100000000000005,"15min":0.52700000000000002},"min":{"1min":0.39500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.96099999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.42899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":0.91800000000000004,"15min":1.0549999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59399999999999997,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.29199999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.91200000000000003},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.59299999999999997,"15min":0.52700000000000002},"min":{"1min":0.40799999999999997,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57599999999999996,"15min":0.54500000000000004},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.63,"15min":0.57499999999999996},"min":{"1min":0.51100000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.68500000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85699999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69899999999999995,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.89400000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.70399999999999996}]},{"osd":5,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51900000000000002,"15min":0.47399999999999998},"min":{"1min":0.39000000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.88800000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52000000000000002,"15min":0.48799999999999999},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85099999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.54000000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.67300000000000004,"15min":0.63300000000000001},"min":{"1min":0.51900000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.66900000000000004,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82999999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.64300000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.498,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93899999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52400000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94799999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.77400000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.64400000000000002,"15min":0.61499999999999999},"min":{"1min":0.48299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.64800000000000002,"15min":0.623},"min":{"1min":0.49199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.747}]},{"osd":9,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.74099999999999999,"15min":0.71599999999999997},"min":{"1min":0.66300000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.746,"15min":0.72399999999999998},"min":{"1min":0.61699999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.93100000000000005,"5min":1.024,"15min":1.024},"last":0.80000000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.76300000000000001,"15min":0.74199999999999999},"min":{"1min":0.626,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94499999999999995,"5min":0.997,"15min":0.997},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.80300000000000005,"15min":0.77800000000000002},"min":{"1min":0.626,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0029999999999999,"5min":1.038,"15min":1.038},"last":0.879}]},{"osd":11,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82999999999999996,"15min":0.83199999999999996},"min":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.85499999999999998},"min":{"1min":0.66400000000000003,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.90800000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6808,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738344,"statfs":{"total":95995035648,"available":95988064256,"internally_reserved":0,"allocated":999424,"data_stored":502929,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41399999999999998,"15min":0.38400000000000001},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.43099999999999999,"15min":0.40899999999999997},"min":{"1min":0.33100000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.56999999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.62,"15min":0.57499999999999996},"min":{"1min":0.47799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79600000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58999999999999997,"15min":0.55600000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77000000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.68999999999999995}]},{"osd":2,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64700000000000002,"15min":0.60899999999999999},"min":{"1min":0.439,"5min":0.379,"15min":0.379},"max":{"1min":0.91000000000000003,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65000000000000002,"15min":0.59399999999999997},"min":{"1min":0.48999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80100000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Wed Oct 11 16:31:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.64000000000000001},"min":{"1min":0.54200000000000004,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82099999999999995,"5min":1.101,"15min":1.101},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71499999999999997,"15min":0.68700000000000006},"min":{"1min":0.624,"5min":0.498,"15min":0.498},"max":{"1min":0.83799999999999997,"5min":1.194,"15min":1.194},"last":0.54300000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.621,"15min":0.63100000000000001},"min":{"1min":0.46300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88600000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92900000000000005,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.78800000000000003,"5min":1.268,"15min":1.268},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.749,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.626}]},{"osd":7,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.57299999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.85699999999999998,"5min":1.119,"15min":1.119},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.623,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95799999999999996,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.745}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:31:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.77600000000000002,"15min":0.80200000000000005},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.90800000000000003,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.77600000000000002,"15min":0.79900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.876,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69799999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.81200000000000006,"15min":0.82499999999999996},"min":{"1min":0.66700000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.028,"5min":1.137,"15min":1.137},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.82199999999999995,"15min":0.83199999999999996},"min":{"1min":0.68400000000000005,"5min":0.627,"15min":0.627},"max":{"1min":0.876,"5min":1.165,"15min":1.165},"last":0.64400000000000002}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78700000000000003,"15min":0.78800000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.90900000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72099999999999997,"15min":0.71799999999999997},"min":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.71199999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6796,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738356,"statfs":{"total":95995035648,"available":95988076544,"internally_reserved":0,"allocated":1052672,"data_stored":574017,"data_compressed":30188,"data_compressed_allocated":40960,"data_compressed_original":102400,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51500000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39000000000000001,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48999999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59799999999999998,"15min":0.56000000000000005},"min":{"1min":0.41899999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81599999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52700000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75700000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.69099999999999995,"15min":0.58399999999999996},"min":{"1min":0.433,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.234,"5min":0.69699999999999995,"15min":0.57599999999999996},"min":{"1min":0.46800000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.0129999999999999,"5min":7.0129999999999999,"15min":7.0129999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67600000000000005,"15min":0.73799999999999999},"min":{"1min":0.52100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78500000000000003,"5min":4.5,"15min":4.5},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.71599999999999997},"min":{"1min":0.51500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76700000000000002,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.57699999999999996}]},{"osd":6,"last update":"Wed Oct 11 16:31:39 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.70799999999999996,"15min":0.65500000000000003},"min":{"1min":0.52300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.9420000000000002,"5min":6.9420000000000002,"15min":6.9420000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.58199999999999996},"min":{"1min":0.52400000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79200000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56200000000000006}]},{"osd":7,"last update":"Wed Oct 11 16:32:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.373,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55400000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58099999999999996,"15min":0.54600000000000004},"min":{"1min":0.42799999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79200000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.40100000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.48999999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76800000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.622}]},{"osd":9,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.64800000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.84899999999999998,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.454,"5min":0.87,"15min":0.76800000000000002},"min":{"1min":0.63800000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":6.968,"5min":6.968,"15min":6.968},"last":0.71399999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.46200000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89200000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.52900000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.94099999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.59999999999999998}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.68300000000000005,"15min":0.68799999999999994},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73899999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1077248,"data_stored":1109323,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.53100000000000003},"min":{"1min":0.434,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.309,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57199999999999995,"15min":0.53700000000000003},"min":{"1min":0.375,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.874,"15min":0.874},"last":0.62},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57999999999999996,"15min":0.55800000000000005},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77100000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55000000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.439,"15min":0.40699999999999997},"min":{"1min":0.34300000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.59799999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.46600000000000003,"15min":0.42599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65100000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.309}]},{"osd":3,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72599999999999998,"5min":1.089,"15min":1.089},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60299999999999998,"15min":0.56599999999999995},"min":{"1min":0.47599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.79700000000000004}]},{"osd":4,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.621,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82399999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.62,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.754,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65600000000000003,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.629},"min":{"1min":0.52400000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.753}]},{"osd":7,"last update":"Wed Oct 11 16:32:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88800000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70499999999999996,"15min":0.69999999999999996},"min":{"1min":0.45200000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57699999999999996}]},{"osd":8,"last update":"Wed Oct 11 16:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68999999999999995},"min":{"1min":0.502,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83699999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.42199999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:31:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.751,"15min":0.751},"min":{"1min":0.59699999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90900000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77300000000000002,"15min":0.76800000000000002},"min":{"1min":0.57699999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.83499999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72899999999999998,"15min":0.71999999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89700000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.745,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":11175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":66945,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":11883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":11569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":67879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:32:24.517 INFO:tasks.fwd_scrub.ceph_manager:clean! 2023-10-11T16:32:24.518 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T16:32:24.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:24 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/856548750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 -- 172.21.15.111:0/3994028415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24100290 msgr2=0x7fdd241006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 --2- 172.21.15.111:0/3994028415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24100290 0x7fdd241006a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdd140098f0 tx=0x7fdd14009c00 comp rx=0 tx=0).stop 2023-10-11T16:32:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 -- 172.21.15.111:0/3994028415 shutdown_connections 2023-10-11T16:32:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 --2- 172.21.15.111:0/3994028415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd24101e10 0x7fdd241086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 --2- 172.21.15.111:0/3994028415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24100290 0x7fdd241006a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 --2- 172.21.15.111:0/3994028415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd24101480 0x7fdd241018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 -- 172.21.15.111:0/3994028415 >> 172.21.15.111:0/3994028415 conn(0x7fdd240fb870 msgr2=0x7fdd240fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:24.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 -- 172.21.15.111:0/3994028415 shutdown_connections 2023-10-11T16:32:24.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.744+0000 7fdd29fff700 1 -- 172.21.15.111:0/3994028415 wait complete. 2023-10-11T16:32:24.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.745+0000 7fdd29fff700 1 Processor -- start 2023-10-11T16:32:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.745+0000 7fdd29fff700 1 -- start start 2023-10-11T16:32:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd29fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd24100290 0x7fdd240748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd29fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd24101480 0x7fdd24072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:24.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd29fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24101e10 0x7fdd24073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd29fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24073bc0 con 0x7fdd24101e10 2023-10-11T16:32:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd29fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24073d30 con 0x7fdd24101480 2023-10-11T16:32:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd29fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd241a0080 con 0x7fdd24100290 2023-10-11T16:32:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd237fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd24100290 0x7fdd240748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:24.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd23fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24101e10 0x7fdd24073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd1bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd24101480 0x7fdd24072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd237fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd24100290 0x7fdd240748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:44708/0 (socket says 172.21.15.111:44708) 2023-10-11T16:32:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd237fe700 1 -- 172.21.15.111:0/2928579365 learned_addr learned my addr 172.21.15.111:0/2928579365 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd23fff700 1 -- 172.21.15.111:0/2928579365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd24100290 msgr2=0x7fdd240748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd23fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd24100290 0x7fdd240748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd23fff700 1 -- 172.21.15.111:0/2928579365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd24101480 msgr2=0x7fdd24072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd23fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd24101480 0x7fdd24072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd23fff700 1 -- 172.21.15.111:0/2928579365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c0095f0 con 0x7fdd24101e10 2023-10-11T16:32:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.746+0000 7fdd237fe700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd24100290 0x7fdd240748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.747+0000 7fdd23fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24101e10 0x7fdd24073420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00efc0 tx=0x7fdd1c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.747+0000 7fdd217fa700 1 -- 172.21.15.111:0/2928579365 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c018070 con 0x7fdd24101e10 2023-10-11T16:32:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.747+0000 7fdd217fa700 1 -- 172.21.15.111:0/2928579365 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd1c00f040 con 0x7fdd24101e10 2023-10-11T16:32:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.747+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd14009520 con 0x7fdd24101e10 2023-10-11T16:32:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.747+0000 7fdd217fa700 1 -- 172.21.15.111:0/2928579365 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00cd40 con 0x7fdd24101e10 2023-10-11T16:32:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.747+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2410bc40 con 0x7fdd24101e10 2023-10-11T16:32:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.748+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd24067a60 con 0x7fdd24101e10 2023-10-11T16:32:24.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.750+0000 7fdd217fa700 1 -- 172.21.15.111:0/2928579365 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fdd1c003a00 con 0x7fdd24101e10 2023-10-11T16:32:24.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.752+0000 7fdd217fa700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd100a11d0 0x7fdd100a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:24.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.752+0000 7fdd217fa700 1 -- 172.21.15.111:0/2928579365 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fdd1c0cc7b0 con 0x7fdd24101e10 2023-10-11T16:32:24.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.752+0000 7fdd237fe700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd100a11d0 0x7fdd100a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:24.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.753+0000 7fdd237fe700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd100a11d0 0x7fdd100a3680 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400c010 tx=0x7fdd14019040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:24.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.756+0000 7fdd217fa700 1 -- 172.21.15.111:0/2928579365 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdd1c095bc0 con 0x7fdd24101e10 2023-10-11T16:32:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.950+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd24060f30 con 0x7fdd100a11d0 2023-10-11T16:32:24.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.962+0000 7fdd217fa700 1 -- 172.21.15.111:0/2928579365 <== mgr.14148 v2:172.21.15.111:6800/2838247405 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427546 (secure 0 0 0) 0x7fdd24060f30 con 0x7fdd100a11d0 2023-10-11T16:32:24.969 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:24.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd100a11d0 msgr2=0x7fdd100a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd100a11d0 0x7fdd100a3680 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400c010 tx=0x7fdd14019040 comp rx=0 tx=0).stop 2023-10-11T16:32:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24101e10 msgr2=0x7fdd24073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24101e10 0x7fdd24073420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00efc0 tx=0x7fdd1c00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 shutdown_connections 2023-10-11T16:32:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fdd100a11d0 0x7fdd100a3680 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd24101e10 0x7fdd24073420 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd24100290 0x7fdd240748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 --2- 172.21.15.111:0/2928579365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd24101480 0x7fdd24072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 >> 172.21.15.111:0/2928579365 conn(0x7fdd240fb870 msgr2=0x7fdd240fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 shutdown_connections 2023-10-11T16:32:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:24.967+0000 7fdd29fff700 1 -- 172.21.15.111:0/2928579365 wait complete. 2023-10-11T16:32:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2023-10-11T16:32:24.988 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2023-10-11T16:32:24.087409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670932,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":786,"num_read_kb":530,"num_write":357,"num_write_kb":3077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1220437,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82464,"kb_used_data":12160,"kb_used_omap":0,"kb_used_meta":70208,"kb_avail":1124859360,"statfs":{"total":1151940427776,"available":1151855984640,"internally_reserved":0,"allocated":12451840,"data_stored":8079545,"data_compressed":395806,"data_compressed_allocated":536576,"data_compressed_original":2592768,"omap_allocated":0,"internal_metadata":71892992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":565,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":47,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100200+0000","last_change":"2023-10-11T16:30:11.670524+0000","last_active":"2023-10-11T16:32:10.100200+0000","last_peered":"2023-10-11T16:32:10.100200+0000","last_clean":"2023-10-11T16:32:10.100200+0000","last_became_active":"2023-10-11T16:30:11.669043+0000","last_became_peered":"2023-10-11T16:30:11.669043+0000","last_unstale":"2023-10-11T16:32:10.100200+0000","last_undegraded":"2023-10-11T16:32:10.100200+0000","last_fullsized":"2023-10-11T16:32:10.100200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:26:00.751881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100937+0000","last_change":"2023-10-11T16:30:11.673705+0000","last_active":"2023-10-11T16:32:10.100937+0000","last_peered":"2023-10-11T16:32:10.100937+0000","last_clean":"2023-10-11T16:32:10.100937+0000","last_became_active":"2023-10-11T16:30:11.673369+0000","last_became_peered":"2023-10-11T16:30:11.673369+0000","last_unstale":"2023-10-11T16:32:10.100937+0000","last_undegraded":"2023-10-11T16:32:10.100937+0000","last_fullsized":"2023-10-11T16:32:10.100937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:51:20.593620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101715+0000","last_change":"2023-10-11T16:30:11.655394+0000","last_active":"2023-10-11T16:32:10.101715+0000","last_peered":"2023-10-11T16:32:10.101715+0000","last_clean":"2023-10-11T16:32:10.101715+0000","last_became_active":"2023-10-11T16:30:11.655043+0000","last_became_peered":"2023-10-11T16:30:11.655043+0000","last_unstale":"2023-10-11T16:32:10.101715+0000","last_undegraded":"2023-10-11T16:32:10.101715+0000","last_fullsized":"2023-10-11T16:32:10.101715+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:20:45.562554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083360+0000","last_change":"2023-10-11T16:30:11.652370+0000","last_active":"2023-10-11T16:32:10.083360+0000","last_peered":"2023-10-11T16:32:10.083360+0000","last_clean":"2023-10-11T16:32:10.083360+0000","last_became_active":"2023-10-11T16:30:11.652006+0000","last_became_peered":"2023-10-11T16:30:11.652006+0000","last_unstale":"2023-10-11T16:32:10.083360+0000","last_undegraded":"2023-10-11T16:32:10.083360+0000","last_fullsized":"2023-10-11T16:32:10.083360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:03:22.258325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.071382+0000","last_change":"2023-10-11T16:30:11.670320+0000","last_active":"2023-10-11T16:32:10.071382+0000","last_peered":"2023-10-11T16:32:10.071382+0000","last_clean":"2023-10-11T16:32:10.071382+0000","last_became_active":"2023-10-11T16:30:11.669720+0000","last_became_peered":"2023-10-11T16:30:11.669720+0000","last_unstale":"2023-10-11T16:32:10.071382+0000","last_undegraded":"2023-10-11T16:32:10.071382+0000","last_fullsized":"2023-10-11T16:32:10.071382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:59:15.949889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.086763+0000","last_change":"2023-10-11T16:30:11.661551+0000","last_active":"2023-10-11T16:32:10.086763+0000","last_peered":"2023-10-11T16:32:10.086763+0000","last_clean":"2023-10-11T16:32:10.086763+0000","last_became_active":"2023-10-11T16:30:11.660905+0000","last_became_peered":"2023-10-11T16:30:11.660905+0000","last_unstale":"2023-10-11T16:32:10.086763+0000","last_undegraded":"2023-10-11T16:32:10.086763+0000","last_fullsized":"2023-10-11T16:32:10.086763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:17:51.767698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.087839+0000","last_change":"2023-10-11T16:30:11.661107+0000","last_active":"2023-10-11T16:32:10.087839+0000","last_peered":"2023-10-11T16:32:10.087839+0000","last_clean":"2023-10-11T16:32:10.087839+0000","last_became_active":"2023-10-11T16:30:11.660792+0000","last_became_peered":"2023-10-11T16:30:11.660792+0000","last_unstale":"2023-10-11T16:32:10.087839+0000","last_undegraded":"2023-10-11T16:32:10.087839+0000","last_fullsized":"2023-10-11T16:32:10.087839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:31:16.375555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.099303+0000","last_change":"2023-10-11T16:30:11.663219+0000","last_active":"2023-10-11T16:32:10.099303+0000","last_peered":"2023-10-11T16:32:10.099303+0000","last_clean":"2023-10-11T16:32:10.099303+0000","last_became_active":"2023-10-11T16:30:11.660177+0000","last_became_peered":"2023-10-11T16:30:11.660177+0000","last_unstale":"2023-10-11T16:32:10.099303+0000","last_undegraded":"2023-10-11T16:32:10.099303+0000","last_fullsized":"2023-10-11T16:32:10.099303+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:14:00.865418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092992+0000","last_change":"2023-10-11T16:30:11.661104+0000","last_active":"2023-10-11T16:32:10.092992+0000","last_peered":"2023-10-11T16:32:10.092992+0000","last_clean":"2023-10-11T16:32:10.092992+0000","last_became_active":"2023-10-11T16:30:11.660797+0000","last_became_peered":"2023-10-11T16:30:11.660797+0000","last_unstale":"2023-10-11T16:32:10.092992+0000","last_undegraded":"2023-10-11T16:32:10.092992+0000","last_fullsized":"2023-10-11T16:32:10.092992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:48:35.792817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.091540+0000","last_change":"2023-10-11T16:30:11.657273+0000","last_active":"2023-10-11T16:32:10.091540+0000","last_peered":"2023-10-11T16:32:10.091540+0000","last_clean":"2023-10-11T16:32:10.091540+0000","last_became_active":"2023-10-11T16:30:11.656982+0000","last_became_peered":"2023-10-11T16:30:11.656982+0000","last_unstale":"2023-10-11T16:32:10.091540+0000","last_undegraded":"2023-10-11T16:32:10.091540+0000","last_fullsized":"2023-10-11T16:32:10.091540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:48:35.706510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.094525+0000","last_change":"2023-10-11T16:30:11.656200+0000","last_active":"2023-10-11T16:32:10.094525+0000","last_peered":"2023-10-11T16:32:10.094525+0000","last_clean":"2023-10-11T16:32:10.094525+0000","last_became_active":"2023-10-11T16:30:11.655518+0000","last_became_peered":"2023-10-11T16:30:11.655518+0000","last_unstale":"2023-10-11T16:32:10.094525+0000","last_undegraded":"2023-10-11T16:32:10.094525+0000","last_fullsized":"2023-10-11T16:32:10.094525+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:39:30.482685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089635+0000","last_change":"2023-10-11T16:30:11.657083+0000","last_active":"2023-10-11T16:32:10.089635+0000","last_peered":"2023-10-11T16:32:10.089635+0000","last_clean":"2023-10-11T16:32:10.089635+0000","last_became_active":"2023-10-11T16:30:11.656755+0000","last_became_peered":"2023-10-11T16:30:11.656755+0000","last_unstale":"2023-10-11T16:32:10.089635+0000","last_undegraded":"2023-10-11T16:32:10.089635+0000","last_fullsized":"2023-10-11T16:32:10.089635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:34:42.597400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089114+0000","last_change":"2023-10-11T16:30:11.661562+0000","last_active":"2023-10-11T16:32:10.089114+0000","last_peered":"2023-10-11T16:32:10.089114+0000","last_clean":"2023-10-11T16:32:10.089114+0000","last_became_active":"2023-10-11T16:30:11.661216+0000","last_became_peered":"2023-10-11T16:30:11.661216+0000","last_unstale":"2023-10-11T16:32:10.089114+0000","last_undegraded":"2023-10-11T16:32:10.089114+0000","last_fullsized":"2023-10-11T16:32:10.089114+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:46:38.920873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100645+0000","last_change":"2023-10-11T16:30:11.657153+0000","last_active":"2023-10-11T16:32:10.100645+0000","last_peered":"2023-10-11T16:32:10.100645+0000","last_clean":"2023-10-11T16:32:10.100645+0000","last_became_active":"2023-10-11T16:30:11.656676+0000","last_became_peered":"2023-10-11T16:30:11.656676+0000","last_unstale":"2023-10-11T16:32:10.100645+0000","last_undegraded":"2023-10-11T16:32:10.100645+0000","last_fullsized":"2023-10-11T16:32:10.100645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:53:32.036585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100988+0000","last_change":"2023-10-11T16:30:11.665071+0000","last_active":"2023-10-11T16:32:10.100988+0000","last_peered":"2023-10-11T16:32:10.100988+0000","last_clean":"2023-10-11T16:32:10.100988+0000","last_became_active":"2023-10-11T16:30:11.664356+0000","last_became_peered":"2023-10-11T16:30:11.664356+0000","last_unstale":"2023-10-11T16:32:10.100988+0000","last_undegraded":"2023-10-11T16:32:10.100988+0000","last_fullsized":"2023-10-11T16:32:10.100988+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:59:32.595236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100326+0000","last_change":"2023-10-11T16:30:11.659888+0000","last_active":"2023-10-11T16:32:10.100326+0000","last_peered":"2023-10-11T16:32:10.100326+0000","last_clean":"2023-10-11T16:32:10.100326+0000","last_became_active":"2023-10-11T16:30:11.659434+0000","last_became_peered":"2023-10-11T16:30:11.659434+0000","last_unstale":"2023-10-11T16:32:10.100326+0000","last_undegraded":"2023-10-11T16:32:10.100326+0000","last_fullsized":"2023-10-11T16:32:10.100326+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:43:51.221018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100337+0000","last_change":"2023-10-11T16:30:11.670551+0000","last_active":"2023-10-11T16:32:10.100337+0000","last_peered":"2023-10-11T16:32:10.100337+0000","last_clean":"2023-10-11T16:32:10.100337+0000","last_became_active":"2023-10-11T16:30:11.669389+0000","last_became_peered":"2023-10-11T16:30:11.669389+0000","last_unstale":"2023-10-11T16:32:10.100337+0000","last_undegraded":"2023-10-11T16:32:10.100337+0000","last_fullsized":"2023-10-11T16:32:10.100337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:40:38.294389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083579+0000","last_change":"2023-10-11T16:30:11.653265+0000","last_active":"2023-10-11T16:32:10.083579+0000","last_peered":"2023-10-11T16:32:10.083579+0000","last_clean":"2023-10-11T16:32:10.083579+0000","last_became_active":"2023-10-11T16:30:11.652948+0000","last_became_peered":"2023-10-11T16:30:11.652948+0000","last_unstale":"2023-10-11T16:32:10.083579+0000","last_undegraded":"2023-10-11T16:32:10.083579+0000","last_fullsized":"2023-10-11T16:32:10.083579+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:56:56.119945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102789+0000","last_change":"2023-10-11T16:30:11.661346+0000","last_active":"2023-10-11T16:32:10.102789+0000","last_peered":"2023-10-11T16:32:10.102789+0000","last_clean":"2023-10-11T16:32:10.102789+0000","last_became_active":"2023-10-11T16:30:11.661028+0000","last_became_peered":"2023-10-11T16:30:11.661028+0000","last_unstale":"2023-10-11T16:32:10.102789+0000","last_undegraded":"2023-10-11T16:32:10.102789+0000","last_fullsized":"2023-10-11T16:32:10.102789+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:24:13.338895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159916+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:31:42.159916+0000","last_peered":"2023-10-11T16:31:42.159916+0000","last_clean":"2023-10-11T16:31:42.159916+0000","last_became_active":"2023-10-11T16:30:16.135097+0000","last_became_peered":"2023-10-11T16:30:16.135097+0000","last_unstale":"2023-10-11T16:31:42.159916+0000","last_undegraded":"2023-10-11T16:31:42.159916+0000","last_fullsized":"2023-10-11T16:31:42.159916+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:50:38.133970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159320+0000","last_change":"2023-10-11T16:30:41.198854+0000","last_active":"2023-10-11T16:31:42.159320+0000","last_peered":"2023-10-11T16:31:42.159320+0000","last_clean":"2023-10-11T16:31:42.159320+0000","last_became_active":"2023-10-11T16:30:41.189677+0000","last_became_peered":"2023-10-11T16:30:41.189677+0000","last_unstale":"2023-10-11T16:31:42.159320+0000","last_undegraded":"2023-10-11T16:31:42.159320+0000","last_fullsized":"2023-10-11T16:31:42.159320+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:02:16.154156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097708+0000","last_change":"2023-10-11T16:30:11.672838+0000","last_active":"2023-10-11T16:32:10.097708+0000","last_peered":"2023-10-11T16:32:10.097708+0000","last_clean":"2023-10-11T16:32:10.097708+0000","last_became_active":"2023-10-11T16:30:11.671161+0000","last_became_peered":"2023-10-11T16:30:11.671161+0000","last_unstale":"2023-10-11T16:32:10.097708+0000","last_undegraded":"2023-10-11T16:32:10.097708+0000","last_fullsized":"2023-10-11T16:32:10.097708+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:45:10.365459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288525+0000","last_change":"2023-10-11T16:30:16.130722+0000","last_active":"2023-10-11T16:31:42.288525+0000","last_peered":"2023-10-11T16:31:42.288525+0000","last_clean":"2023-10-11T16:31:42.288525+0000","last_became_active":"2023-10-11T16:30:16.130328+0000","last_became_peered":"2023-10-11T16:30:16.130328+0000","last_unstale":"2023-10-11T16:31:42.288525+0000","last_undegraded":"2023-10-11T16:31:42.288525+0000","last_fullsized":"2023-10-11T16:31:42.288525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:37:11.027756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534803+0000","last_change":"2023-10-11T16:30:11.131387+0000","last_active":"2023-10-11T16:31:42.534803+0000","last_peered":"2023-10-11T16:31:42.534803+0000","last_clean":"2023-10-11T16:31:42.534803+0000","last_became_active":"2023-10-11T16:30:11.130985+0000","last_became_peered":"2023-10-11T16:30:11.130985+0000","last_unstale":"2023-10-11T16:31:42.534803+0000","last_undegraded":"2023-10-11T16:31:42.534803+0000","last_fullsized":"2023-10-11T16:31:42.534803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:29:20.287428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102278+0000","last_change":"2023-10-11T16:30:11.658793+0000","last_active":"2023-10-11T16:32:10.102278+0000","last_peered":"2023-10-11T16:32:10.102278+0000","last_clean":"2023-10-11T16:32:10.102278+0000","last_became_active":"2023-10-11T16:30:11.655093+0000","last_became_peered":"2023-10-11T16:30:11.655093+0000","last_unstale":"2023-10-11T16:32:10.102278+0000","last_undegraded":"2023-10-11T16:32:10.102278+0000","last_fullsized":"2023-10-11T16:32:10.102278+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:23:33.919634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288642+0000","last_change":"2023-10-11T16:30:16.135678+0000","last_active":"2023-10-11T16:31:42.288642+0000","last_peered":"2023-10-11T16:31:42.288642+0000","last_clean":"2023-10-11T16:31:42.288642+0000","last_became_active":"2023-10-11T16:30:16.135257+0000","last_became_peered":"2023-10-11T16:30:16.135257+0000","last_unstale":"2023-10-11T16:31:42.288642+0000","last_undegraded":"2023-10-11T16:31:42.288642+0000","last_fullsized":"2023-10-11T16:31:42.288642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:42:02.132395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462596+0000","last_change":"2023-10-11T16:30:11.118163+0000","last_active":"2023-10-11T16:31:42.462596+0000","last_peered":"2023-10-11T16:31:42.462596+0000","last_clean":"2023-10-11T16:31:42.462596+0000","last_became_active":"2023-10-11T16:30:11.117898+0000","last_became_peered":"2023-10-11T16:30:11.117898+0000","last_unstale":"2023-10-11T16:31:42.462596+0000","last_undegraded":"2023-10-11T16:31:42.462596+0000","last_fullsized":"2023-10-11T16:31:42.462596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:38:07.419529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092994+0000","last_change":"2023-10-11T16:30:11.659772+0000","last_active":"2023-10-11T16:32:10.092994+0000","last_peered":"2023-10-11T16:32:10.092994+0000","last_clean":"2023-10-11T16:32:10.092994+0000","last_became_active":"2023-10-11T16:30:11.659185+0000","last_became_peered":"2023-10-11T16:30:11.659185+0000","last_unstale":"2023-10-11T16:32:10.092994+0000","last_undegraded":"2023-10-11T16:32:10.092994+0000","last_fullsized":"2023-10-11T16:32:10.092994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:44:22.080004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288644+0000","last_change":"2023-10-11T16:30:16.130852+0000","last_active":"2023-10-11T16:31:42.288644+0000","last_peered":"2023-10-11T16:31:42.288644+0000","last_clean":"2023-10-11T16:31:42.288644+0000","last_became_active":"2023-10-11T16:30:16.130353+0000","last_became_peered":"2023-10-11T16:30:16.130353+0000","last_unstale":"2023-10-11T16:31:42.288644+0000","last_undegraded":"2023-10-11T16:31:42.288644+0000","last_fullsized":"2023-10-11T16:31:42.288644+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:01:40.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462545+0000","last_change":"2023-10-11T16:30:11.117940+0000","last_active":"2023-10-11T16:31:42.462545+0000","last_peered":"2023-10-11T16:31:42.462545+0000","last_clean":"2023-10-11T16:31:42.462545+0000","last_became_active":"2023-10-11T16:30:11.117351+0000","last_became_peered":"2023-10-11T16:30:11.117351+0000","last_unstale":"2023-10-11T16:31:42.462545+0000","last_undegraded":"2023-10-11T16:31:42.462545+0000","last_fullsized":"2023-10-11T16:31:42.462545+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:32:50.524951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083771+0000","last_change":"2023-10-11T16:30:41.195736+0000","last_active":"2023-10-11T16:32:10.083771+0000","last_peered":"2023-10-11T16:32:10.083771+0000","last_clean":"2023-10-11T16:32:10.083771+0000","last_became_active":"2023-10-11T16:30:41.195352+0000","last_became_peered":"2023-10-11T16:30:41.195352+0000","last_unstale":"2023-10-11T16:32:10.083771+0000","last_undegraded":"2023-10-11T16:32:10.083771+0000","last_fullsized":"2023-10-11T16:32:10.083771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:29:40.859051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160178+0000","last_change":"2023-10-11T16:30:16.126145+0000","last_active":"2023-10-11T16:31:42.160178+0000","last_peered":"2023-10-11T16:31:42.160178+0000","last_clean":"2023-10-11T16:31:42.160178+0000","last_became_active":"2023-10-11T16:30:16.125766+0000","last_became_peered":"2023-10-11T16:30:16.125766+0000","last_unstale":"2023-10-11T16:31:42.160178+0000","last_undegraded":"2023-10-11T16:31:42.160178+0000","last_fullsized":"2023-10-11T16:31:42.160178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:55:21.244206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'10","reported_seq":148,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:19.340286+0000","last_change":"2023-10-11T16:30:11.115374+0000","last_active":"2023-10-11T16:32:19.340286+0000","last_peered":"2023-10-11T16:32:19.340286+0000","last_clean":"2023-10-11T16:32:19.340286+0000","last_became_active":"2023-10-11T16:30:11.115091+0000","last_became_peered":"2023-10-11T16:30:11.115091+0000","last_unstale":"2023-10-11T16:32:19.340286+0000","last_undegraded":"2023-10-11T16:32:19.340286+0000","last_fullsized":"2023-10-11T16:32:19.340286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:37:47.462085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11029,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":14,"num_write":7,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.078744+0000","last_change":"2023-10-11T16:30:11.654365+0000","last_active":"2023-10-11T16:32:10.078744+0000","last_peered":"2023-10-11T16:32:10.078744+0000","last_clean":"2023-10-11T16:32:10.078744+0000","last_became_active":"2023-10-11T16:30:11.654073+0000","last_became_peered":"2023-10-11T16:30:11.654073+0000","last_unstale":"2023-10-11T16:32:10.078744+0000","last_undegraded":"2023-10-11T16:32:10.078744+0000","last_fullsized":"2023-10-11T16:32:10.078744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:49:22.705791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160227+0000","last_change":"2023-10-11T16:30:16.136016+0000","last_active":"2023-10-11T16:31:42.160227+0000","last_peered":"2023-10-11T16:31:42.160227+0000","last_clean":"2023-10-11T16:31:42.160227+0000","last_became_active":"2023-10-11T16:30:16.135196+0000","last_became_peered":"2023-10-11T16:30:16.135196+0000","last_unstale":"2023-10-11T16:31:42.160227+0000","last_undegraded":"2023-10-11T16:31:42.160227+0000","last_fullsized":"2023-10-11T16:31:42.160227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:18:51.568546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285229+0000","last_change":"2023-10-11T16:30:11.118355+0000","last_active":"2023-10-11T16:31:42.285229+0000","last_peered":"2023-10-11T16:31:42.285229+0000","last_clean":"2023-10-11T16:31:42.285229+0000","last_became_active":"2023-10-11T16:30:11.118075+0000","last_became_peered":"2023-10-11T16:30:11.118075+0000","last_unstale":"2023-10-11T16:31:42.285229+0000","last_undegraded":"2023-10-11T16:31:42.285229+0000","last_fullsized":"2023-10-11T16:31:42.285229+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:13:18.888864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.085765+0000","last_change":"2023-10-11T16:30:11.659049+0000","last_active":"2023-10-11T16:32:10.085765+0000","last_peered":"2023-10-11T16:32:10.085765+0000","last_clean":"2023-10-11T16:32:10.085765+0000","last_became_active":"2023-10-11T16:30:11.658748+0000","last_became_peered":"2023-10-11T16:30:11.658748+0000","last_unstale":"2023-10-11T16:32:10.085765+0000","last_undegraded":"2023-10-11T16:32:10.085765+0000","last_fullsized":"2023-10-11T16:32:10.085765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:46:58.254928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.164266+0000","last_change":"2023-10-11T16:30:16.136779+0000","last_active":"2023-10-11T16:31:42.164266+0000","last_peered":"2023-10-11T16:31:42.164266+0000","last_clean":"2023-10-11T16:31:42.164266+0000","last_became_active":"2023-10-11T16:30:16.136246+0000","last_became_peered":"2023-10-11T16:30:16.136246+0000","last_unstale":"2023-10-11T16:31:42.164266+0000","last_undegraded":"2023-10-11T16:31:42.164266+0000","last_fullsized":"2023-10-11T16:31:42.164266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:20:42.850191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285323+0000","last_change":"2023-10-11T16:30:11.121958+0000","last_active":"2023-10-11T16:31:42.285323+0000","last_peered":"2023-10-11T16:31:42.285323+0000","last_clean":"2023-10-11T16:31:42.285323+0000","last_became_active":"2023-10-11T16:30:11.120222+0000","last_became_peered":"2023-10-11T16:30:11.120222+0000","last_unstale":"2023-10-11T16:31:42.285323+0000","last_undegraded":"2023-10-11T16:31:42.285323+0000","last_fullsized":"2023-10-11T16:31:42.285323+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:11:41.038914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101540+0000","last_change":"2023-10-11T16:30:11.667386+0000","last_active":"2023-10-11T16:32:10.101540+0000","last_peered":"2023-10-11T16:32:10.101540+0000","last_clean":"2023-10-11T16:32:10.101540+0000","last_became_active":"2023-10-11T16:30:11.659772+0000","last_became_peered":"2023-10-11T16:30:11.659772+0000","last_unstale":"2023-10-11T16:32:10.101540+0000","last_undegraded":"2023-10-11T16:32:10.101540+0000","last_fullsized":"2023-10-11T16:32:10.101540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:04:56.860908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161225+0000","last_change":"2023-10-11T16:30:16.141157+0000","last_active":"2023-10-11T16:31:42.161225+0000","last_peered":"2023-10-11T16:31:42.161225+0000","last_clean":"2023-10-11T16:31:42.161225+0000","last_became_active":"2023-10-11T16:30:16.140669+0000","last_became_peered":"2023-10-11T16:30:16.140669+0000","last_unstale":"2023-10-11T16:31:42.161225+0000","last_undegraded":"2023-10-11T16:31:42.161225+0000","last_fullsized":"2023-10-11T16:31:42.161225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:03:40.600507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288380+0000","last_change":"2023-10-11T16:30:11.112740+0000","last_active":"2023-10-11T16:31:42.288380+0000","last_peered":"2023-10-11T16:31:42.288380+0000","last_clean":"2023-10-11T16:31:42.288380+0000","last_became_active":"2023-10-11T16:30:11.112334+0000","last_became_peered":"2023-10-11T16:30:11.112334+0000","last_unstale":"2023-10-11T16:31:42.288380+0000","last_undegraded":"2023-10-11T16:31:42.288380+0000","last_fullsized":"2023-10-11T16:31:42.288380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:37:32.836177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088886+0000","last_change":"2023-10-11T16:31:42.286928+0000","last_active":"2023-10-11T16:32:10.088886+0000","last_peered":"2023-10-11T16:32:10.088886+0000","last_clean":"2023-10-11T16:32:10.088886+0000","last_became_active":"2023-10-11T16:31:42.286409+0000","last_became_peered":"2023-10-11T16:31:42.286409+0000","last_unstale":"2023-10-11T16:32:10.088886+0000","last_undegraded":"2023-10-11T16:32:10.088886+0000","last_fullsized":"2023-10-11T16:32:10.088886+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:16:50.559256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533799+0000","last_change":"2023-10-11T16:30:16.128501+0000","last_active":"2023-10-11T16:31:42.533799+0000","last_peered":"2023-10-11T16:31:42.533799+0000","last_clean":"2023-10-11T16:31:42.533799+0000","last_became_active":"2023-10-11T16:30:16.128154+0000","last_became_peered":"2023-10-11T16:30:16.128154+0000","last_unstale":"2023-10-11T16:31:42.533799+0000","last_undegraded":"2023-10-11T16:31:42.533799+0000","last_fullsized":"2023-10-11T16:31:42.533799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:53:33.531388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":139,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:21.342444+0000","last_change":"2023-10-11T16:30:11.127248+0000","last_active":"2023-10-11T16:32:21.342444+0000","last_peered":"2023-10-11T16:32:21.342444+0000","last_clean":"2023-10-11T16:32:21.342444+0000","last_became_active":"2023-10-11T16:30:11.126997+0000","last_became_peered":"2023-10-11T16:30:11.126997+0000","last_unstale":"2023-10-11T16:32:21.342444+0000","last_undegraded":"2023-10-11T16:32:21.342444+0000","last_fullsized":"2023-10-11T16:32:21.342444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:05:22.346902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092896+0000","last_change":"2023-10-11T16:30:11.660416+0000","last_active":"2023-10-11T16:32:10.092896+0000","last_peered":"2023-10-11T16:32:10.092896+0000","last_clean":"2023-10-11T16:32:10.092896+0000","last_became_active":"2023-10-11T16:30:11.660108+0000","last_became_peered":"2023-10-11T16:30:11.660108+0000","last_unstale":"2023-10-11T16:32:10.092896+0000","last_undegraded":"2023-10-11T16:32:10.092896+0000","last_fullsized":"2023-10-11T16:32:10.092896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:36:02.019758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.157442+0000","last_change":"2023-10-11T16:30:16.132107+0000","last_active":"2023-10-11T16:31:42.157442+0000","last_peered":"2023-10-11T16:31:42.157442+0000","last_clean":"2023-10-11T16:31:42.157442+0000","last_became_active":"2023-10-11T16:30:16.130178+0000","last_became_peered":"2023-10-11T16:30:16.130178+0000","last_unstale":"2023-10-11T16:31:42.157442+0000","last_undegraded":"2023-10-11T16:31:42.157442+0000","last_fullsized":"2023-10-11T16:31:42.157442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:20:41.440369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161173+0000","last_change":"2023-10-11T16:30:11.112370+0000","last_active":"2023-10-11T16:31:42.161173+0000","last_peered":"2023-10-11T16:31:42.161173+0000","last_clean":"2023-10-11T16:31:42.161173+0000","last_became_active":"2023-10-11T16:30:11.112068+0000","last_became_peered":"2023-10-11T16:30:11.112068+0000","last_unstale":"2023-10-11T16:31:42.161173+0000","last_undegraded":"2023-10-11T16:31:42.161173+0000","last_fullsized":"2023-10-11T16:31:42.161173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:01:48.986542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101928+0000","last_change":"2023-10-11T16:31:42.464453+0000","last_active":"2023-10-11T16:32:10.101928+0000","last_peered":"2023-10-11T16:32:10.101928+0000","last_clean":"2023-10-11T16:32:10.101928+0000","last_became_active":"2023-10-11T16:31:42.463997+0000","last_became_peered":"2023-10-11T16:31:42.463997+0000","last_unstale":"2023-10-11T16:32:10.101928+0000","last_undegraded":"2023-10-11T16:32:10.101928+0000","last_fullsized":"2023-10-11T16:32:10.101928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:44:47.706246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,6],"acting":[0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288834+0000","last_change":"2023-10-11T16:30:16.127082+0000","last_active":"2023-10-11T16:31:42.288834+0000","last_peered":"2023-10-11T16:31:42.288834+0000","last_clean":"2023-10-11T16:31:42.288834+0000","last_became_active":"2023-10-11T16:30:16.126647+0000","last_became_peered":"2023-10-11T16:30:16.126647+0000","last_unstale":"2023-10-11T16:31:42.288834+0000","last_undegraded":"2023-10-11T16:31:42.288834+0000","last_fullsized":"2023-10-11T16:31:42.288834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:13:12.138603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289000+0000","last_change":"2023-10-11T16:30:11.124803+0000","last_active":"2023-10-11T16:31:42.289000+0000","last_peered":"2023-10-11T16:31:42.289000+0000","last_clean":"2023-10-11T16:31:42.289000+0000","last_became_active":"2023-10-11T16:30:11.124424+0000","last_became_peered":"2023-10-11T16:30:11.124424+0000","last_unstale":"2023-10-11T16:31:42.289000+0000","last_undegraded":"2023-10-11T16:31:42.289000+0000","last_fullsized":"2023-10-11T16:31:42.289000+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:15:55.186995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092874+0000","last_change":"2023-10-11T16:30:11.660760+0000","last_active":"2023-10-11T16:32:10.092874+0000","last_peered":"2023-10-11T16:32:10.092874+0000","last_clean":"2023-10-11T16:32:10.092874+0000","last_became_active":"2023-10-11T16:30:11.660450+0000","last_became_peered":"2023-10-11T16:30:11.660450+0000","last_unstale":"2023-10-11T16:32:10.092874+0000","last_undegraded":"2023-10-11T16:32:10.092874+0000","last_fullsized":"2023-10-11T16:32:10.092874+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:23:25.760798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158398+0000","last_change":"2023-10-11T16:30:16.133362+0000","last_active":"2023-10-11T16:31:42.158398+0000","last_peered":"2023-10-11T16:31:42.158398+0000","last_clean":"2023-10-11T16:31:42.158398+0000","last_became_active":"2023-10-11T16:30:16.131403+0000","last_became_peered":"2023-10-11T16:30:16.131403+0000","last_unstale":"2023-10-11T16:31:42.158398+0000","last_undegraded":"2023-10-11T16:31:42.158398+0000","last_fullsized":"2023-10-11T16:31:42.158398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:04:51.440115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":303,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:22.343291+0000","last_change":"2023-10-11T16:30:41.788994+0000","last_active":"2023-10-11T16:32:22.343291+0000","last_peered":"2023-10-11T16:32:22.343291+0000","last_clean":"2023-10-11T16:32:22.343291+0000","last_became_active":"2023-10-11T16:30:41.189949+0000","last_became_peered":"2023-10-11T16:30:41.189949+0000","last_unstale":"2023-10-11T16:32:22.343291+0000","last_undegraded":"2023-10-11T16:32:22.343291+0000","last_fullsized":"2023-10-11T16:32:22.343291+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:58:15.728137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66349,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":217,"num_read_kb":174,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38515,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.082731+0000","last_change":"2023-10-11T16:30:11.658931+0000","last_active":"2023-10-11T16:32:10.082731+0000","last_peered":"2023-10-11T16:32:10.082731+0000","last_clean":"2023-10-11T16:32:10.082731+0000","last_became_active":"2023-10-11T16:30:11.658580+0000","last_became_peered":"2023-10-11T16:30:11.658580+0000","last_unstale":"2023-10-11T16:32:10.082731+0000","last_undegraded":"2023-10-11T16:32:10.082731+0000","last_fullsized":"2023-10-11T16:32:10.082731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:04:39.079541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289353+0000","last_change":"2023-10-11T16:30:16.132480+0000","last_active":"2023-10-11T16:31:42.289353+0000","last_peered":"2023-10-11T16:31:42.289353+0000","last_clean":"2023-10-11T16:31:42.289353+0000","last_became_active":"2023-10-11T16:30:16.132105+0000","last_became_peered":"2023-10-11T16:30:16.132105+0000","last_unstale":"2023-10-11T16:31:42.289353+0000","last_undegraded":"2023-10-11T16:31:42.289353+0000","last_fullsized":"2023-10-11T16:31:42.289353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:31:29.246174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160086+0000","last_change":"2023-10-11T16:30:11.119520+0000","last_active":"2023-10-11T16:31:42.160086+0000","last_peered":"2023-10-11T16:31:42.160086+0000","last_clean":"2023-10-11T16:31:42.160086+0000","last_became_active":"2023-10-11T16:30:11.119194+0000","last_became_peered":"2023-10-11T16:30:11.119194+0000","last_unstale":"2023-10-11T16:31:42.160086+0000","last_undegraded":"2023-10-11T16:31:42.160086+0000","last_fullsized":"2023-10-11T16:31:42.160086+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:39:49.561867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.104692+0000","last_change":"2023-10-11T16:30:11.669554+0000","last_active":"2023-10-11T16:32:10.104692+0000","last_peered":"2023-10-11T16:32:10.104692+0000","last_clean":"2023-10-11T16:32:10.104692+0000","last_became_active":"2023-10-11T16:30:11.668326+0000","last_became_peered":"2023-10-11T16:30:11.668326+0000","last_unstale":"2023-10-11T16:32:10.104692+0000","last_undegraded":"2023-10-11T16:32:10.104692+0000","last_fullsized":"2023-10-11T16:32:10.104692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:17:46.544148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.163839+0000","last_change":"2023-10-11T16:30:16.135867+0000","last_active":"2023-10-11T16:31:42.163839+0000","last_peered":"2023-10-11T16:31:42.163839+0000","last_clean":"2023-10-11T16:31:42.163839+0000","last_became_active":"2023-10-11T16:30:16.135476+0000","last_became_peered":"2023-10-11T16:30:16.135476+0000","last_unstale":"2023-10-11T16:31:42.163839+0000","last_undegraded":"2023-10-11T16:31:42.163839+0000","last_fullsized":"2023-10-11T16:31:42.163839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:36:11.677512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288800+0000","last_change":"2023-10-11T16:30:11.120159+0000","last_active":"2023-10-11T16:31:42.288800+0000","last_peered":"2023-10-11T16:31:42.288800+0000","last_clean":"2023-10-11T16:31:42.288800+0000","last_became_active":"2023-10-11T16:30:11.119818+0000","last_became_peered":"2023-10-11T16:30:11.119818+0000","last_unstale":"2023-10-11T16:31:42.288800+0000","last_undegraded":"2023-10-11T16:31:42.288800+0000","last_fullsized":"2023-10-11T16:31:42.288800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:55:39.104456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.091980+0000","last_change":"2023-10-11T16:30:11.658170+0000","last_active":"2023-10-11T16:32:10.091980+0000","last_peered":"2023-10-11T16:32:10.091980+0000","last_clean":"2023-10-11T16:32:10.091980+0000","last_became_active":"2023-10-11T16:30:11.657880+0000","last_became_peered":"2023-10-11T16:30:11.657880+0000","last_unstale":"2023-10-11T16:32:10.091980+0000","last_undegraded":"2023-10-11T16:32:10.091980+0000","last_fullsized":"2023-10-11T16:32:10.091980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:26:20.985716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158790+0000","last_change":"2023-10-11T16:30:16.135840+0000","last_active":"2023-10-11T16:31:42.158790+0000","last_peered":"2023-10-11T16:31:42.158790+0000","last_clean":"2023-10-11T16:31:42.158790+0000","last_became_active":"2023-10-11T16:30:16.135425+0000","last_became_peered":"2023-10-11T16:30:16.135425+0000","last_unstale":"2023-10-11T16:31:42.158790+0000","last_undegraded":"2023-10-11T16:31:42.158790+0000","last_fullsized":"2023-10-11T16:31:42.158790+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:05:47.475689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":135,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:18.337663+0000","last_change":"2023-10-11T16:30:11.114852+0000","last_active":"2023-10-11T16:32:18.337663+0000","last_peered":"2023-10-11T16:32:18.337663+0000","last_clean":"2023-10-11T16:32:18.337663+0000","last_became_active":"2023-10-11T16:30:11.114505+0000","last_became_peered":"2023-10-11T16:30:11.114505+0000","last_unstale":"2023-10-11T16:32:18.337663+0000","last_undegraded":"2023-10-11T16:32:18.337663+0000","last_fullsized":"2023-10-11T16:32:18.337663+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:37:12.900675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093444+0000","last_change":"2023-10-11T16:30:11.656993+0000","last_active":"2023-10-11T16:32:10.093444+0000","last_peered":"2023-10-11T16:32:10.093444+0000","last_clean":"2023-10-11T16:32:10.093444+0000","last_became_active":"2023-10-11T16:30:11.656658+0000","last_became_peered":"2023-10-11T16:30:11.656658+0000","last_unstale":"2023-10-11T16:32:10.093444+0000","last_undegraded":"2023-10-11T16:32:10.093444+0000","last_fullsized":"2023-10-11T16:32:10.093444+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:27:01.920127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285928+0000","last_change":"2023-10-11T16:30:16.135460+0000","last_active":"2023-10-11T16:31:42.285928+0000","last_peered":"2023-10-11T16:31:42.285928+0000","last_clean":"2023-10-11T16:31:42.285928+0000","last_became_active":"2023-10-11T16:30:16.135006+0000","last_became_peered":"2023-10-11T16:30:16.135006+0000","last_unstale":"2023-10-11T16:31:42.285928+0000","last_undegraded":"2023-10-11T16:31:42.285928+0000","last_fullsized":"2023-10-11T16:31:42.285928+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:08:54.198092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462035+0000","last_change":"2023-10-11T16:30:11.126515+0000","last_active":"2023-10-11T16:31:42.462035+0000","last_peered":"2023-10-11T16:31:42.462035+0000","last_clean":"2023-10-11T16:31:42.462035+0000","last_became_active":"2023-10-11T16:30:11.126235+0000","last_became_peered":"2023-10-11T16:30:11.126235+0000","last_unstale":"2023-10-11T16:31:42.462035+0000","last_undegraded":"2023-10-11T16:31:42.462035+0000","last_fullsized":"2023-10-11T16:31:42.462035+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:55:08.925417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.092273+0000","last_change":"2023-10-11T16:30:11.657941+0000","last_active":"2023-10-11T16:32:10.092273+0000","last_peered":"2023-10-11T16:32:10.092273+0000","last_clean":"2023-10-11T16:32:10.092273+0000","last_became_active":"2023-10-11T16:30:11.657553+0000","last_became_peered":"2023-10-11T16:30:11.657553+0000","last_unstale":"2023-10-11T16:32:10.092273+0000","last_undegraded":"2023-10-11T16:32:10.092273+0000","last_fullsized":"2023-10-11T16:32:10.092273+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:15:28.155402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288455+0000","last_change":"2023-10-11T16:30:16.128099+0000","last_active":"2023-10-11T16:31:42.288455+0000","last_peered":"2023-10-11T16:31:42.288455+0000","last_clean":"2023-10-11T16:31:42.288455+0000","last_became_active":"2023-10-11T16:30:16.127667+0000","last_became_peered":"2023-10-11T16:30:16.127667+0000","last_unstale":"2023-10-11T16:31:42.288455+0000","last_undegraded":"2023-10-11T16:31:42.288455+0000","last_fullsized":"2023-10-11T16:31:42.288455+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:28:12.958080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462997+0000","last_change":"2023-10-11T16:30:11.119093+0000","last_active":"2023-10-11T16:31:42.462997+0000","last_peered":"2023-10-11T16:31:42.462997+0000","last_clean":"2023-10-11T16:31:42.462997+0000","last_became_active":"2023-10-11T16:30:11.118756+0000","last_became_peered":"2023-10-11T16:30:11.118756+0000","last_unstale":"2023-10-11T16:31:42.462997+0000","last_undegraded":"2023-10-11T16:31:42.462997+0000","last_fullsized":"2023-10-11T16:31:42.462997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:03:09.115056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.080956+0000","last_change":"2023-10-11T16:31:42.287141+0000","last_active":"2023-10-11T16:32:10.080956+0000","last_peered":"2023-10-11T16:32:10.080956+0000","last_clean":"2023-10-11T16:32:10.080956+0000","last_became_active":"2023-10-11T16:31:42.286758+0000","last_became_peered":"2023-10-11T16:31:42.286758+0000","last_unstale":"2023-10-11T16:32:10.080956+0000","last_undegraded":"2023-10-11T16:32:10.080956+0000","last_fullsized":"2023-10-11T16:32:10.080956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:40:33.920330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462187+0000","last_change":"2023-10-11T16:30:16.171200+0000","last_active":"2023-10-11T16:31:42.462187+0000","last_peered":"2023-10-11T16:31:42.462187+0000","last_clean":"2023-10-11T16:31:42.462187+0000","last_became_active":"2023-10-11T16:30:16.170702+0000","last_became_peered":"2023-10-11T16:30:16.170702+0000","last_unstale":"2023-10-11T16:31:42.462187+0000","last_undegraded":"2023-10-11T16:31:42.462187+0000","last_fullsized":"2023-10-11T16:31:42.462187+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:43.818267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.158964+0000","last_change":"2023-10-11T16:30:11.127287+0000","last_active":"2023-10-11T16:31:42.158964+0000","last_peered":"2023-10-11T16:31:42.158964+0000","last_clean":"2023-10-11T16:31:42.158964+0000","last_became_active":"2023-10-11T16:30:11.125949+0000","last_became_peered":"2023-10-11T16:30:11.125949+0000","last_unstale":"2023-10-11T16:31:42.158964+0000","last_undegraded":"2023-10-11T16:31:42.158964+0000","last_fullsized":"2023-10-11T16:31:42.158964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:16:18.715348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093724+0000","last_change":"2023-10-11T16:30:11.660745+0000","last_active":"2023-10-11T16:32:10.093724+0000","last_peered":"2023-10-11T16:32:10.093724+0000","last_clean":"2023-10-11T16:32:10.093724+0000","last_became_active":"2023-10-11T16:30:11.660395+0000","last_became_peered":"2023-10-11T16:30:11.660395+0000","last_unstale":"2023-10-11T16:32:10.093724+0000","last_undegraded":"2023-10-11T16:32:10.093724+0000","last_fullsized":"2023-10-11T16:32:10.093724+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:03:22.304558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285744+0000","last_change":"2023-10-11T16:30:16.136552+0000","last_active":"2023-10-11T16:31:42.285744+0000","last_peered":"2023-10-11T16:31:42.285744+0000","last_clean":"2023-10-11T16:31:42.285744+0000","last_became_active":"2023-10-11T16:30:16.134581+0000","last_became_peered":"2023-10-11T16:30:16.134581+0000","last_unstale":"2023-10-11T16:31:42.285744+0000","last_undegraded":"2023-10-11T16:31:42.285744+0000","last_fullsized":"2023-10-11T16:31:42.285744+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:45:09.560952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160923+0000","last_change":"2023-10-11T16:30:11.121421+0000","last_active":"2023-10-11T16:31:42.160923+0000","last_peered":"2023-10-11T16:31:42.160923+0000","last_clean":"2023-10-11T16:31:42.160923+0000","last_became_active":"2023-10-11T16:30:11.121093+0000","last_became_peered":"2023-10-11T16:30:11.121093+0000","last_unstale":"2023-10-11T16:31:42.160923+0000","last_undegraded":"2023-10-11T16:31:42.160923+0000","last_fullsized":"2023-10-11T16:31:42.160923+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:11:25.625028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.095113+0000","last_change":"2023-10-11T16:30:11.661934+0000","last_active":"2023-10-11T16:32:10.095113+0000","last_peered":"2023-10-11T16:32:10.095113+0000","last_clean":"2023-10-11T16:32:10.095113+0000","last_became_active":"2023-10-11T16:30:11.660684+0000","last_became_peered":"2023-10-11T16:30:11.660684+0000","last_unstale":"2023-10-11T16:32:10.095113+0000","last_undegraded":"2023-10-11T16:32:10.095113+0000","last_fullsized":"2023-10-11T16:32:10.095113+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:33:24.458841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.286057+0000","last_change":"2023-10-11T16:30:16.170774+0000","last_active":"2023-10-11T16:31:42.286057+0000","last_peered":"2023-10-11T16:31:42.286057+0000","last_clean":"2023-10-11T16:31:42.286057+0000","last_became_active":"2023-10-11T16:30:16.170243+0000","last_became_peered":"2023-10-11T16:30:16.170243+0000","last_unstale":"2023-10-11T16:31:42.286057+0000","last_undegraded":"2023-10-11T16:31:42.286057+0000","last_fullsized":"2023-10-11T16:31:42.286057+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:23:48.290443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083891+0000","last_change":"2023-10-11T16:30:11.661665+0000","last_active":"2023-10-11T16:32:10.083891+0000","last_peered":"2023-10-11T16:32:10.083891+0000","last_clean":"2023-10-11T16:32:10.083891+0000","last_became_active":"2023-10-11T16:30:11.661351+0000","last_became_peered":"2023-10-11T16:30:11.661351+0000","last_unstale":"2023-10-11T16:32:10.083891+0000","last_undegraded":"2023-10-11T16:32:10.083891+0000","last_fullsized":"2023-10-11T16:32:10.083891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:18:07.940669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":140,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:21.341422+0000","last_change":"2023-10-11T16:30:11.125295+0000","last_active":"2023-10-11T16:32:21.341422+0000","last_peered":"2023-10-11T16:32:21.341422+0000","last_clean":"2023-10-11T16:32:21.341422+0000","last_became_active":"2023-10-11T16:30:11.124998+0000","last_became_peered":"2023-10-11T16:30:11.124998+0000","last_unstale":"2023-10-11T16:32:21.341422+0000","last_undegraded":"2023-10-11T16:32:21.341422+0000","last_fullsized":"2023-10-11T16:32:21.341422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:13:34.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161339+0000","last_change":"2023-10-11T16:30:16.125645+0000","last_active":"2023-10-11T16:31:42.161339+0000","last_peered":"2023-10-11T16:31:42.161339+0000","last_clean":"2023-10-11T16:31:42.161339+0000","last_became_active":"2023-10-11T16:30:16.125230+0000","last_became_peered":"2023-10-11T16:30:16.125230+0000","last_unstale":"2023-10-11T16:31:42.161339+0000","last_undegraded":"2023-10-11T16:31:42.161339+0000","last_fullsized":"2023-10-11T16:31:42.161339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:05:48.188359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101162+0000","last_change":"2023-10-11T16:30:11.664723+0000","last_active":"2023-10-11T16:32:10.101162+0000","last_peered":"2023-10-11T16:32:10.101162+0000","last_clean":"2023-10-11T16:32:10.101162+0000","last_became_active":"2023-10-11T16:30:11.664438+0000","last_became_peered":"2023-10-11T16:30:11.664438+0000","last_unstale":"2023-10-11T16:32:10.101162+0000","last_undegraded":"2023-10-11T16:32:10.101162+0000","last_fullsized":"2023-10-11T16:32:10.101162+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:12:34.720396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288780+0000","last_change":"2023-10-11T16:30:11.119850+0000","last_active":"2023-10-11T16:31:42.288780+0000","last_peered":"2023-10-11T16:31:42.288780+0000","last_clean":"2023-10-11T16:31:42.288780+0000","last_became_active":"2023-10-11T16:30:11.119520+0000","last_became_peered":"2023-10-11T16:30:11.119520+0000","last_unstale":"2023-10-11T16:31:42.288780+0000","last_undegraded":"2023-10-11T16:31:42.288780+0000","last_fullsized":"2023-10-11T16:31:42.288780+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:17:09.996158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160483+0000","last_change":"2023-10-11T16:29:06.136004+0000","last_active":"2023-10-11T16:31:42.160483+0000","last_peered":"2023-10-11T16:31:42.160483+0000","last_clean":"2023-10-11T16:31:42.160483+0000","last_became_active":"2023-10-11T16:29:06.135452+0000","last_became_peered":"2023-10-11T16:29:06.135452+0000","last_unstale":"2023-10-11T16:31:42.160483+0000","last_undegraded":"2023-10-11T16:31:42.160483+0000","last_fullsized":"2023-10-11T16:31:42.160483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:25:56.629200+0000","last_clean_scrub_stamp":"2023-10-11T16:25:56.629200+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:46:36.114850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289135+0000","last_change":"2023-10-11T16:30:16.171530+0000","last_active":"2023-10-11T16:31:42.289135+0000","last_peered":"2023-10-11T16:31:42.289135+0000","last_clean":"2023-10-11T16:31:42.289135+0000","last_became_active":"2023-10-11T16:30:16.170956+0000","last_became_peered":"2023-10-11T16:30:16.170956+0000","last_unstale":"2023-10-11T16:31:42.289135+0000","last_undegraded":"2023-10-11T16:31:42.289135+0000","last_fullsized":"2023-10-11T16:31:42.289135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:30:07.695611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.074567+0000","last_change":"2023-10-11T16:30:11.658546+0000","last_active":"2023-10-11T16:32:10.074567+0000","last_peered":"2023-10-11T16:32:10.074567+0000","last_clean":"2023-10-11T16:32:10.074567+0000","last_became_active":"2023-10-11T16:30:11.657484+0000","last_became_peered":"2023-10-11T16:30:11.657484+0000","last_unstale":"2023-10-11T16:32:10.074567+0000","last_undegraded":"2023-10-11T16:32:10.074567+0000","last_fullsized":"2023-10-11T16:32:10.074567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:33:14.082898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":44,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.160623+0000","last_change":"2023-10-11T16:30:41.938949+0000","last_active":"2023-10-11T16:31:42.160623+0000","last_peered":"2023-10-11T16:31:42.160623+0000","last_clean":"2023-10-11T16:31:42.160623+0000","last_became_active":"2023-10-11T16:30:41.192902+0000","last_became_peered":"2023-10-11T16:30:41.192902+0000","last_unstale":"2023-10-11T16:31:42.160623+0000","last_undegraded":"2023-10-11T16:31:42.160623+0000","last_fullsized":"2023-10-11T16:31:42.160623+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:57:28.224909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.163836+0000","last_change":"2023-10-11T16:30:16.130641+0000","last_active":"2023-10-11T16:31:42.163836+0000","last_peered":"2023-10-11T16:31:42.163836+0000","last_clean":"2023-10-11T16:31:42.163836+0000","last_became_active":"2023-10-11T16:30:16.130205+0000","last_became_peered":"2023-10-11T16:30:16.130205+0000","last_unstale":"2023-10-11T16:31:42.163836+0000","last_undegraded":"2023-10-11T16:31:42.163836+0000","last_fullsized":"2023-10-11T16:31:42.163836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T23:42:22.315441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.164029+0000","last_change":"2023-10-11T16:30:41.205550+0000","last_active":"2023-10-11T16:31:42.164029+0000","last_peered":"2023-10-11T16:31:42.164029+0000","last_clean":"2023-10-11T16:31:42.164029+0000","last_became_active":"2023-10-11T16:30:41.190345+0000","last_became_peered":"2023-10-11T16:30:41.190345+0000","last_unstale":"2023-10-11T16:31:42.164029+0000","last_undegraded":"2023-10-11T16:31:42.164029+0000","last_fullsized":"2023-10-11T16:31:42.164029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:32:00.140298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.074906+0000","last_change":"2023-10-11T16:30:11.652399+0000","last_active":"2023-10-11T16:32:10.074906+0000","last_peered":"2023-10-11T16:32:10.074906+0000","last_clean":"2023-10-11T16:32:10.074906+0000","last_became_active":"2023-10-11T16:30:11.651719+0000","last_became_peered":"2023-10-11T16:30:11.651719+0000","last_unstale":"2023-10-11T16:32:10.074906+0000","last_undegraded":"2023-10-11T16:32:10.074906+0000","last_fullsized":"2023-10-11T16:32:10.074906+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:57:43.652224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534261+0000","last_change":"2023-10-11T16:30:16.171627+0000","last_active":"2023-10-11T16:31:42.534261+0000","last_peered":"2023-10-11T16:31:42.534261+0000","last_clean":"2023-10-11T16:31:42.534261+0000","last_became_active":"2023-10-11T16:30:16.170946+0000","last_became_peered":"2023-10-11T16:30:16.170946+0000","last_unstale":"2023-10-11T16:31:42.534261+0000","last_undegraded":"2023-10-11T16:31:42.534261+0000","last_fullsized":"2023-10-11T16:31:42.534261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:13:32.227680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.564394+0000","last_change":"2023-10-11T16:30:11.122494+0000","last_active":"2023-10-11T16:32:10.564394+0000","last_peered":"2023-10-11T16:32:10.564394+0000","last_clean":"2023-10-11T16:32:10.564394+0000","last_became_active":"2023-10-11T16:30:11.120435+0000","last_became_peered":"2023-10-11T16:30:11.120435+0000","last_unstale":"2023-10-11T16:32:10.564394+0000","last_undegraded":"2023-10-11T16:32:10.564394+0000","last_fullsized":"2023-10-11T16:32:10.564394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:02:28.868871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.101410+0000","last_change":"2023-10-11T16:30:11.662478+0000","last_active":"2023-10-11T16:32:10.101410+0000","last_peered":"2023-10-11T16:32:10.101410+0000","last_clean":"2023-10-11T16:32:10.101410+0000","last_became_active":"2023-10-11T16:30:11.660078+0000","last_became_peered":"2023-10-11T16:30:11.660078+0000","last_unstale":"2023-10-11T16:32:10.101410+0000","last_undegraded":"2023-10-11T16:32:10.101410+0000","last_fullsized":"2023-10-11T16:32:10.101410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:50:14.165387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.159765+0000","last_change":"2023-10-11T16:30:16.122558+0000","last_active":"2023-10-11T16:31:42.159765+0000","last_peered":"2023-10-11T16:31:42.159765+0000","last_clean":"2023-10-11T16:31:42.159765+0000","last_became_active":"2023-10-11T16:30:16.122075+0000","last_became_peered":"2023-10-11T16:30:16.122075+0000","last_unstale":"2023-10-11T16:31:42.159765+0000","last_undegraded":"2023-10-11T16:31:42.159765+0000","last_fullsized":"2023-10-11T16:31:42.159765+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:04:09.873784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289422+0000","last_change":"2023-10-11T16:30:11.115078+0000","last_active":"2023-10-11T16:31:42.289422+0000","last_peered":"2023-10-11T16:31:42.289422+0000","last_clean":"2023-10-11T16:31:42.289422+0000","last_became_active":"2023-10-11T16:30:11.114679+0000","last_became_peered":"2023-10-11T16:30:11.114679+0000","last_unstale":"2023-10-11T16:31:42.289422+0000","last_undegraded":"2023-10-11T16:31:42.289422+0000","last_fullsized":"2023-10-11T16:31:42.289422+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:41:03.781734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100197+0000","last_change":"2023-10-11T16:31:42.290019+0000","last_active":"2023-10-11T16:32:10.100197+0000","last_peered":"2023-10-11T16:32:10.100197+0000","last_clean":"2023-10-11T16:32:10.100197+0000","last_became_active":"2023-10-11T16:31:42.289721+0000","last_became_peered":"2023-10-11T16:31:42.289721+0000","last_unstale":"2023-10-11T16:32:10.100197+0000","last_undegraded":"2023-10-11T16:32:10.100197+0000","last_fullsized":"2023-10-11T16:32:10.100197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:04:10.132863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289138+0000","last_change":"2023-10-11T16:30:41.193489+0000","last_active":"2023-10-11T16:31:42.289138+0000","last_peered":"2023-10-11T16:31:42.289138+0000","last_clean":"2023-10-11T16:31:42.289138+0000","last_became_active":"2023-10-11T16:30:41.193060+0000","last_became_peered":"2023-10-11T16:30:41.193060+0000","last_unstale":"2023-10-11T16:31:42.289138+0000","last_undegraded":"2023-10-11T16:31:42.289138+0000","last_fullsized":"2023-10-11T16:31:42.289138+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T16:51:45.983921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533782+0000","last_change":"2023-10-11T16:30:11.122508+0000","last_active":"2023-10-11T16:31:42.533782+0000","last_peered":"2023-10-11T16:31:42.533782+0000","last_clean":"2023-10-11T16:31:42.533782+0000","last_became_active":"2023-10-11T16:30:11.122198+0000","last_became_peered":"2023-10-11T16:30:11.122198+0000","last_unstale":"2023-10-11T16:31:42.533782+0000","last_undegraded":"2023-10-11T16:31:42.533782+0000","last_fullsized":"2023-10-11T16:31:42.533782+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:43:17.566817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.057040+0000","last_change":"2023-10-11T16:30:11.654260+0000","last_active":"2023-10-11T16:32:10.057040+0000","last_peered":"2023-10-11T16:32:10.057040+0000","last_clean":"2023-10-11T16:32:10.057040+0000","last_became_active":"2023-10-11T16:30:11.653940+0000","last_became_peered":"2023-10-11T16:30:11.653940+0000","last_unstale":"2023-10-11T16:32:10.057040+0000","last_undegraded":"2023-10-11T16:32:10.057040+0000","last_fullsized":"2023-10-11T16:32:10.057040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:20:18.016311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462851+0000","last_change":"2023-10-11T16:30:41.404891+0000","last_active":"2023-10-11T16:31:42.462851+0000","last_peered":"2023-10-11T16:31:42.462851+0000","last_clean":"2023-10-11T16:31:42.462851+0000","last_became_active":"2023-10-11T16:30:41.404396+0000","last_became_peered":"2023-10-11T16:30:41.404396+0000","last_unstale":"2023-10-11T16:31:42.462851+0000","last_undegraded":"2023-10-11T16:31:42.462851+0000","last_fullsized":"2023-10-11T16:31:42.462851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:41:36.636884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533885+0000","last_change":"2023-10-11T16:30:11.120578+0000","last_active":"2023-10-11T16:31:42.533885+0000","last_peered":"2023-10-11T16:31:42.533885+0000","last_clean":"2023-10-11T16:31:42.533885+0000","last_became_active":"2023-10-11T16:30:11.120232+0000","last_became_peered":"2023-10-11T16:30:11.120232+0000","last_unstale":"2023-10-11T16:31:42.533885+0000","last_undegraded":"2023-10-11T16:31:42.533885+0000","last_fullsized":"2023-10-11T16:31:42.533885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:11:17.333111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097932+0000","last_change":"2023-10-11T16:30:11.647048+0000","last_active":"2023-10-11T16:32:10.097932+0000","last_peered":"2023-10-11T16:32:10.097932+0000","last_clean":"2023-10-11T16:32:10.097932+0000","last_became_active":"2023-10-11T16:30:11.646617+0000","last_became_peered":"2023-10-11T16:30:11.646617+0000","last_unstale":"2023-10-11T16:32:10.097932+0000","last_undegraded":"2023-10-11T16:32:10.097932+0000","last_fullsized":"2023-10-11T16:32:10.097932+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:14:58.964194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289262+0000","last_change":"2023-10-11T16:30:16.127150+0000","last_active":"2023-10-11T16:31:42.289262+0000","last_peered":"2023-10-11T16:31:42.289262+0000","last_clean":"2023-10-11T16:31:42.289262+0000","last_became_active":"2023-10-11T16:30:16.126599+0000","last_became_peered":"2023-10-11T16:30:16.126599+0000","last_unstale":"2023-10-11T16:31:42.289262+0000","last_undegraded":"2023-10-11T16:31:42.289262+0000","last_fullsized":"2023-10-11T16:31:42.289262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:43:35.192572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:12.146327+0000","last_change":"2023-10-11T16:30:11.119517+0000","last_active":"2023-10-11T16:32:12.146327+0000","last_peered":"2023-10-11T16:32:12.146327+0000","last_clean":"2023-10-11T16:32:12.146327+0000","last_became_active":"2023-10-11T16:30:11.119204+0000","last_became_peered":"2023-10-11T16:30:11.119204+0000","last_unstale":"2023-10-11T16:32:12.146327+0000","last_undegraded":"2023-10-11T16:32:12.146327+0000","last_fullsized":"2023-10-11T16:32:12.146327+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:52:56.087178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.102288+0000","last_change":"2023-10-11T16:30:11.662395+0000","last_active":"2023-10-11T16:32:10.102288+0000","last_peered":"2023-10-11T16:32:10.102288+0000","last_clean":"2023-10-11T16:32:10.102288+0000","last_became_active":"2023-10-11T16:30:11.659692+0000","last_became_peered":"2023-10-11T16:30:11.659692+0000","last_unstale":"2023-10-11T16:32:10.102288+0000","last_undegraded":"2023-10-11T16:32:10.102288+0000","last_fullsized":"2023-10-11T16:32:10.102288+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:24:38.182167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288761+0000","last_change":"2023-10-11T16:30:16.171277+0000","last_active":"2023-10-11T16:31:42.288761+0000","last_peered":"2023-10-11T16:31:42.288761+0000","last_clean":"2023-10-11T16:31:42.288761+0000","last_became_active":"2023-10-11T16:30:16.170747+0000","last_became_peered":"2023-10-11T16:30:16.170747+0000","last_unstale":"2023-10-11T16:31:42.288761+0000","last_undegraded":"2023-10-11T16:31:42.288761+0000","last_fullsized":"2023-10-11T16:31:42.288761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:29:21.139277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.462324+0000","last_change":"2023-10-11T16:30:11.118208+0000","last_active":"2023-10-11T16:31:42.462324+0000","last_peered":"2023-10-11T16:31:42.462324+0000","last_clean":"2023-10-11T16:31:42.462324+0000","last_became_active":"2023-10-11T16:30:11.117882+0000","last_became_peered":"2023-10-11T16:30:11.117882+0000","last_unstale":"2023-10-11T16:31:42.462324+0000","last_undegraded":"2023-10-11T16:31:42.462324+0000","last_fullsized":"2023-10-11T16:31:42.462324+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T19:50:35.582480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088149+0000","last_change":"2023-10-11T16:31:42.286822+0000","last_active":"2023-10-11T16:32:10.088149+0000","last_peered":"2023-10-11T16:32:10.088149+0000","last_clean":"2023-10-11T16:32:10.088149+0000","last_became_active":"2023-10-11T16:31:42.286327+0000","last_became_peered":"2023-10-11T16:31:42.286327+0000","last_unstale":"2023-10-11T16:32:10.088149+0000","last_undegraded":"2023-10-11T16:32:10.088149+0000","last_fullsized":"2023-10-11T16:32:10.088149+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:21:36.527593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.533754+0000","last_change":"2023-10-11T16:30:41.200104+0000","last_active":"2023-10-11T16:31:42.533754+0000","last_peered":"2023-10-11T16:31:42.533754+0000","last_clean":"2023-10-11T16:31:42.533754+0000","last_became_active":"2023-10-11T16:30:41.199637+0000","last_became_peered":"2023-10-11T16:30:41.199637+0000","last_unstale":"2023-10-11T16:31:42.533754+0000","last_undegraded":"2023-10-11T16:31:42.533754+0000","last_fullsized":"2023-10-11T16:31:42.533754+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:07:59.931450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.534011+0000","last_change":"2023-10-11T16:30:41.202718+0000","last_active":"2023-10-11T16:31:42.534011+0000","last_peered":"2023-10-11T16:31:42.534011+0000","last_clean":"2023-10-11T16:31:42.534011+0000","last_became_active":"2023-10-11T16:30:41.190741+0000","last_became_peered":"2023-10-11T16:30:41.190741+0000","last_unstale":"2023-10-11T16:31:42.534011+0000","last_undegraded":"2023-10-11T16:31:42.534011+0000","last_fullsized":"2023-10-11T16:31:42.534011+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:13:39.958791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.094846+0000","last_change":"2023-10-11T16:30:11.647441+0000","last_active":"2023-10-11T16:32:10.094846+0000","last_peered":"2023-10-11T16:32:10.094846+0000","last_clean":"2023-10-11T16:32:10.094846+0000","last_became_active":"2023-10-11T16:30:11.647149+0000","last_became_peered":"2023-10-11T16:30:11.647149+0000","last_unstale":"2023-10-11T16:32:10.094846+0000","last_undegraded":"2023-10-11T16:32:10.094846+0000","last_fullsized":"2023-10-11T16:32:10.094846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:44:05.420593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.161952+0000","last_change":"2023-10-11T16:30:41.190477+0000","last_active":"2023-10-11T16:31:42.161952+0000","last_peered":"2023-10-11T16:31:42.161952+0000","last_clean":"2023-10-11T16:31:42.161952+0000","last_became_active":"2023-10-11T16:30:41.189929+0000","last_became_peered":"2023-10-11T16:30:41.189929+0000","last_unstale":"2023-10-11T16:31:42.161952+0000","last_undegraded":"2023-10-11T16:31:42.161952+0000","last_fullsized":"2023-10-11T16:31:42.161952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:14:01.200276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.289168+0000","last_change":"2023-10-11T16:30:11.120350+0000","last_active":"2023-10-11T16:31:42.289168+0000","last_peered":"2023-10-11T16:31:42.289168+0000","last_clean":"2023-10-11T16:31:42.289168+0000","last_became_active":"2023-10-11T16:30:11.120047+0000","last_became_peered":"2023-10-11T16:30:11.120047+0000","last_unstale":"2023-10-11T16:31:42.289168+0000","last_undegraded":"2023-10-11T16:31:42.289168+0000","last_fullsized":"2023-10-11T16:31:42.289168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:21:13.844885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.093645+0000","last_change":"2023-10-11T16:30:11.665412+0000","last_active":"2023-10-11T16:32:10.093645+0000","last_peered":"2023-10-11T16:32:10.093645+0000","last_clean":"2023-10-11T16:32:10.093645+0000","last_became_active":"2023-10-11T16:30:11.665087+0000","last_became_peered":"2023-10-11T16:30:11.665087+0000","last_unstale":"2023-10-11T16:32:10.093645+0000","last_undegraded":"2023-10-11T16:32:10.093645+0000","last_fullsized":"2023-10-11T16:32:10.093645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:07:18.219144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.288217+0000","last_change":"2023-10-11T16:30:16.126202+0000","last_active":"2023-10-11T16:31:42.288217+0000","last_peered":"2023-10-11T16:31:42.288217+0000","last_clean":"2023-10-11T16:31:42.288217+0000","last_became_active":"2023-10-11T16:30:16.125801+0000","last_became_peered":"2023-10-11T16:30:16.125801+0000","last_unstale":"2023-10-11T16:31:42.288217+0000","last_undegraded":"2023-10-11T16:31:42.288217+0000","last_fullsized":"2023-10-11T16:31:42.288217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:15.094322+0000","last_clean_scrub_stamp":"2023-10-11T16:30:15.094322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:07:28.047642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:31:42.285779+0000","last_change":"2023-10-11T16:30:11.122852+0000","last_active":"2023-10-11T16:31:42.285779+0000","last_peered":"2023-10-11T16:31:42.285779+0000","last_clean":"2023-10-11T16:31:42.285779+0000","last_became_active":"2023-10-11T16:30:11.122494+0000","last_became_peered":"2023-10-11T16:30:11.122494+0000","last_unstale":"2023-10-11T16:31:42.285779+0000","last_undegraded":"2023-10-11T16:31:42.285779+0000","last_fullsized":"2023-10-11T16:31:42.285779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:10.084575+0000","last_clean_scrub_stamp":"2023-10-11T16:30:10.084575+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:43:48.783409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.088126+0000","last_change":"2023-10-11T16:30:11.644720+0000","last_active":"2023-10-11T16:32:10.088126+0000","last_peered":"2023-10-11T16:32:10.088126+0000","last_clean":"2023-10-11T16:32:10.088126+0000","last_became_active":"2023-10-11T16:30:11.644407+0000","last_became_peered":"2023-10-11T16:30:11.644407+0000","last_unstale":"2023-10-11T16:32:10.088126+0000","last_undegraded":"2023-10-11T16:32:10.088126+0000","last_fullsized":"2023-10-11T16:32:10.088126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T04:29:09.209568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.085185+0000","last_change":"2023-10-11T16:30:11.671012+0000","last_active":"2023-10-11T16:32:10.085185+0000","last_peered":"2023-10-11T16:32:10.085185+0000","last_clean":"2023-10-11T16:32:10.085185+0000","last_became_active":"2023-10-11T16:30:11.670717+0000","last_became_peered":"2023-10-11T16:30:11.670717+0000","last_unstale":"2023-10-11T16:32:10.085185+0000","last_undegraded":"2023-10-11T16:32:10.085185+0000","last_fullsized":"2023-10-11T16:32:10.085185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:00:29.421748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.089406+0000","last_change":"2023-10-11T16:30:11.660272+0000","last_active":"2023-10-11T16:32:10.089406+0000","last_peered":"2023-10-11T16:32:10.089406+0000","last_clean":"2023-10-11T16:32:10.089406+0000","last_became_active":"2023-10-11T16:30:11.660062+0000","last_became_peered":"2023-10-11T16:30:11.660062+0000","last_unstale":"2023-10-11T16:32:10.089406+0000","last_undegraded":"2023-10-11T16:32:10.089406+0000","last_fullsized":"2023-10-11T16:32:10.089406+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:41:24.461888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.104714+0000","last_change":"2023-10-11T16:30:11.669112+0000","last_active":"2023-10-11T16:32:10.104714+0000","last_peered":"2023-10-11T16:32:10.104714+0000","last_clean":"2023-10-11T16:32:10.104714+0000","last_became_active":"2023-10-11T16:30:11.668181+0000","last_became_peered":"2023-10-11T16:30:11.668181+0000","last_unstale":"2023-10-11T16:32:10.104714+0000","last_undegraded":"2023-10-11T16:32:10.104714+0000","last_fullsized":"2023-10-11T16:32:10.104714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T22:25:45.955319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.100277+0000","last_change":"2023-10-11T16:30:11.666695+0000","last_active":"2023-10-11T16:32:10.100277+0000","last_peered":"2023-10-11T16:32:10.100277+0000","last_clean":"2023-10-11T16:32:10.100277+0000","last_became_active":"2023-10-11T16:30:11.661704+0000","last_became_peered":"2023-10-11T16:30:11.661704+0000","last_unstale":"2023-10-11T16:32:10.100277+0000","last_undegraded":"2023-10-11T16:32:10.100277+0000","last_fullsized":"2023-10-11T16:32:10.100277+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T18:50:42.892510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.087129+0000","last_change":"2023-10-11T16:30:11.657773+0000","last_active":"2023-10-11T16:32:10.087129+0000","last_peered":"2023-10-11T16:32:10.087129+0000","last_clean":"2023-10-11T16:32:10.087129+0000","last_became_active":"2023-10-11T16:30:11.657456+0000","last_became_peered":"2023-10-11T16:30:11.657456+0000","last_unstale":"2023-10-11T16:32:10.087129+0000","last_undegraded":"2023-10-11T16:32:10.087129+0000","last_fullsized":"2023-10-11T16:32:10.087129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T00:21:58.187591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.081932+0000","last_change":"2023-10-11T16:30:11.671077+0000","last_active":"2023-10-11T16:32:10.081932+0000","last_peered":"2023-10-11T16:32:10.081932+0000","last_clean":"2023-10-11T16:32:10.081932+0000","last_became_active":"2023-10-11T16:30:11.670595+0000","last_became_peered":"2023-10-11T16:30:11.670595+0000","last_unstale":"2023-10-11T16:32:10.081932+0000","last_undegraded":"2023-10-11T16:32:10.081932+0000","last_fullsized":"2023-10-11T16:32:10.081932+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T01:38:42.257752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.082937+0000","last_change":"2023-10-11T16:30:11.652771+0000","last_active":"2023-10-11T16:32:10.082937+0000","last_peered":"2023-10-11T16:32:10.082937+0000","last_clean":"2023-10-11T16:32:10.082937+0000","last_became_active":"2023-10-11T16:30:11.652461+0000","last_became_peered":"2023-10-11T16:30:11.652461+0000","last_unstale":"2023-10-11T16:32:10.082937+0000","last_undegraded":"2023-10-11T16:32:10.082937+0000","last_fullsized":"2023-10-11T16:32:10.082937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T21:53:03.240456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.098142+0000","last_change":"2023-10-11T16:30:11.671013+0000","last_active":"2023-10-11T16:32:10.098142+0000","last_peered":"2023-10-11T16:32:10.098142+0000","last_clean":"2023-10-11T16:32:10.098142+0000","last_became_active":"2023-10-11T16:30:11.670685+0000","last_became_peered":"2023-10-11T16:30:11.670685+0000","last_unstale":"2023-10-11T16:32:10.098142+0000","last_undegraded":"2023-10-11T16:32:10.098142+0000","last_fullsized":"2023-10-11T16:32:10.098142+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T02:14:19.917814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.083854+0000","last_change":"2023-10-11T16:30:11.671942+0000","last_active":"2023-10-11T16:32:10.083854+0000","last_peered":"2023-10-11T16:32:10.083854+0000","last_clean":"2023-10-11T16:32:10.083854+0000","last_became_active":"2023-10-11T16:30:11.671654+0000","last_became_peered":"2023-10-11T16:30:11.671654+0000","last_unstale":"2023-10-11T16:32:10.083854+0000","last_undegraded":"2023-10-11T16:32:10.083854+0000","last_fullsized":"2023-10-11T16:32:10.083854+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:59:39.214839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.097729+0000","last_change":"2023-10-11T16:30:11.659143+0000","last_active":"2023-10-11T16:32:10.097729+0000","last_peered":"2023-10-11T16:32:10.097729+0000","last_clean":"2023-10-11T16:32:10.097729+0000","last_became_active":"2023-10-11T16:30:11.658844+0000","last_became_peered":"2023-10-11T16:30:11.658844+0000","last_unstale":"2023-10-11T16:32:10.097729+0000","last_undegraded":"2023-10-11T16:32:10.097729+0000","last_fullsized":"2023-10-11T16:32:10.097729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T20:47:58.913211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.105194+0000","last_change":"2023-10-11T16:30:11.669029+0000","last_active":"2023-10-11T16:32:10.105194+0000","last_peered":"2023-10-11T16:32:10.105194+0000","last_clean":"2023-10-11T16:32:10.105194+0000","last_became_active":"2023-10-11T16:30:11.668762+0000","last_became_peered":"2023-10-11T16:30:11.668762+0000","last_unstale":"2023-10-11T16:32:10.105194+0000","last_undegraded":"2023-10-11T16:32:10.105194+0000","last_fullsized":"2023-10-11T16:32:10.105194+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-13T03:54:53.313375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.095040+0000","last_change":"2023-10-11T16:30:11.648240+0000","last_active":"2023-10-11T16:32:10.095040+0000","last_peered":"2023-10-11T16:32:10.095040+0000","last_clean":"2023-10-11T16:32:10.095040+0000","last_became_active":"2023-10-11T16:30:11.647870+0000","last_became_peered":"2023-10-11T16:30:11.647870+0000","last_unstale":"2023-10-11T16:32:10.095040+0000","last_undegraded":"2023-10-11T16:32:10.095040+0000","last_fullsized":"2023-10-11T16:32:10.095040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:18:45.218633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-11T16:32:10.080249+0000","last_change":"2023-10-11T16:31:42.463155+0000","last_active":"2023-10-11T16:32:10.080249+0000","last_peered":"2023-10-11T16:32:10.080249+0000","last_clean":"2023-10-11T16:32:10.080249+0000","last_became_active":"2023-10-11T16:31:42.462678+0000","last_became_peered":"2023-10-11T16:31:42.462678+0000","last_unstale":"2023-10-11T16:32:10.080249+0000","last_undegraded":"2023-10-11T16:32:10.080249+0000","last_fullsized":"2023-10-11T16:32:10.080249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T16:30:11.090383+0000","last_clean_scrub_stamp":"2023-10-11T16:30:11.090383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T17:50:11.499350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":246962,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80318,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":650,"num_read_kb":411,"num_write":174,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":39701,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":240954,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449126,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":1003520,"data_stored":508258,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56000000000000005,"15min":0.56200000000000006},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80700000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.443},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59099999999999997,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":1.204,"15min":1.204},"last":0.41999999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61499999999999999},"min":{"1min":0.42399999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72799999999999998,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73699999999999999,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.52500000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.34799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.33000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.57099999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.39100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.46999999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79200000000000004,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70399999999999996,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60499999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64600000000000002,"15min":0.66000000000000003},"min":{"1min":0.39400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48499999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.50600000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76500000000000001,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.70799999999999996},"min":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.006,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":3.75}]},{"osd":7,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83199999999999996,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.621},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.73599999999999999,"15min":0.74299999999999999},"min":{"1min":0.54500000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.86099999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.64000000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:32:14 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68799999999999994,"15min":0.68899999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.871,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.753},"min":{"1min":0.52000000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96799999999999997,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.69499999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.78700000000000003,"15min":0.76900000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.96,"5min":2.96,"15min":2.96},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.79700000000000004,"15min":0.80100000000000005},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.86499999999999999,"5min":1.286,"15min":1.286},"last":0.71999999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:31:24 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.86599999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76500000000000001,"15min":0.77100000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92100000000000004,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.65600000000000003}]}]},{"osd":10,"up_from":71,"seq":304942678059,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6696,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738456,"statfs":{"total":95995035648,"available":95988178944,"internally_reserved":0,"allocated":1146880,"data_stored":1161223,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.496},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.33200000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.59799999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.49299999999999999,"15min":0.51100000000000001},"min":{"1min":0.318,"5min":0.314,"15min":0.314},"max":{"1min":0.56000000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52300000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.60999999999999999,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.747,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64900000000000002,"15min":0.66300000000000003},"min":{"1min":0.41299999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71499999999999997,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70099999999999996,"5min":0.995,"15min":0.995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.64300000000000002,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.745,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.62}]},{"osd":4,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.38300000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69299999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.438},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55100000000000005,"15min":0.55600000000000005},"min":{"1min":0.36899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76000000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.47899999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65800000000000003,"15min":0.66500000000000004},"min":{"1min":0.443,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.71099999999999997,"15min":0.72099999999999997},"min":{"1min":0.48699999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.76200000000000001,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.72099999999999997,"15min":0.74099999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.84499999999999997,"5min":1.085,"15min":1.085},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.42199999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82499999999999996,"5min":1.03,"15min":1.03},"last":0.53900000000000003}]},{"osd":7,"last update":"Wed Oct 11 16:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51200000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.45700000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90200000000000002,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.83999999999999997,"15min":0.877},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.84799999999999998,"5min":2.46,"15min":2.46},"last":0.751}]},{"osd":9,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.79200000000000004,"15min":0.81699999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.85099999999999998,"5min":1.286,"15min":1.286},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.79900000000000004,"15min":0.81499999999999995},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92600000000000005,"5min":1.347,"15min":1.347},"last":0.66000000000000003}]},{"osd":11,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.745,"15min":0.745},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.876,"5min":1.073,"15min":1.073},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90100000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.83399999999999996}]}]},{"osd":9,"up_from":66,"seq":283467841582,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":1015808,"data_stored":511986,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.496}]},{"osd":1,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":3.3719999999999999,"5min":3.3719999999999999,"15min":3.3719999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69699999999999995,"15min":0.71099999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80000000000000004,"15min":0.76700000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.198,"5min":7.056,"15min":7.056},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.80300000000000005,"15min":0.78200000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.2250000000000001,"5min":7.1529999999999996,"15min":7.1529999999999996},"last":0.44800000000000001}]},{"osd":3,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69499999999999995,"15min":0.72799999999999998},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.379,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.758,"5min":2.758,"15min":2.758},"last":0.63100000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.286,"5min":2.286,"15min":2.286},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77800000000000002,"15min":0.79600000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.58299999999999996}]},{"osd":5,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.81699999999999995,"15min":0.84499999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.83099999999999996,"15min":0.84999999999999998},"min":{"1min":0.48299999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.6480000000000001,"5min":2.6480000000000001,"15min":2.6480000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.61299999999999999}]},{"osd":7,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86699999999999999,"15min":0.89500000000000002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88200000000000001,"15min":0.89100000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6789999999999998,"5min":2.6789999999999998,"15min":2.6789999999999998},"last":0.51000000000000001}]},{"osd":8,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.96599999999999997,"15min":0.95299999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":3.387,"5min":7.1070000000000002,"15min":7.1070000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.96699999999999997,"15min":0.96499999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.4860000000000002,"5min":7.1260000000000003,"15min":7.1260000000000003},"last":0.77400000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.84899999999999998,"15min":0.88300000000000001},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1699999999999999,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.92900000000000005,"15min":0.95899999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.3300000000000001,"5min":2.706,"15min":2.706},"last":0.75600000000000001}]},{"osd":11,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.88300000000000001,"15min":0.90400000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.995,"5min":2.46,"15min":2.46},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.95099999999999996,"15min":0.98499999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.79600000000000004}]}]},{"osd":8,"up_from":58,"seq":249108103220,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6556,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738596,"statfs":{"total":95995035648,"available":95988322304,"internally_reserved":0,"allocated":1003520,"data_stored":507457,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91600000000000004,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":0.996,"15min":0.996},"last":0.45200000000000001}]},{"osd":1,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82599999999999996,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.63}]},{"osd":2,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.34399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.44600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.77400000000000002,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.629,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0649999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.57699999999999996}]},{"osd":4,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82599999999999996,"5min":1.028,"15min":1.028},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.45200000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90900000000000003,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.36599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82499999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55900000000000005,"15min":0.54500000000000004},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61299999999999999}]},{"osd":6,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.47499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.179,"15min":1.179},"last":0.874},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.499,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.751}]},{"osd":7,"last update":"Wed Oct 11 16:31:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.68899999999999995,"15min":0.68100000000000005},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.95299999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.69099999999999995,"15min":0.68300000000000005},"min":{"1min":0.56599999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.99399999999999999,"5min":1.101,"15min":1.101},"last":0.73499999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.73299999999999998,"15min":0.69899999999999995},"min":{"1min":0.69799999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.74299999999999999,"15min":0.72499999999999998},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.83899999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.74399999999999999,"15min":0.71199999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0449999999999999,"5min":1.069,"15min":1.069},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.72199999999999998,"15min":0.69599999999999995},"min":{"1min":0.57699999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0549999999999999,"5min":1.083,"15min":1.083},"last":0.80300000000000005}]},{"osd":11,"last update":"Wed Oct 11 16:32:14 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.51900000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.012,"5min":1.048,"15min":1.048},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.78800000000000003,"15min":0.77800000000000002},"min":{"1min":0.59599999999999997,"5min":0.436,"15min":0.436},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.81699999999999995}]}]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6744,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738408,"statfs":{"total":95995035648,"available":95988129792,"internally_reserved":0,"allocated":1064960,"data_stored":1113105,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79700000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.46500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.70799999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.66100000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.45400000000000001},"min":{"1min":0.29199999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63900000000000001,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.374}]},{"osd":2,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.38500000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77700000000000002,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.57599999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.80700000000000005,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.628}]},{"osd":3,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58099999999999996,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.621,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80900000000000005,"5min":1.103,"15min":1.103},"last":0.51500000000000001}]},{"osd":4,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.71099999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.501,"15min":0.5},"min":{"1min":0.29399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45000000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.45700000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91800000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66100000000000003,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86399999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68700000000000006}]},{"osd":6,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.53900000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.083,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.67900000000000005,"15min":0.66300000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.71899999999999997}]},{"osd":8,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71199999999999997,"15min":0.70399999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98399999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.72599999999999998,"15min":0.72799999999999998},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77800000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.77700000000000002,"15min":0.77600000000000002},"min":{"1min":0.63500000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.079,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.77000000000000002,"15min":0.77300000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.60499999999999998}]},{"osd":10,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70699999999999996,"15min":0.69499999999999995},"min":{"1min":0.48499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.054,"5min":1.071,"15min":1.071},"last":0.751},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73099999999999998},"min":{"1min":0.47699999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.99199999999999999,"5min":1.099,"15min":1.099},"last":0.79500000000000004}]},{"osd":11,"last update":"Wed Oct 11 16:31:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.82899999999999996}]}]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6544,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738608,"statfs":{"total":95995035648,"available":95988334592,"internally_reserved":0,"allocated":991232,"data_stored":503066,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.47899999999999998,"15min":0.44500000000000001},"min":{"1min":0.32700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.59399999999999997,"15min":0.54100000000000004},"min":{"1min":0.39800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.73099999999999998}]},{"osd":1,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.60399999999999998,"15min":0.56999999999999995},"min":{"1min":0.435,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.63500000000000001,"15min":0.59499999999999997},"min":{"1min":0.44,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.71799999999999997,"15min":0.67700000000000005},"min":{"1min":0.48699999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.48699999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.47799999999999998}]},{"osd":3,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.39100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.374,"5min":0.317,"15min":0.317},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55900000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.73699999999999999,"15min":0.70699999999999996},"min":{"1min":0.52000000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.69299999999999995,"15min":0.66000000000000003},"min":{"1min":0.52800000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.64200000000000002}]},{"osd":5,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70899999999999996,"15min":0.66900000000000004},"min":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.79800000000000004,"15min":0.75800000000000001},"min":{"1min":0.65000000000000002,"5min":0.434,"15min":0.434},"max":{"1min":2.4009999999999998,"5min":2.4009999999999998,"15min":2.4009999999999998},"last":0.79000000000000004}]},{"osd":7,"last update":"Wed Oct 11 16:31:59 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.79700000000000004,"15min":0.76500000000000001},"min":{"1min":0.53700000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":2.4780000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.73999999999999999,"15min":0.72399999999999998},"min":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.67600000000000005}]},{"osd":8,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.752},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.875,"15min":0.874},"min":{"1min":0.60099999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.77500000000000002}]},{"osd":9,"last update":"Wed Oct 11 16:31:46 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84599999999999997,"15min":0.872},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.89500000000000002,"15min":0.90700000000000003},"min":{"1min":0.57899999999999996,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.3719999999999999,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":0.80600000000000005}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.66200000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.871,"15min":0.875},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.71499999999999997}]},{"osd":11,"last update":"Wed Oct 11 16:31:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.5640000000000001,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":0.73899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":569576,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55100000000000005,"15min":0.61299999999999999},"min":{"1min":0.39200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":1.8520000000000001},"last":0.433},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.52400000000000002,"15min":0.53800000000000003},"min":{"1min":0.41499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83099999999999996,"5min":0.872,"15min":0.89800000000000002},"last":0.40999999999999998}]},{"osd":2,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.57299999999999995,"15min":0.52300000000000002},"min":{"1min":0.51500000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.56599999999999995,"15min":0.52600000000000002},"min":{"1min":0.495,"5min":0.32900000000000001,"15min":0.307},"max":{"1min":0.89300000000000002,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61299999999999999,"15min":0.57999999999999996},"min":{"1min":0.48899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.88800000000000001,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.60799999999999998,"15min":0.57199999999999995},"min":{"1min":0.53000000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53700000000000003}]},{"osd":4,"last update":"Wed Oct 11 16:32:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.47399999999999998,"15min":0.42699999999999999},"min":{"1min":0.34300000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.752,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.54300000000000004,"15min":0.46600000000000003},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.91000000000000003,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.51100000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.59599999999999997,"15min":0.51200000000000001},"min":{"1min":0.505,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.64900000000000002,"15min":0.55400000000000005},"min":{"1min":0.58199999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.85999999999999999,"5min":0.997,"15min":0.997},"last":0.80800000000000005}]},{"osd":6,"last update":"Wed Oct 11 16:31:38 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.67900000000000005,"15min":0.627},"min":{"1min":0.60199999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":2.3580000000000001},"last":0.747},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.65900000000000003,"15min":0.63500000000000001},"min":{"1min":0.59499999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.58299999999999996}]},{"osd":7,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.57999999999999996,"15min":0.51600000000000001},"min":{"1min":0.47699999999999998,"5min":0.253,"15min":0.253},"max":{"1min":1.004,"5min":1.127,"15min":1.127},"last":0.621},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.66900000000000004,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90700000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.69799999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.73199999999999998,"15min":0.67800000000000005},"min":{"1min":0.65500000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.70499999999999996,"15min":0.64500000000000002},"min":{"1min":0.67700000000000005,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.99199999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.68100000000000005}]},{"osd":9,"last update":"Wed Oct 11 16:31:47 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.746,"15min":0.72199999999999998},"min":{"1min":0.70699999999999996,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97099999999999997,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.79200000000000004,"15min":0.77000000000000002},"min":{"1min":0.67000000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.98699999999999999,"5min":1.179,"15min":1.179},"last":0.82799999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.71299999999999997,"15min":0.67800000000000005},"min":{"1min":0.48999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96099999999999997,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.76000000000000001,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93799999999999994,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.76600000000000001}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.84199999999999997,"15min":0.83699999999999997},"min":{"1min":0.629,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.024,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.86399999999999999,"15min":0.86199999999999999},"min":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.97599999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.84299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7464,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737688,"statfs":{"total":95995035648,"available":95987392512,"internally_reserved":0,"allocated":1015808,"data_stored":504454,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.41599999999999998,"5min":0.33100000000000002,"15min":0.309},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.60999999999999999,"15min":0.54400000000000004},"min":{"1min":0.54500000000000004,"5min":0.38500000000000001,"15min":0.33800000000000002},"max":{"1min":0.82499999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Oct 11 16:32:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.65300000000000002,"15min":0.59499999999999997},"min":{"1min":0.46200000000000002,"5min":0.38500000000000001,"15min":0.35899999999999999},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66700000000000004,"15min":0.61299999999999999},"min":{"1min":0.54300000000000004,"5min":0.42599999999999999,"15min":0.40500000000000003},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.69399999999999995}]},{"osd":3,"last update":"Wed Oct 11 16:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.42299999999999999}]},{"osd":4,"last update":"Wed Oct 11 16:31:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61599999999999999,"15min":0.56200000000000006},"min":{"1min":0.41499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64800000000000002,"15min":0.59699999999999998},"min":{"1min":0.47399999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81899999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76400000000000001}]},{"osd":5,"last update":"Wed Oct 11 16:32:19 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64300000000000002,"15min":0.61299999999999999},"min":{"1min":0.504,"5min":0.377,"15min":0.377},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.68200000000000005,"15min":0.64600000000000002},"min":{"1min":0.5,"5min":0.436,"15min":0.436},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.80000000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.626},"min":{"1min":0.441,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.63,"15min":0.60399999999999998},"min":{"1min":0.374,"5min":0.37,"15min":0.37},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Wed Oct 11 16:32:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.76800000000000002,"15min":0.77300000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.56299999999999994,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0840000000000001,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.77300000000000002,"15min":0.753},"min":{"1min":0.63500000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":3.073,"5min":3.073,"15min":3.073},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75800000000000001,"15min":0.76200000000000001},"min":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.73199999999999998}]},{"osd":9,"last update":"Wed Oct 11 16:31:50 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.80600000000000005,"15min":0.83899999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.80400000000000005,"15min":0.82899999999999996},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.52100000000000002}]},{"osd":10,"last update":"Wed Oct 11 16:32:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.81399999999999995,"15min":0.81100000000000005},"min":{"1min":0.69499999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.82299999999999995,"15min":0.82599999999999996},"min":{"1min":0.624,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63500000000000001}]},{"osd":11,"last update":"Wed Oct 11 16:31:28 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.81000000000000005,"15min":0.79900000000000004},"min":{"1min":0.64100000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.70399999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.78700000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7592,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737560,"statfs":{"total":95995035648,"available":95987261440,"internally_reserved":0,"allocated":1015808,"data_stored":514151,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55100000000000005,"15min":0.52700000000000002},"min":{"1min":0.39500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.96099999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.42899999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":0.91800000000000004,"15min":1.0549999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59399999999999997,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.29199999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.91200000000000003},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.59299999999999997,"15min":0.52700000000000002},"min":{"1min":0.40799999999999997,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57599999999999996,"15min":0.54500000000000004},"min":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.63,"15min":0.57499999999999996},"min":{"1min":0.51100000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.68500000000000005}]},{"osd":4,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.70299999999999996,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85699999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69899999999999995,"15min":0.70199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.89400000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.70399999999999996}]},{"osd":5,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51900000000000002,"15min":0.47399999999999998},"min":{"1min":0.39000000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.88800000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.52000000000000002,"15min":0.48799999999999999},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85099999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.54000000000000004}]},{"osd":6,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.67300000000000004,"15min":0.63300000000000001},"min":{"1min":0.51900000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.66900000000000004,"15min":0.64300000000000002},"min":{"1min":0.47099999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82999999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.64300000000000002}]},{"osd":7,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.498,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93899999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52400000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94799999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.77400000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.64400000000000002,"15min":0.61499999999999999},"min":{"1min":0.48299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.95299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.64800000000000002,"15min":0.623},"min":{"1min":0.49199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93600000000000005,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.747}]},{"osd":9,"last update":"Wed Oct 11 16:31:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.74099999999999999,"15min":0.71599999999999997},"min":{"1min":0.66300000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.746,"15min":0.72399999999999998},"min":{"1min":0.61699999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.93100000000000005,"5min":1.024,"15min":1.024},"last":0.80000000000000004}]},{"osd":10,"last update":"Wed Oct 11 16:32:08 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.76300000000000001,"15min":0.74199999999999999},"min":{"1min":0.626,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94499999999999995,"5min":0.997,"15min":0.997},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.80300000000000005,"15min":0.77800000000000002},"min":{"1min":0.626,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0029999999999999,"5min":1.038,"15min":1.038},"last":0.879}]},{"osd":11,"last update":"Wed Oct 11 16:32:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82999999999999996,"15min":0.83199999999999996},"min":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.85299999999999998,"15min":0.85499999999999998},"min":{"1min":0.66400000000000003,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.90800000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6808,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738344,"statfs":{"total":95995035648,"available":95988064256,"internally_reserved":0,"allocated":999424,"data_stored":502929,"data_compressed":23488,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41399999999999998,"15min":0.38400000000000001},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.43099999999999999,"15min":0.40899999999999997},"min":{"1min":0.33100000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.56999999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.62,"15min":0.57499999999999996},"min":{"1min":0.47799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79600000000000004,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58999999999999997,"15min":0.55600000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77000000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.68999999999999995}]},{"osd":2,"last update":"Wed Oct 11 16:31:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.64700000000000002,"15min":0.60899999999999999},"min":{"1min":0.439,"5min":0.379,"15min":0.379},"max":{"1min":0.91000000000000003,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.65000000000000002,"15min":0.59399999999999997},"min":{"1min":0.48999999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.80100000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Wed Oct 11 16:31:54 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.64000000000000001},"min":{"1min":0.54200000000000004,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82099999999999995,"5min":1.101,"15min":1.101},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71499999999999997,"15min":0.68700000000000006},"min":{"1min":0.624,"5min":0.498,"15min":0.498},"max":{"1min":0.83799999999999997,"5min":1.194,"15min":1.194},"last":0.54300000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.621,"15min":0.63100000000000001},"min":{"1min":0.46300000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88600000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92900000000000005,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Wed Oct 11 16:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61699999999999999,"15min":0.60999999999999999},"min":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.78800000000000003,"5min":1.268,"15min":1.268},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.749,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.626}]},{"osd":7,"last update":"Wed Oct 11 16:32:03 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.67700000000000005,"15min":0.68300000000000005},"min":{"1min":0.57299999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.85699999999999998,"5min":1.119,"15min":1.119},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.623,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95799999999999996,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.745}]},{"osd":8,"last update":"Wed Oct 11 16:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.627},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69099999999999995}]},{"osd":9,"last update":"Wed Oct 11 16:31:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.77600000000000002,"15min":0.80200000000000005},"min":{"1min":0.55200000000000005,"5min":0.496,"15min":0.496},"max":{"1min":0.90800000000000003,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.77600000000000002,"15min":0.79900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.876,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.69799999999999995}]},{"osd":10,"last update":"Wed Oct 11 16:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.81200000000000006,"15min":0.82499999999999996},"min":{"1min":0.66700000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.028,"5min":1.137,"15min":1.137},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.82199999999999995,"15min":0.83199999999999996},"min":{"1min":0.68400000000000005,"5min":0.627,"15min":0.627},"max":{"1min":0.876,"5min":1.165,"15min":1.165},"last":0.64400000000000002}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78700000000000003,"15min":0.78800000000000003},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":0.90900000000000003,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.72099999999999997,"15min":0.71799999999999997},"min":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.71199999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6796,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738356,"statfs":{"total":95995035648,"available":95988076544,"internally_reserved":0,"allocated":1052672,"data_stored":574017,"data_compressed":30188,"data_compressed_allocated":40960,"data_compressed_original":102400,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:29:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.497,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78800000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.38500000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.80800000000000005,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.378}]},{"osd":1,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39000000000000001,"15min":0.378},"min":{"1min":0.28299999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51500000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39000000000000001,"15min":0.38600000000000001},"min":{"1min":0.30299999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48999999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59799999999999998,"15min":0.56000000000000005},"min":{"1min":0.41899999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81599999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52700000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75700000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.52700000000000002}]},{"osd":3,"last update":"Wed Oct 11 16:31:56 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":0.69099999999999995,"15min":0.58399999999999996},"min":{"1min":0.433,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":6.9900000000000002,"5min":6.9900000000000002,"15min":6.9900000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.234,"5min":0.69699999999999995,"15min":0.57599999999999996},"min":{"1min":0.46800000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.0129999999999999,"5min":7.0129999999999999,"15min":7.0129999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67600000000000005,"15min":0.73799999999999999},"min":{"1min":0.52100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78500000000000003,"5min":4.5,"15min":4.5},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.71599999999999997},"min":{"1min":0.51500000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76700000000000002,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.57699999999999996}]},{"osd":6,"last update":"Wed Oct 11 16:31:39 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.70799999999999996,"15min":0.65500000000000003},"min":{"1min":0.52300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.9420000000000002,"5min":6.9420000000000002,"15min":6.9420000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.58199999999999996},"min":{"1min":0.52400000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79200000000000004,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56200000000000006}]},{"osd":7,"last update":"Wed Oct 11 16:32:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.373,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55400000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58099999999999996,"15min":0.54600000000000004},"min":{"1min":0.42799999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79200000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.40100000000000002}]},{"osd":8,"last update":"Wed Oct 11 16:31:23 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63500000000000001,"15min":0.60299999999999998},"min":{"1min":0.48999999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89100000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40799999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76800000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.622}]},{"osd":9,"last update":"Wed Oct 11 16:31:42 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.64800000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.84899999999999998,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.454,"5min":0.87,"15min":0.76800000000000002},"min":{"1min":0.63800000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":6.968,"5min":6.968,"15min":6.968},"last":0.71399999999999997}]},{"osd":10,"last update":"Wed Oct 11 16:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.46200000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.89200000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.73899999999999999,"15min":0.72299999999999998},"min":{"1min":0.52900000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.94099999999999995,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.59999999999999998}]},{"osd":11,"last update":"Wed Oct 11 16:32:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.68300000000000005,"15min":0.68799999999999994},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.73899999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1077248,"data_stored":1109323,"data_compressed":59238,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.53100000000000003},"min":{"1min":0.434,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55600000000000005,"15min":0.54000000000000004},"min":{"1min":0.309,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.52800000000000002}]},{"osd":1,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57199999999999995,"15min":0.53700000000000003},"min":{"1min":0.375,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.874,"15min":0.874},"last":0.62},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57999999999999996,"15min":0.55800000000000005},"min":{"1min":0.374,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77100000000000002,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55000000000000004}]},{"osd":2,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.439,"15min":0.40699999999999997},"min":{"1min":0.34300000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.59799999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.46600000000000003,"15min":0.42599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65100000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.309}]},{"osd":3,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.40100000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72599999999999998,"5min":1.089,"15min":1.089},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.60299999999999998,"15min":0.56599999999999995},"min":{"1min":0.47599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.79700000000000004}]},{"osd":4,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.621,"15min":0.57899999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82399999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.62,"15min":0.58699999999999997},"min":{"1min":0.374,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.754,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Wed Oct 11 16:31:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65600000000000003,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.629},"min":{"1min":0.52400000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.753}]},{"osd":7,"last update":"Wed Oct 11 16:32:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69599999999999995,"15min":0.71099999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88800000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70499999999999996,"15min":0.69999999999999996},"min":{"1min":0.45200000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57699999999999996}]},{"osd":8,"last update":"Wed Oct 11 16:31:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.68600000000000005,"15min":0.68999999999999995},"min":{"1min":0.502,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83699999999999997,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65600000000000003,"15min":0.65100000000000002},"min":{"1min":0.47899999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.42199999999999999}]},{"osd":9,"last update":"Wed Oct 11 16:31:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.751,"15min":0.751},"min":{"1min":0.59699999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90900000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.77300000000000002,"15min":0.76800000000000002},"min":{"1min":0.57699999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.83499999999999996}]},{"osd":10,"last update":"Wed Oct 11 16:32:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.72899999999999998,"15min":0.71999999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89700000000000002,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.745,"15min":0.73299999999999998},"min":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34106,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35750,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":66931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":11175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":66945,"data_compressed":6700,"data_compressed_allocated":8192,"data_compressed_original":36864,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":11883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":11569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":67879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-11T16:32:24.998 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:32:25.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 -- 172.21.15.111:0/913448951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c1002b0 msgr2=0x7f1f6c1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 --2- 172.21.15.111:0/913448951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c1002b0 0x7f1f6c1006c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009990 tx=0x7f1f5c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 -- 172.21.15.111:0/913448951 shutdown_connections 2023-10-11T16:32:25.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 --2- 172.21.15.111:0/913448951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f6c101e30 0x7f1f6c1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 --2- 172.21.15.111:0/913448951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c1002b0 0x7f1f6c1006c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 --2- 172.21.15.111:0/913448951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f6c1014a0 0x7f1f6c1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 -- 172.21.15.111:0/913448951 >> 172.21.15.111:0/913448951 conn(0x7f1f6c0fb890 msgr2=0x7f1f6c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:25.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 -- 172.21.15.111:0/913448951 shutdown_connections 2023-10-11T16:32:25.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.226+0000 7f1f723ff700 1 -- 172.21.15.111:0/913448951 wait complete. 2023-10-11T16:32:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.227+0000 7f1f723ff700 1 Processor -- start 2023-10-11T16:32:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.227+0000 7f1f723ff700 1 -- start start 2023-10-11T16:32:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.227+0000 7f1f723ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f6c1002b0 0x7f1f6c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.227+0000 7f1f723ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f6c1014a0 0x7f1f6c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.227+0000 7f1f723ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c101e30 0x7f1f6c19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.227+0000 7f1f723ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c19fd60 con 0x7f1f6c101e30 2023-10-11T16:32:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.227+0000 7f1f723ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c19fea0 con 0x7f1f6c1014a0 2023-10-11T16:32:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.227+0000 7f1f723ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c1a0030 con 0x7f1f6c1002b0 2023-10-11T16:32:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c101e30 0x7f1f6c19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f6b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f6c1014a0 0x7f1f6c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c101e30 0x7f1f6c19f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47130/0 (socket says 172.21.15.111:47130) 2023-10-11T16:32:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f6b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f6c1014a0 0x7f1f6c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46806/0 (socket says 172.21.15.111:46806) 2023-10-11T16:32:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 -- 172.21.15.111:0/1277731914 learned_addr learned my addr 172.21.15.111:0/1277731914 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f6bfff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f6c1002b0 0x7f1f6c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:25.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 -- 172.21.15.111:0/1277731914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f6c1002b0 msgr2=0x7f1f6c1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f6c1002b0 0x7f1f6c1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 -- 172.21.15.111:0/1277731914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f6c1014a0 msgr2=0x7f1f6c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f6c1014a0 0x7f1f6c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 -- 172.21.15.111:0/1277731914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c0095f0 con 0x7f1f6c101e30 2023-10-11T16:32:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f7099c700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c101e30 0x7f1f6c19f790 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400ef30 tx=0x7f1f6400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.228+0000 7f1f697fa700 1 -- 172.21.15.111:0/1277731914 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f64018070 con 0x7f1f6c101e30 2023-10-11T16:32:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.229+0000 7f1f697fa700 1 -- 172.21.15.111:0/1277731914 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f6400f040 con 0x7f1f6c101e30 2023-10-11T16:32:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.229+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c10bc80 con 0x7f1f6c101e30 2023-10-11T16:32:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.229+0000 7f1f697fa700 1 -- 172.21.15.111:0/1277731914 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6400cd00 con 0x7f1f6c101e30 2023-10-11T16:32:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.229+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c10baf0 con 0x7f1f6c101e30 2023-10-11T16:32:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.230+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c192910 con 0x7f1f6c101e30 2023-10-11T16:32:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.232+0000 7f1f697fa700 1 -- 172.21.15.111:0/1277731914 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f1f64003a00 con 0x7f1f6c101e30 2023-10-11T16:32:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.233+0000 7f1f697fa700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1f540a1150 0x7f1f540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.233+0000 7f1f697fa700 1 -- 172.21.15.111:0/1277731914 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f1f640cd150 con 0x7f1f6c101e30 2023-10-11T16:32:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.233+0000 7f1f6bfff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1f540a1150 0x7f1f540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.234+0000 7f1f6bfff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1f540a1150 0x7f1f540a3600 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00b2e0 tx=0x7f1f5c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.235+0000 7f1f697fa700 1 -- 172.21.15.111:0/1277731914 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1f64096500 con 0x7f1f6c101e30 2023-10-11T16:32:25.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:25 smithi111 ceph-mon[98527]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:25.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:25 smithi111 ceph-mon[98527]: pgmap v343: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:25.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:25 smithi111 ceph-mon[98527]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:25.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.435+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f6c02cfe0 con 0x7f1f6c101e30 2023-10-11T16:32:25.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.437+0000 7f1f697fa700 1 -- 172.21.15.111:0/1277731914 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24870 (secure 0 0 0) 0x7f1f64099b20 con 0x7f1f6c101e30 2023-10-11T16:32:25.439 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:25.439 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":91,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:31:42.150221+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:32:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1f540a1150 msgr2=0x7f1f540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1f540a1150 0x7f1f540a3600 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c00b2e0 tx=0x7f1f5c005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c101e30 msgr2=0x7f1f6c19f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c101e30 0x7f1f6c19f790 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400ef30 tx=0x7f1f6400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 shutdown_connections 2023-10-11T16:32:25.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1f540a1150 0x7f1f540a3600 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f6c101e30 0x7f1f6c19f790 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f6c1002b0 0x7f1f6c1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 --2- 172.21.15.111:0/1277731914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f6c1014a0 0x7f1f6c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 >> 172.21.15.111:0/1277731914 conn(0x7f1f6c0fb890 msgr2=0x7f1f6c106db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:25.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 shutdown_connections 2023-10-11T16:32:25.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.439+0000 7f1f723ff700 1 -- 172.21.15.111:0/1277731914 wait complete. 2023-10-11T16:32:25.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:25 smithi150 ceph-mon[132217]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:25.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:25 smithi150 ceph-mon[132217]: pgmap v343: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:25.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:25 smithi150 ceph-mon[132217]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:25.460 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:32:25.461 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:32:25.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:25 smithi130 ceph-mon[131859]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:25.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:25 smithi130 ceph-mon[131859]: pgmap v343: 129 pgs: 129 active+clean; 655 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-10-11T16:32:25.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:25 smithi130 ceph-mon[131859]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:25.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 -- 172.21.15.111:0/3508614573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 msgr2=0x7fc5ac101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/3508614573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 0x7fc5ac101900 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc594009990 tx=0x7fc594009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 -- 172.21.15.111:0/3508614573 shutdown_connections 2023-10-11T16:32:25.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/3508614573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac101e40 0x7fc5ac108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/3508614573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5ac1002c0 0x7fc5ac1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/3508614573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 0x7fc5ac101900 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 -- 172.21.15.111:0/3508614573 >> 172.21.15.111:0/3508614573 conn(0x7fc5ac0fb8a0 msgr2=0x7fc5ac0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:25.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 -- 172.21.15.111:0/3508614573 shutdown_connections 2023-10-11T16:32:25.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.659+0000 7fc5b09ff700 1 -- 172.21.15.111:0/3508614573 wait complete. 2023-10-11T16:32:25.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.660+0000 7fc5b09ff700 1 Processor -- start 2023-10-11T16:32:25.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.660+0000 7fc5b09ff700 1 -- start start 2023-10-11T16:32:25.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5b09ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac1002c0 0x7fc5ac198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:25.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5b09ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 0x7fc5ac198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:25.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5b09ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5ac101e40 0x7fc5ac19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:25.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5b09ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac19fdc0 con 0x7fc5ac1014b0 2023-10-11T16:32:25.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5b09ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac19ff00 con 0x7fc5ac101e40 2023-10-11T16:32:25.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5b09ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac1a0090 con 0x7fc5ac1002c0 2023-10-11T16:32:25.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 0x7fc5ac198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:25.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5aa59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac1002c0 0x7fc5ac198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:25.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 0x7fc5ac198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47146/0 (socket says 172.21.15.111:47146) 2023-10-11T16:32:25.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5a3fff700 1 -- 172.21.15.111:0/1862797424 learned_addr learned my addr 172.21.15.111:0/1862797424 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:25.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5aad9d700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5ac101e40 0x7fc5ac19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:25.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5a3fff700 1 -- 172.21.15.111:0/1862797424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac1002c0 msgr2=0x7fc5ac198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5a3fff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac1002c0 0x7fc5ac198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5a3fff700 1 -- 172.21.15.111:0/1862797424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5ac101e40 msgr2=0x7fc5ac19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5a3fff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5ac101e40 0x7fc5ac19f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5a3fff700 1 -- 172.21.15.111:0/1862797424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5940095f0 con 0x7fc5ac1014b0 2023-10-11T16:32:25.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.661+0000 7fc5aa59c700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac1002c0 0x7fc5ac198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:25.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.662+0000 7fc5a3fff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 0x7fc5ac198780 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc594004760 tx=0x7fc594004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:25.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.662+0000 7fc5a37fe700 1 -- 172.21.15.111:0/1862797424 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59401d070 con 0x7fc5ac1014b0 2023-10-11T16:32:25.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.662+0000 7fc5a37fe700 1 -- 172.21.15.111:0/1862797424 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc594004830 con 0x7fc5ac1014b0 2023-10-11T16:32:25.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.662+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ac10bc70 con 0x7fc5ac1014b0 2023-10-11T16:32:25.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.662+0000 7fc5a37fe700 1 -- 172.21.15.111:0/1862797424 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594005350 con 0x7fc5ac1014b0 2023-10-11T16:32:25.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.662+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ac10bae0 con 0x7fc5ac1014b0 2023-10-11T16:32:25.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.663+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ac067b00 con 0x7fc5ac1014b0 2023-10-11T16:32:25.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.665+0000 7fc5a37fe700 1 -- 172.21.15.111:0/1862797424 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fc5940054f0 con 0x7fc5ac1014b0 2023-10-11T16:32:25.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.667+0000 7fc5a37fe700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5980a11d0 0x7fc5980a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:25.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.667+0000 7fc5a37fe700 1 -- 172.21.15.111:0/1862797424 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fc5940ce2c0 con 0x7fc5ac1014b0 2023-10-11T16:32:25.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.668+0000 7fc5aa59c700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5980a11d0 0x7fc5980a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:25.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.668+0000 7fc5aa59c700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5980a11d0 0x7fc5980a3680 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac101310 tx=0x7fc59c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:25.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.670+0000 7fc5a37fe700 1 -- 172.21.15.111:0/1862797424 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc5940976d0 con 0x7fc5ac1014b0 2023-10-11T16:32:25.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.868+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5ac1a0b10 con 0x7fc5ac1014b0 2023-10-11T16:32:25.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.869+0000 7fc5a37fe700 1 -- 172.21.15.111:0/1862797424 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc59409acf0 con 0x7fc5ac1014b0 2023-10-11T16:32:25.871 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:32:25.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5980a11d0 msgr2=0x7fc5980a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5980a11d0 0x7fc5980a3680 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac101310 tx=0x7fc59c009260 comp rx=0 tx=0).stop 2023-10-11T16:32:25.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 msgr2=0x7fc5ac198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:25.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 0x7fc5ac198780 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc594004760 tx=0x7fc594004790 comp rx=0 tx=0).stop 2023-10-11T16:32:25.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 shutdown_connections 2023-10-11T16:32:25.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fc5980a11d0 0x7fc5980a3680 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5ac101e40 0x7fc5ac19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac1002c0 0x7fc5ac198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 --2- 172.21.15.111:0/1862797424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5ac1014b0 0x7fc5ac198780 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:25.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 >> 172.21.15.111:0/1862797424 conn(0x7fc5ac0fb8a0 msgr2=0x7fc5ac0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:25.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 shutdown_connections 2023-10-11T16:32:25.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:25.871+0000 7fc5b09ff700 1 -- 172.21.15.111:0/1862797424 wait complete. 2023-10-11T16:32:25.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:32:26.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 -- 172.21.15.111:0/1119257834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41014b0 msgr2=0x7f82e4101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/1119257834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41014b0 0x7f82e4101900 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009990 tx=0x7f82d4009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 -- 172.21.15.111:0/1119257834 shutdown_connections 2023-10-11T16:32:26.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/1119257834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e4101e40 0x7f82e4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/1119257834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82e41002c0 0x7f82e41006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/1119257834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e41014b0 0x7f82e4101900 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 -- 172.21.15.111:0/1119257834 >> 172.21.15.111:0/1119257834 conn(0x7f82e40fb8a0 msgr2=0x7f82e40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:26.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 -- 172.21.15.111:0/1119257834 shutdown_connections 2023-10-11T16:32:26.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.092+0000 7f82ec3ff700 1 -- 172.21.15.111:0/1119257834 wait complete. 2023-10-11T16:32:26.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.093+0000 7f82ec3ff700 1 Processor -- start 2023-10-11T16:32:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.093+0000 7f82ec3ff700 1 -- start start 2023-10-11T16:32:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ec3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82e41002c0 0x7f82e40748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ec3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e41014b0 0x7f82e4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ec3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4101e40 0x7f82e4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ec3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4073bc0 con 0x7f82e4101e40 2023-10-11T16:32:26.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ec3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e4073d30 con 0x7f82e41002c0 2023-10-11T16:32:26.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ec3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e41a00b0 con 0x7f82e41014b0 2023-10-11T16:32:26.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea19b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82e41002c0 0x7f82e40748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea19b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82e41002c0 0x7f82e40748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46852/0 (socket says 172.21.15.111:46852) 2023-10-11T16:32:26.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea99c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4101e40 0x7f82e4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea19b700 1 -- 172.21.15.111:0/3699753110 learned_addr learned my addr 172.21.15.111:0/3699753110 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:26.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82e999a700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e41014b0 0x7f82e4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea19b700 1 -- 172.21.15.111:0/3699753110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e41014b0 msgr2=0x7f82e4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea19b700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e41014b0 0x7f82e4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea19b700 1 -- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4101e40 msgr2=0x7f82e4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea19b700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4101e40 0x7f82e4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.094+0000 7f82ea19b700 1 -- 172.21.15.111:0/3699753110 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d40095f0 con 0x7f82e41002c0 2023-10-11T16:32:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.095+0000 7f82ea19b700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82e41002c0 0x7f82e40748a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f82e000e970 tx=0x7f82e000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:26.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.095+0000 7f82db7fe700 1 -- 172.21.15.111:0/3699753110 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0009790 con 0x7f82e41002c0 2023-10-11T16:32:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.095+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e410bc70 con 0x7f82e41002c0 2023-10-11T16:32:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.095+0000 7f82db7fe700 1 -- 172.21.15.111:0/3699753110 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82e000eec0 con 0x7f82e41002c0 2023-10-11T16:32:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.095+0000 7f82db7fe700 1 -- 172.21.15.111:0/3699753110 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0003cf0 con 0x7f82e41002c0 2023-10-11T16:32:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.095+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e410bae0 con 0x7f82e41002c0 2023-10-11T16:32:26.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.096+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e41071f0 con 0x7f82e41002c0 2023-10-11T16:32:26.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.098+0000 7f82db7fe700 1 -- 172.21.15.111:0/3699753110 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f82e000ccd0 con 0x7f82e41002c0 2023-10-11T16:32:26.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.099+0000 7f82db7fe700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f82d00a1150 0x7f82d00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.100+0000 7f82db7fe700 1 -- 172.21.15.111:0/3699753110 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f82e0014070 con 0x7f82e41002c0 2023-10-11T16:32:26.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.100+0000 7f82e999a700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f82d00a1150 0x7f82d00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.100+0000 7f82e999a700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f82d00a1150 0x7f82d00a3600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f82d400b2e0 tx=0x7f82d4005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:26.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.102+0000 7f82db7fe700 1 -- 172.21.15.111:0/3699753110 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f82e00963f0 con 0x7f82e41002c0 2023-10-11T16:32:26.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.303+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f82e4061700 con 0x7f82e41002c0 2023-10-11T16:32:26.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.304+0000 7f82db7fe700 1 -- 172.21.15.111:0/3699753110 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f82e0099a10 con 0x7f82e41002c0 2023-10-11T16:32:26.305 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:26.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f82d00a1150 msgr2=0x7f82d00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f82d00a1150 0x7f82d00a3600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f82d400b2e0 tx=0x7f82d4005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82e41002c0 msgr2=0x7f82e40748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82e41002c0 0x7f82e40748a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f82e000e970 tx=0x7f82e000ec80 comp rx=0 tx=0).stop 2023-10-11T16:32:26.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 shutdown_connections 2023-10-11T16:32:26.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f82d00a1150 0x7f82d00a3600 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82e4101e40 0x7f82e4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82e41002c0 0x7f82e40748a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 --2- 172.21.15.111:0/3699753110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e41014b0 0x7f82e4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 >> 172.21.15.111:0/3699753110 conn(0x7f82e40fb8a0 msgr2=0x7f82e40fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:26.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 shutdown_connections 2023-10-11T16:32:26.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.306+0000 7f82ec3ff700 1 -- 172.21.15.111:0/3699753110 wait complete. 2023-10-11T16:32:26.326 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:32:26.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:26 smithi150 ceph-mon[132217]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:26.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:26 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1277731914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:26.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:26 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1862797424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:26.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 -- 172.21.15.111:0/4162328924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d4101e50 msgr2=0x7f27d4108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/4162328924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d4101e50 0x7f27d4108710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00b0a0 tx=0x7f27cc00b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 -- 172.21.15.111:0/4162328924 shutdown_connections 2023-10-11T16:32:26.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/4162328924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d4101e50 0x7f27d4108710 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/4162328924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27d41002d0 0x7f27d41006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/4162328924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d41014c0 0x7f27d4101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 -- 172.21.15.111:0/4162328924 >> 172.21.15.111:0/4162328924 conn(0x7f27d40fb8b0 msgr2=0x7f27d40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:26.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 -- 172.21.15.111:0/4162328924 shutdown_connections 2023-10-11T16:32:26.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.529+0000 7f27dc5ff700 1 -- 172.21.15.111:0/4162328924 wait complete. 2023-10-11T16:32:26.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.530+0000 7f27dc5ff700 1 Processor -- start 2023-10-11T16:32:26.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.530+0000 7f27dc5ff700 1 -- start start 2023-10-11T16:32:26.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.530+0000 7f27dc5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d41002d0 0x7f27d4198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.530+0000 7f27dc5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d41014c0 0x7f27d41988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.530+0000 7f27dc5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27d4101e50 0x7f27d419f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.530+0000 7f27dc5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d419ff10 con 0x7f27d41002d0 2023-10-11T16:32:26.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.530+0000 7f27dc5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d41a0050 con 0x7f27d4101e50 2023-10-11T16:32:26.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.530+0000 7f27dc5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d41a01e0 con 0x7f27d41014c0 2023-10-11T16:32:26.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d41002d0 0x7f27d4198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d41002d0 0x7f27d4198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47186/0 (socket says 172.21.15.111:47186) 2023-10-11T16:32:26.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 -- 172.21.15.111:0/835515942 learned_addr learned my addr 172.21.15.111:0/835515942 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:26.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27d9b9a700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d41014c0 0x7f27d41988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27dab9c700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27d4101e50 0x7f27d419f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 -- 172.21.15.111:0/835515942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d41014c0 msgr2=0x7f27d41988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d41014c0 0x7f27d41988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 -- 172.21.15.111:0/835515942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27d4101e50 msgr2=0x7f27d419f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27d4101e50 0x7f27d419f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 -- 172.21.15.111:0/835515942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc009d00 con 0x7f27d41002d0 2023-10-11T16:32:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27dab9c700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27d4101e50 0x7f27d419f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:26.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27da39b700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d41002d0 0x7f27d4198390 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f27d000e970 tx=0x7f27d000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.531+0000 7f27cb7fe700 1 -- 172.21.15.111:0/835515942 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0009790 con 0x7f27d41002d0 2023-10-11T16:32:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.532+0000 7f27cb7fe700 1 -- 172.21.15.111:0/835515942 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27d000eec0 con 0x7f27d41002d0 2023-10-11T16:32:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.532+0000 7f27cb7fe700 1 -- 172.21.15.111:0/835515942 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0003cf0 con 0x7f27d41002d0 2023-10-11T16:32:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.532+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d410bca0 con 0x7f27d41002d0 2023-10-11T16:32:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.532+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d410bb10 con 0x7f27d41002d0 2023-10-11T16:32:26.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.533+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d41a08e0 con 0x7f27d41002d0 2023-10-11T16:32:26.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.534+0000 7f27cb7fe700 1 -- 172.21.15.111:0/835515942 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f27d000ccd0 con 0x7f27d41002d0 2023-10-11T16:32:26.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.536+0000 7f27cb7fe700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f27c00a1150 0x7f27c00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.536+0000 7f27cb7fe700 1 -- 172.21.15.111:0/835515942 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f27d0014070 con 0x7f27d41002d0 2023-10-11T16:32:26.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.536+0000 7f27d9b9a700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f27c00a1150 0x7f27c00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.537+0000 7f27d9b9a700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f27c00a1150 0x7f27c00a3600 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009c60 tx=0x7f27c4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:26.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.540+0000 7f27cb7fe700 1 -- 172.21.15.111:0/835515942 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f27d00962b0 con 0x7f27d41002d0 2023-10-11T16:32:26.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:26 smithi130 ceph-mon[131859]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:26.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:26 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1277731914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:26.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:26 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1862797424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:26.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:26 smithi111 ceph-mon[98527]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-11T16:32:26.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:26 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1277731914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:26.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:26 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1862797424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:32:26.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.737+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f27d41a0e90 con 0x7f27d41002d0 2023-10-11T16:32:26.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.738+0000 7f27cb7fe700 1 -- 172.21.15.111:0/835515942 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f27d00998d0 con 0x7f27d41002d0 2023-10-11T16:32:26.740 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:32:26.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.740+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f27c00a1150 msgr2=0x7f27c00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.740+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f27c00a1150 0x7f27c00a3600 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009c60 tx=0x7f27c4009260 comp rx=0 tx=0).stop 2023-10-11T16:32:26.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.740+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d41002d0 msgr2=0x7f27d4198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.740+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d41002d0 0x7f27d4198390 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f27d000e970 tx=0x7f27d000ec80 comp rx=0 tx=0).stop 2023-10-11T16:32:26.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.740+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 shutdown_connections 2023-10-11T16:32:26.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.741+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f27c00a1150 0x7f27c00a3600 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.741+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27d4101e50 0x7f27d419f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.741+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27d41002d0 0x7f27d4198390 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.741+0000 7f27dc5ff700 1 --2- 172.21.15.111:0/835515942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27d41014c0 0x7f27d41988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.741+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 >> 172.21.15.111:0/835515942 conn(0x7f27d40fb8b0 msgr2=0x7f27d4106cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:26.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.741+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 shutdown_connections 2023-10-11T16:32:26.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.741+0000 7f27dc5ff700 1 -- 172.21.15.111:0/835515942 wait complete. 2023-10-11T16:32:26.761 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:32:26.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1720385344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b81014b0 msgr2=0x7f03b8101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1720385344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b81014b0 0x7f03b8101900 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f03a8009990 tx=0x7f03a8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1720385344 shutdown_connections 2023-10-11T16:32:26.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1720385344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03b8101e40 0x7f03b8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1720385344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b81002c0 0x7f03b81006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1720385344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b81014b0 0x7f03b8101900 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1720385344 >> 172.21.15.111:0/1720385344 conn(0x7f03b80fb8a0 msgr2=0x7f03b80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:26.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1720385344 shutdown_connections 2023-10-11T16:32:26.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.961+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1720385344 wait complete. 2023-10-11T16:32:26.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.962+0000 7f03bf7ff700 1 Processor -- start 2023-10-11T16:32:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.962+0000 7f03bf7ff700 1 -- start start 2023-10-11T16:32:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.962+0000 7f03bf7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03b81002c0 0x7f03b8198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.962+0000 7f03bf7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b81014b0 0x7f03b8198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.962+0000 7f03bf7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b8101e40 0x7f03b819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.962+0000 7f03bf7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b819fdc0 con 0x7f03b8101e40 2023-10-11T16:32:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.962+0000 7f03bf7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b819ff00 con 0x7f03b81002c0 2023-10-11T16:32:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.962+0000 7f03bf7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b81a0090 con 0x7f03b81014b0 2023-10-11T16:32:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b8101e40 0x7f03b819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b8101e40 0x7f03b819f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47204/0 (socket says 172.21.15.111:47204) 2023-10-11T16:32:26.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 -- 172.21.15.111:0/1281360237 learned_addr learned my addr 172.21.15.111:0/1281360237 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:26.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bd59b700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03b81002c0 0x7f03b8198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bcd9a700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b81014b0 0x7f03b8198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 -- 172.21.15.111:0/1281360237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b81014b0 msgr2=0x7f03b8198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b81014b0 0x7f03b8198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 -- 172.21.15.111:0/1281360237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03b81002c0 msgr2=0x7f03b8198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:26.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03b81002c0 0x7f03b8198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:26.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 -- 172.21.15.111:0/1281360237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a80095f0 con 0x7f03b8101e40 2023-10-11T16:32:26.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bd59b700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03b81002c0 0x7f03b8198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:26.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03bdd9c700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b8101e40 0x7f03b819f7f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f03b000ef30 tx=0x7f03b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:26.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.963+0000 7f03ae7fc700 1 -- 172.21.15.111:0/1281360237 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0018070 con 0x7f03b8101e40 2023-10-11T16:32:26.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.964+0000 7f03ae7fc700 1 -- 172.21.15.111:0/1281360237 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03b000f040 con 0x7f03b8101e40 2023-10-11T16:32:26.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.964+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b810bc70 con 0x7f03b8101e40 2023-10-11T16:32:26.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.964+0000 7f03ae7fc700 1 -- 172.21.15.111:0/1281360237 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b000cd00 con 0x7f03b8101e40 2023-10-11T16:32:26.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.964+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b810bae0 con 0x7f03b8101e40 2023-10-11T16:32:26.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.965+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b81a0770 con 0x7f03b8101e40 2023-10-11T16:32:26.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.966+0000 7f03ae7fc700 1 -- 172.21.15.111:0/1281360237 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f03b0003a00 con 0x7f03b8101e40 2023-10-11T16:32:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.968+0000 7f03ae7fc700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03a40a11d0 0x7f03a40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.968+0000 7f03ae7fc700 1 -- 172.21.15.111:0/1281360237 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f03b00cc760 con 0x7f03b8101e40 2023-10-11T16:32:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.968+0000 7f03bd59b700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03a40a11d0 0x7f03a40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.968+0000 7f03bd59b700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03a40a11d0 0x7f03a40a3680 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f03b8101310 tx=0x7f03b4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:26.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:26.971+0000 7f03ae7fc700 1 -- 172.21.15.111:0/1281360237 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f03b0095b70 con 0x7f03b8101e40 2023-10-11T16:32:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.176+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f03b81a0d20 con 0x7f03b8101e40 2023-10-11T16:32:27.178 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.177+0000 7f03ae7fc700 1 -- 172.21.15.111:0/1281360237 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f03b0099190 con 0x7f03b8101e40 2023-10-11T16:32:27.179 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:32:27.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03a40a11d0 msgr2=0x7f03a40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03a40a11d0 0x7f03a40a3680 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f03b8101310 tx=0x7f03b4009260 comp rx=0 tx=0).stop 2023-10-11T16:32:27.181 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b8101e40 msgr2=0x7f03b819f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b8101e40 0x7f03b819f7f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f03b000ef30 tx=0x7f03b000c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 shutdown_connections 2023-10-11T16:32:27.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f03a40a11d0 0x7f03a40a3680 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b8101e40 0x7f03b819f7f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.182 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03b81002c0 0x7f03b8198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 --2- 172.21.15.111:0/1281360237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b81014b0 0x7f03b8198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.179+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 >> 172.21.15.111:0/1281360237 conn(0x7f03b80fb8a0 msgr2=0x7f03b8106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.180+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 shutdown_connections 2023-10-11T16:32:27.183 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.180+0000 7f03bf7ff700 1 -- 172.21.15.111:0/1281360237 wait complete. 2023-10-11T16:32:27.200 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:32:27.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 -- 172.21.15.111:0/1756631460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 msgr2=0x7f1b0c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 --2- 172.21.15.111:0/1756631460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 0x7f1b0c0739c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009990 tx=0x7f1b00009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 -- 172.21.15.111:0/1756631460 shutdown_connections 2023-10-11T16:32:27.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 --2- 172.21.15.111:0/1756631460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b0c108170 0x7f1b0c10a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 --2- 172.21.15.111:0/1756631460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c074b30 0x7f1b0c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 --2- 172.21.15.111:0/1756631460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 0x7f1b0c0739c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 -- 172.21.15.111:0/1756631460 >> 172.21.15.111:0/1756631460 conn(0x7f1b0c0fb890 msgr2=0x7f1b0c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:27.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 -- 172.21.15.111:0/1756631460 shutdown_connections 2023-10-11T16:32:27.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.397+0000 7f1b121ff700 1 -- 172.21.15.111:0/1756631460 wait complete. 2023-10-11T16:32:27.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b121ff700 1 Processor -- start 2023-10-11T16:32:27.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b121ff700 1 -- start start 2023-10-11T16:32:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b121ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 0x7f1b0c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b121ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b0c074b30 0x7f1b0c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b121ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c108170 0x7f1b0c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b121ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c19fd80 con 0x7f1b0c073550 2023-10-11T16:32:27.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b121ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c19fec0 con 0x7f1b0c074b30 2023-10-11T16:32:27.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b121ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c1a0050 con 0x7f1b0c108170 2023-10-11T16:32:27.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.398+0000 7f1b0b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 0x7f1b0c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:27.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 0x7f1b0c198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47210/0 (socket says 172.21.15.111:47210) 2023-10-11T16:32:27.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0b7fe700 1 -- 172.21.15.111:0/3721067569 learned_addr learned my addr 172.21.15.111:0/3721067569 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:27.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0affd700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b0c074b30 0x7f1b0c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0bfff700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c108170 0x7f1b0c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0b7fe700 1 -- 172.21.15.111:0/3721067569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c108170 msgr2=0x7f1b0c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0b7fe700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c108170 0x7f1b0c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0b7fe700 1 -- 172.21.15.111:0/3721067569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b0c074b30 msgr2=0x7f1b0c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0b7fe700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b0c074b30 0x7f1b0c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0b7fe700 1 -- 172.21.15.111:0/3721067569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b000095f0 con 0x7f1b0c073550 2023-10-11T16:32:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b0b7fe700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 0x7f1b0c198200 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00e970 tx=0x7f1afc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b08ff9700 1 -- 172.21.15.111:0/3721067569 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc009790 con 0x7f1b0c073550 2023-10-11T16:32:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b08ff9700 1 -- 172.21.15.111:0/3721067569 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1afc00c7e0 con 0x7f1b0c073550 2023-10-11T16:32:27.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0c10de90 con 0x7f1b0c073550 2023-10-11T16:32:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b08ff9700 1 -- 172.21.15.111:0/3721067569 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc0054e0 con 0x7f1b0c073550 2023-10-11T16:32:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.399+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0c10dd00 con 0x7f1b0c073550 2023-10-11T16:32:27.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.400+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0c106820 con 0x7f1b0c073550 2023-10-11T16:32:27.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.402+0000 7f1b08ff9700 1 -- 172.21.15.111:0/3721067569 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f1afc00c950 con 0x7f1b0c073550 2023-10-11T16:32:27.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.404+0000 7f1b08ff9700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1af40a11d0 0x7f1af40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:27.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.404+0000 7f1b08ff9700 1 -- 172.21.15.111:0/3721067569 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f1afc014070 con 0x7f1b0c073550 2023-10-11T16:32:27.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.404+0000 7f1b0affd700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1af40a11d0 0x7f1af40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:27.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.405+0000 7f1b0affd700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1af40a11d0 0x7f1af40a3680 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000b2e0 tx=0x7f1b00005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:27.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.407+0000 7f1b08ff9700 1 -- 172.21.15.111:0/3721067569 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1afc096260 con 0x7f1b0c073550 2023-10-11T16:32:27.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:27 smithi150 ceph-mon[132217]: pgmap v344: 129 pgs: 129 active+clean; 657 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:32:27.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:27 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3699753110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:27.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:27 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/835515942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:27.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:27 smithi130 ceph-mon[131859]: pgmap v344: 129 pgs: 129 active+clean; 657 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:32:27.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:27 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3699753110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:27.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:27 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/835515942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:27.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:27 smithi111 ceph-mon[98527]: pgmap v344: 129 pgs: 129 active+clean; 657 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2023-10-11T16:32:27.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:27 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3699753110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:32:27.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:27 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/835515942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:32:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.650+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b0c02cfe0 con 0x7f1b0c073550 2023-10-11T16:32:27.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.651+0000 7f1b08ff9700 1 -- 172.21.15.111:0/3721067569 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f1afc099880 con 0x7f1b0c073550 2023-10-11T16:32:27.653 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:27.653 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:30:51.227688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:32:27.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.653+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1af40a11d0 msgr2=0x7f1af40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.653+0000 7f1b121ff700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1af40a11d0 0x7f1af40a3680 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000b2e0 tx=0x7f1b00005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.653+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 msgr2=0x7f1b0c198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.653+0000 7f1b121ff700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 0x7f1b0c198200 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00e970 tx=0x7f1afc00ec80 comp rx=0 tx=0).stop 2023-10-11T16:32:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.654+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 shutdown_connections 2023-10-11T16:32:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.654+0000 7f1b121ff700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1af40a11d0 0x7f1af40a3680 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.654+0000 7f1b121ff700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b0c108170 0x7f1b0c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.654+0000 7f1b121ff700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b0c073550 0x7f1b0c198200 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.654+0000 7f1b121ff700 1 --2- 172.21.15.111:0/3721067569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1b0c074b30 0x7f1b0c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.654+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 >> 172.21.15.111:0/3721067569 conn(0x7f1b0c0fb890 msgr2=0x7f1b0c0fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:27.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.654+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 shutdown_connections 2023-10-11T16:32:27.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.654+0000 7f1b121ff700 1 -- 172.21.15.111:0/3721067569 wait complete. 2023-10-11T16:32:27.658 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2023-10-11T16:32:27.676 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:32:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.870+0000 7fd2359ff700 1 -- 172.21.15.111:0/283689490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd230101e30 msgr2=0x7fd2301086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.870+0000 7fd2359ff700 1 --2- 172.21.15.111:0/283689490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd230101e30 0x7fd2301086f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd22800b0a0 tx=0x7fd22800b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.871+0000 7fd2359ff700 1 -- 172.21.15.111:0/283689490 shutdown_connections 2023-10-11T16:32:27.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.871+0000 7fd2359ff700 1 --2- 172.21.15.111:0/283689490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd230101e30 0x7fd2301086f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.871+0000 7fd2359ff700 1 --2- 172.21.15.111:0/283689490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2301002b0 0x7fd2301006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.871+0000 7fd2359ff700 1 --2- 172.21.15.111:0/283689490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2301014a0 0x7fd2301018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.871+0000 7fd2359ff700 1 -- 172.21.15.111:0/283689490 >> 172.21.15.111:0/283689490 conn(0x7fd2300fb890 msgr2=0x7fd2300fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:27.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.871+0000 7fd2359ff700 1 -- 172.21.15.111:0/283689490 shutdown_connections 2023-10-11T16:32:27.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.871+0000 7fd2359ff700 1 -- 172.21.15.111:0/283689490 wait complete. 2023-10-11T16:32:27.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd2359ff700 1 Processor -- start 2023-10-11T16:32:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd2359ff700 1 -- start start 2023-10-11T16:32:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd2359ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2301002b0 0x7fd2301982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd2359ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2301014a0 0x7fd230198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd2359ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd230101e30 0x7fd23019f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:27.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd2359ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23019fe70 con 0x7fd2301014a0 2023-10-11T16:32:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd2359ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23019ffb0 con 0x7fd2301002b0 2023-10-11T16:32:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd2359ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2301a0140 con 0x7fd230101e30 2023-10-11T16:32:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd22e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2301014a0 0x7fd230198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd22e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2301014a0 0x7fd230198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47238/0 (socket says 172.21.15.111:47238) 2023-10-11T16:32:27.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.872+0000 7fd22e7fc700 1 -- 172.21.15.111:0/1967782531 learned_addr learned my addr 172.21.15.111:0/1967782531 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd22effd700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2301002b0 0x7fd2301982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd22f7fe700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd230101e30 0x7fd23019f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd22e7fc700 1 -- 172.21.15.111:0/1967782531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd230101e30 msgr2=0x7fd23019f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd22e7fc700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd230101e30 0x7fd23019f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd22e7fc700 1 -- 172.21.15.111:0/1967782531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2301002b0 msgr2=0x7fd2301982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd22e7fc700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2301002b0 0x7fd2301982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd22e7fc700 1 -- 172.21.15.111:0/1967782531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd228009d00 con 0x7fd2301014a0 2023-10-11T16:32:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd22e7fc700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2301014a0 0x7fd230198830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd22400e9a0 tx=0x7fd22400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd2349fd700 1 -- 172.21.15.111:0/1967782531 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224009790 con 0x7fd2301014a0 2023-10-11T16:32:27.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd2349fd700 1 -- 172.21.15.111:0/1967782531 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd224010430 con 0x7fd2301014a0 2023-10-11T16:32:27.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23010bc80 con 0x7fd2301014a0 2023-10-11T16:32:27.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd2349fd700 1 -- 172.21.15.111:0/1967782531 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224018680 con 0x7fd2301014a0 2023-10-11T16:32:27.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.873+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23010baf0 con 0x7fd2301014a0 2023-10-11T16:32:27.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.874+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2301a0830 con 0x7fd2301014a0 2023-10-11T16:32:27.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.876+0000 7fd2349fd700 1 -- 172.21.15.111:0/1967782531 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fd2240105a0 con 0x7fd2301014a0 2023-10-11T16:32:27.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.878+0000 7fd2349fd700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd2180a1150 0x7fd2180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:27.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.878+0000 7fd2349fd700 1 -- 172.21.15.111:0/1967782531 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fd224014070 con 0x7fd2301014a0 2023-10-11T16:32:27.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.878+0000 7fd22effd700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd2180a1150 0x7fd2180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:27.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.879+0000 7fd22effd700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd2180a1150 0x7fd2180a3600 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd230101300 tx=0x7fd220009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:27.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:27.881+0000 7fd2349fd700 1 -- 172.21.15.111:0/1967782531 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd224051730 con 0x7fd2301014a0 2023-10-11T16:32:28.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.081+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2301a0ed0 con 0x7fd2301014a0 2023-10-11T16:32:28.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.082+0000 7fd2349fd700 1 -- 172.21.15.111:0/1967782531 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24870 (secure 0 0 0) 0x7fd224096190 con 0x7fd2301014a0 2023-10-11T16:32:28.084 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:28.085 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":91,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:31:42.150221+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:32:28.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd2180a1150 msgr2=0x7fd2180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd2180a1150 0x7fd2180a3600 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd230101300 tx=0x7fd220009260 comp rx=0 tx=0).stop 2023-10-11T16:32:28.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2301014a0 msgr2=0x7fd230198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2301014a0 0x7fd230198830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd22400e9a0 tx=0x7fd22400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 shutdown_connections 2023-10-11T16:32:28.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fd2180a1150 0x7fd2180a3600 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd230101e30 0x7fd23019f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd2301002b0 0x7fd2301982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 --2- 172.21.15.111:0/1967782531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2301014a0 0x7fd230198830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 >> 172.21.15.111:0/1967782531 conn(0x7fd2300fb890 msgr2=0x7fd230106d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 shutdown_connections 2023-10-11T16:32:28.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.085+0000 7fd2359ff700 1 -- 172.21.15.111:0/1967782531 wait complete. 2023-10-11T16:32:28.106 DEBUG:tasks.fwd_scrub:Yielding 2023-10-11T16:32:28.106 INFO:teuthology.run_tasks:Running task exec... 2023-10-11T16:32:28.117 INFO:teuthology.task.exec:Executing custom commands... 2023-10-11T16:32:28.118 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:32:28.118 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2023-10-11T16:32:28.119 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2023-10-11T16:32:28.119 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:32:28.120 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:32:28.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 -- 172.21.15.111:0/2373859089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c073550 msgr2=0x7f205c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 --2- 172.21.15.111:0/2373859089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c073550 0x7f205c0739c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f204c009990 tx=0x7f204c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 -- 172.21.15.111:0/2373859089 shutdown_connections 2023-10-11T16:32:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 --2- 172.21.15.111:0/2373859089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f205c108180 0x7f205c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 --2- 172.21.15.111:0/2373859089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c074b30 0x7f205c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 --2- 172.21.15.111:0/2373859089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c073550 0x7f205c0739c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 -- 172.21.15.111:0/2373859089 >> 172.21.15.111:0/2373859089 conn(0x7f205c0fb8e0 msgr2=0x7f205c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 -- 172.21.15.111:0/2373859089 shutdown_connections 2023-10-11T16:32:28.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.320+0000 7f20643ff700 1 -- 172.21.15.111:0/2373859089 wait complete. 2023-10-11T16:32:28.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.322+0000 7f20643ff700 1 Processor -- start 2023-10-11T16:32:28.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.322+0000 7f20643ff700 1 -- start start 2023-10-11T16:32:28.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.322+0000 7f20643ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c073550 0x7f205c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.322+0000 7f20643ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f205c074b30 0x7f205c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f20643ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c108180 0x7f205c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f20643ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c19fd80 con 0x7f205c074b30 2023-10-11T16:32:28.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f20643ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c19fec0 con 0x7f205c073550 2023-10-11T16:32:28.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f20643ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205c1a0050 con 0x7f205c108180 2023-10-11T16:32:28.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206199a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f205c074b30 0x7f205c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206299c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c108180 0x7f205c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206199a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f205c074b30 0x7f205c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47250/0 (socket says 172.21.15.111:47250) 2023-10-11T16:32:28.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206199a700 1 -- 172.21.15.111:0/545454472 learned_addr learned my addr 172.21.15.111:0/545454472 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206219b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c073550 0x7f205c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206299c700 1 -- 172.21.15.111:0/545454472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c073550 msgr2=0x7f205c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206299c700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c073550 0x7f205c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206299c700 1 -- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f205c074b30 msgr2=0x7f205c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206299c700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f205c074b30 0x7f205c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206299c700 1 -- 172.21.15.111:0/545454472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2054009c40 con 0x7f205c108180 2023-10-11T16:32:28.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206199a700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f205c074b30 0x7f205c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:28.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.323+0000 7f206299c700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c108180 0x7f205c19f7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f205400efb0 tx=0x7f205400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.324+0000 7f20537fe700 1 -- 172.21.15.111:0/545454472 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2054003a00 con 0x7f205c108180 2023-10-11T16:32:28.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.324+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c0095f0 con 0x7f205c108180 2023-10-11T16:32:28.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.324+0000 7f20537fe700 1 -- 172.21.15.111:0/545454472 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f205400f040 con 0x7f205c108180 2023-10-11T16:32:28.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.324+0000 7f20537fe700 1 -- 172.21.15.111:0/545454472 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2054004550 con 0x7f205c108180 2023-10-11T16:32:28.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.324+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205c10dde0 con 0x7f205c108180 2023-10-11T16:32:28.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.325+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205c106830 con 0x7f205c108180 2023-10-11T16:32:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.327+0000 7f20537fe700 1 -- 172.21.15.111:0/545454472 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f2054003bf0 con 0x7f205c108180 2023-10-11T16:32:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.328+0000 7f20537fe700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f20480a1150 0x7f20480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.328+0000 7f20537fe700 1 -- 172.21.15.111:0/545454472 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f20540cc910 con 0x7f205c108180 2023-10-11T16:32:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.329+0000 7f206219b700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f20480a1150 0x7f20480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.329+0000 7f206219b700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f20480a1150 0x7f20480a3600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f205c074f00 tx=0x7f2058009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.330+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/1618073060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b781002d0 msgr2=0x7f0b781006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.330+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/1618073060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b781002d0 0x7f0b781006e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009990 tx=0x7f0b68009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.331+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/1618073060 shutdown_connections 2023-10-11T16:32:28.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.331+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/1618073060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b78101e50 0x7f0b78108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.331+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/1618073060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b781002d0 0x7f0b781006e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.331+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/1618073060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b781014c0 0x7f0b78101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.331+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/1618073060 >> 172.21.15.111:0/1618073060 conn(0x7f0b780fb8d0 msgr2=0x7f0b780fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.331+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/1618073060 shutdown_connections 2023-10-11T16:32:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.331+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/1618073060 wait complete. 2023-10-11T16:32:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.332+0000 7f0b7d5ff700 1 Processor -- start 2023-10-11T16:32:28.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.332+0000 7f0b7d5ff700 1 -- start start 2023-10-11T16:32:28.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b7d5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b781002d0 0x7f0b781960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f20537fe700 1 -- 172.21.15.111:0/545454472 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2054095d20 con 0x7f205c108180 2023-10-11T16:32:28.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b7d5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b781014c0 0x7f0b781965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b7d5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b78101e50 0x7f0b7819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b7d5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7819dc30 con 0x7f0b781002d0 2023-10-11T16:32:28.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b7d5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7819dd70 con 0x7f0b78101e50 2023-10-11T16:32:28.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b7d5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7819df00 con 0x7f0b781014c0 2023-10-11T16:32:28.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b76ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b781002d0 0x7f0b781960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b76ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b781002d0 0x7f0b781960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47264/0 (socket says 172.21.15.111:47264) 2023-10-11T16:32:28.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b76ffd700 1 -- 172.21.15.111:0/2224697354 learned_addr learned my addr 172.21.15.111:0/2224697354 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b767fc700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b781014c0 0x7f0b781965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b777fe700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b78101e50 0x7f0b7819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b76ffd700 1 -- 172.21.15.111:0/2224697354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b781014c0 msgr2=0x7f0b781965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b76ffd700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b781014c0 0x7f0b781965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b76ffd700 1 -- 172.21.15.111:0/2224697354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b78101e50 msgr2=0x7f0b7819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b76ffd700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b78101e50 0x7f0b7819d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.333+0000 7f0b76ffd700 1 -- 172.21.15.111:0/2224697354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b70009c40 con 0x7f0b781002d0 2023-10-11T16:32:28.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.334+0000 7f0b76ffd700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b781002d0 0x7f0b781960b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0b68004730 tx=0x7f0b68004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.334+0000 7f0b5ffff700 1 -- 172.21.15.111:0/2224697354 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6801d070 con 0x7f0b781002d0 2023-10-11T16:32:28.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.334+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/2224697354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b680095f0 con 0x7f0b781002d0 2023-10-11T16:32:28.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.334+0000 7f0b5ffff700 1 -- 172.21.15.111:0/2224697354 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b68021410 con 0x7f0b781002d0 2023-10-11T16:32:28.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.334+0000 7f0b5ffff700 1 -- 172.21.15.111:0/2224697354 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68017650 con 0x7f0b781002d0 2023-10-11T16:32:28.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.334+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/2224697354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7810bca0 con 0x7f0b781002d0 2023-10-11T16:32:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.338+0000 7f0b90ff3b80 1 Processor -- start 2023-10-11T16:32:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b90ff3b80 1 -- start start 2023-10-11T16:32:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b5ffff700 1 -- 172.21.15.111:0/2224697354 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7f0b68021810 con 0x7f0b781002d0 2023-10-11T16:32:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc51090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc51bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc52600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78db560e0 con 0x55a78dc50c80 2023-10-11T16:32:28.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dba8920 con 0x55a78dc515d0 2023-10-11T16:32:28.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dba1db0 con 0x55a78dc52190 2023-10-11T16:32:28.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc51090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b76ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc51bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc51090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47274/0 (socket says 172.21.15.111:47274) 2023-10-11T16:32:28.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b777fe700 1 -- 172.21.15.111:0/663287524 learned_addr learned my addr 172.21.15.111:0/663287524 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.339+0000 7f0b767fc700 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc52600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b777fe700 1 -- 172.21.15.111:0/663287524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 msgr2=0x55a78dc52600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b777fe700 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc52600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b777fe700 1 -- 172.21.15.111:0/663287524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 msgr2=0x55a78dc51bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b777fe700 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc51bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b777fe700 1 -- 172.21.15.111:0/663287524 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a78da236b0 con 0x55a78dc50c80 2023-10-11T16:32:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b76ffd700 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc51bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b777fe700 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc51090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000e730 tx=0x7f0b70009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26a62da835926dc server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b5d7fa700 1 -- 172.21.15.111:0/663287524 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7000e040 con 0x55a78dc50c80 2023-10-11T16:32:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b5d7fa700 1 -- 172.21.15.111:0/663287524 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b7000f040 con 0x55a78dc50c80 2023-10-11T16:32:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b5d7fa700 1 -- 172.21.15.111:0/663287524 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70007700 con 0x55a78dc50c80 2023-10-11T16:32:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.340+0000 7f0b5ffff700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b600a1150 0x7f0b600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b5ffff700 1 -- 172.21.15.111:0/2224697354 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f0b680ce0e0 con 0x7f0b781002d0 2023-10-11T16:32:28.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b76ffd700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b600a1150 0x7f0b600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/663287524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 msgr2=0x55a78dc51090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc51090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000e730 tx=0x7f0b70009920 comp rx=0 tx=0).stop 2023-10-11T16:32:28.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/663287524 shutdown_connections 2023-10-11T16:32:28.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc52600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc51090 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/663287524 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc51bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.341+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/663287524 >> 172.21.15.111:0/663287524 conn(0x55a78dc463e0 msgr2=0x55a78dc4b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.342+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/663287524 shutdown_connections 2023-10-11T16:32:28.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.342+0000 7f0b76ffd700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b600a1150 0x7f0b600a3600 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55a78dc4c530 tx=0x7f0b680d1b80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.342+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/663287524 wait complete. 2023-10-11T16:32:28.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.342+0000 7f0b90ff3b80 1 -- start start 2023-10-11T16:32:28.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.342+0000 7f0b90ff3b80 1 Processor -- start 2023-10-11T16:32:28.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc50c80 0x55a78dc5a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc515d0 0x55a78dc5aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc52190 0x55a78dc61b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78db43960 con 0x55a78dc515d0 2023-10-11T16:32:28.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78db43600 con 0x55a78dc52190 2023-10-11T16:32:28.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78da259a0 con 0x55a78dc50c80 2023-10-11T16:32:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b90ff3b80 10 client.0 _ll_register_callbacks cb 0x55a78db61800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc50c80 0x55a78dc5a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc50c80 0x55a78dc5a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:47272/0 (socket says 172.21.15.111:47272) 2023-10-11T16:32:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b777fe700 1 -- 172.21.15.111:0/2378556546 learned_addr learned my addr 172.21.15.111:0/2378556546 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.343+0000 7f0b767fc700 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc515d0 0x55a78dc5aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.344+0000 7f0b777fe700 1 -- 172.21.15.111:0/2378556546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc52190 msgr2=0x55a78dc61b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.344+0000 7f0b777fe700 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc52190 0x55a78dc61b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.344+0000 7f0b777fe700 1 -- 172.21.15.111:0/2378556546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc515d0 msgr2=0x55a78dc5aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.344+0000 7f0b777fe700 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc515d0 0x55a78dc5aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.344+0000 7f0b777fe700 1 -- 172.21.15.111:0/2378556546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7000e410 con 0x55a78dc50c80 2023-10-11T16:32:28.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.344+0000 7f0b767fc700 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc515d0 0x55a78dc5aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:28.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.344+0000 7f0b777fe700 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc50c80 0x55a78dc5a5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0b70004a00 tx=0x7f0b70004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e58bd246e711d4d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.344+0000 7f0b5aff5700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:32:28.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b5aff5700 1 -- 172.21.15.111:0/2378556546 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70003c00 con 0x55a78dc50c80 2023-10-11T16:32:28.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b5aff5700 1 -- 172.21.15.111:0/2378556546 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f0b70003d60 con 0x55a78dc50c80 2023-10-11T16:32:28.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/2378556546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a78dc43ba0 con 0x55a78dc50c80 2023-10-11T16:32:28.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b5aff5700 1 -- 172.21.15.111:0/2378556546 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b70007500 con 0x55a78dc50c80 2023-10-11T16:32:28.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 10 client.25528 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:28.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/2378556546 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a78db68210 con 0x55a78dc50c80 2023-10-11T16:32:28.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b5aff5700 1 -- 172.21.15.111:0/2378556546 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b70007b20 con 0x55a78dc50c80 2023-10-11T16:32:28.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b5aff5700 1 -- 172.21.15.111:0/2378556546 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f0b70015930 con 0x55a78dc50c80 2023-10-11T16:32:28.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 10 client.25528 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:28.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 10 client.25528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:32:28.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 20 client.25528 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:28.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a78dc66240 0x55a78dc68650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 4 client.25528 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:32:28.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.345+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/2378556546 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a78dc43ba0 con 0x55a78dc66240 2023-10-11T16:32:28.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.346+0000 7f0b767fc700 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a78dc66240 0x55a78dc68650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.346+0000 7f0b767fc700 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a78dc66240 0x55a78dc68650 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9a3a7aca81f6ebe server_cookie=610cbfae93c75fbc in_seq=0 out_seq=0 2023-10-11T16:32:28.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.346+0000 7f0b5aff5700 10 client.25528 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:28.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b5aff5700 1 -- 172.21.15.111:0/2378556546 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a78dc43ba0 con 0x55a78dc66240 2023-10-11T16:32:28.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b5aff5700 10 client.25528 handle_command_reply: tid=0 2023-10-11T16:32:28.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 client.25528 shutdown 2023-10-11T16:32:28.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/2378556546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc50c80 msgr2=0x55a78dc5a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc50c80 0x55a78dc5a5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0b70004a00 tx=0x7f0b70004ad0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/2378556546 shutdown_connections 2023-10-11T16:32:28.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a78dc66240 0x55a78dc68650 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc52190 0x55a78dc61b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc50c80 0x55a78dc5a5a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b5aff5700 0 client.25528 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:28.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/2378556546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc515d0 0x55a78dc5aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/2378556546 >> 172.21.15.111:0/2378556546 conn(0x55a78dc463e0 msgr2=0x55a78dc4ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/2378556546 shutdown_connections 2023-10-11T16:32:28.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.347+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/2378556546 wait complete. 2023-10-11T16:32:28.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.348+0000 7f0b90ff3b80 20 client.25528 trim_cache size 0 max 16384 2023-10-11T16:32:28.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.365+0000 7f0b90ff3b80 1 Processor -- start 2023-10-11T16:32:28.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.365+0000 7f0b90ff3b80 1 -- start start 2023-10-11T16:32:28.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.365+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc54fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.365+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc61ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.365+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc554e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.365+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dc52010 con 0x55a78dc50c80 2023-10-11T16:32:28.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.365+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dc59a50 con 0x55a78dc515d0 2023-10-11T16:32:28.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.365+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dc55b60 con 0x55a78dc52190 2023-10-11T16:32:28.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc54fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc54fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47296/0 (socket says 172.21.15.111:47296) 2023-10-11T16:32:28.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 -- 172.21.15.111:0/4242389865 learned_addr learned my addr 172.21.15.111:0/4242389865 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b767fc700 1 --2- 172.21.15.111:0/4242389865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc61ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 -- 172.21.15.111:0/4242389865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 msgr2=0x55a78dc554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 --2- 172.21.15.111:0/4242389865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 -- 172.21.15.111:0/4242389865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 msgr2=0x55a78dc61ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 --2- 172.21.15.111:0/4242389865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc61ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 -- 172.21.15.111:0/4242389865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a78dc47b80 con 0x55a78dc50c80 2023-10-11T16:32:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b777fe700 1 --2- 172.21.15.111:0/4242389865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc54fa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0b70022040 tx=0x7f0b70004620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ced7460df33b38d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b5dffb700 1 -- 172.21.15.111:0/4242389865 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70003c00 con 0x55a78dc50c80 2023-10-11T16:32:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b5dffb700 1 -- 172.21.15.111:0/4242389865 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b70004bf0 con 0x55a78dc50c80 2023-10-11T16:32:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.366+0000 7f0b5dffb700 1 -- 172.21.15.111:0/4242389865 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70005110 con 0x55a78dc50c80 2023-10-11T16:32:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/4242389865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 msgr2=0x55a78dc54fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/4242389865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc54fa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0b70022040 tx=0x7f0b70004620 comp rx=0 tx=0).stop 2023-10-11T16:32:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/4242389865 shutdown_connections 2023-10-11T16:32:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/4242389865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/4242389865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc54fa0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/4242389865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc61ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/4242389865 >> 172.21.15.111:0/4242389865 conn(0x55a78dc463e0 msgr2=0x55a78dc6cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/4242389865 shutdown_connections 2023-10-11T16:32:28.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.367+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/4242389865 wait complete. 2023-10-11T16:32:28.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 1 -- start start 2023-10-11T16:32:28.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 1 Processor -- start 2023-10-11T16:32:28.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc61590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc61ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc5e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dc5e850 con 0x55a78dc50c80 2023-10-11T16:32:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dc5e9c0 con 0x55a78dc515d0 2023-10-11T16:32:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dc5eb80 con 0x55a78dc52190 2023-10-11T16:32:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.368+0000 7f0b90ff3b80 10 client.0 _ll_register_callbacks cb 0x7f0b680d1a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:28.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc61590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc61590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47300/0 (socket says 172.21.15.111:47300) 2023-10-11T16:32:28.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 -- 172.21.15.111:0/1713363880 learned_addr learned my addr 172.21.15.111:0/1713363880 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b767fc700 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc61ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc5e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 -- 172.21.15.111:0/1713363880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 msgr2=0x55a78dc5e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc5e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 -- 172.21.15.111:0/1713363880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 msgr2=0x55a78dc61ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc61ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 -- 172.21.15.111:0/1713363880 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a78db68210 con 0x55a78dc50c80 2023-10-11T16:32:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b777fe700 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc61590 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0b70005090 tx=0x7f0b70004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181fdb7cc686c388 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b5aff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.369+0000 7f0b5aff5700 1 -- 172.21.15.111:0/1713363880 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70015660 con 0x55a78dc50c80 2023-10-11T16:32:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b5aff5700 1 -- 172.21.15.111:0/1713363880 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f0b70003a00 con 0x55a78dc50c80 2023-10-11T16:32:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/1713363880 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b7000f040 con 0x55a78dc50c80 2023-10-11T16:32:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b5aff5700 1 -- 172.21.15.111:0/1713363880 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b70042080 con 0x55a78dc50c80 2023-10-11T16:32:28.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 10 client.15468 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:28.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/1713363880 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a78db67f60 con 0x55a78dc50c80 2023-10-11T16:32:28.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b5aff5700 1 -- 172.21.15.111:0/1713363880 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b7000f040 con 0x55a78dc50c80 2023-10-11T16:32:28.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b5aff5700 1 -- 172.21.15.111:0/1713363880 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f0b700055d0 con 0x55a78dc50c80 2023-10-11T16:32:28.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 10 client.15468 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 10 client.15468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:32:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 20 client.15468 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a78dcbb510 0x55a78dcbd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 4 client.15468 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:32:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.370+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/1713363880 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a78db64120 con 0x55a78dcbb510 2023-10-11T16:32:28.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.371+0000 7f0b767fc700 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a78dcbb510 0x55a78dcbd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.371+0000 7f0b5aff5700 10 client.15468 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:28.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.371+0000 7f0b767fc700 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a78dcbb510 0x55a78dcbd920 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f6c034cbaea71ad server_cookie=85753c1191a2a5ff in_seq=0 out_seq=0 2023-10-11T16:32:28.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.372+0000 7f0b5aff5700 1 -- 172.21.15.111:0/1713363880 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a78db64120 con 0x55a78dcbb510 2023-10-11T16:32:28.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.372+0000 7f0b5aff5700 10 client.15468 handle_command_reply: tid=0 2023-10-11T16:32:28.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.372+0000 7f0b90ff3b80 1 client.15468 shutdown 2023-10-11T16:32:28.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/1713363880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 msgr2=0x55a78dc61590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc61590 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0b70005090 tx=0x7f0b70004590 comp rx=0 tx=0).stop 2023-10-11T16:32:28.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/1713363880 shutdown_connections 2023-10-11T16:32:28.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a78dcbb510 0x55a78dcbd920 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a78dc52190 0x55a78dc5e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a78dc50c80 0x55a78dc61590 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 --2- 172.21.15.111:0/1713363880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a78dc515d0 0x55a78dc61ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b5aff5700 0 client.15468 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:28.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/1713363880 >> 172.21.15.111:0/1713363880 conn(0x55a78dc463e0 msgr2=0x55a78dc58490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/1713363880 shutdown_connections 2023-10-11T16:32:28.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 1 -- 172.21.15.111:0/1713363880 wait complete. 2023-10-11T16:32:28.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.373+0000 7f0b90ff3b80 20 client.15468 trim_cache size 0 max 16384 2023-10-11T16:32:28.388 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:32:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:32:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "df1e45e3-c41d-410b-a694-c7f875f07db0", 2023-10-11T16:32:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:32:28.388 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:32:28.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b600a1150 msgr2=0x7f0b600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b600a1150 0x7f0b600a3600 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55a78dc4c530 tx=0x7f0b680d1b80 comp rx=0 tx=0).stop 2023-10-11T16:32:28.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b781002d0 msgr2=0x7f0b781960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b781002d0 0x7f0b781960b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0b68004730 tx=0x7f0b68004760 comp rx=0 tx=0).stop 2023-10-11T16:32:28.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/2224697354 shutdown_connections 2023-10-11T16:32:28.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f0b600a1150 0x7f0b600a3600 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b78101e50 0x7f0b7819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b781002d0 0x7f0b781960b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 --2- 172.21.15.111:0/2224697354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b781014c0 0x7f0b781965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/2224697354 >> 172.21.15.111:0/2224697354 conn(0x7f0b780fb8d0 msgr2=0x7f0b780fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/2224697354 shutdown_connections 2023-10-11T16:32:28.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.374+0000 7f0b7d5ff700 1 -- 172.21.15.111:0/2224697354 wait complete. 2023-10-11T16:32:28.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:28 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1281360237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:28.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:28 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3721067569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:28.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:28 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1967782531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:28.393 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:32:28.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:28 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1281360237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:28.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:28 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3721067569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:28.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:28 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1967782531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:28.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1281360237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:32:28.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3721067569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:32:28.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:28 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1967782531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:32:28.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 -- 172.21.15.111:0/2258759582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c101e50 msgr2=0x7fac6c108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 --2- 172.21.15.111:0/2258759582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c101e50 0x7fac6c108710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fac6400b0a0 tx=0x7fac6400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 -- 172.21.15.111:0/2258759582 shutdown_connections 2023-10-11T16:32:28.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 --2- 172.21.15.111:0/2258759582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c101e50 0x7fac6c108710 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 --2- 172.21.15.111:0/2258759582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac6c1002d0 0x7fac6c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 --2- 172.21.15.111:0/2258759582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6c1014c0 0x7fac6c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 -- 172.21.15.111:0/2258759582 >> 172.21.15.111:0/2258759582 conn(0x7fac6c0fb8b0 msgr2=0x7fac6c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 -- 172.21.15.111:0/2258759582 shutdown_connections 2023-10-11T16:32:28.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.595+0000 7fac71dff700 1 -- 172.21.15.111:0/2258759582 wait complete. 2023-10-11T16:32:28.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.596+0000 7fac71dff700 1 Processor -- start 2023-10-11T16:32:28.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.596+0000 7fac71dff700 1 -- start start 2023-10-11T16:32:28.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.596+0000 7fac71dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c1002d0 0x7fac6c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.596+0000 7fac71dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6c1014c0 0x7fac6c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.596+0000 7fac71dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac6c101e50 0x7fac6c19f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.596+0000 7fac71dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c19ff20 con 0x7fac6c1002d0 2023-10-11T16:32:28.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.596+0000 7fac71dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1a0060 con 0x7fac6c1014c0 2023-10-11T16:32:28.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.596+0000 7fac71dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1a01f0 con 0x7fac6c101e50 2023-10-11T16:32:28.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c1002d0 0x7fac6c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6c1014c0 0x7fac6c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c1002d0 0x7fac6c1983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47316/0 (socket says 172.21.15.111:47316) 2023-10-11T16:32:28.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6b7fe700 1 -- 172.21.15.111:0/2747976292 learned_addr learned my addr 172.21.15.111:0/2747976292 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac6c101e50 0x7fac6c19f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6bfff700 1 -- 172.21.15.111:0/2747976292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6c1014c0 msgr2=0x7fac6c1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6c1014c0 0x7fac6c1988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6bfff700 1 -- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c1002d0 msgr2=0x7fac6c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c1002d0 0x7fac6c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6bfff700 1 -- 172.21.15.111:0/2747976292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac64009d00 con 0x7fac6c101e50 2023-10-11T16:32:28.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.597+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac6c101e50 0x7fac6c19f950 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fac6400b070 tx=0x7fac6400bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.598+0000 7fac68ff9700 1 -- 172.21.15.111:0/2747976292 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6400e040 con 0x7fac6c101e50 2023-10-11T16:32:28.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.598+0000 7fac71dff700 1 -- 172.21.15.111:0/2747976292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c10bca0 con 0x7fac6c101e50 2023-10-11T16:32:28.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.598+0000 7fac68ff9700 1 -- 172.21.15.111:0/2747976292 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac64009480 con 0x7fac6c101e50 2023-10-11T16:32:28.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.598+0000 7fac68ff9700 1 -- 172.21.15.111:0/2747976292 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac640052a0 con 0x7fac6c101e50 2023-10-11T16:32:28.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.598+0000 7fac71dff700 1 -- 172.21.15.111:0/2747976292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c10bb10 con 0x7fac6c101e50 2023-10-11T16:32:28.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.601+0000 7fac68ff9700 1 -- 172.21.15.111:0/2747976292 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 18) v1 ==== 135716+0+0 (secure 0 0 0) 0x7fac6401d480 con 0x7fac6c101e50 2023-10-11T16:32:28.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.602+0000 7fac8571fb80 1 Processor -- start 2023-10-11T16:32:28.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.602+0000 7fac8571fb80 1 -- start start 2023-10-11T16:32:28.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.602+0000 7fac68ff9700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac540a11d0 0x7fac540a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac68ff9700 1 -- 172.21.15.111:0/2747976292 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fac640cdd80 con 0x7fac6c101e50 2023-10-11T16:32:28.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 0x55ede19a7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a8110 0x55ede19a8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac8571fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede18ac060 con 0x55ede19a8110 2023-10-11T16:32:28.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac8571fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede18fe8a0 con 0x55ede19a6c00 2023-10-11T16:32:28.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac8571fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede18f7d30 con 0x55ede19a7550 2023-10-11T16:32:28.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6affd700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac540a11d0 0x7fac540a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a8110 0x55ede19a8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 0x55ede19a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a8110 0x55ede19a8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47328/0 (socket says 172.21.15.111:47328) 2023-10-11T16:32:28.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6bfff700 1 -- 172.21.15.111:0/2802918409 learned_addr learned my addr 172.21.15.111:0/2802918409 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6bfff700 1 -- 172.21.15.111:0/2802918409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 msgr2=0x55ede19a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2802918409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6bfff700 1 -- 172.21.15.111:0/2802918409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 msgr2=0x55ede19a7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2802918409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 0x55ede19a7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.603+0000 7fac6bfff700 1 -- 172.21.15.111:0/2802918409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ede1779630 con 0x55ede19a8110 2023-10-11T16:32:28.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.604+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2802918409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a8110 0x55ede19a8580 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fac6409ad80 tx=0x7fac640099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1851d784eddca server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.604+0000 7fac6affd700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac540a11d0 0x7fac540a3680 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fac60009d80 tx=0x7fac600098c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.604+0000 7fac51ffb700 1 -- 172.21.15.111:0/2802918409 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac64011390 con 0x55ede19a8110 2023-10-11T16:32:28.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.604+0000 7fac51ffb700 1 -- 172.21.15.111:0/2802918409 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac6409adf0 con 0x55ede19a8110 2023-10-11T16:32:28.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.604+0000 7fac51ffb700 1 -- 172.21.15.111:0/2802918409 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6409a2e0 con 0x55ede19a8110 2023-10-11T16:32:28.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 -- 172.21.15.111:0/2802918409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a8110 msgr2=0x55ede19a8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2802918409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a8110 0x55ede19a8580 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fac6409ad80 tx=0x7fac640099e0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 -- 172.21.15.111:0/2802918409 shutdown_connections 2023-10-11T16:32:28.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2802918409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a8110 0x55ede19a8580 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2802918409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 0x55ede19a7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2802918409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 -- 172.21.15.111:0/2802918409 >> 172.21.15.111:0/2802918409 conn(0x55ede199c360 msgr2=0x55ede19a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 -- 172.21.15.111:0/2802918409 shutdown_connections 2023-10-11T16:32:28.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.605+0000 7fac8571fb80 1 -- 172.21.15.111:0/2802918409 wait complete. 2023-10-11T16:32:28.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.606+0000 7fac8571fb80 1 -- start start 2023-10-11T16:32:28.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.606+0000 7fac8571fb80 1 Processor -- start 2023-10-11T16:32:28.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 0x55ede19b0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a7550 0x55ede19b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a8110 0x55ede19b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac8571fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede18998e0 con 0x55ede19a7550 2023-10-11T16:32:28.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac8571fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede1899580 con 0x55ede19a6c00 2023-10-11T16:32:28.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac8571fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede177b920 con 0x55ede19a8110 2023-10-11T16:32:28.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac8571fb80 10 client.0 _ll_register_callbacks cb 0x55ede18b7780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:28.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a7550 0x55ede19b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 0x55ede19b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a7550 0x55ede19b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47330/0 (socket says 172.21.15.111:47330) 2023-10-11T16:32:28.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6affd700 1 -- 172.21.15.111:0/2688690246 learned_addr learned my addr 172.21.15.111:0/2688690246 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a8110 0x55ede19b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6affd700 1 -- 172.21.15.111:0/2688690246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a8110 msgr2=0x55ede19b77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6affd700 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a8110 0x55ede19b77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6affd700 1 -- 172.21.15.111:0/2688690246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 msgr2=0x55ede19b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6affd700 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 0x55ede19b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.607+0000 7fac6affd700 1 -- 172.21.15.111:0/2688690246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c10bb10 con 0x55ede19a7550 2023-10-11T16:32:28.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac6affd700 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a7550 0x55ede19b07b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fac60010700 tx=0x7fac600047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246d2f0fe6a531a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac4affd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:28.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac4affd700 1 -- 172.21.15.111:0/2688690246 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6001f690 con 0x55ede19a7550 2023-10-11T16:32:28.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac4affd700 1 -- 172.21.15.111:0/2688690246 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fac6001f7f0 con 0x55ede19a7550 2023-10-11T16:32:28.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac8571fb80 1 -- 172.21.15.111:0/2688690246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ede1999b20 con 0x55ede19a7550 2023-10-11T16:32:28.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac4affd700 1 -- 172.21.15.111:0/2688690246 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac6001f9b0 con 0x55ede19a7550 2023-10-11T16:32:28.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac8571fb80 10 client.15480 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:28.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac8571fb80 1 -- 172.21.15.111:0/2688690246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ede18be190 con 0x55ede19a7550 2023-10-11T16:32:28.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.608+0000 7fac4affd700 1 -- 172.21.15.111:0/2688690246 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ede1999b20 con 0x55ede19a7550 2023-10-11T16:32:28.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.609+0000 7fac4affd700 1 -- 172.21.15.111:0/2688690246 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fac6001eac0 con 0x55ede19a7550 2023-10-11T16:32:28.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.609+0000 7fac8571fb80 10 client.15480 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:28.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.609+0000 7fac8571fb80 10 client.15480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:32:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.609+0000 7fac8571fb80 20 client.15480 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.609+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ede19bc150 0x55ede19be560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.609+0000 7fac8571fb80 4 client.15480 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:32:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.609+0000 7fac8571fb80 1 -- 172.21.15.111:0/2688690246 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ede18ba0a0 con 0x55ede19bc150 2023-10-11T16:32:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.609+0000 7fac6b7fe700 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ede19bc150 0x55ede19be560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.610+0000 7fac6b7fe700 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ede19bc150 0x55ede19be560 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=110f0f7fa3490c81 server_cookie=5c9878ae49d59385 in_seq=0 out_seq=0 2023-10-11T16:32:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.610+0000 7fac4affd700 10 client.15480 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.610+0000 7fac4affd700 1 -- 172.21.15.111:0/2688690246 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ede18ba0a0 con 0x55ede19bc150 2023-10-11T16:32:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.610+0000 7fac4affd700 10 client.15480 handle_command_reply: tid=0 2023-10-11T16:32:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.610+0000 7fac8571fb80 1 client.15480 shutdown 2023-10-11T16:32:28.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 -- 172.21.15.111:0/2688690246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a7550 msgr2=0x55ede19b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a7550 0x55ede19b07b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fac60010700 tx=0x7fac600047c0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 -- 172.21.15.111:0/2688690246 shutdown_connections 2023-10-11T16:32:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ede19bc150 0x55ede19be560 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a8110 0x55ede19b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a6c00 0x55ede19b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2688690246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a7550 0x55ede19b07b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 -- 172.21.15.111:0/2688690246 >> 172.21.15.111:0/2688690246 conn(0x55ede199c360 msgr2=0x55ede19a0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac4affd700 0 client.15480 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 -- 172.21.15.111:0/2688690246 shutdown_connections 2023-10-11T16:32:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 1 -- 172.21.15.111:0/2688690246 wait complete. 2023-10-11T16:32:28.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.611+0000 7fac8571fb80 20 client.15480 trim_cache size 0 max 16384 2023-10-11T16:32:28.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.615+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f205c061950 con 0x7f205c108180 2023-10-11T16:32:28.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.628+0000 7fac8571fb80 1 Processor -- start 2023-10-11T16:32:28.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac8571fb80 1 -- start start 2023-10-11T16:32:28.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19ae470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac8571fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede19a7f90 con 0x55ede19a6c00 2023-10-11T16:32:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac8571fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede19b1170 con 0x55ede19a8110 2023-10-11T16:32:28.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac8571fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede19afde0 con 0x55ede19a7550 2023-10-11T16:32:28.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac6affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac6affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:47336/0 (socket says 172.21.15.111:47336) 2023-10-11T16:32:28.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac6affd700 1 -- 172.21.15.111:0/3268014830 learned_addr learned my addr 172.21.15.111:0/3268014830 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19ae470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47336/0 (socket says 172.21.15.111:47336) 2023-10-11T16:32:28.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.629+0000 7fac6bfff700 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac6b7fe700 1 -- 172.21.15.111:0/3268014830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 msgr2=0x55ede19b7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac6b7fe700 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac6b7fe700 1 -- 172.21.15.111:0/3268014830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 msgr2=0x55ede19aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac6b7fe700 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19aaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac6b7fe700 1 -- 172.21.15.111:0/3268014830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ede18bdee0 con 0x55ede19a6c00 2023-10-11T16:32:28.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac6bfff700 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:28.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac6b7fe700 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19ae470 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55ede199c080 tx=0x7fac5c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b3513a7c69c7be server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac527fc700 1 -- 172.21.15.111:0/3268014830 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c017040 con 0x55ede19a6c00 2023-10-11T16:32:28.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac527fc700 1 -- 172.21.15.111:0/3268014830 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac5c018040 con 0x55ede19a6c00 2023-10-11T16:32:28.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.630+0000 7fac527fc700 1 -- 172.21.15.111:0/3268014830 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c0215a0 con 0x55ede19a6c00 2023-10-11T16:32:28.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 -- 172.21.15.111:0/3268014830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 msgr2=0x55ede19ae470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19ae470 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55ede199c080 tx=0x7fac5c009260 comp rx=0 tx=0).stop 2023-10-11T16:32:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 -- 172.21.15.111:0/3268014830 shutdown_connections 2023-10-11T16:32:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19ae470 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 --2- 172.21.15.111:0/3268014830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 -- 172.21.15.111:0/3268014830 >> 172.21.15.111:0/3268014830 conn(0x55ede199c360 msgr2=0x55ede19be160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 -- 172.21.15.111:0/3268014830 shutdown_connections 2023-10-11T16:32:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 -- 172.21.15.111:0/3268014830 wait complete. 2023-10-11T16:32:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.631+0000 7fac8571fb80 1 -- start start 2023-10-11T16:32:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac8571fb80 1 Processor -- start 2023-10-11T16:32:28.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac8571fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac8571fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede19b6e20 con 0x55ede19a6c00 2023-10-11T16:32:28.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac8571fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede19b6f90 con 0x55ede19a8110 2023-10-11T16:32:28.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac8571fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ede19b7150 con 0x55ede19a7550 2023-10-11T16:32:28.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac8571fb80 10 client.0 _ll_register_callbacks cb 0x7fac600104c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:28.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac6affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac6b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47340/0 (socket says 172.21.15.111:47340) 2023-10-11T16:32:28.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac6b7fe700 1 -- 172.21.15.111:0/2252895234 learned_addr learned my addr 172.21.15.111:0/2252895234 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:28.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.632+0000 7fac6bfff700 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac6b7fe700 1 -- 172.21.15.111:0/2252895234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 msgr2=0x55ede19b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac6b7fe700 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac6b7fe700 1 -- 172.21.15.111:0/2252895234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 msgr2=0x55ede19b68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:28.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac6b7fe700 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19b68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac6b7fe700 1 -- 172.21.15.111:0/2252895234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ede1779630 con 0x55ede19a6c00 2023-10-11T16:32:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac6b7fe700 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19b3380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0096b0 tx=0x7fac5c0122d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7e7817b406d412 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:28.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac4affd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac4affd700 1 -- 172.21.15.111:0/2252895234 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c004050 con 0x55ede19a6c00 2023-10-11T16:32:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac8571fb80 1 -- 172.21.15.111:0/2252895234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac6409adf0 con 0x55ede19a6c00 2023-10-11T16:32:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac4affd700 1 -- 172.21.15.111:0/2252895234 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7fac5c0041b0 con 0x55ede19a6c00 2023-10-11T16:32:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac4affd700 1 -- 172.21.15.111:0/2252895234 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac5c004370 con 0x55ede19a6c00 2023-10-11T16:32:28.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac8571fb80 10 client.15486 fetch_fsmap learned FSMap version 26 2023-10-11T16:32:28.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.633+0000 7fac8571fb80 1 -- 172.21.15.111:0/2252895234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fac5c009d00 con 0x55ede19a6c00 2023-10-11T16:32:28.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac4affd700 1 -- 172.21.15.111:0/2252895234 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac6409adf0 con 0x55ede19a6c00 2023-10-11T16:32:28.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac4affd700 1 -- 172.21.15.111:0/2252895234 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fac5c005380 con 0x55ede19a6c00 2023-10-11T16:32:28.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac8571fb80 10 client.15486 fetch_fsmap finished waiting for FSMap version 26 2023-10-11T16:32:28.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac8571fb80 10 client.15486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:32:28.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac8571fb80 20 client.15486 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:28.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ede1a10ee0 0x55ede1a132f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:28.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac8571fb80 4 client.15486 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:32:28.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac8571fb80 1 -- 172.21.15.111:0/2252895234 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ede18c6660 con 0x55ede1a10ee0 2023-10-11T16:32:28.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac6affd700 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ede1a10ee0 0x55ede1a132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac4affd700 10 client.15486 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.634+0000 7fac6affd700 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ede1a10ee0 0x55ede1a132f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a8f7fbc3c06b66 server_cookie=ea779d207bfecf89 in_seq=0 out_seq=0 2023-10-11T16:32:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac4affd700 1 -- 172.21.15.111:0/2252895234 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ede18c6660 con 0x55ede1a10ee0 2023-10-11T16:32:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac4affd700 10 client.15486 handle_command_reply: tid=0 2023-10-11T16:32:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 client.15486 shutdown 2023-10-11T16:32:28.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 -- 172.21.15.111:0/2252895234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 msgr2=0x55ede19b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19b3380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0096b0 tx=0x7fac5c0122d0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 -- 172.21.15.111:0/2252895234 shutdown_connections 2023-10-11T16:32:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ede1a10ee0 0x55ede1a132f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ede19a8110 0x55ede19b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ede19a6c00 0x55ede19b3380 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac4affd700 0 client.15486 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:28.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 --2- 172.21.15.111:0/2252895234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ede19a7550 0x55ede19b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 -- 172.21.15.111:0/2252895234 >> 172.21.15.111:0/2252895234 conn(0x55ede199c360 msgr2=0x55ede19ae470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 -- 172.21.15.111:0/2252895234 shutdown_connections 2023-10-11T16:32:28.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 1 -- 172.21.15.111:0/2252895234 wait complete. 2023-10-11T16:32:28.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.635+0000 7fac8571fb80 20 client.15486 trim_cache size 0 max 16384 2023-10-11T16:32:28.651 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:32:28.651 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:32:28.651 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:32:28.651 INFO:teuthology.orchestra.run.smithi111.stdout: "df1e45e3-c41d-410b-a694-c7f875f07db0": { 2023-10-11T16:32:28.652 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:32:28.652 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "df1e45e3-c41d-410b-a694-c7f875f07db0", 2023-10-11T16:32:28.652 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:32:28.652 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:32:28.652 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:32:28.653 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:32:28.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 -- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac540a11d0 msgr2=0x7fac540a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac540a11d0 0x7fac540a3680 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fac60009d80 tx=0x7fac600098c0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 -- 172.21.15.111:0/2747976292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac6c101e50 msgr2=0x7fac6c19f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac6c101e50 0x7fac6c19f950 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fac6400b070 tx=0x7fac6400bc80 comp rx=0 tx=0).stop 2023-10-11T16:32:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 -- 172.21.15.111:0/2747976292 shutdown_connections 2023-10-11T16:32:28.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7fac540a11d0 0x7fac540a3680 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac6c101e50 0x7fac6c19f950 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac6c1002d0 0x7fac6c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 --2- 172.21.15.111:0/2747976292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac6c1014c0 0x7fac6c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:28.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 -- 172.21.15.111:0/2747976292 >> 172.21.15.111:0/2747976292 conn(0x7fac6c0fb8b0 msgr2=0x7fac6c106cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:28.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 -- 172.21.15.111:0/2747976292 shutdown_connections 2023-10-11T16:32:28.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:28.636+0000 7fac71dff700 1 -- 172.21.15.111:0/2747976292 wait complete. 2023-10-11T16:32:28.657 INFO:tasks.cephfs.filesystem:scrub status for tag:df1e45e3-c41d-410b-a694-c7f875f07db0 - {'path': '/', 'tag': 'df1e45e3-c41d-410b-a694-c7f875f07db0', 'options': 'recursive,force'} 2023-10-11T16:32:29.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.199+0000 7f20537fe700 1 -- 172.21.15.111:0/545454472 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mgrmap(e 19) v1 ==== 135733+0+0 (secure 0 0 0) 0x7f20540960f0 con 0x7f205c108180 2023-10-11T16:32:29.201 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.200+0000 7f20537fe700 1 -- 172.21.15.111:0/545454472 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f2054076040 con 0x7f205c108180 2023-10-11T16:32:29.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f20480a1150 msgr2=0x7f20480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f20480a1150 0x7f20480a3600 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f205c074f00 tx=0x7f2058009260 comp rx=0 tx=0).stop 2023-10-11T16:32:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c108180 msgr2=0x7f205c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c108180 0x7f205c19f7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f205400efb0 tx=0x7f205400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 shutdown_connections 2023-10-11T16:32:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f20480a1150 0x7f20480a3600 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205c108180 0x7f205c19f7b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f205c073550 0x7f205c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 --2- 172.21.15.111:0/545454472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f205c074b30 0x7f205c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 >> 172.21.15.111:0/545454472 conn(0x7f205c0fb8e0 msgr2=0x7f205c0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 shutdown_connections 2023-10-11T16:32:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.204+0000 7f20643ff700 1 -- 172.21.15.111:0/545454472 wait complete. 2023-10-11T16:32:29.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2023-10-11T16:32:29.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.444+0000 7f8320dff700 1 -- 172.21.15.111:0/2168743816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c074b30 msgr2=0x7f831c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.444+0000 7f8320dff700 1 --2- 172.21.15.111:0/2168743816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c074b30 0x7f831c072f80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f830c009990 tx=0x7f830c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.444+0000 7f8320dff700 1 -- 172.21.15.111:0/2168743816 shutdown_connections 2023-10-11T16:32:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.444+0000 7f8320dff700 1 --2- 172.21.15.111:0/2168743816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f831c10c890 0x7f831c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.444+0000 7f8320dff700 1 --2- 172.21.15.111:0/2168743816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c074b30 0x7f831c072f80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.444+0000 7f8320dff700 1 --2- 172.21.15.111:0/2168743816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c073550 0x7f831c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.444+0000 7f8320dff700 1 -- 172.21.15.111:0/2168743816 >> 172.21.15.111:0/2168743816 conn(0x7f831c0fb8e0 msgr2=0x7f831c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:29.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.445+0000 7f8320dff700 1 -- 172.21.15.111:0/2168743816 shutdown_connections 2023-10-11T16:32:29.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.445+0000 7f8320dff700 1 -- 172.21.15.111:0/2168743816 wait complete. 2023-10-11T16:32:29.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.445+0000 7f8320dff700 1 Processor -- start 2023-10-11T16:32:29.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f8320dff700 1 -- start start 2023-10-11T16:32:29.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f8320dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c073550 0x7f831c19c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f8320dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c074b30 0x7f831c19cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f8320dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f831c10c890 0x7f831c1a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f8320dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c1a4170 con 0x7f831c073550 2023-10-11T16:32:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f8320dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c1a42b0 con 0x7f831c10c890 2023-10-11T16:32:29.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f8320dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c1a4440 con 0x7f831c074b30 2023-10-11T16:32:29.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f831a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c073550 0x7f831c19c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:29.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f831ad9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f831c10c890 0x7f831c1a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:29.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f8319d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c074b30 0x7f831c19cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:29.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f831a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c073550 0x7f831c19c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47382/0 (socket says 172.21.15.111:47382) 2023-10-11T16:32:29.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.446+0000 7f831a59c700 1 -- 172.21.15.111:0/673308988 learned_addr learned my addr 172.21.15.111:0/673308988 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:29.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f831ad9d700 1 -- 172.21.15.111:0/673308988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c074b30 msgr2=0x7f831c19cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f831ad9d700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c074b30 0x7f831c19cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f831ad9d700 1 -- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c073550 msgr2=0x7f831c19c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f831ad9d700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c073550 0x7f831c19c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f831ad9d700 1 -- 172.21.15.111:0/673308988 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c0095f0 con 0x7f831c10c890 2023-10-11T16:32:29.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f831ad9d700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f831c10c890 0x7f831c1a3ba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f831400efc0 tx=0x7f831400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:29.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f830b7fe700 1 -- 172.21.15.111:0/673308988 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314018070 con 0x7f831c10c890 2023-10-11T16:32:29.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831c112210 con 0x7f831c10c890 2023-10-11T16:32:29.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f830b7fe700 1 -- 172.21.15.111:0/673308988 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f831400f040 con 0x7f831c10c890 2023-10-11T16:32:29.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f830b7fe700 1 -- 172.21.15.111:0/673308988 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831400cd40 con 0x7f831c10c890 2023-10-11T16:32:29.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.447+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831c112080 con 0x7f831c10c890 2023-10-11T16:32:29.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.449+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831c1023e0 con 0x7f831c10c890 2023-10-11T16:32:29.456 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: ignoring --setuser ceph since I am not root 2023-10-11T16:32:29.456 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: ignoring --setgroup ceph since I am not root 2023-10-11T16:32:29.456 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mgr[134250]: -- 172.21.15.150:0/2854119592 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562ebf505520 con 0x562ebf4dac00 2023-10-11T16:32:29.456 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:29.373+0000 7f97e526d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T16:32:29.456 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:29.452+0000 7f97e526d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T16:32:29.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: pgmap v345: 129 pgs: 129 active+clean; 661 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 7 op/s 2023-10-11T16:32:29.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:32:29.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:32:29.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:32:29.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/545454472' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-11T16:32:29.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-11T16:32:29.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.450+0000 7f830b7fe700 1 -- 172.21.15.111:0/673308988 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 19) v1 ==== 135733+0+0 (secure 0 0 0) 0x7f8314003a00 con 0x7f831c10c890 2023-10-11T16:32:29.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.451+0000 7f830b7fe700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f83040a1220 0x7f83040a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:29.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.451+0000 7f831a59c700 1 -- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f83040a1220 msgr2=0x7f83040a36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:29.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.451+0000 7f830b7fe700 1 -- 172.21.15.111:0/673308988 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f83140cc870 con 0x7f831c10c890 2023-10-11T16:32:29.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.452+0000 7f831a59c700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f83040a1220 0x7f83040a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:32:29.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.455+0000 7f830b7fe700 1 -- 172.21.15.111:0/673308988 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8314095c20 con 0x7f831c10c890 2023-10-11T16:32:29.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: ignoring --setuser ceph since I am not root 2023-10-11T16:32:29.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: ignoring --setgroup ceph since I am not root 2023-10-11T16:32:29.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mgr[98801]: -- 172.21.15.111:0/3074118759 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5588dd159520 con 0x5588dd131000 2023-10-11T16:32:29.489 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: pgmap v345: 129 pgs: 129 active+clean; 661 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 7 op/s 2023-10-11T16:32:29.489 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:32:29.490 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.490 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:32:29.490 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:32:29.490 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.490 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.491 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/545454472' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-11T16:32:29.491 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-11T16:32:29.535 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: pgmap v345: 129 pgs: 129 active+clean; 661 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 7 op/s 2023-10-11T16:32:29.535 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:32:29.535 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.536 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:32:29.536 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:32:29.536 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.536 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:32:29.536 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/545454472' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-11T16:32:29.536 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-11T16:32:29.537 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: ignoring --setuser ceph since I am not root 2023-10-11T16:32:29.537 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: ignoring --setgroup ceph since I am not root 2023-10-11T16:32:29.537 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-mgr[133340]: -- 172.21.15.130:0/953555899 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f0e6949520 con 0x55f0e691f000 2023-10-11T16:32:29.537 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:29.365+0000 7fca535a0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T16:32:29.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:29.443+0000 7fca535a0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T16:32:29.538 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:29.534+0000 7fca535a0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T16:32:29.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.653+0000 7f831a59c700 1 -- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f83040a1220 msgr2=0x7f83040a36d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:29.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.653+0000 7f831a59c700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f83040a1220 0x7f83040a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-11T16:32:29.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.696+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f831c061b20 con 0x7f831c10c890 2023-10-11T16:32:29.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.704+0000 7f830b7fe700 1 -- 172.21.15.111:0/673308988 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8314099240 con 0x7f831c10c890 2023-10-11T16:32:29.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f83040a1220 msgr2=0x7f83040a36d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:32:29.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f83040a1220 0x7f83040a36d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f831c10c890 msgr2=0x7f831c1a3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f831c10c890 0x7f831c1a3ba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f831400efc0 tx=0x7f831400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 shutdown_connections 2023-10-11T16:32:29.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f83040a1220 0x7f83040a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f831c10c890 0x7f831c1a3ba0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f831c073550 0x7f831c19c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 --2- 172.21.15.111:0/673308988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f831c074b30 0x7f831c19cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.706+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 >> 172.21.15.111:0/673308988 conn(0x7f831c0fb8e0 msgr2=0x7f831c0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:29.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.707+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 shutdown_connections 2023-10-11T16:32:29.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.707+0000 7f8320dff700 1 -- 172.21.15.111:0/673308988 wait complete. 2023-10-11T16:32:29.727 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2023-10-11T16:32:29.788 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:29.486+0000 7f1d021f1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T16:32:29.789 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:29.609+0000 7f1d021f1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T16:32:29.789 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:29.725+0000 7f1d021f1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T16:32:29.797 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:29.627+0000 7fca535a0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T16:32:29.806 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:29.543+0000 7f97e526d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T16:32:29.806 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:29.635+0000 7f97e526d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T16:32:29.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 -- 172.21.15.111:0/2905820408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a5c073550 msgr2=0x7f1a5c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 --2- 172.21.15.111:0/2905820408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a5c073550 0x7f1a5c0739c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1a500098f0 tx=0x7f1a50009c00 comp rx=0 tx=0).stop 2023-10-11T16:32:29.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 -- 172.21.15.111:0/2905820408 shutdown_connections 2023-10-11T16:32:29.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 --2- 172.21.15.111:0/2905820408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c10c890 0x7f1a5c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 --2- 172.21.15.111:0/2905820408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a5c074b30 0x7f1a5c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 --2- 172.21.15.111:0/2905820408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a5c073550 0x7f1a5c0739c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 -- 172.21.15.111:0/2905820408 >> 172.21.15.111:0/2905820408 conn(0x7f1a5c0fb8e0 msgr2=0x7f1a5c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:29.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 -- 172.21.15.111:0/2905820408 shutdown_connections 2023-10-11T16:32:29.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.925+0000 7f1a61bff700 1 -- 172.21.15.111:0/2905820408 wait complete. 2023-10-11T16:32:29.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.926+0000 7f1a61bff700 1 Processor -- start 2023-10-11T16:32:29.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.926+0000 7f1a61bff700 1 -- start start 2023-10-11T16:32:29.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.926+0000 7f1a61bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a5c073550 0x7f1a5c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:29.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.926+0000 7f1a61bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a5c074b30 0x7f1a5c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:29.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a61bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c10c890 0x7f1a5c1a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:29.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a61bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c1a41e0 con 0x7f1a5c073550 2023-10-11T16:32:29.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a61bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c1a4320 con 0x7f1a5c074b30 2023-10-11T16:32:29.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a61bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c1a44b0 con 0x7f1a5c10c890 2023-10-11T16:32:29.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c10c890 0x7f1a5c1a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:29.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a5c074b30 0x7f1a5c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:29.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a5c073550 0x7f1a5c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:29.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c10c890 0x7f1a5c1a3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:47392/0 (socket says 172.21.15.111:47392) 2023-10-11T16:32:29.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5bfff700 1 -- 172.21.15.111:0/1843378612 learned_addr learned my addr 172.21.15.111:0/1843378612 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:29.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5affd700 1 -- 172.21.15.111:0/1843378612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c10c890 msgr2=0x7f1a5c1a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5affd700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c10c890 0x7f1a5c1a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5affd700 1 -- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a5c073550 msgr2=0x7f1a5c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:29.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5affd700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a5c073550 0x7f1a5c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:29.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5affd700 1 -- 172.21.15.111:0/1843378612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c0095f0 con 0x7f1a5c074b30 2023-10-11T16:32:29.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5bfff700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c10c890 0x7f1a5c1a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:29.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.927+0000 7f1a5affd700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a5c074b30 0x7f1a5c19cba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a50000c00 tx=0x7f1a5000b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:29.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.928+0000 7f1a58ff9700 1 -- 172.21.15.111:0/1843378612 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5001c070 con 0x7f1a5c074b30 2023-10-11T16:32:29.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.928+0000 7f1a58ff9700 1 -- 172.21.15.111:0/1843378612 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a5000f7d0 con 0x7f1a5c074b30 2023-10-11T16:32:29.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.928+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a50009520 con 0x7f1a5c074b30 2023-10-11T16:32:29.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.928+0000 7f1a58ff9700 1 -- 172.21.15.111:0/1843378612 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500039b0 con 0x7f1a5c074b30 2023-10-11T16:32:29.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.928+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c112230 con 0x7f1a5c074b30 2023-10-11T16:32:29.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.929+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c1023e0 con 0x7f1a5c074b30 2023-10-11T16:32:29.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.931+0000 7f1a58ff9700 1 -- 172.21.15.111:0/1843378612 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 19) v1 ==== 135733+0+0 (secure 0 0 0) 0x7f1a50020030 con 0x7f1a5c074b30 2023-10-11T16:32:29.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.932+0000 7f1a58ff9700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1a440a11a0 0x7f1a440a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.932+0000 7f1a5b7fe700 1 -- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1a440a11a0 msgr2=0x7f1a440a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.932+0000 7f1a58ff9700 1 -- 172.21.15.111:0/1843378612 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f1a500cdca0 con 0x7f1a5c074b30 2023-10-11T16:32:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.932+0000 7f1a5b7fe700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1a440a11a0 0x7f1a440a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:32:29.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:29.936+0000 7f1a58ff9700 1 -- 172.21.15.111:0/1843378612 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1a50097050 con 0x7f1a5c074b30 2023-10-11T16:32:30.081 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:29 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:29.852+0000 7f1d021f1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T16:32:30.081 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:29.796+0000 7fca535a0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T16:32:30.113 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:29.806+0000 7f97e526d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T16:32:30.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.133+0000 7f1a5b7fe700 1 -- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1a440a11a0 msgr2=0x7f1a440a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:30.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.133+0000 7f1a5b7fe700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1a440a11a0 0x7f1a440a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-11T16:32:30.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.138+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f1a5c02cfe0 con 0x7f1a5c074b30 2023-10-11T16:32:30.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.145+0000 7f1a58ff9700 1 -- 172.21.15.111:0/1843378612 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1a5009a670 con 0x7f1a5c074b30 2023-10-11T16:32:30.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1a440a11a0 msgr2=0x7f1a440a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:32:30.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1a440a11a0 0x7f1a440a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a5c074b30 msgr2=0x7f1a5c19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:30.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a5c074b30 0x7f1a5c19cba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a50000c00 tx=0x7f1a5000b790 comp rx=0 tx=0).stop 2023-10-11T16:32:30.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 shutdown_connections 2023-10-11T16:32:30.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f1a440a11a0 0x7f1a440a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c10c890 0x7f1a5c1a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a5c073550 0x7f1a5c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 --2- 172.21.15.111:0/1843378612 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a5c074b30 0x7f1a5c19cba0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.147+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 >> 172.21.15.111:0/1843378612 conn(0x7f1a5c0fb8e0 msgr2=0x7f1a5c0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:30.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.148+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 shutdown_connections 2023-10-11T16:32:30.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.148+0000 7f1a61bff700 1 -- 172.21.15.111:0/1843378612 wait complete. 2023-10-11T16:32:30.170 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2023-10-11T16:32:30.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.368+0000 7f369f7ff700 1 -- 172.21.15.111:0/2301901773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3698101450 msgr2=0x7f36981018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:30.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.368+0000 7f369f7ff700 1 --2- 172.21.15.111:0/2301901773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3698101450 0x7f36981018a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3688009990 tx=0x7f3688009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.368+0000 7f369f7ff700 1 -- 172.21.15.111:0/2301901773 shutdown_connections 2023-10-11T16:32:30.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.368+0000 7f369f7ff700 1 --2- 172.21.15.111:0/2301901773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3698101de0 0x7f36981086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.368+0000 7f369f7ff700 1 --2- 172.21.15.111:0/2301901773 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3698100260 0x7f3698100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.368+0000 7f369f7ff700 1 --2- 172.21.15.111:0/2301901773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3698101450 0x7f36981018a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.368+0000 7f369f7ff700 1 -- 172.21.15.111:0/2301901773 >> 172.21.15.111:0/2301901773 conn(0x7f36980fb880 msgr2=0x7f36980fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.369+0000 7f369f7ff700 1 -- 172.21.15.111:0/2301901773 shutdown_connections 2023-10-11T16:32:30.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.369+0000 7f369f7ff700 1 -- 172.21.15.111:0/2301901773 wait complete. 2023-10-11T16:32:30.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369f7ff700 1 Processor -- start 2023-10-11T16:32:30.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369f7ff700 1 -- start start 2023-10-11T16:32:30.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369f7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3698100260 0x7f3698195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:30.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369f7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3698101450 0x7f36981964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:30.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369f7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3698101de0 0x7f369819d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:30.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369f7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369819db00 con 0x7f3698101450 2023-10-11T16:32:30.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369f7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369819dc40 con 0x7f3698100260 2023-10-11T16:32:30.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369f7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369819ddd0 con 0x7f3698101de0 2023-10-11T16:32:30.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369d59b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3698100260 0x7f3698195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:30.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369cd9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3698101450 0x7f36981964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:30.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369d59b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3698100260 0x7f3698195f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:47106/0 (socket says 172.21.15.111:47106) 2023-10-11T16:32:30.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369d59b700 1 -- 172.21.15.111:0/2855830675 learned_addr learned my addr 172.21.15.111:0/2855830675 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:30.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.370+0000 7f369dd9c700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3698101de0 0x7f369819d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:30.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369cd9a700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3698101de0 msgr2=0x7f369819d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:30.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369cd9a700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3698101de0 0x7f369819d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369cd9a700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3698100260 msgr2=0x7f3698195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369cd9a700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3698100260 0x7f3698195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369cd9a700 1 -- 172.21.15.111:0/2855830675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36880095f0 con 0x7f3698101450 2023-10-11T16:32:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3698100260 0x7f3698195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:30.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369cd9a700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3698101450 0x7f36981964c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f368800b680 tx=0x7f3688009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368801d070 con 0x7f3698101450 2023-10-11T16:32:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369810bbb0 con 0x7f3698101450 2023-10-11T16:32:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.371+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369810ba20 con 0x7f3698101450 2023-10-11T16:32:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.372+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36880173f0 con 0x7f3698101450 2023-10-11T16:32:30.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.372+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368802a610 con 0x7f3698101450 2023-10-11T16:32:30.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.372+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c005320 con 0x7f3698101450 2023-10-11T16:32:30.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.376+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 135733+0+0 (secure 0 0 0) 0x7f368800fad0 con 0x7f3698101450 2023-10-11T16:32:30.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.377+0000 7f368e7fc700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 0x7f36840a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:30.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.377+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9210+0+0 (secure 0 0 0) 0x7f36880cdf60 con 0x7f3698101450 2023-10-11T16:32:30.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.377+0000 7f369d59b700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 msgr2=0x7f36840a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:30.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.377+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 0x7f36840a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-11T16:32:30.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.380+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3688097370 con 0x7f3698101450 2023-10-11T16:32:30.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:30.079+0000 7f1d021f1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T16:32:30.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-10-11T16:32:30.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-mon[98527]: mgrmap e19: x(active, since 8m), standbys: z, y 2023-10-11T16:32:30.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' 2023-10-11T16:32:30.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' 2023-10-11T16:32:30.419 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:30.104+0000 7fca535a0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T16:32:30.419 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:30.193+0000 7fca535a0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T16:32:30.420 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:30.322+0000 7fca535a0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T16:32:30.420 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-10-11T16:32:30.421 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-mon[131859]: mgrmap e19: x(active, since 8m), standbys: z, y 2023-10-11T16:32:30.421 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' 2023-10-11T16:32:30.422 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' 2023-10-11T16:32:30.431 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-10-11T16:32:30.431 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-mon[132217]: mgrmap e19: x(active, since 8m), standbys: z, y 2023-10-11T16:32:30.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' 2023-10-11T16:32:30.432 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' 2023-10-11T16:32:30.432 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:30.112+0000 7f97e526d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T16:32:30.432 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:30.202+0000 7f97e526d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T16:32:30.433 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:30.333+0000 7f97e526d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T16:32:30.433 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:30.430+0000 7f97e526d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T16:32:30.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.578+0000 7f369d59b700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 msgr2=0x7f36840a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:30.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.578+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 0x7f36840a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-11T16:32:30.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.610+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 --> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f367c000bf0 con 0x7f36840a11a0 2023-10-11T16:32:30.770 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:30.488+0000 7f1d021f1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T16:32:30.770 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:30.604+0000 7f1d021f1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T16:32:30.831 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:30.417+0000 7fca535a0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T16:32:30.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.979+0000 7f369d59b700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 msgr2=0x7f36840a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:30.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:30.979+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 0x7f36840a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-11T16:32:31.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:30.768+0000 7f1d021f1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T16:32:31.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:30 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:30.891+0000 7f1d021f1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T16:32:31.202 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:30.855+0000 7f97e526d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T16:32:31.331 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:30.837+0000 7fca535a0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T16:32:31.702 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:31.426+0000 7f97e526d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T16:32:31.702 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:31.597+0000 7f97e526d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T16:32:31.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:31.780+0000 7f369d59b700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 msgr2=0x7f36840a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:31.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:31.780+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 0x7f36840a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-11T16:32:31.833 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:31.399+0000 7fca535a0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T16:32:31.834 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:31.569+0000 7fca535a0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T16:32:31.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:31 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:31.424+0000 7f1d021f1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T16:32:32.581 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:32 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:32.330+0000 7fca535a0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:32:32.582 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:32 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:32.421+0000 7fca535a0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T16:32:32.628 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:32.364+0000 7f97e526d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:32:32.628 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:32.455+0000 7f97e526d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T16:32:32.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:32 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:32.163+0000 7f1d021f1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T16:32:32.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:32 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:32.374+0000 7f1d021f1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T16:32:32.864 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:32 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:32.589+0000 7fca535a0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:32:32.904 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:32.627+0000 7f97e526d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:32:33.202 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:32.904+0000 7f97e526d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T16:32:33.203 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:33.131+0000 7f97e526d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T16:32:33.239 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:32 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:32.863+0000 7fca535a0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T16:32:33.239 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:33 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:33.091+0000 7fca535a0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T16:32:33.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:33.382+0000 7f369d59b700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 msgr2=0x7f36840a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:33.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:33.382+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 0x7f36840a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-11T16:32:33.499 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:33 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:33.238+0000 7fca535a0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T16:32:33.543 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:33.281+0000 7f97e526d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T16:32:33.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:33 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:33.312+0000 7f1d021f1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:32:33.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:33 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:33.431+0000 7f1d021f1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T16:32:33.831 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:33 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:33.498+0000 7fca535a0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T16:32:33.952 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:33.542+0000 7f97e526d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T16:32:34.013 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:33 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:33.653+0000 7f1d021f1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T16:32:34.289 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:34.011+0000 7f1d021f1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T16:32:34.519 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:34 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:34.198+0000 7fca535a0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T16:32:34.520 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:34 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:34.328+0000 7fca535a0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T16:32:34.570 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:34.248+0000 7f97e526d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T16:32:34.570 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:34.378+0000 7f97e526d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T16:32:34.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:34.287+0000 7f1d021f1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T16:32:34.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:34.477+0000 7f1d021f1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T16:32:34.831 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:34 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:34.518+0000 7fca535a0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T16:32:34.832 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:34 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:34.631+0000 7fca535a0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T16:32:34.898 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:34.569+0000 7f97e526d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T16:32:34.899 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:34.682+0000 7f97e526d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T16:32:35.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:34.820+0000 7f1d021f1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T16:32:35.331 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:34 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:34.846+0000 7fca535a0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T16:32:36.081 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:32:35 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y[133315]: 2023-10-11T16:32:35.704+0000 7fca535a0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T16:32:36.090 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:34.897+0000 7f97e526d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T16:32:36.090 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:32:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z[134151]: 2023-10-11T16:32:35.761+0000 7f97e526d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T16:32:36.091 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:35 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:35.756+0000 7f1d021f1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T16:32:36.091 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:35 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:35.929+0000 7f1d021f1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T16:32:36.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:36.103+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 20) v1 ==== 135733+0+0 (secure 0 0 0) 0x7f3688097740 con 0x7f3698101450 2023-10-11T16:32:36.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:36.184+0000 7f1d021f1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T16:32:36.385 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:36.331+0000 7f1d021f1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T16:32:36.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: Standby manager daemon y restarted 2023-10-11T16:32:36.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: Standby manager daemon y started 2023-10-11T16:32:36.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-11T16:32:36.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:32:36.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-11T16:32:36.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:32:36.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: Standby manager daemon z restarted 2023-10-11T16:32:36.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: Standby manager daemon z started 2023-10-11T16:32:36.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-11T16:32:36.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:32:36.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-11T16:32:36.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-mon[98527]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:32:36.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: Standby manager daemon y restarted 2023-10-11T16:32:36.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: Standby manager daemon y started 2023-10-11T16:32:36.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-11T16:32:36.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:32:36.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-11T16:32:36.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:32:36.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: Standby manager daemon z restarted 2023-10-11T16:32:36.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: Standby manager daemon z started 2023-10-11T16:32:36.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-11T16:32:36.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:32:36.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-11T16:32:36.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:36 smithi150 ceph-mon[132217]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:32:36.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: Standby manager daemon y restarted 2023-10-11T16:32:36.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: Standby manager daemon y started 2023-10-11T16:32:36.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-11T16:32:36.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:32:36.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-11T16:32:36.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.130:0/1586596012' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:32:36.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: Standby manager daemon z restarted 2023-10-11T16:32:36.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: Standby manager daemon z started 2023-10-11T16:32:36.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-11T16:32:36.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-11T16:32:36.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-11T16:32:36.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:36 smithi130 ceph-mon[131859]: from='mgr.? 172.21.15.150:0/1169126370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-11T16:32:36.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:36.585+0000 7f369d59b700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 msgr2=0x7f36840a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2838247405 2023-10-11T16:32:36.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:36.585+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 0x7f36840a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-10-11T16:32:36.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:36.614+0000 7f1d021f1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T16:32:37.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-mon[98527]: mgrmap e20: x(active, since 8m), standbys: y, z 2023-10-11T16:32:37.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:37 smithi150 ceph-mon[132217]: mgrmap e20: x(active, since 8m), standbys: y, z 2023-10-11T16:32:37.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:37 smithi130 ceph-mon[131859]: mgrmap e20: x(active, since 8m), standbys: y, z 2023-10-11T16:32:37.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:37.692+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 21) v1 ==== 135400+0+0 (secure 0 0 0) 0x7f3688062af0 con 0x7f3698101450 2023-10-11T16:32:37.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:37.693+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 msgr2=0x7f36840a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:32:37.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:37.693+0000 7f368e7fc700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/2838247405,v1:172.21.15.111:6801/2838247405] conn(0x7f36840a11a0 0x7f36840a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:37.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.678+0000 7f1d021f1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T16:32:37.993 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.912+0000 7f1c6164f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.993 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.912+0000 7f1c6164f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.993 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.912+0000 7f1c6164f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.993 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.912+0000 7f1c6164f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.994 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.912+0000 7f1c6164f700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.994 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.913+0000 7f1c5ce46700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.994 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.913+0000 7f1c5ce46700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.994 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.913+0000 7f1c5ce46700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.994 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.913+0000 7f1c5ce46700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:37.995 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:32:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:32:37.913+0000 7f1c5ce46700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:32:38.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: Active manager daemon x restarted 2023-10-11T16:32:38.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: Activating manager daemon x 2023-10-11T16:32:38.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: osdmap e92: 12 total, 12 up, 12 in 2023-10-11T16:32:38.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: mgrmap e21: x(active, starting, since 0.0121992s), standbys: y, z 2023-10-11T16:32:38.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:32:38.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:32:38.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:32:38.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-11T16:32:38.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-11T16:32:38.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-11T16:32:38.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-11T16:32:38.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-11T16:32:38.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-11T16:32:38.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:32:38.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:32:38.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:32:38.389 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:32:38.390 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:32:38.390 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:32:38.390 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-11T16:32:38.390 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-11T16:32:38.390 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-11T16:32:38.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:32:38.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:32:38.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:32:38.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:32:38.391 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:32:38.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:32:38.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:32:38.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:32:38.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:32:38.392 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:32:38.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:32:38.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:32:38.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-11T16:32:38.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-11T16:32:38.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-11T16:32:38.393 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: Manager daemon x is now available 2023-10-11T16:32:38.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:38.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:38.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-11T16:32:38.394 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:38 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-11T16:32:38.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: Active manager daemon x restarted 2023-10-11T16:32:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: Activating manager daemon x 2023-10-11T16:32:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: osdmap e92: 12 total, 12 up, 12 in 2023-10-11T16:32:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: mgrmap e21: x(active, starting, since 0.0121992s), standbys: y, z 2023-10-11T16:32:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:32:38.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:32:38.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:32:38.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-11T16:32:38.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-11T16:32:38.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-11T16:32:38.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-11T16:32:38.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-11T16:32:38.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-11T16:32:38.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:32:38.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:32:38.455 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:32:38.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:32:38.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:32:38.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:32:38.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-11T16:32:38.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-11T16:32:38.456 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-11T16:32:38.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:32:38.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:32:38.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:32:38.457 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:32:38.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:32:38.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:32:38.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:32:38.458 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:32:38.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:32:38.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:32:38.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:32:38.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:32:38.459 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-11T16:32:38.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-11T16:32:38.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-11T16:32:38.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: Manager daemon x is now available 2023-10-11T16:32:38.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:38.460 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:38.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-11T16:32:38.461 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:38 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-11T16:32:38.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: Active manager daemon x restarted 2023-10-11T16:32:38.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: Activating manager daemon x 2023-10-11T16:32:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: osdmap e92: 12 total, 12 up, 12 in 2023-10-11T16:32:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: mgrmap e21: x(active, starting, since 0.0121992s), standbys: y, z 2023-10-11T16:32:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-11T16:32:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-11T16:32:38.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-11T16:32:38.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-11T16:32:38.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-11T16:32:38.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-11T16:32:38.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-11T16:32:38.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-11T16:32:38.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-11T16:32:38.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:32:38.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:32:38.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:32:38.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:32:38.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:32:38.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:32:38.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-11T16:32:38.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-11T16:32:38.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-11T16:32:38.586 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-11T16:32:38.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-11T16:32:38.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-11T16:32:38.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-11T16:32:38.587 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-11T16:32:38.588 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-11T16:32:38.589 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-11T16:32:38.589 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-11T16:32:38.589 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-11T16:32:38.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-11T16:32:38.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-11T16:32:38.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-11T16:32:38.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-11T16:32:38.590 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-11T16:32:38.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-11T16:32:38.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: Manager daemon x is now available 2023-10-11T16:32:38.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:38.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:38.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-11T16:32:38.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:38 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-11T16:32:38.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.700+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mon.0 v2:172.21.15.111:3300/0 9 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f3688017560 con 0x7f3698101450 2023-10-11T16:32:38.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.701+0000 7f368e7fc700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f368401b4a0 0x7f368401b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:38.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.701+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 --> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f367c000bf0 con 0x7f368401b4a0 2023-10-11T16:32:38.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.705+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f368401b4a0 0x7f368401b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:38.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.711+0000 7f369d59b700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f368401b4a0 0x7f368401b870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3694009fd0 tx=0x7f3694009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:38.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.775+0000 7f368e7fc700 1 -- 172.21.15.111:0/2855830675 <== mgr.25544 v2:172.21.15.111:6800/1233945144 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f367c000bf0 con 0x7f368401b4a0 2023-10-11T16:32:38.777 INFO:teuthology.orchestra.run.smithi111.stdout:Schedule set for path / 2023-10-11T16:32:38.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f368401b4a0 msgr2=0x7f368401b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:38.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369f7ff700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f368401b4a0 0x7f368401b870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3694009fd0 tx=0x7f3694009260 comp rx=0 tx=0).stop 2023-10-11T16:32:38.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3698101450 msgr2=0x7f36981964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:38.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369f7ff700 1 --2- 172.21.15.111:0/2855830675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3698101450 0x7f36981964c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f368800b680 tx=0x7f3688009580 comp rx=0 tx=0).stop 2023-10-11T16:32:38.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369dd9c700 1 -- 172.21.15.111:0/2855830675 reap_dead start 2023-10-11T16:32:38.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 shutdown_connections 2023-10-11T16:32:38.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 >> 172.21.15.111:0/2855830675 conn(0x7f36980fb880 msgr2=0x7f3698104930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:38.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 shutdown_connections 2023-10-11T16:32:38.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:38.778+0000 7f369f7ff700 1 -- 172.21.15.111:0/2855830675 wait complete. 2023-10-11T16:32:38.802 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2023-10-11T16:32:39.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 -- 172.21.15.111:0/727215195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238100cb0 msgr2=0x7fa238103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/727215195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238100cb0 0x7fa238103090 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa228009990 tx=0x7fa228009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 -- 172.21.15.111:0/727215195 shutdown_connections 2023-10-11T16:32:39.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/727215195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa238105ef0 0x7fa238108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/727215195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238100cb0 0x7fa238103090 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/727215195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2381035d0 0x7fa2381059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 -- 172.21.15.111:0/727215195 >> 172.21.15.111:0/727215195 conn(0x7fa2380fa750 msgr2=0x7fa2380fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:39.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 -- 172.21.15.111:0/727215195 shutdown_connections 2023-10-11T16:32:39.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.031+0000 7fa23e5ff700 1 -- 172.21.15.111:0/727215195 wait complete. 2023-10-11T16:32:39.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.032+0000 7fa23e5ff700 1 Processor -- start 2023-10-11T16:32:39.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.032+0000 7fa23e5ff700 1 -- start start 2023-10-11T16:32:39.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.032+0000 7fa23e5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa238100cb0 0x7fa2381981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:39.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.032+0000 7fa23e5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2381035d0 0x7fa238198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:39.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.032+0000 7fa23e5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238105ef0 0x7fa23819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:39.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.032+0000 7fa23e5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23819fd60 con 0x7fa238105ef0 2023-10-11T16:32:39.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.032+0000 7fa23e5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23819fea0 con 0x7fa2381035d0 2023-10-11T16:32:39.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.032+0000 7fa23e5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2381a0030 con 0x7fa238100cb0 2023-10-11T16:32:39.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238105ef0 0x7fa23819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:39.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa237fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa238100cb0 0x7fa2381981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:39.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238105ef0 0x7fa23819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37366/0 (socket says 172.21.15.111:37366) 2023-10-11T16:32:39.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 -- 172.21.15.111:0/455026613 learned_addr learned my addr 172.21.15.111:0/455026613 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:39.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa2377fe700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2381035d0 0x7fa238198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:39.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 -- 172.21.15.111:0/455026613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa238100cb0 msgr2=0x7fa2381981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa238100cb0 0x7fa2381981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 -- 172.21.15.111:0/455026613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2381035d0 msgr2=0x7fa238198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2381035d0 0x7fa238198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 -- 172.21.15.111:0/455026613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22c009520 con 0x7fa238105ef0 2023-10-11T16:32:39.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa237fff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa238100cb0 0x7fa2381981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:39.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa2377fe700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2381035d0 0x7fa238198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:39.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.033+0000 7fa23cb9c700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238105ef0 0x7fa23819f790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa23000ef30 tx=0x7fa23000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:39.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.034+0000 7fa2357fa700 1 -- 172.21.15.111:0/455026613 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa230018070 con 0x7fa238105ef0 2023-10-11T16:32:39.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.034+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2280095f0 con 0x7fa238105ef0 2023-10-11T16:32:39.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.034+0000 7fa2357fa700 1 -- 172.21.15.111:0/455026613 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa23000f040 con 0x7fa238105ef0 2023-10-11T16:32:39.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.034+0000 7fa2357fa700 1 -- 172.21.15.111:0/455026613 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23000cdb0 con 0x7fa238105ef0 2023-10-11T16:32:39.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.034+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23810bbe0 con 0x7fa238105ef0 2023-10-11T16:32:39.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.035+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa238107ae0 con 0x7fa238105ef0 2023-10-11T16:32:39.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.037+0000 7fa2357fa700 1 -- 172.21.15.111:0/455026613 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7fa230003a00 con 0x7fa238105ef0 2023-10-11T16:32:39.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.038+0000 7fa2357fa700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2200a0f50 0x7fa2200a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:39.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.038+0000 7fa2357fa700 1 -- 172.21.15.111:0/455026613 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa2300cc5d0 con 0x7fa238105ef0 2023-10-11T16:32:39.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.038+0000 7fa237fff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2200a0f50 0x7fa2200a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:39.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.039+0000 7fa237fff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2200a0f50 0x7fa2200a3400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa22800b2e0 tx=0x7fa228005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:39.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.042+0000 7fa2357fa700 1 -- 172.21.15.111:0/455026613 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa230095b90 con 0x7fa238105ef0 2023-10-11T16:32:39.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.284+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 --> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7fa238060f30 con 0x7fa2200a0f50 2023-10-11T16:32:39.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.338+0000 7fa2357fa700 1 -- 172.21.15.111:0/455026613 <== mgr.25544 v2:172.21.15.111:6800/1233945144 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fa238060f30 con 0x7fa2200a0f50 2023-10-11T16:32:39.340 INFO:teuthology.orchestra.run.smithi111.stdout:Retention added to path / 2023-10-11T16:32:39.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2200a0f50 msgr2=0x7fa2200a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2200a0f50 0x7fa2200a3400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa22800b2e0 tx=0x7fa228005fb0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238105ef0 msgr2=0x7fa23819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238105ef0 0x7fa23819f790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa23000ef30 tx=0x7fa23000c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 shutdown_connections 2023-10-11T16:32:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2200a0f50 0x7fa2200a3400 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa238105ef0 0x7fa23819f790 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa238100cb0 0x7fa2381981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 --2- 172.21.15.111:0/455026613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2381035d0 0x7fa238198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 >> 172.21.15.111:0/455026613 conn(0x7fa2380fa750 msgr2=0x7fa2380ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 shutdown_connections 2023-10-11T16:32:39.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.341+0000 7fa23e5ff700 1 -- 172.21.15.111:0/455026613 wait complete. 2023-10-11T16:32:39.363 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-10-11T16:32:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.562+0000 7f53633ff700 1 -- 172.21.15.111:0/1368432552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1014b0 msgr2=0x7f535c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.562+0000 7f53633ff700 1 --2- 172.21.15.111:0/1368432552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1014b0 0x7f535c101900 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f534c009990 tx=0x7f534c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.562+0000 7f53633ff700 1 -- 172.21.15.111:0/1368432552 shutdown_connections 2023-10-11T16:32:39.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.562+0000 7f53633ff700 1 --2- 172.21.15.111:0/1368432552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f535c101e40 0x7f535c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.562+0000 7f53633ff700 1 --2- 172.21.15.111:0/1368432552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f535c1002c0 0x7f535c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.562+0000 7f53633ff700 1 --2- 172.21.15.111:0/1368432552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1014b0 0x7f535c101900 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.562+0000 7f53633ff700 1 -- 172.21.15.111:0/1368432552 >> 172.21.15.111:0/1368432552 conn(0x7f535c0fb8e0 msgr2=0x7f535c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.562+0000 7f53633ff700 1 -- 172.21.15.111:0/1368432552 shutdown_connections 2023-10-11T16:32:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.563+0000 7f53633ff700 1 -- 172.21.15.111:0/1368432552 wait complete. 2023-10-11T16:32:39.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.563+0000 7f53633ff700 1 Processor -- start 2023-10-11T16:32:39.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f53633ff700 1 -- start start 2023-10-11T16:32:39.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f53633ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1002c0 0x7f535c1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:39.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f53633ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f535c1014b0 0x7f535c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:39.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f53633ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f535c101e40 0x7f535c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:39.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f53633ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c19fd50 con 0x7f535c1014b0 2023-10-11T16:32:39.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f53633ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c19fe90 con 0x7f535c1002c0 2023-10-11T16:32:39.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f53633ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c1a0020 con 0x7f535c101e40 2023-10-11T16:32:39.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f536099a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f535c1014b0 0x7f535c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:39.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f536099a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f535c1014b0 0x7f535c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37390/0 (socket says 172.21.15.111:37390) 2023-10-11T16:32:39.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f536099a700 1 -- 172.21.15.111:0/2980601050 learned_addr learned my addr 172.21.15.111:0/2980601050 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:39.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f536119b700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1002c0 0x7f535c1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:39.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.564+0000 7f536199c700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f535c101e40 0x7f535c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:39.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f536119b700 1 -- 172.21.15.111:0/2980601050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f535c101e40 msgr2=0x7f535c19f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f536119b700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f535c101e40 0x7f535c19f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f536119b700 1 -- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f535c1014b0 msgr2=0x7f535c198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f536119b700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f535c1014b0 0x7f535c198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f536119b700 1 -- 172.21.15.111:0/2980601050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c0095f0 con 0x7f535c1002c0 2023-10-11T16:32:39.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f536099a700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f535c1014b0 0x7f535c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:39.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f536119b700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1002c0 0x7f535c1981d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f535800e9c0 tx=0x7f535800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:39.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f53527fc700 1 -- 172.21.15.111:0/2980601050 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358004220 con 0x7f535c1002c0 2023-10-11T16:32:39.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f53527fc700 1 -- 172.21.15.111:0/2980601050 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5358004380 con 0x7f535c1002c0 2023-10-11T16:32:39.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f53527fc700 1 -- 172.21.15.111:0/2980601050 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358003be0 con 0x7f535c1002c0 2023-10-11T16:32:39.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535c10bc10 con 0x7f535c1002c0 2023-10-11T16:32:39.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.565+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535c10ba80 con 0x7f535c1002c0 2023-10-11T16:32:39.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.566+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535c1a0770 con 0x7f535c1002c0 2023-10-11T16:32:39.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.568+0000 7f53527fc700 1 -- 172.21.15.111:0/2980601050 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f5358003d40 con 0x7f535c1002c0 2023-10-11T16:32:39.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.569+0000 7f53527fc700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f53480a0fd0 0x7f53480a3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:39.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.570+0000 7f53527fc700 1 -- 172.21.15.111:0/2980601050 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f5358014070 con 0x7f535c1002c0 2023-10-11T16:32:39.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.570+0000 7f536099a700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f53480a0fd0 0x7f53480a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:39.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.570+0000 7f536099a700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f53480a0fd0 0x7f53480a3480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f534c006010 tx=0x7f534c00b260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:39.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.574+0000 7f53527fc700 1 -- 172.21.15.111:0/2980601050 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5358096160 con 0x7f535c1002c0 2023-10-11T16:32:39.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.710+0000 7f53527fc700 1 -- 172.21.15.111:0/2980601050 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mgrmap(e 23) v1 ==== 135892+0+0 (secure 0 0 0) 0x7f5358096530 con 0x7f535c1002c0 2023-10-11T16:32:39.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.818+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 --> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f535c1a0d20 con 0x7f53480a0fd0 2023-10-11T16:32:39.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.827+0000 7f53527fc700 1 -- 172.21.15.111:0/2980601050 <== mgr.25544 v2:172.21.15.111:6800/1233945144 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7f535c1a0d20 con 0x7f53480a0fd0 2023-10-11T16:32:39.829 INFO:teuthology.orchestra.run.smithi111.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-10-11T00:00:00", "created": "2023-10-11T16:32:38", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2023-10-11T16:32:39.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f53480a0fd0 msgr2=0x7f53480a3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f53480a0fd0 0x7f53480a3480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f534c006010 tx=0x7f534c00b260 comp rx=0 tx=0).stop 2023-10-11T16:32:39.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1002c0 msgr2=0x7f535c1981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:39.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1002c0 0x7f535c1981d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f535800e9c0 tx=0x7f535800ecd0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 shutdown_connections 2023-10-11T16:32:39.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f53480a0fd0 0x7f53480a3480 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f535c101e40 0x7f535c19f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f535c1002c0 0x7f535c1981d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 --2- 172.21.15.111:0/2980601050 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f535c1014b0 0x7f535c198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:39.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 >> 172.21.15.111:0/2980601050 conn(0x7f535c0fb8e0 msgr2=0x7f535c0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:39.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 shutdown_connections 2023-10-11T16:32:39.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:39.829+0000 7f53633ff700 1 -- 172.21.15.111:0/2980601050 wait complete. 2023-10-11T16:32:39.850 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-10-11T16:32:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:39 smithi150 ceph-mon[132217]: [11/Oct/2023:16:32:38] ENGINE Bus STARTING 2023-10-11T16:32:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:39 smithi150 ceph-mon[132217]: [11/Oct/2023:16:32:38] ENGINE Serving on https://172.21.15.111:7150 2023-10-11T16:32:39.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:39 smithi150 ceph-mon[132217]: [11/Oct/2023:16:32:38] ENGINE Serving on http://172.21.15.111:8765 2023-10-11T16:32:39.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:39 smithi150 ceph-mon[132217]: [11/Oct/2023:16:32:38] ENGINE Bus STARTED 2023-10-11T16:32:39.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:39 smithi150 ceph-mon[132217]: mgrmap e22: x(active, since 1.02052s), standbys: y, z 2023-10-11T16:32:39.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:39 smithi150 ceph-mon[132217]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:39.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:39 smithi150 ceph-mon[132217]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:39.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:39 smithi150 ceph-mon[132217]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:40.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 -- 172.21.15.111:0/3231804825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 msgr2=0x7f725c0fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:40.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 --2- 172.21.15.111:0/3231804825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 0x7f725c0fed70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f724c009990 tx=0x7f724c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 -- 172.21.15.111:0/3231804825 shutdown_connections 2023-10-11T16:32:40.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 --2- 172.21.15.111:0/3231804825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f725c0ffc60 0x7f725c1000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 --2- 172.21.15.111:0/3231804825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 0x7f725c0fed70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 --2- 172.21.15.111:0/3231804825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f725c0ff2b0 0x7f725c0ff720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 -- 172.21.15.111:0/3231804825 >> 172.21.15.111:0/3231804825 conn(0x7f725c0fa530 msgr2=0x7f725c0fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:40.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 -- 172.21.15.111:0/3231804825 shutdown_connections 2023-10-11T16:32:40.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.058+0000 7f726139d700 1 -- 172.21.15.111:0/3231804825 wait complete. 2023-10-11T16:32:40.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.059+0000 7f726139d700 1 Processor -- start 2023-10-11T16:32:40.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f726139d700 1 -- start start 2023-10-11T16:32:40.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f726139d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 0x7f725c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:40.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f726139d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f725c0ff2b0 0x7f725c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:40.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f726139d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f725c0ffc60 0x7f725c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:40.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f726139d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c19fbb0 con 0x7f725c0fe960 2023-10-11T16:32:40.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f726139d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c19fcf0 con 0x7f725c0ffc60 2023-10-11T16:32:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f726139d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c19fe80 con 0x7f725c0ff2b0 2023-10-11T16:32:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f725bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 0x7f725c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f725bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 0x7f725c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37406/0 (socket says 172.21.15.111:37406) 2023-10-11T16:32:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f725bfff700 1 -- 172.21.15.111:0/25399748 learned_addr learned my addr 172.21.15.111:0/25399748 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:40.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.060+0000 7f725b7fe700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f725c0ff2b0 0x7f725c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f7260b9c700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f725c0ffc60 0x7f725c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f725bfff700 1 -- 172.21.15.111:0/25399748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f725c0ff2b0 msgr2=0x7f725c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f725bfff700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f725c0ff2b0 0x7f725c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f725bfff700 1 -- 172.21.15.111:0/25399748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f725c0ffc60 msgr2=0x7f725c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:40.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f725bfff700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f725c0ffc60 0x7f725c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f725bfff700 1 -- 172.21.15.111:0/25399748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c0095f0 con 0x7f725c0fe960 2023-10-11T16:32:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f725b7fe700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f725c0ff2b0 0x7f725c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f725bfff700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 0x7f725c198030 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f724c00b680 tx=0x7f724c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f72597fa700 1 -- 172.21.15.111:0/25399748 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c01d070 con 0x7f725c0fe960 2023-10-11T16:32:40.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f72597fa700 1 -- 172.21.15.111:0/25399748 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f724c021410 con 0x7f725c0fe960 2023-10-11T16:32:40.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.061+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725c10dc00 con 0x7f725c0fe960 2023-10-11T16:32:40.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.062+0000 7f72597fa700 1 -- 172.21.15.111:0/25399748 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c017650 con 0x7f725c0fe960 2023-10-11T16:32:40.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.062+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725c10da70 con 0x7f725c0fe960 2023-10-11T16:32:40.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.063+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725c109740 con 0x7f725c0fe960 2023-10-11T16:32:40.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.065+0000 7f72597fa700 1 -- 172.21.15.111:0/25399748 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 135892+0+0 (secure 0 0 0) 0x7f724c021580 con 0x7f725c0fe960 2023-10-11T16:32:40.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.067+0000 7f72597fa700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f72440a13d0 0x7f72440a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:40.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.067+0000 7f72597fa700 1 -- 172.21.15.111:0/25399748 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f724c0ce690 con 0x7f725c0fe960 2023-10-11T16:32:40.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.068+0000 7f725b7fe700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f72440a13d0 0x7f72440a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:40.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.068+0000 7f725b7fe700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f72440a13d0 0x7f72440a3880 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7250011490 tx=0x7f7250009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:40.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.071+0000 7f72597fa700 1 -- 172.21.15.111:0/25399748 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f724c0975b0 con 0x7f725c0fe960 2023-10-11T16:32:40.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:39 smithi130 ceph-mon[131859]: [11/Oct/2023:16:32:38] ENGINE Bus STARTING 2023-10-11T16:32:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:39 smithi130 ceph-mon[131859]: [11/Oct/2023:16:32:38] ENGINE Serving on https://172.21.15.111:7150 2023-10-11T16:32:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:39 smithi130 ceph-mon[131859]: [11/Oct/2023:16:32:38] ENGINE Serving on http://172.21.15.111:8765 2023-10-11T16:32:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:39 smithi130 ceph-mon[131859]: [11/Oct/2023:16:32:38] ENGINE Bus STARTED 2023-10-11T16:32:40.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:39 smithi130 ceph-mon[131859]: mgrmap e22: x(active, since 1.02052s), standbys: y, z 2023-10-11T16:32:40.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:39 smithi130 ceph-mon[131859]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:40.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:39 smithi130 ceph-mon[131859]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:40.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:39 smithi130 ceph-mon[131859]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:40.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:39 smithi111 ceph-mon[98527]: [11/Oct/2023:16:32:38] ENGINE Bus STARTING 2023-10-11T16:32:40.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:39 smithi111 ceph-mon[98527]: [11/Oct/2023:16:32:38] ENGINE Serving on https://172.21.15.111:7150 2023-10-11T16:32:40.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:39 smithi111 ceph-mon[98527]: [11/Oct/2023:16:32:38] ENGINE Serving on http://172.21.15.111:8765 2023-10-11T16:32:40.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:39 smithi111 ceph-mon[98527]: [11/Oct/2023:16:32:38] ENGINE Bus STARTED 2023-10-11T16:32:40.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:39 smithi111 ceph-mon[98527]: mgrmap e22: x(active, since 1.02052s), standbys: y, z 2023-10-11T16:32:40.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:39 smithi111 ceph-mon[98527]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:40.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:39 smithi111 ceph-mon[98527]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:40.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:39 smithi111 ceph-mon[98527]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:40.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.327+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 --> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f725c0619c0 con 0x7f72440a13d0 2023-10-11T16:32:40.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.333+0000 7f72597fa700 1 -- 172.21.15.111:0/25399748 <== mgr.25544 v2:172.21.15.111:6800/1233945144 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f725c0619c0 con 0x7f72440a13d0 2023-10-11T16:32:40.335 INFO:teuthology.orchestra.run.smithi111.stdout:/ 1M 6M3h 2023-10-11T16:32:40.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f72440a13d0 msgr2=0x7f72440a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:40.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f72440a13d0 0x7f72440a3880 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7250011490 tx=0x7f7250009260 comp rx=0 tx=0).stop 2023-10-11T16:32:40.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 msgr2=0x7f725c198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:40.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 0x7f725c198030 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f724c00b680 tx=0x7f724c005740 comp rx=0 tx=0).stop 2023-10-11T16:32:40.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 shutdown_connections 2023-10-11T16:32:40.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f72440a13d0 0x7f72440a3880 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f725c0ffc60 0x7f725c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f725c0fe960 0x7f725c198030 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 --2- 172.21.15.111:0/25399748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f725c0ff2b0 0x7f725c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:40.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 >> 172.21.15.111:0/25399748 conn(0x7f725c0fa530 msgr2=0x7f725c0fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:40.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 shutdown_connections 2023-10-11T16:32:40.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:40.336+0000 7f726139d700 1 -- 172.21.15.111:0/25399748 wait complete. 2023-10-11T16:32:40.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2023-10-11T16:32:40.440 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2023-10-11T16:32:40.451 INFO:teuthology.run_tasks:Running task background_exec... 2023-10-11T16:32:40.494 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:32:40.495 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-10-11T16:32:40.500 INFO:teuthology.run_tasks:Running task workunit... 2023-10-11T16:32:40.589 INFO:tasks.workunit:Pulling workunits from ref 3f7a9d091654926539d989dd4c83dfee517abc9e 2023-10-11T16:32:40.590 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-11T16:32:40.590 INFO:tasks.workunit:timeout=3h 2023-10-11T16:32:40.590 INFO:tasks.workunit:cleanup=True 2023-10-11T16:32:40.591 DEBUG:teuthology.orchestra.run.smithi111:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-11T16:32:40.638 INFO:teuthology.orchestra.run.smithi111.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-11T16:32:40.638 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-11T16:32:40.638 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2023-10-11T16:32:40.639 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T16:32:40.639 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:cephfs_t:s0 2023-10-11T16:32:40.639 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2023-10-11 16:30:39.622253188 +0000 2023-10-11T16:32:40.639 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2023-10-11 16:30:39.622253188 +0000 2023-10-11T16:32:40.639 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2023-10-11 16:32:18.213869548 +0000 2023-10-11T16:32:40.640 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2023-10-11 16:30:39.622253188 +0000 2023-10-11T16:32:40.640 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-11T16:32:40.640 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-11T16:32:40.759 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/chrisphoffman/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3f7a9d091654926539d989dd4c83dfee517abc9e 2023-10-11T16:32:40.810 INFO:tasks.workunit.client.0.smithi111.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-11T16:32:41.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:40 smithi130 ceph-mon[131859]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:41.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:40 smithi130 ceph-mon[131859]: mgrmap e23: x(active, since 2s), standbys: y, z 2023-10-11T16:32:41.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:40 smithi130 ceph-mon[131859]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:41.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:40 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:32:41.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:40 smithi111 ceph-mon[98527]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:41.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:40 smithi111 ceph-mon[98527]: mgrmap e23: x(active, since 2s), standbys: y, z 2023-10-11T16:32:41.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:40 smithi111 ceph-mon[98527]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:41.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:40 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:32:41.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:40 smithi150 ceph-mon[132217]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:41.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:40 smithi150 ceph-mon[132217]: mgrmap e23: x(active, since 2s), standbys: y, z 2023-10-11T16:32:41.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:40 smithi150 ceph-mon[132217]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:41.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:40 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:32:42.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:41 smithi130 ceph-mon[131859]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:42.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:41 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:32:42.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:41 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:41 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:41 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:41 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:41 smithi111 ceph-mon[98527]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:42.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:41 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:32:42.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:41 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:41 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:41 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:41 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:41 smithi150 ceph-mon[132217]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:32:42.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:41 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:32:42.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:41 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:41 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:41 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:42.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:41 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi130 to 196.2M 2023-10-11T16:32:43.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: Unable to set osd_memory_target on smithi130 to 205751705: error parsing value: Value '205751705' is below minimum 939524096 2023-10-11T16:32:43.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi150 to 196.2M 2023-10-11T16:32:43.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:42 smithi111 ceph-mon[98527]: Unable to set osd_memory_target on smithi150 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-11T16:32:43.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:43.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi130 to 196.2M 2023-10-11T16:32:43.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: Unable to set osd_memory_target on smithi130 to 205751705: error parsing value: Value '205751705' is below minimum 939524096 2023-10-11T16:32:43.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi150 to 196.2M 2023-10-11T16:32:43.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:42 smithi150 ceph-mon[132217]: Unable to set osd_memory_target on smithi150 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-11T16:32:43.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:43.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi130 to 196.2M 2023-10-11T16:32:43.333 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: Unable to set osd_memory_target on smithi130 to 205751705: error parsing value: Value '205751705' is below minimum 939524096 2023-10-11T16:32:43.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:43.334 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:43.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi150 to 196.2M 2023-10-11T16:32:43.335 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:42 smithi130 ceph-mon[131859]: Unable to set osd_memory_target on smithi150 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-11T16:32:44.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:44.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:44.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi111 to 221.9M 2023-10-11T16:32:44.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Unable to set osd_memory_target on smithi111 to 232767897: error parsing value: Value '232767897' is below minimum 939524096 2023-10-11T16:32:44.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:32:44.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:32:44.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:32:44.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Updating smithi111:/etc/ceph/ceph.conf 2023-10-11T16:32:44.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:32:44.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:32:44.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:44.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:44.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:44 smithi130 ceph-mon[131859]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:44.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:44.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:44.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi111 to 221.9M 2023-10-11T16:32:44.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Unable to set osd_memory_target on smithi111 to 232767897: error parsing value: Value '232767897' is below minimum 939524096 2023-10-11T16:32:44.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:32:44.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:32:44.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:32:44.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Updating smithi111:/etc/ceph/ceph.conf 2023-10-11T16:32:44.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:32:44.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:32:44.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:44.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:44.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:44 smithi111 ceph-mon[98527]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:44.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:44.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:44.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:32:44.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi111 to 221.9M 2023-10-11T16:32:44.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Unable to set osd_memory_target on smithi111 to 232767897: error parsing value: Value '232767897' is below minimum 939524096 2023-10-11T16:32:44.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:32:44.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:32:44.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:32:44.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Updating smithi111:/etc/ceph/ceph.conf 2023-10-11T16:32:44.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Updating smithi130:/etc/ceph/ceph.conf 2023-10-11T16:32:44.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Updating smithi150:/etc/ceph/ceph.conf 2023-10-11T16:32:44.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:32:44.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-11T16:32:44.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:44.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:44 smithi150 ceph-mon[132217]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:45 smithi150 ceph-mon[132217]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:45 smithi150 ceph-mon[132217]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:45 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:45 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:45 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:45 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:45.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:45 smithi150 ceph-mon[132217]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:45.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:45 smithi150 ceph-mon[132217]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:46.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:45 smithi130 ceph-mon[131859]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:45 smithi130 ceph-mon[131859]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:45 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:45 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:45 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:46.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:45 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:46.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:45 smithi130 ceph-mon[131859]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:46.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:45 smithi130 ceph-mon[131859]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:46.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:45 smithi111 ceph-mon[98527]: Updating smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:46.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:45 smithi111 ceph-mon[98527]: Updating smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:45 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:45 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:45 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:45 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:46.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:45 smithi111 ceph-mon[98527]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2023-10-11T16:32:46.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:45 smithi111 ceph-mon[98527]: Updating smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-11T16:32:47.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 50 op/s 2023-10-11T16:32:47.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:32:47.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:32:47.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:32:47.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:47.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:32:47.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:32:47.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:32:47.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:32:47.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:32:47.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:46 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:32:47.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 50 op/s 2023-10-11T16:32:47.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:32:47.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:32:47.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:32:47.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:47.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:32:47.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:32:47.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:32:47.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:32:47.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:32:47.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:46 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:32:47.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 50 op/s 2023-10-11T16:32:47.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:32:47.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:32:47.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:32:47.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:32:47.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:32:47.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:32:47.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:32:47.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:32:47.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:32:47.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:32:47.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:46 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:32:49.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:48 smithi130 ceph-mon[131859]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 102 KiB/s wr, 39 op/s 2023-10-11T16:32:49.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:48 smithi111 ceph-mon[98527]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 102 KiB/s wr, 39 op/s 2023-10-11T16:32:49.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:48 smithi150 ceph-mon[132217]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 102 KiB/s wr, 39 op/s 2023-10-11T16:32:51.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:51 smithi111 ceph-mon[98527]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 KiB/s wr, 35 op/s 2023-10-11T16:32:51.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:51 smithi150 ceph-mon[132217]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 KiB/s wr, 35 op/s 2023-10-11T16:32:51.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:51 smithi130 ceph-mon[131859]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 84 KiB/s wr, 35 op/s 2023-10-11T16:32:52.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:52 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:32:52.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:52 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:32:52.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:52 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:32:53.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:53 smithi111 ceph-mon[98527]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-11T16:32:53.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:53 smithi150 ceph-mon[132217]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-11T16:32:53.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:53 smithi130 ceph-mon[131859]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-11T16:32:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 -- 172.21.15.111:0/599852296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0c101460 msgr2=0x7f3b0c101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 --2- 172.21.15.111:0/599852296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0c101460 0x7f3b0c101830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009990 tx=0x7f3afc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 -- 172.21.15.111:0/599852296 shutdown_connections 2023-10-11T16:32:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 --2- 172.21.15.111:0/599852296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b0c10a7a0 0x7f3b0c10cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 --2- 172.21.15.111:0/599852296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0c101460 0x7f3b0c101830 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 --2- 172.21.15.111:0/599852296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b0c101d70 0x7f3b0c10a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 -- 172.21.15.111:0/599852296 >> 172.21.15.111:0/599852296 conn(0x7f3b0c0763a0 msgr2=0x7f3b0c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 -- 172.21.15.111:0/599852296 shutdown_connections 2023-10-11T16:32:53.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.743+0000 7f3b119ff700 1 -- 172.21.15.111:0/599852296 wait complete. 2023-10-11T16:32:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.744+0000 7f3b119ff700 1 Processor -- start 2023-10-11T16:32:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b119ff700 1 -- start start 2023-10-11T16:32:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b119ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b0c101460 0x7f3b0c0ff0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b119ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0c101d70 0x7f3b0c0ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b119ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b0c10a7a0 0x7f3b0c1a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b119ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c0ffe20 con 0x7f3b0c101460 2023-10-11T16:32:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b119ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c1a4ad0 con 0x7f3b0c10a7a0 2023-10-11T16:32:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b119ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c1a4c10 con 0x7f3b0c101d70 2023-10-11T16:32:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b0c101460 0x7f3b0c0ff0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b0c101460 0x7f3b0c0ff0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56086/0 (socket says 172.21.15.111:56086) 2023-10-11T16:32:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.745+0000 7f3b0affd700 1 -- 172.21.15.111:0/3887022572 learned_addr learned my addr 172.21.15.111:0/3887022572 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:53.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b0a7fc700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0c101d70 0x7f3b0c0ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b0b7fe700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b0c10a7a0 0x7f3b0c1a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b0affd700 1 -- 172.21.15.111:0/3887022572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0c101d70 msgr2=0x7f3b0c0ff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b0affd700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0c101d70 0x7f3b0c0ff5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b0affd700 1 -- 172.21.15.111:0/3887022572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b0c10a7a0 msgr2=0x7f3b0c1a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b0affd700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b0c10a7a0 0x7f3b0c1a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b0affd700 1 -- 172.21.15.111:0/3887022572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0095f0 con 0x7f3b0c101460 2023-10-11T16:32:53.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b0affd700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b0c101460 0x7f3b0c0ff0b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3afc000c00 tx=0x7f3afc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:53.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b109fd700 1 -- 172.21.15.111:0/3887022572 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc01d070 con 0x7f3b0c101460 2023-10-11T16:32:53.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b119ff700 1 -- 172.21.15.111:0/3887022572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0c110260 con 0x7f3b0c101460 2023-10-11T16:32:53.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b109fd700 1 -- 172.21.15.111:0/3887022572 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3afc022470 con 0x7f3b0c101460 2023-10-11T16:32:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.746+0000 7f3b109fd700 1 -- 172.21.15.111:0/3887022572 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc017650 con 0x7f3b0c101460 2023-10-11T16:32:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.747+0000 7f3b119ff700 1 -- 172.21.15.111:0/3887022572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0c1100d0 con 0x7f3b0c101460 2023-10-11T16:32:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.750+0000 7f3b109fd700 1 -- 172.21.15.111:0/3887022572 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 135892+0+0 (secure 0 0 0) 0x7f3afc0225e0 con 0x7f3b0c101460 2023-10-11T16:32:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.751+0000 7f3b2529fb80 1 Processor -- start 2023-10-11T16:32:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.751+0000 7f3b2529fb80 1 -- start start 2023-10-11T16:32:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.751+0000 7f3b109fd700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3af40a1350 0x7f3af40a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b109fd700 1 -- 172.21.15.111:0/3887022572 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f3afc0ce630 con 0x7f3b0c101460 2023-10-11T16:32:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a2a4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a1aa000 con 0x56005a1006b0 2023-10-11T16:32:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a1fc840 con 0x56005a2aab90 2023-10-11T16:32:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a1f5cd0 con 0x56005a2aa720 2023-10-11T16:32:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b0b7fe700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3af40a1350 0x7f3af40a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b0a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a2a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2aa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56088/0 (socket says 172.21.15.111:56088) 2023-10-11T16:32:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b0affd700 1 -- 172.21.15.111:0/2545698288 learned_addr learned my addr 172.21.15.111:0/2545698288 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b0affd700 1 -- 172.21.15.111:0/2545698288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 msgr2=0x56005a2a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b0affd700 1 --2- 172.21.15.111:0/2545698288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a2a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.752+0000 7f3b0affd700 1 -- 172.21.15.111:0/2545698288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 msgr2=0x56005a2a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.753+0000 7f3b0affd700 1 --2- 172.21.15.111:0/2545698288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.753+0000 7f3b0affd700 1 -- 172.21.15.111:0/2545698288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56005a0775d0 con 0x56005a1006b0 2023-10-11T16:32:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.753+0000 7f3b0b7fe700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3af40a1350 0x7f3af40a3800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c1011c0 tx=0x7f3b04009920 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.753+0000 7f3b0affd700 1 --2- 172.21.15.111:0/2545698288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2aa1e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0098e0 tx=0x7f3afc09ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ae350809812373 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.753+0000 7f3af1ffb700 1 -- 172.21.15.111:0/2545698288 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc09f2a0 con 0x56005a1006b0 2023-10-11T16:32:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.753+0000 7f3af1ffb700 1 -- 172.21.15.111:0/2545698288 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3afc0a3630 con 0x56005a1006b0 2023-10-11T16:32:53.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.753+0000 7f3af1ffb700 1 -- 172.21.15.111:0/2545698288 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0ce480 con 0x56005a1006b0 2023-10-11T16:32:53.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2545698288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 msgr2=0x56005a2aa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2545698288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2aa1e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0098e0 tx=0x7f3afc09ae80 comp rx=0 tx=0).stop 2023-10-11T16:32:53.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2545698288 shutdown_connections 2023-10-11T16:32:53.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2545698288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2a4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2545698288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2aa1e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2545698288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a2a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2545698288 >> 172.21.15.111:0/2545698288 conn(0x56005a299f50 msgr2=0x56005a29f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:53.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2545698288 shutdown_connections 2023-10-11T16:32:53.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.754+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2545698288 wait complete. 2023-10-11T16:32:53.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 1 -- start start 2023-10-11T16:32:53.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 1 Processor -- start 2023-10-11T16:32:53.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a2b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2ada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a197880 con 0x56005a1006b0 2023-10-11T16:32:53.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a197520 con 0x56005a2aab90 2023-10-11T16:32:53.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a0798c0 con 0x56005a2aa720 2023-10-11T16:32:53.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b2529fb80 10 client.0 _ll_register_callbacks cb 0x56005a1b5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:53.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.755+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56100/0 (socket says 172.21.15.111:56100) 2023-10-11T16:32:53.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0affd700 1 -- 172.21.15.111:0/2299400754 learned_addr learned my addr 172.21.15.111:0/2299400754 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:53.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0a7fc700 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a2b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0b7fe700 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0affd700 1 -- 172.21.15.111:0/2299400754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 msgr2=0x56005a2b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:53.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0affd700 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a2b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0affd700 1 -- 172.21.15.111:0/2299400754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 msgr2=0x56005a2ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:53.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0affd700 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2ada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0affd700 1 -- 172.21.15.111:0/2299400754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0c1100d0 con 0x56005a1006b0 2023-10-11T16:32:53.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b0affd700 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b37f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0cb9f0 tx=0x7f3afc0cbf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd4a44f1b8efb57 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:53.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3aeaffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:53.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3aeaffd700 1 -- 172.21.15.111:0/2299400754 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc09f2a0 con 0x56005a1006b0 2023-10-11T16:32:53.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3aeaffd700 1 -- 172.21.15.111:0/2299400754 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f3afc01bbe0 con 0x56005a1006b0 2023-10-11T16:32:53.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.756+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2299400754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56005a297710 con 0x56005a1006b0 2023-10-11T16:32:53.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3aeaffd700 1 -- 172.21.15.111:0/2299400754 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3afc01bda0 con 0x56005a1006b0 2023-10-11T16:32:53.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3b2529fb80 10 client.15576 fetch_fsmap learned FSMap version 27 2023-10-11T16:32:53.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2299400754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56005a1bc130 con 0x56005a1006b0 2023-10-11T16:32:53.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3aeaffd700 1 -- 172.21.15.111:0/2299400754 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56005a297710 con 0x56005a1006b0 2023-10-11T16:32:53.773 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3aeaffd700 1 -- 172.21.15.111:0/2299400754 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f3afc09a7d0 con 0x56005a1006b0 2023-10-11T16:32:53.773 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3b2529fb80 10 client.15576 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:32:53.773 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3b2529fb80 10 client.15576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:32:53.773 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3b2529fb80 20 client.15576 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:53.773 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56005a2bd7c0 0x56005a2bfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3b2529fb80 4 client.15576 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:32:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.757+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2299400754 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56005a1b8040 con 0x56005a2bd7c0 2023-10-11T16:32:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.758+0000 7f3b0a7fc700 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56005a2bd7c0 0x56005a2bfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.758+0000 7f3b0a7fc700 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56005a2bd7c0 0x56005a2bfbd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76e792c75f0e4661 server_cookie=bd948ab04feb382c in_seq=0 out_seq=0 2023-10-11T16:32:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.758+0000 7f3aeaffd700 10 client.15576 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3aeaffd700 1 -- 172.21.15.111:0/2299400754 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56005a1b8040 con 0x56005a2bd7c0 2023-10-11T16:32:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3aeaffd700 10 client.15576 handle_command_reply: tid=0 2023-10-11T16:32:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 client.15576 shutdown 2023-10-11T16:32:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2299400754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 msgr2=0x56005a2b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.775 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b37f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0cb9f0 tx=0x7f3afc0cbf10 comp rx=0 tx=0).stop 2023-10-11T16:32:53.776 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2299400754 shutdown_connections 2023-10-11T16:32:53.776 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56005a2bd7c0 0x56005a2bfbd0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.776 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.776 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b37f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.776 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/2299400754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a2b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2299400754 >> 172.21.15.111:0/2299400754 conn(0x56005a299f50 msgr2=0x56005a29e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:53.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3aeaffd700 0 client.15576 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:53.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.759+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2299400754 shutdown_connections 2023-10-11T16:32:53.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.760+0000 7f3b2529fb80 1 -- 172.21.15.111:0/2299400754 wait complete. 2023-10-11T16:32:53.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.760+0000 7f3b2529fb80 20 client.15576 trim_cache size 0 max 16384 2023-10-11T16:32:53.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.775+0000 7f3b2529fb80 1 Processor -- start 2023-10-11T16:32:53.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.775+0000 7f3b2529fb80 1 -- start start 2023-10-11T16:32:53.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a29ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2a6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a29bfd0 con 0x56005a1006b0 2023-10-11T16:32:53.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a299b70 con 0x56005a2aab90 2023-10-11T16:32:53.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a2aeea0 con 0x56005a2aa720 2023-10-11T16:32:53.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b0b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2a6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.782 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b0a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a29ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.782 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2ab6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56116/0 (socket says 172.21.15.111:56116) 2023-10-11T16:32:53.782 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.776+0000 7f3b0affd700 1 -- 172.21.15.111:0/3274448420 learned_addr learned my addr 172.21.15.111:0/3274448420 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:53.782 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3b0b7fe700 1 -- 172.21.15.111:0/3274448420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 msgr2=0x56005a29ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:53.782 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3b0b7fe700 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a29ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3b0b7fe700 1 -- 172.21.15.111:0/3274448420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 msgr2=0x56005a2ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:53.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3b0b7fe700 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3b0b7fe700 1 -- 172.21.15.111:0/3274448420 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56005a1bbe80 con 0x56005a2aab90 2023-10-11T16:32:53.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3b0a7fc700 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a29ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:53.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3b0affd700 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:53.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3b0b7fe700 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2a6c10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400fcb0 tx=0x7f3b0401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c3b386a16fd3d4f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3af27fc700 1 -- 172.21.15.111:0/3274448420 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b04027070 con 0x56005a2aab90 2023-10-11T16:32:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3af27fc700 1 -- 172.21.15.111:0/3274448420 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b0400f040 con 0x56005a2aab90 2023-10-11T16:32:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.777+0000 7f3af27fc700 1 -- 172.21.15.111:0/3274448420 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0400e040 con 0x56005a2aab90 2023-10-11T16:32:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.778+0000 7f3b2529fb80 1 -- 172.21.15.111:0/3274448420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 msgr2=0x56005a2a6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.778+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2a6c10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400fcb0 tx=0x7f3b0401a000 comp rx=0 tx=0).stop 2023-10-11T16:32:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.778+0000 7f3b2529fb80 1 -- 172.21.15.111:0/3274448420 shutdown_connections 2023-10-11T16:32:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.778+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aab90 0x56005a2a6c10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.778+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.778+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/3274448420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aa720 0x56005a29ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.778+0000 7f3b2529fb80 1 -- 172.21.15.111:0/3274448420 >> 172.21.15.111:0/3274448420 conn(0x56005a299f50 msgr2=0x56005a2aaf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.779+0000 7f3b2529fb80 1 -- 172.21.15.111:0/3274448420 shutdown_connections 2023-10-11T16:32:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.779+0000 7f3b2529fb80 1 -- 172.21.15.111:0/3274448420 wait complete. 2023-10-11T16:32:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.779+0000 7f3b2529fb80 1 -- start start 2023-10-11T16:32:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.779+0000 7f3b2529fb80 1 Processor -- start 2023-10-11T16:32:53.788 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aa720 0x56005a2b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b2529fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aab90 0x56005a2b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a310680 con 0x56005a1006b0 2023-10-11T16:32:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a3107c0 con 0x56005a2aa720 2023-10-11T16:32:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b2529fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005a310930 con 0x56005a2aab90 2023-10-11T16:32:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b2529fb80 10 client.0 _ll_register_callbacks cb 0x7f3b00009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0a7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aa720 0x56005a2b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56122/0 (socket says 172.21.15.111:56122) 2023-10-11T16:32:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0affd700 1 -- 172.21.15.111:0/725535127 learned_addr learned my addr 172.21.15.111:0/725535127 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0b7fe700 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aab90 0x56005a2b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0a7fc700 1 -- 172.21.15.111:0/725535127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aab90 msgr2=0x56005a2b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0a7fc700 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aab90 0x56005a2b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0a7fc700 1 -- 172.21.15.111:0/725535127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 msgr2=0x56005a2b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0a7fc700 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.780+0000 7f3b0a7fc700 1 -- 172.21.15.111:0/725535127 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56005a1bc130 con 0x56005a2aa720 2023-10-11T16:32:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.781+0000 7f3b0affd700 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.781+0000 7f3b0a7fc700 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aa720 0x56005a2b18a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x56005a2c0480 tx=0x7f3b0001e520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6511ff02ecb3efc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.781+0000 7f3aeaffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:32:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.781+0000 7f3aeaffd700 1 -- 172.21.15.111:0/725535127 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00017030 con 0x56005a2aa720 2023-10-11T16:32:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.781+0000 7f3b2529fb80 1 -- 172.21.15.111:0/725535127 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3afc0a3630 con 0x56005a2aa720 2023-10-11T16:32:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.781+0000 7f3aeaffd700 1 -- 172.21.15.111:0/725535127 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f3b00004450 con 0x56005a2aa720 2023-10-11T16:32:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.781+0000 7f3aeaffd700 1 -- 172.21.15.111:0/725535127 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b00003680 con 0x56005a2aa720 2023-10-11T16:32:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b2529fb80 10 client.25628 fetch_fsmap learned FSMap version 27 2023-10-11T16:32:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b2529fb80 1 -- 172.21.15.111:0/725535127 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f3b00005ce0 con 0x56005a2aa720 2023-10-11T16:32:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3aeaffd700 1 -- 172.21.15.111:0/725535127 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3afc0a3630 con 0x56005a2aa720 2023-10-11T16:32:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3aeaffd700 1 -- 172.21.15.111:0/725535127 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f3b00027930 con 0x56005a2aa720 2023-10-11T16:32:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b2529fb80 10 client.25628 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:32:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b2529fb80 10 client.25628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:32:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b2529fb80 20 client.25628 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56005a312120 0x56005a314530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b2529fb80 4 client.25628 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:32:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b2529fb80 1 -- 172.21.15.111:0/725535127 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56005a1c4600 con 0x56005a312120 2023-10-11T16:32:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.782+0000 7f3b0affd700 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56005a312120 0x56005a314530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.783+0000 7f3aeaffd700 10 client.25628 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.783+0000 7f3b0affd700 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56005a312120 0x56005a314530 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baac8fc84377f621 server_cookie=9822837ca34560bb in_seq=0 out_seq=0 2023-10-11T16:32:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3aeaffd700 1 -- 172.21.15.111:0/725535127 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56005a1c4600 con 0x56005a312120 2023-10-11T16:32:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3aeaffd700 10 client.25628 handle_command_reply: tid=0 2023-10-11T16:32:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 client.25628 shutdown 2023-10-11T16:32:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 -- 172.21.15.111:0/725535127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aa720 msgr2=0x56005a2b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aa720 0x56005a2b18a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x56005a2c0480 tx=0x7f3b0001e520 comp rx=0 tx=0).stop 2023-10-11T16:32:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 -- 172.21.15.111:0/725535127 shutdown_connections 2023-10-11T16:32:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56005a312120 0x56005a314530 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56005a2aab90 0x56005a2b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56005a1006b0 0x56005a2b1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 --2- 172.21.15.111:0/725535127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56005a2aa720 0x56005a2b18a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3aeaffd700 0 client.25628 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 -- 172.21.15.111:0/725535127 >> 172.21.15.111:0/725535127 conn(0x56005a299f50 msgr2=0x56005a2ad2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 -- 172.21.15.111:0/725535127 shutdown_connections 2023-10-11T16:32:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.794+0000 7f3b2529fb80 1 -- 172.21.15.111:0/725535127 wait complete. 2023-10-11T16:32:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b2529fb80 20 client.25628 trim_cache size 0 max 16384 2023-10-11T16:32:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:32:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:32:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:32:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:32:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 -- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3af40a1350 msgr2=0x7f3af40a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3af40a1350 0x7f3af40a3800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c1011c0 tx=0x7f3b04009920 comp rx=0 tx=0).stop 2023-10-11T16:32:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 -- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b0c101460 msgr2=0x7f3b0c0ff0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b0c101460 0x7f3b0c0ff0b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3afc000c00 tx=0x7f3afc005740 comp rx=0 tx=0).stop 2023-10-11T16:32:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 -- 172.21.15.111:0/3887022572 shutdown_connections 2023-10-11T16:32:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3af40a1350 0x7f3af40a3800 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b0c10a7a0 0x7f3b0c1a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b0c101460 0x7f3b0c0ff0b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 --2- 172.21.15.111:0/3887022572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0c101d70 0x7f3b0c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.795+0000 7f3b119ff700 1 -- 172.21.15.111:0/3887022572 >> 172.21.15.111:0/3887022572 conn(0x7f3b0c0763a0 msgr2=0x7f3b0c10b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.796+0000 7f3b119ff700 1 -- 172.21.15.111:0/3887022572 shutdown_connections 2023-10-11T16:32:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:32:53.796+0000 7f3b119ff700 1 -- 172.21.15.111:0/3887022572 wait complete. 2023-10-11T16:32:55.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:55 smithi111 ceph-mon[98527]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-11T16:32:55.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:55 smithi150 ceph-mon[132217]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-11T16:32:55.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:55 smithi130 ceph-mon[131859]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-11T16:32:57.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:57 smithi111 ceph-mon[98527]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2023-10-11T16:32:57.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:57 smithi150 ceph-mon[132217]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2023-10-11T16:32:57.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:57 smithi130 ceph-mon[131859]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2023-10-11T16:32:58.658 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:32:58.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 -- 172.21.15.111:0/4072662292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ffdf0 msgr2=0x7fd818103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 --2- 172.21.15.111:0/4072662292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ffdf0 0x7fd818103db0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009990 tx=0x7fd80c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 -- 172.21.15.111:0/4072662292 shutdown_connections 2023-10-11T16:32:58.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 --2- 172.21.15.111:0/4072662292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104490 0x7fd81810c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 --2- 172.21.15.111:0/4072662292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8180ff450 0x7fd8180ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 --2- 172.21.15.111:0/4072662292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ffdf0 0x7fd818103db0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 -- 172.21.15.111:0/4072662292 >> 172.21.15.111:0/4072662292 conn(0x7fd8180770b0 msgr2=0x7fd8180774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 -- 172.21.15.111:0/4072662292 shutdown_connections 2023-10-11T16:32:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.916+0000 7fd81cdff700 1 -- 172.21.15.111:0/4072662292 wait complete. 2023-10-11T16:32:58.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.917+0000 7fd81cdff700 1 Processor -- start 2023-10-11T16:32:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.918+0000 7fd81cdff700 1 -- start start 2023-10-11T16:32:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.918+0000 7fd81cdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ff450 0x7fd8181985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.918+0000 7fd81cdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8180ffdf0 0x7fd818198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.918+0000 7fd81cdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104490 0x7fd81819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.918+0000 7fd81cdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81819d5c0 con 0x7fd8180ffdf0 2023-10-11T16:32:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.918+0000 7fd81cdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81819d730 con 0x7fd8180ff450 2023-10-11T16:32:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.918+0000 7fd81cdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81819d8f0 con 0x7fd818104490 2023-10-11T16:32:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd81659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ff450 0x7fd8181985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd81659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ff450 0x7fd8181985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45980/0 (socket says 172.21.15.111:45980) 2023-10-11T16:32:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd81659c700 1 -- 172.21.15.111:0/105195012 learned_addr learned my addr 172.21.15.111:0/105195012 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd815d9b700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8180ffdf0 0x7fd818198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd816d9d700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104490 0x7fd81819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd81659c700 1 -- 172.21.15.111:0/105195012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104490 msgr2=0x7fd81819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd81659c700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104490 0x7fd81819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd81659c700 1 -- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8180ffdf0 msgr2=0x7fd818198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd81659c700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8180ffdf0 0x7fd818198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd81659c700 1 -- 172.21.15.111:0/105195012 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c0095f0 con 0x7fd8180ff450 2023-10-11T16:32:58.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.919+0000 7fd816d9d700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104490 0x7fd81819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:58.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.920+0000 7fd81659c700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ff450 0x7fd8181985e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd80800e9a0 tx=0x7fd80800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:58.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.920+0000 7fd8077fe700 1 -- 172.21.15.111:0/105195012 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808009790 con 0x7fd8180ff450 2023-10-11T16:32:58.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.920+0000 7fd8077fe700 1 -- 172.21.15.111:0/105195012 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd80800cbd0 con 0x7fd8180ff450 2023-10-11T16:32:58.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.920+0000 7fd81cdff700 1 -- 172.21.15.111:0/105195012 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd818110350 con 0x7fd8180ff450 2023-10-11T16:32:58.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.920+0000 7fd81cdff700 1 -- 172.21.15.111:0/105195012 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8181101c0 con 0x7fd8180ff450 2023-10-11T16:32:58.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.920+0000 7fd8077fe700 1 -- 172.21.15.111:0/105195012 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808018680 con 0x7fd8180ff450 2023-10-11T16:32:58.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.923+0000 7fd8077fe700 1 -- 172.21.15.111:0/105195012 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 23) v1 ==== 135892+0+0 (secure 0 0 0) 0x7fd80800cd40 con 0x7fd8180ff450 2023-10-11T16:32:58.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.926+0000 7fd8077fe700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8000a13d0 0x7fd8000a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.926+0000 7fd8077fe700 1 -- 172.21.15.111:0/105195012 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd808014070 con 0x7fd8180ff450 2023-10-11T16:32:58.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.926+0000 7fd815d9b700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8000a13d0 0x7fd8000a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.926+0000 7fd830649b80 1 Processor -- start 2023-10-11T16:32:58.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.927+0000 7fd830649b80 1 -- start start 2023-10-11T16:32:58.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.927+0000 7fd830649b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd253740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.927+0000 7fd830649b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253c80 0x55fcbd257b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.927+0000 7fd830649b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd258510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.927+0000 7fd830649b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd150060 con 0x55fcbd253370 2023-10-11T16:32:58.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.927+0000 7fd830649b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd1a28a0 con 0x55fcbd253c80 2023-10-11T16:32:58.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.927+0000 7fd830649b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd19bd30 con 0x55fcbd2580c0 2023-10-11T16:32:58.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd81659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd253740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd81659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd253740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33546/0 (socket says 172.21.15.111:33546) 2023-10-11T16:32:58.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd81659c700 1 -- 172.21.15.111:0/513292483 learned_addr learned my addr 172.21.15.111:0/513292483 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:58.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd816d9d700 1 --2- 172.21.15.111:0/513292483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd258510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd81659c700 1 -- 172.21.15.111:0/513292483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 msgr2=0x55fcbd258510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:58.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd81659c700 1 --2- 172.21.15.111:0/513292483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd258510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd81659c700 1 -- 172.21.15.111:0/513292483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253c80 msgr2=0x55fcbd257b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:32:58.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd81659c700 1 --2- 172.21.15.111:0/513292483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253c80 0x55fcbd257b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd815d9b700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8000a13d0 0x7fd8000a3880 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00b2e0 tx=0x7fd80c005fd0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:58.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.928+0000 7fd81659c700 1 -- 172.21.15.111:0/513292483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcbd01d630 con 0x55fcbd253370 2023-10-11T16:32:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.929+0000 7fd81659c700 1 --2- 172.21.15.111:0/513292483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd253740 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd808099c30 tx=0x7fd808099920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d203069e43947f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.929+0000 7fd804ff9700 1 -- 172.21.15.111:0/513292483 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8080a2380 con 0x55fcbd253370 2023-10-11T16:32:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.929+0000 7fd804ff9700 1 -- 172.21.15.111:0/513292483 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8080982d0 con 0x55fcbd253370 2023-10-11T16:32:58.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.929+0000 7fd804ff9700 1 -- 172.21.15.111:0/513292483 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8080a1ef0 con 0x55fcbd253370 2023-10-11T16:32:58.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 -- 172.21.15.111:0/513292483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 msgr2=0x55fcbd253740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 --2- 172.21.15.111:0/513292483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd253740 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd808099c30 tx=0x7fd808099920 comp rx=0 tx=0).stop 2023-10-11T16:32:58.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 -- 172.21.15.111:0/513292483 shutdown_connections 2023-10-11T16:32:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 --2- 172.21.15.111:0/513292483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd258510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 --2- 172.21.15.111:0/513292483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd253740 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 --2- 172.21.15.111:0/513292483 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253c80 0x55fcbd257b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 -- 172.21.15.111:0/513292483 >> 172.21.15.111:0/513292483 conn(0x55fcbd240360 msgr2=0x55fcbd245b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 -- 172.21.15.111:0/513292483 shutdown_connections 2023-10-11T16:32:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.930+0000 7fd830649b80 1 -- 172.21.15.111:0/513292483 wait complete. 2023-10-11T16:32:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.931+0000 7fd830649b80 1 -- start start 2023-10-11T16:32:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.931+0000 7fd830649b80 1 Processor -- start 2023-10-11T16:32:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.932+0000 7fd830649b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd25f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.932+0000 7fd830649b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd25ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.932+0000 7fd830649b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd25d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.932+0000 7fd830649b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd13d8e0 con 0x55fcbd253c80 2023-10-11T16:32:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.932+0000 7fd830649b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd13d580 con 0x55fcbd253370 2023-10-11T16:32:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.932+0000 7fd830649b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd01f920 con 0x55fcbd2580c0 2023-10-11T16:32:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.932+0000 7fd830649b80 10 client.0 _ll_register_callbacks cb 0x55fcbd15b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:58.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd81659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd25f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd81659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd25f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45998/0 (socket says 172.21.15.111:45998) 2023-10-11T16:32:58.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd81659c700 1 -- 172.21.15.111:0/3191789861 learned_addr learned my addr 172.21.15.111:0/3191789861 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:58.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd816d9d700 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd25d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd815d9b700 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd25ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd81659c700 1 -- 172.21.15.111:0/3191789861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 msgr2=0x55fcbd25d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:58.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd81659c700 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd25d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd81659c700 1 -- 172.21.15.111:0/3191789861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 msgr2=0x55fcbd25ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:58.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd81659c700 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd25ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd81659c700 1 -- 172.21.15.111:0/3191789861 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8181101c0 con 0x55fcbd253370 2023-10-11T16:32:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.933+0000 7fd815d9b700 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd25ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.934+0000 7fd81659c700 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd25f7e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd80800e970 tx=0x7fd80809a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476a65109679f81f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:58.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.934+0000 7fd7fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:32:58.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.934+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191789861 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8080a2380 con 0x55fcbd253370 2023-10-11T16:32:58.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.934+0000 7fd830649b80 1 -- 172.21.15.111:0/3191789861 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fcbd23db20 con 0x55fcbd253370 2023-10-11T16:32:58.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.938+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191789861 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd808094b40 con 0x55fcbd253370 2023-10-11T16:32:58.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.938+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191789861 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8080b8260 con 0x55fcbd253370 2023-10-11T16:32:58.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.938+0000 7fd830649b80 10 client.25637 fetch_fsmap learned FSMap version 27 2023-10-11T16:32:58.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.938+0000 7fd830649b80 1 -- 172.21.15.111:0/3191789861 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fcbd162190 con 0x55fcbd253370 2023-10-11T16:32:58.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.938+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191789861 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8080ca020 con 0x55fcbd253370 2023-10-11T16:32:58.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.938+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191789861 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd808098e90 con 0x55fcbd253370 2023-10-11T16:32:58.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.939+0000 7fd830649b80 10 client.25637 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:32:58.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.939+0000 7fd830649b80 10 client.25637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:32:58.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.939+0000 7fd830649b80 20 client.25637 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:58.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.939+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fcbd26cc00 0x55fcbd26f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.939+0000 7fd830649b80 4 client.25637 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:32:58.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.939+0000 7fd830649b80 1 -- 172.21.15.111:0/3191789861 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fcbd15e0a0 con 0x55fcbd26cc00 2023-10-11T16:32:58.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.939+0000 7fd815d9b700 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fcbd26cc00 0x55fcbd26f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.940+0000 7fd815d9b700 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fcbd26cc00 0x55fcbd26f010 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=968faea70bed297 server_cookie=496b7f53bb3bda8b in_seq=0 out_seq=0 2023-10-11T16:32:58.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.940+0000 7fd7fdffb700 10 client.25637 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:58.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.940+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191789861 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55fcbd15e0a0 con 0x55fcbd26cc00 2023-10-11T16:32:58.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.940+0000 7fd7fdffb700 10 client.25637 handle_command_reply: tid=0 2023-10-11T16:32:58.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.940+0000 7fd830649b80 1 client.25637 shutdown 2023-10-11T16:32:58.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 -- 172.21.15.111:0/3191789861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 msgr2=0x55fcbd25f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd25f7e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd80800e970 tx=0x7fd80809a440 comp rx=0 tx=0).stop 2023-10-11T16:32:58.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 -- 172.21.15.111:0/3191789861 shutdown_connections 2023-10-11T16:32:58.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fcbd26cc00 0x55fcbd26f010 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd25d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd7fdffb700 0 client.25637 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:58.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd25f7e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191789861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd25ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 -- 172.21.15.111:0/3191789861 >> 172.21.15.111:0/3191789861 conn(0x55fcbd240360 msgr2=0x55fcbd244de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:58.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 -- 172.21.15.111:0/3191789861 shutdown_connections 2023-10-11T16:32:58.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 1 -- 172.21.15.111:0/3191789861 wait complete. 2023-10-11T16:32:58.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.941+0000 7fd830649b80 20 client.25637 trim_cache size 0 max 16384 2023-10-11T16:32:58.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.974+0000 7fd830649b80 1 Processor -- start 2023-10-11T16:32:58.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.974+0000 7fd830649b80 1 -- start start 2023-10-11T16:32:58.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd830649b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd2641b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd830649b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd257140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd830649b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd257680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd830649b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd254090 con 0x55fcbd253c80 2023-10-11T16:32:58.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd830649b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd25afc0 con 0x55fcbd253370 2023-10-11T16:32:58.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd830649b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd257cb0 con 0x55fcbd2580c0 2023-10-11T16:32:58.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd81659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd2641b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd816d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd257680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd81659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd2641b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46014/0 (socket says 172.21.15.111:46014) 2023-10-11T16:32:58.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd81659c700 1 -- 172.21.15.111:0/2121900201 learned_addr learned my addr 172.21.15.111:0/2121900201 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:58.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.975+0000 7fd815d9b700 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd257140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd816d9d700 1 -- 172.21.15.111:0/2121900201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 msgr2=0x55fcbd2641b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:58.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd816d9d700 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd2641b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd816d9d700 1 -- 172.21.15.111:0/2121900201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 msgr2=0x55fcbd257140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:58.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd816d9d700 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd257140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd816d9d700 1 -- 172.21.15.111:0/2121900201 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcbd161ee0 con 0x55fcbd2580c0 2023-10-11T16:32:58.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd815d9b700 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd257140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:32:58.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd816d9d700 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd257680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd810007550 tx=0x7fd81000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcd53d32b3f798a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:58.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd8057fa700 1 -- 172.21.15.111:0/2121900201 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81000e040 con 0x55fcbd2580c0 2023-10-11T16:32:58.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd8057fa700 1 -- 172.21.15.111:0/2121900201 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd81000f040 con 0x55fcbd2580c0 2023-10-11T16:32:58.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.976+0000 7fd8057fa700 1 -- 172.21.15.111:0/2121900201 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810017410 con 0x55fcbd2580c0 2023-10-11T16:32:58.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.977+0000 7fd830649b80 1 -- 172.21.15.111:0/2121900201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 msgr2=0x55fcbd257680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.977+0000 7fd830649b80 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd257680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd810007550 tx=0x7fd81000b040 comp rx=0 tx=0).stop 2023-10-11T16:32:58.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.977+0000 7fd830649b80 1 -- 172.21.15.111:0/2121900201 shutdown_connections 2023-10-11T16:32:58.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.977+0000 7fd830649b80 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd2580c0 0x55fcbd257680 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.977+0000 7fd830649b80 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd253370 0x55fcbd2641b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.977+0000 7fd830649b80 1 --2- 172.21.15.111:0/2121900201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253c80 0x55fcbd257140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.977+0000 7fd830649b80 1 -- 172.21.15.111:0/2121900201 >> 172.21.15.111:0/2121900201 conn(0x55fcbd240360 msgr2=0x55fcbd2693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:58.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.978+0000 7fd830649b80 1 -- 172.21.15.111:0/2121900201 shutdown_connections 2023-10-11T16:32:58.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.978+0000 7fd830649b80 1 -- 172.21.15.111:0/2121900201 wait complete. 2023-10-11T16:32:58.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.978+0000 7fd830649b80 1 -- start start 2023-10-11T16:32:58.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.978+0000 7fd830649b80 1 Processor -- start 2023-10-11T16:32:58.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.979+0000 7fd830649b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd260580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.979+0000 7fd830649b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd253c80 0x55fcbd260ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.979+0000 7fd830649b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd2580c0 0x55fcbd2b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.979+0000 7fd830649b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd2b9020 con 0x55fcbd253370 2023-10-11T16:32:58.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.979+0000 7fd830649b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd2b9160 con 0x55fcbd2580c0 2023-10-11T16:32:58.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.979+0000 7fd830649b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcbd2b9320 con 0x55fcbd253c80 2023-10-11T16:32:58.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.979+0000 7fd830649b80 10 client.0 _ll_register_callbacks cb 0x55fcbd15b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:58.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd816d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd2580c0 0x55fcbd2b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd816d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd2580c0 0x55fcbd2b8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46018/0 (socket says 172.21.15.111:46018) 2023-10-11T16:32:58.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd816d9d700 1 -- 172.21.15.111:0/3191713785 learned_addr learned my addr 172.21.15.111:0/3191713785 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:58.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd815d9b700 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd253c80 0x55fcbd260ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd81659c700 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd260580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd816d9d700 1 -- 172.21.15.111:0/3191713785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd253c80 msgr2=0x55fcbd260ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:58.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd816d9d700 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd253c80 0x55fcbd260ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd816d9d700 1 -- 172.21.15.111:0/3191713785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 msgr2=0x55fcbd260580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:58.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd816d9d700 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd260580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.980+0000 7fd816d9d700 1 -- 172.21.15.111:0/3191713785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd810009250 con 0x55fcbd2580c0 2023-10-11T16:32:58.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd815d9b700 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd253c80 0x55fcbd260ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:58.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd816d9d700 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd2580c0 0x55fcbd2b8a50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd810007860 tx=0x7fd810004820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=395b6b1bdc3d083a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:58.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd7fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:32:58.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191713785 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81000e030 con 0x55fcbd2580c0 2023-10-11T16:32:58.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191713785 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd810004ce0 con 0x55fcbd2580c0 2023-10-11T16:32:58.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd830649b80 1 -- 172.21.15.111:0/3191713785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8080982d0 con 0x55fcbd2580c0 2023-10-11T16:32:58.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191713785 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd810024070 con 0x55fcbd2580c0 2023-10-11T16:32:58.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd830649b80 10 client.25643 fetch_fsmap learned FSMap version 27 2023-10-11T16:32:58.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.981+0000 7fd830649b80 1 -- 172.21.15.111:0/3191713785 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fcbcfb2c60 con 0x55fcbd2580c0 2023-10-11T16:32:58.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.982+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191713785 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8080982d0 con 0x55fcbd2580c0 2023-10-11T16:32:58.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.982+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191713785 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd81001d030 con 0x55fcbd2580c0 2023-10-11T16:32:58.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.982+0000 7fd830649b80 10 client.25643 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:32:58.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.982+0000 7fd830649b80 10 client.25643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:32:58.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.982+0000 7fd830649b80 20 client.25643 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:58.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.982+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fcbd2c1bc0 0x55fcbd2c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:58.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.982+0000 7fd830649b80 4 client.25643 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:32:58.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.982+0000 7fd830649b80 1 -- 172.21.15.111:0/3191713785 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fcbd16a660 con 0x55fcbd2c1bc0 2023-10-11T16:32:58.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.983+0000 7fd81659c700 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fcbd2c1bc0 0x55fcbd2c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:58.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.983+0000 7fd7fdffb700 10 client.25643 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:58.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.983+0000 7fd81659c700 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fcbd2c1bc0 0x55fcbd2c3fd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28a95518677ecf97 server_cookie=344dbceae627a32 in_seq=0 out_seq=0 2023-10-11T16:32:58.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.983+0000 7fd7fdffb700 1 -- 172.21.15.111:0/3191713785 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fcbd16a660 con 0x55fcbd2c1bc0 2023-10-11T16:32:58.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.983+0000 7fd7fdffb700 10 client.25643 handle_command_reply: tid=0 2023-10-11T16:32:58.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.983+0000 7fd830649b80 1 client.25643 shutdown 2023-10-11T16:32:58.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 -- 172.21.15.111:0/3191713785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd2580c0 msgr2=0x55fcbd2b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd2580c0 0x55fcbd2b8a50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd810007860 tx=0x7fd810004820 comp rx=0 tx=0).stop 2023-10-11T16:32:58.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 -- 172.21.15.111:0/3191713785 shutdown_connections 2023-10-11T16:32:58.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fcbd2c1bc0 0x55fcbd2c3fd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fcbd2580c0 0x55fcbd2b8a50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fcbd253370 0x55fcbd260580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 --2- 172.21.15.111:0/3191713785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fcbd253c80 0x55fcbd260ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 -- 172.21.15.111:0/3191713785 >> 172.21.15.111:0/3191713785 conn(0x55fcbd240360 msgr2=0x55fcbd257140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:58.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd7fdffb700 0 client.25643 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:32:58.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 -- 172.21.15.111:0/3191713785 shutdown_connections 2023-10-11T16:32:58.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.984+0000 7fd830649b80 1 -- 172.21.15.111:0/3191713785 wait complete. 2023-10-11T16:32:58.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.985+0000 7fd830649b80 20 client.25643 trim_cache size 0 max 16384 2023-10-11T16:32:58.997 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:32:58.997 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:32:58.997 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:32:58.997 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:32:58.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 -- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8000a13d0 msgr2=0x7fd8000a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8000a13d0 0x7fd8000a3880 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00b2e0 tx=0x7fd80c005fd0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 -- 172.21.15.111:0/105195012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ff450 msgr2=0x7fd8181985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:58.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ff450 0x7fd8181985e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd80800e9a0 tx=0x7fd80800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 -- 172.21.15.111:0/105195012 shutdown_connections 2023-10-11T16:32:58.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8000a13d0 0x7fd8000a3880 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:58.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd818104490 0x7fd81819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8180ff450 0x7fd8181985e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 --2- 172.21.15.111:0/105195012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8180ffdf0 0x7fd818198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.986+0000 7fd81cdff700 1 -- 172.21.15.111:0/105195012 >> 172.21.15.111:0/105195012 conn(0x7fd8180770b0 msgr2=0x7fd8180fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.987+0000 7fd81cdff700 1 -- 172.21.15.111:0/105195012 shutdown_connections 2023-10-11T16:32:59.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:58.987+0000 7fd81cdff700 1 -- 172.21.15.111:0/105195012 wait complete. 2023-10-11T16:32:59.008 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:32:59.008 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:32:59.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.250+0000 7fd8217ff700 1 -- 172.21.15.111:0/707036992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 msgr2=0x7fd81c107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.250+0000 7fd8217ff700 1 --2- 172.21.15.111:0/707036992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 0x7fd81c107ec0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009990 tx=0x7fd80c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.250+0000 7fd8217ff700 1 -- 172.21.15.111:0/707036992 shutdown_connections 2023-10-11T16:32:59.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.250+0000 7fd8217ff700 1 --2- 172.21.15.111:0/707036992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd81c1133b0 0x7fd81c115790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.250+0000 7fd8217ff700 1 --2- 172.21.15.111:0/707036992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 0x7fd81c107ec0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.250+0000 7fd8217ff700 1 --2- 172.21.15.111:0/707036992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c108400 0x7fd81c112e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.250+0000 7fd8217ff700 1 -- 172.21.15.111:0/707036992 >> 172.21.15.111:0/707036992 conn(0x7fd81c06c550 msgr2=0x7fd81c0716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.250+0000 7fd8217ff700 1 -- 172.21.15.111:0/707036992 shutdown_connections 2023-10-11T16:32:59.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.251+0000 7fd8217ff700 1 -- 172.21.15.111:0/707036992 wait complete. 2023-10-11T16:32:59.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.252+0000 7fd8217ff700 1 Processor -- start 2023-10-11T16:32:59.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd8217ff700 1 -- start start 2023-10-11T16:32:59.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd8217ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 0x7fd81c112c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd8217ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd81c108400 0x7fd81c10ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd8217ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1133b0 0x7fd81c10f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd8217ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c10f910 con 0x7fd81c107af0 2023-10-11T16:32:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd8217ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c10fa80 con 0x7fd81c1133b0 2023-10-11T16:32:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd8217ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c1b17d0 con 0x7fd81c108400 2023-10-11T16:32:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd81affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 0x7fd81c112c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1133b0 0x7fd81c10f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.253+0000 7fd81affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 0x7fd81c112c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33588/0 (socket says 172.21.15.111:33588) 2023-10-11T16:32:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81affd700 1 -- 172.21.15.111:0/737701936 learned_addr learned my addr 172.21.15.111:0/737701936 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd813fff700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd81c108400 0x7fd81c10ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81b7fe700 1 -- 172.21.15.111:0/737701936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd81c108400 msgr2=0x7fd81c10ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81b7fe700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd81c108400 0x7fd81c10ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81b7fe700 1 -- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 msgr2=0x7fd81c112c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81b7fe700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 0x7fd81c112c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81b7fe700 1 -- 172.21.15.111:0/737701936 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c0095f0 con 0x7fd81c1133b0 2023-10-11T16:32:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81affd700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 0x7fd81c112c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:59.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.254+0000 7fd81b7fe700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1133b0 0x7fd81c10f190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd81400efc0 tx=0x7fd81400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.255+0000 7fd818ff9700 1 -- 172.21.15.111:0/737701936 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814018070 con 0x7fd81c1133b0 2023-10-11T16:32:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.255+0000 7fd818ff9700 1 -- 172.21.15.111:0/737701936 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd81400f040 con 0x7fd81c1133b0 2023-10-11T16:32:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.255+0000 7fd818ff9700 1 -- 172.21.15.111:0/737701936 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81400cd40 con 0x7fd81c1133b0 2023-10-11T16:32:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.255+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c1098c0 con 0x7fd81c1133b0 2023-10-11T16:32:59.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.255+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c109730 con 0x7fd81c1133b0 2023-10-11T16:32:59.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.257+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c109a50 con 0x7fd81c1133b0 2023-10-11T16:32:59.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.260+0000 7fd818ff9700 1 -- 172.21.15.111:0/737701936 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 23) v1 ==== 135892+0+0 (secure 0 0 0) 0x7fd814003a00 con 0x7fd81c1133b0 2023-10-11T16:32:59.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.263+0000 7fd818ff9700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8080a13d0 0x7fd8080a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.263+0000 7fd818ff9700 1 -- 172.21.15.111:0/737701936 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd8140ccd50 con 0x7fd81c1133b0 2023-10-11T16:32:59.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.263+0000 7fd81affd700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8080a13d0 0x7fd8080a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.264+0000 7fd81affd700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8080a13d0 0x7fd8080a3880 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd80c005b20 tx=0x7fd80c005a90 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:59.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.266+0000 7fd818ff9700 1 -- 172.21.15.111:0/737701936 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd814095c70 con 0x7fd81c1133b0 2023-10-11T16:32:59.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:32:59 smithi111 ceph-mon[98527]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2023-10-11T16:32:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:32:59 smithi150 ceph-mon[132217]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2023-10-11T16:32:59.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:32:59 smithi130 ceph-mon[131859]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2023-10-11T16:32:59.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.649+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd81c067a60 con 0x7fd81c1133b0 2023-10-11T16:32:59.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.650+0000 7fd818ff9700 1 -- 172.21.15.111:0/737701936 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10821 (secure 0 0 0) 0x7fd814099290 con 0x7fd81c1133b0 2023-10-11T16:32:59.651 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:32:59.651 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:32:51.094845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:32:59.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8080a13d0 msgr2=0x7fd8080a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8080a13d0 0x7fd8080a3880 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd80c005b20 tx=0x7fd80c005a90 comp rx=0 tx=0).stop 2023-10-11T16:32:59.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1133b0 msgr2=0x7fd81c10f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1133b0 0x7fd81c10f190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd81400efc0 tx=0x7fd81400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 shutdown_connections 2023-10-11T16:32:59.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8080a13d0 0x7fd8080a3880 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd81c1133b0 0x7fd81c10f190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd81c107af0 0x7fd81c112c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 --2- 172.21.15.111:0/737701936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd81c108400 0x7fd81c10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.653+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 >> 172.21.15.111:0/737701936 conn(0x7fd81c06c550 msgr2=0x7fd81c06ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.656+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 shutdown_connections 2023-10-11T16:32:59.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.656+0000 7fd8217ff700 1 -- 172.21.15.111:0/737701936 wait complete. 2023-10-11T16:32:59.658 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 27 2023-10-11T16:32:59.681 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:32:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/1408635847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34100cc0 msgr2=0x7f2f34101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/1408635847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34100cc0 0x7f2f34101090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2f300099c0 tx=0x7f2f30009cd0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/1408635847 shutdown_connections 2023-10-11T16:32:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/1408635847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f34105620 0x7f2f3410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/1408635847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34100cc0 0x7f2f34101090 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/1408635847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f34101660 0x7f2f34104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/1408635847 >> 172.21.15.111:0/1408635847 conn(0x7f2f34077560 msgr2=0x7f2f34077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/1408635847 shutdown_connections 2023-10-11T16:32:59.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.913+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/1408635847 wait complete. 2023-10-11T16:32:59.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.915+0000 7f2f3b3ff700 1 Processor -- start 2023-10-11T16:32:59.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.915+0000 7f2f3b3ff700 1 -- start start 2023-10-11T16:32:59.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3b3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f34100cc0 0x7f2f341986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3b3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f34101660 0x7f2f34198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3b3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34105620 0x7f2f3419bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3b3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3419c5b0 con 0x7f2f34105620 2023-10-11T16:32:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3b3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3419c720 con 0x7f2f34101660 2023-10-11T16:32:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3b3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3419c8e0 con 0x7f2f34100cc0 2023-10-11T16:32:59.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3919b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f34100cc0 0x7f2f341986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3919b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f34100cc0 0x7f2f341986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40810/0 (socket says 172.21.15.111:40810) 2023-10-11T16:32:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3919b700 1 -- 172.21.15.111:0/414837666 learned_addr learned my addr 172.21.15.111:0/414837666 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3899a700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f34101660 0x7f2f34198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3919b700 1 -- 172.21.15.111:0/414837666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f34101660 msgr2=0x7f2f34198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3919b700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f34101660 0x7f2f34198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3999c700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34105620 0x7f2f3419bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.916+0000 7f2f3919b700 1 -- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34105620 msgr2=0x7f2f3419bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f3919b700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34105620 0x7f2f3419bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f3919b700 1 -- 172.21.15.111:0/414837666 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300095f0 con 0x7f2f34100cc0 2023-10-11T16:32:59.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f3999c700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34105620 0x7f2f3419bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:32:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f3919b700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f34100cc0 0x7f2f341986c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2f300098e0 tx=0x7f2f3000f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f2a7fc700 1 -- 172.21.15.111:0/414837666 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3001d070 con 0x7f2f34100cc0 2023-10-11T16:32:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/414837666 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f341103d0 con 0x7f2f34100cc0 2023-10-11T16:32:59.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f2a7fc700 1 -- 172.21.15.111:0/414837666 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f3000fc30 con 0x7f2f34100cc0 2023-10-11T16:32:59.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/414837666 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f34110240 con 0x7f2f34100cc0 2023-10-11T16:32:59.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.917+0000 7f2f2a7fc700 1 -- 172.21.15.111:0/414837666 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30017650 con 0x7f2f34100cc0 2023-10-11T16:32:59.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.923+0000 7f2f2a7fc700 1 -- 172.21.15.111:0/414837666 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 23) v1 ==== 135892+0+0 (secure 0 0 0) 0x7f2f30022470 con 0x7f2f34100cc0 2023-10-11T16:32:59.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.925+0000 7f2f4ed9bb80 1 Processor -- start 2023-10-11T16:32:59.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.925+0000 7f2f4ed9bb80 1 -- start start 2023-10-11T16:32:59.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.925+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bc370 0x5586434bc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f2a7fc700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2f200a13d0 0x7f2f200a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 0x5586434c1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586433b9060 con 0x5586434bcc80 2023-10-11T16:32:59.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55864340b8a0 con 0x5586434c10c0 2023-10-11T16:32:59.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558643404d30 con 0x5586434bc370 2023-10-11T16:32:59.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f3999c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bc370 0x5586434bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f3999c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bc370 0x5586434bc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40824/0 (socket says 172.21.15.111:40824) 2023-10-11T16:32:59.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.926+0000 7f2f3999c700 1 -- 172.21.15.111:0/1067883917 learned_addr learned my addr 172.21.15.111:0/1067883917 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:59.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.927+0000 7f2f2a7fc700 1 -- 172.21.15.111:0/414837666 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2f300ce120 con 0x7f2f34100cc0 2023-10-11T16:32:59.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.927+0000 7f2f3999c700 1 -- 172.21.15.111:0/1067883917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 msgr2=0x5586434c1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:59.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.927+0000 7f2f3999c700 1 --2- 172.21.15.111:0/1067883917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 0x5586434c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.927+0000 7f2f3999c700 1 -- 172.21.15.111:0/1067883917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 msgr2=0x5586434c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:59.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.927+0000 7f2f3999c700 1 --2- 172.21.15.111:0/1067883917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.927+0000 7f2f3999c700 1 -- 172.21.15.111:0/1067883917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558643286630 con 0x5586434bc370 2023-10-11T16:32:59.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.927+0000 7f2f3999c700 1 --2- 172.21.15.111:0/1067883917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bc370 0x5586434bc740 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009520 tx=0x7f2f2c0077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5687a64a6d4f9064 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:59.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.927+0000 7f2f3899a700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2f200a13d0 0x7f2f200a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.928+0000 7f2f1f7fe700 1 -- 172.21.15.111:0/1067883917 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0155f0 con 0x5586434bc370 2023-10-11T16:32:59.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.928+0000 7f2f1f7fe700 1 -- 172.21.15.111:0/1067883917 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f2c011040 con 0x5586434bc370 2023-10-11T16:32:59.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.928+0000 7f2f1f7fe700 1 -- 172.21.15.111:0/1067883917 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0139b0 con 0x5586434bc370 2023-10-11T16:32:59.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1067883917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bc370 msgr2=0x5586434bc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f3899a700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2f200a13d0 0x7f2f200a3880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2f34199cd0 tx=0x7f2f24009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:59.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1067883917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bc370 0x5586434bc740 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009520 tx=0x7f2f2c0077e0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1067883917 shutdown_connections 2023-10-11T16:32:59.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1067883917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 0x5586434c1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1067883917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bc370 0x5586434bc740 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1067883917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1067883917 >> 172.21.15.111:0/1067883917 conn(0x5586434a9360 msgr2=0x5586434aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1067883917 shutdown_connections 2023-10-11T16:32:59.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.929+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1067883917 wait complete. 2023-10-11T16:32:59.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.930+0000 7f2f4ed9bb80 1 -- start start 2023-10-11T16:32:59.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.930+0000 7f2f4ed9bb80 1 Processor -- start 2023-10-11T16:32:59.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586433a68e0 con 0x5586434bcc80 2023-10-11T16:32:59.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586433a6580 con 0x5586434bc370 2023-10-11T16:32:59.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558643288920 con 0x5586434c10c0 2023-10-11T16:32:59.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f4ed9bb80 10 client.0 _ll_register_callbacks cb 0x5586433c4780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:59.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f3919b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f3919b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46098/0 (socket says 172.21.15.111:46098) 2023-10-11T16:32:59.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.931+0000 7f2f3919b700 1 -- 172.21.15.111:0/3469530208 learned_addr learned my addr 172.21.15.111:0/3469530208 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:59.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.932+0000 7f2f3999c700 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.932+0000 7f2f3899a700 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.932+0000 7f2f3919b700 1 -- 172.21.15.111:0/3469530208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 msgr2=0x5586434c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:59.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.932+0000 7f2f3919b700 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.932+0000 7f2f3919b700 1 -- 172.21.15.111:0/3469530208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 msgr2=0x5586434c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:59.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.932+0000 7f2f3919b700 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.932+0000 7f2f3919b700 1 -- 172.21.15.111:0/3469530208 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2c009110 con 0x5586434bc370 2023-10-11T16:32:59.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.933+0000 7f2f3919b700 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434c5c70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009990 tx=0x7f2f3009b240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133885887aabc853 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:59.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.933+0000 7f2f1cff9700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:32:59.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.933+0000 7f2f1cff9700 1 -- 172.21.15.111:0/3469530208 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3000fda0 con 0x5586434bc370 2023-10-11T16:32:59.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.933+0000 7f2f1cff9700 1 -- 172.21.15.111:0/3469530208 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2f3009ad00 con 0x5586434bc370 2023-10-11T16:32:59.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.933+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/3469530208 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5586434a6b20 con 0x5586434bc370 2023-10-11T16:32:59.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.934+0000 7f2f1cff9700 1 -- 172.21.15.111:0/3469530208 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f3009ed30 con 0x5586434bc370 2023-10-11T16:32:59.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.934+0000 7f2f4ed9bb80 10 client.25655 fetch_fsmap learned FSMap version 27 2023-10-11T16:32:59.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.934+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/3469530208 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5586433cb190 con 0x5586434bc370 2023-10-11T16:32:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.934+0000 7f2f1cff9700 1 -- 172.21.15.111:0/3469530208 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5586434a6b20 con 0x5586434bc370 2023-10-11T16:32:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.935+0000 7f2f1cff9700 1 -- 172.21.15.111:0/3469530208 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f2f300cb650 con 0x5586434bc370 2023-10-11T16:32:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.935+0000 7f2f4ed9bb80 10 client.25655 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:32:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.935+0000 7f2f4ed9bb80 10 client.25655 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:32:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.935+0000 7f2f4ed9bb80 20 client.25655 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.935+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5586434d5e10 0x5586434d8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.935+0000 7f2f4ed9bb80 4 client.25655 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:32:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.935+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/3469530208 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5586433c70a0 con 0x5586434d5e10 2023-10-11T16:32:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.935+0000 7f2f3899a700 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5586434d5e10 0x5586434d8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.936+0000 7f2f3899a700 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5586434d5e10 0x5586434d8220 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab0b70a24bf43177 server_cookie=314d5587d320072f in_seq=0 out_seq=0 2023-10-11T16:32:59.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.936+0000 7f2f1cff9700 10 client.25655 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:32:59.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f1cff9700 1 -- 172.21.15.111:0/3469530208 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5586433c70a0 con 0x5586434d5e10 2023-10-11T16:32:59.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f1cff9700 10 client.25655 handle_command_reply: tid=0 2023-10-11T16:32:59.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 client.25655 shutdown 2023-10-11T16:32:59.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/3469530208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 msgr2=0x5586434c5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434c5c70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009990 tx=0x7f2f3009b240 comp rx=0 tx=0).stop 2023-10-11T16:32:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/3469530208 shutdown_connections 2023-10-11T16:32:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5586434d5e10 0x5586434d8220 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434c5c70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/3469530208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.937+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/3469530208 >> 172.21.15.111:0/3469530208 conn(0x5586434a9360 msgr2=0x5586434c2570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.938+0000 7f2f1cff9700 0 client.25655 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:32:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.938+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/3469530208 shutdown_connections 2023-10-11T16:32:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.938+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/3469530208 wait complete. 2023-10-11T16:32:59.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.938+0000 7f2f4ed9bb80 20 client.25655 trim_cache size 0 max 16384 2023-10-11T16:32:59.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.964+0000 7f2f4ed9bb80 1 Processor -- start 2023-10-11T16:32:59.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.964+0000 7f2f4ed9bb80 1 -- start start 2023-10-11T16:32:59.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434be9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586434a6660 con 0x5586434bcc80 2023-10-11T16:32:59.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586434d2b40 con 0x5586434bc370 2023-10-11T16:32:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586434c1930 con 0x5586434c10c0 2023-10-11T16:32:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f3999c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f3999c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40842/0 (socket says 172.21.15.111:40842) 2023-10-11T16:32:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f3919b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434be9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f3999c700 1 -- 172.21.15.111:0/2941155181 learned_addr learned my addr 172.21.15.111:0/2941155181 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:59.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.965+0000 7f2f3899a700 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f3919b700 1 -- 172.21.15.111:0/2941155181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 msgr2=0x5586434c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:59.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f3919b700 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f3919b700 1 -- 172.21.15.111:0/2941155181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 msgr2=0x5586434cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:59.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f3919b700 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f3919b700 1 -- 172.21.15.111:0/2941155181 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586434aab00 con 0x5586434bc370 2023-10-11T16:32:59.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f3999c700 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:59.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f3919b700 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434be9b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2f3009aa00 tx=0x7f2f30005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74a61b82b84c419 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:59.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f1ffff700 1 -- 172.21.15.111:0/2941155181 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f300cb2e0 con 0x5586434bc370 2023-10-11T16:32:59.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f1ffff700 1 -- 172.21.15.111:0/2941155181 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f3000fda0 con 0x5586434bc370 2023-10-11T16:32:59.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.966+0000 7f2f1ffff700 1 -- 172.21.15.111:0/2941155181 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f300cb650 con 0x5586434bc370 2023-10-11T16:32:59.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/2941155181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 msgr2=0x5586434be9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434be9b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2f3009aa00 tx=0x7f2f30005ef0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/2941155181 shutdown_connections 2023-10-11T16:32:59.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434c10c0 0x5586434c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434bc370 0x5586434be9b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/2941155181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bcc80 0x5586434cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/2941155181 >> 172.21.15.111:0/2941155181 conn(0x5586434a9360 msgr2=0x5586434bf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/2941155181 shutdown_connections 2023-10-11T16:32:59.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.967+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/2941155181 wait complete. 2023-10-11T16:32:59.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.968+0000 7f2f4ed9bb80 1 -- start start 2023-10-11T16:32:59.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.968+0000 7f2f4ed9bb80 1 Processor -- start 2023-10-11T16:32:59.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.969+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bc370 0x5586435211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.969+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bcc80 0x558643521730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.969+0000 7f2f4ed9bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 0x5586434cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.969+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586434cc900 con 0x5586434bc370 2023-10-11T16:32:59.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.969+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586434cca70 con 0x5586434c10c0 2023-10-11T16:32:59.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.969+0000 7f2f4ed9bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586434ccc30 con 0x5586434bcc80 2023-10-11T16:32:59.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.969+0000 7f2f4ed9bb80 10 client.0 _ll_register_callbacks cb 0x7f2f30097130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:32:59.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bc370 0x5586435211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bc370 0x5586435211f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33646/0 (socket says 172.21.15.111:33646) 2023-10-11T16:32:59.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3919b700 1 -- 172.21.15.111:0/1880892917 learned_addr learned my addr 172.21.15.111:0/1880892917 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:32:59.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3899a700 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bcc80 0x558643521730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3999c700 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 0x5586434cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3919b700 1 -- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bcc80 msgr2=0x558643521730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:59.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3919b700 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bcc80 0x558643521730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3919b700 1 -- 172.21.15.111:0/1880892917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 msgr2=0x5586434cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:32:59.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3919b700 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 0x5586434cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3919b700 1 -- 172.21.15.111:0/1880892917 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558643286630 con 0x5586434bc370 2023-10-11T16:32:59.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.970+0000 7f2f3899a700 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bcc80 0x558643521730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:32:59.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.971+0000 7f2f3919b700 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bc370 0x5586435211f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000c00 tx=0x7f2f3009b240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb683f20e473e625 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:32:59.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.971+0000 7f2f1cff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:32:59.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.971+0000 7f2f1cff9700 1 -- 172.21.15.111:0/1880892917 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f300b3440 con 0x5586434bc370 2023-10-11T16:32:59.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.971+0000 7f2f1cff9700 1 -- 172.21.15.111:0/1880892917 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2f3009ed30 con 0x5586434bc370 2023-10-11T16:32:59.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.971+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1880892917 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2f2c011040 con 0x5586434bc370 2023-10-11T16:32:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.972+0000 7f2f1cff9700 1 -- 172.21.15.111:0/1880892917 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f300cb2e0 con 0x5586434bc370 2023-10-11T16:32:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.972+0000 7f2f4ed9bb80 10 client.15618 fetch_fsmap learned FSMap version 27 2023-10-11T16:32:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.972+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1880892917 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5586433cb190 con 0x5586434bc370 2023-10-11T16:32:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.972+0000 7f2f1cff9700 1 -- 172.21.15.111:0/1880892917 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f3007e020 con 0x5586434bc370 2023-10-11T16:32:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.973+0000 7f2f1cff9700 1 -- 172.21.15.111:0/1880892917 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f2f30099d80 con 0x5586434bc370 2023-10-11T16:32:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.973+0000 7f2f4ed9bb80 10 client.15618 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:32:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.973+0000 7f2f4ed9bb80 10 client.15618 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:32:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.973+0000 7f2f4ed9bb80 20 client.15618 populate_metadata read hostname 'smithi111' 2023-10-11T16:32:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.973+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55864353abf0 0x55864353cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:32:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.973+0000 7f2f4ed9bb80 4 client.15618 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:32:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.973+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1880892917 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5586433d3660 con 0x55864353abf0 2023-10-11T16:32:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.974+0000 7f2f3899a700 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55864353abf0 0x55864353cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:32:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f3899a700 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55864353abf0 0x55864353cfd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f8a058d9ebe5b96 server_cookie=2a354c6a798344e1 in_seq=0 out_seq=0 2023-10-11T16:32:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f1cff9700 10 client.15618 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:32:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f1cff9700 1 -- 172.21.15.111:0/1880892917 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5586433d3660 con 0x55864353abf0 2023-10-11T16:32:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f1cff9700 10 client.15618 handle_command_reply: tid=0 2023-10-11T16:32:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 client.15618 shutdown 2023-10-11T16:32:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1880892917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bc370 msgr2=0x5586435211f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bc370 0x5586435211f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000c00 tx=0x7f2f3009b240 comp rx=0 tx=0).stop 2023-10-11T16:32:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1880892917 shutdown_connections 2023-10-11T16:32:59.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55864353abf0 0x55864353cfd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5586434c10c0 0x5586434cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5586434bc370 0x5586435211f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 --2- 172.21.15.111:0/1880892917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5586434bcc80 0x558643521730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1880892917 >> 172.21.15.111:0/1880892917 conn(0x5586434a9360 msgr2=0x5586434c61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.975+0000 7f2f1cff9700 0 client.15618 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:32:59.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.976+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1880892917 shutdown_connections 2023-10-11T16:32:59.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.976+0000 7f2f4ed9bb80 1 -- 172.21.15.111:0/1880892917 wait complete. 2023-10-11T16:32:59.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.976+0000 7f2f4ed9bb80 20 client.15618 trim_cache size 0 max 16384 2023-10-11T16:32:59.994 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:32:59.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2f200a13d0 msgr2=0x7f2f200a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2f200a13d0 0x7f2f200a3880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2f34199cd0 tx=0x7f2f24009260 comp rx=0 tx=0).stop 2023-10-11T16:32:59.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/414837666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f34100cc0 msgr2=0x7f2f341986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:32:59.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f34100cc0 0x7f2f341986c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2f300098e0 tx=0x7f2f3000f740 comp rx=0 tx=0).stop 2023-10-11T16:32:59.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/414837666 shutdown_connections 2023-10-11T16:32:59.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2f200a13d0 0x7f2f200a3880 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f34105620 0x7f2f3419bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f34100cc0 0x7f2f341986c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 --2- 172.21.15.111:0/414837666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f34101660 0x7f2f34198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:32:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/414837666 >> 172.21.15.111:0/414837666 conn(0x7f2f34077560 msgr2=0x7f2f340feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:32:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/414837666 shutdown_connections 2023-10-11T16:32:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:32:59.977+0000 7f2f3b3ff700 1 -- 172.21.15.111:0/414837666 wait complete. 2023-10-11T16:33:00.007 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:33:00.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.342+0000 7fa1bedff700 1 -- 172.21.15.111:0/3751892153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b80746a0 msgr2=0x7fa1b8072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.342+0000 7fa1bedff700 1 --2- 172.21.15.111:0/3751892153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b80746a0 0x7fa1b8072d20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009990 tx=0x7fa1ac009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.343+0000 7fa1bedff700 1 -- 172.21.15.111:0/3751892153 shutdown_connections 2023-10-11T16:33:00.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.343+0000 7fa1bedff700 1 --2- 172.21.15.111:0/3751892153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1b810cec0 0x7fa1b810f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.343+0000 7fa1bedff700 1 --2- 172.21.15.111:0/3751892153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b80746a0 0x7fa1b8072d20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.343+0000 7fa1bedff700 1 --2- 172.21.15.111:0/3751892153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b8074b00 0x7fa1b8073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.343+0000 7fa1bedff700 1 -- 172.21.15.111:0/3751892153 >> 172.21.15.111:0/3751892153 conn(0x7fa1b8077590 msgr2=0x7fa1b8077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.345+0000 7fa1bedff700 1 -- 172.21.15.111:0/3751892153 shutdown_connections 2023-10-11T16:33:00.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.345+0000 7fa1bedff700 1 -- 172.21.15.111:0/3751892153 wait complete. 2023-10-11T16:33:00.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.346+0000 7fa1bedff700 1 Processor -- start 2023-10-11T16:33:00.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.346+0000 7fa1bedff700 1 -- start start 2023-10-11T16:33:00.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.347+0000 7fa1bedff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1b80746a0 0x7fa1b810c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.347+0000 7fa1bedff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b8074b00 0x7fa1b810a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.347+0000 7fa1bedff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b810cec0 0x7fa1b810afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.347+0000 7fa1bedff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b810b750 con 0x7fa1b80746a0 2023-10-11T16:33:00.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.347+0000 7fa1bedff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b81a8ee0 con 0x7fa1b810cec0 2023-10-11T16:33:00.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.347+0000 7fa1bedff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b81a9050 con 0x7fa1b8074b00 2023-10-11T16:33:00.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.348+0000 7fa1b7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b8074b00 0x7fa1b810a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.348+0000 7fa1b7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b8074b00 0x7fa1b810a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40860/0 (socket says 172.21.15.111:40860) 2023-10-11T16:33:00.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.348+0000 7fa1b7fff700 1 -- 172.21.15.111:0/1807678589 learned_addr learned my addr 172.21.15.111:0/1807678589 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.348+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b810cec0 0x7fa1b810afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.348+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1b80746a0 0x7fa1b810c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.348+0000 7fa1bd39c700 1 -- 172.21.15.111:0/1807678589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b8074b00 msgr2=0x7fa1b810a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.348+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b8074b00 0x7fa1b810a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.348+0000 7fa1bd39c700 1 -- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1b80746a0 msgr2=0x7fa1b810c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.349+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1b80746a0 0x7fa1b810c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.349+0000 7fa1bd39c700 1 -- 172.21.15.111:0/1807678589 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0095f0 con 0x7fa1b810cec0 2023-10-11T16:33:00.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.349+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1b80746a0 0x7fa1b810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:00.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.349+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b810cec0 0x7fa1b810afc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0007ee0 tx=0x7fa1b000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.349+0000 7fa1b5ffb700 1 -- 172.21.15.111:0/1807678589 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b000ddd0 con 0x7fa1b810cec0 2023-10-11T16:33:00.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.350+0000 7fa1b5ffb700 1 -- 172.21.15.111:0/1807678589 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1b000f040 con 0x7fa1b810cec0 2023-10-11T16:33:00.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.350+0000 7fa1b5ffb700 1 -- 172.21.15.111:0/1807678589 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0005a00 con 0x7fa1b810cec0 2023-10-11T16:33:00.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.350+0000 7fa1bedff700 1 -- 172.21.15.111:0/1807678589 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b8101710 con 0x7fa1b810cec0 2023-10-11T16:33:00.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.350+0000 7fa1bedff700 1 -- 172.21.15.111:0/1807678589 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b8101580 con 0x7fa1b810cec0 2023-10-11T16:33:00.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.355+0000 7fa1b5ffb700 1 -- 172.21.15.111:0/1807678589 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 23) v1 ==== 135892+0+0 (secure 0 0 0) 0x7fa1b0004ad0 con 0x7fa1b810cec0 2023-10-11T16:33:00.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.356+0000 7fa1d261fb80 1 Processor -- start 2023-10-11T16:33:00.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.357+0000 7fa1d261fb80 1 -- start start 2023-10-11T16:33:00.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.357+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.357+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 0x562fea5f5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.357+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f58b0 0x562fea5f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.357+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea4f6060 con 0x562fea5f58b0 2023-10-11T16:33:00.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.357+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea5488a0 con 0x562fea5f0b60 2023-10-11T16:33:00.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.357+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea541d30 con 0x562fea5f1470 2023-10-11T16:33:00.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5f0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46144/0 (socket says 172.21.15.111:46144) 2023-10-11T16:33:00.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 -- 172.21.15.111:0/4153866558 learned_addr learned my addr 172.21.15.111:0/4153866558 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 0x562fea5f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f58b0 0x562fea5f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 -- 172.21.15.111:0/4153866558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 msgr2=0x562fea5f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 0x562fea5f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 -- 172.21.15.111:0/4153866558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f58b0 msgr2=0x562fea5f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f58b0 0x562fea5f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.358+0000 7fa1b7fff700 1 -- 172.21.15.111:0/4153866558 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fea3c3630 con 0x562fea5f0b60 2023-10-11T16:33:00.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.359+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5f0f30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40046e0 tx=0x7fa1a4009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eab79fce4fde5e71 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.359+0000 7fa1b5ffb700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1a80a1410 0x7fa1a80a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.359+0000 7fa1a2ffd700 1 -- 172.21.15.111:0/4153866558 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4004830 con 0x562fea5f0b60 2023-10-11T16:33:00.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.359+0000 7fa1a2ffd700 1 -- 172.21.15.111:0/4153866558 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1a4017070 con 0x562fea5f0b60 2023-10-11T16:33:00.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.359+0000 7fa1a2ffd700 1 -- 172.21.15.111:0/4153866558 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a401d630 con 0x562fea5f0b60 2023-10-11T16:33:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.359+0000 7fa1b5ffb700 1 -- 172.21.15.111:0/1807678589 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa1b00ccb40 con 0x7fa1b810cec0 2023-10-11T16:33:00.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.360+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1a80a1410 0x7fa1a80a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.360+0000 7fa1d261fb80 1 -- 172.21.15.111:0/4153866558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 msgr2=0x562fea5f0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.360+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5f0f30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40046e0 tx=0x7fa1a4009640 comp rx=0 tx=0).stop 2023-10-11T16:33:00.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.360+0000 7fa1d261fb80 1 -- 172.21.15.111:0/4153866558 shutdown_connections 2023-10-11T16:33:00.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.361+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f58b0 0x562fea5f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.361+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5f0f30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.361+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/4153866558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 0x562fea5f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.361+0000 7fa1d261fb80 1 -- 172.21.15.111:0/4153866558 >> 172.21.15.111:0/4153866558 conn(0x562fea5e6360 msgr2=0x562fea5ebb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.361+0000 7fa1d261fb80 1 -- 172.21.15.111:0/4153866558 shutdown_connections 2023-10-11T16:33:00.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.361+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1a80a1410 0x7fa1a80a38c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00b2e0 tx=0x7fa1ac005fd0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.361+0000 7fa1d261fb80 1 -- 172.21.15.111:0/4153866558 wait complete. 2023-10-11T16:33:00.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.362+0000 7fa1d261fb80 1 -- start start 2023-10-11T16:33:00.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.362+0000 7fa1d261fb80 1 Processor -- start 2023-10-11T16:33:00.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.363+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5fd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.363+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f1470 0x562fea5fa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea5faa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea4e38e0 con 0x562fea5f1470 2023-10-11T16:33:00.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea4e3580 con 0x562fea5f0b60 2023-10-11T16:33:00.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea3c5920 con 0x562fea5f58b0 2023-10-11T16:33:00.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1bcb9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5fd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1d261fb80 10 client.0 _ll_register_callbacks cb 0x562fea501780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:00.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1bcb9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5fd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46158/0 (socket says 172.21.15.111:46158) 2023-10-11T16:33:00.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1bcb9b700 1 -- 172.21.15.111:0/3534645802 learned_addr learned my addr 172.21.15.111:0/3534645802 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f1470 0x562fea5fa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.364+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea5faa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.365+0000 7fa1bcb9b700 1 -- 172.21.15.111:0/3534645802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 msgr2=0x562fea5faa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.365+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea5faa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.365+0000 7fa1bcb9b700 1 -- 172.21.15.111:0/3534645802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f1470 msgr2=0x562fea5fa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.365+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f1470 0x562fea5fa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.365+0000 7fa1bcb9b700 1 -- 172.21.15.111:0/3534645802 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a4019040 con 0x562fea5f0b60 2023-10-11T16:33:00.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.365+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f1470 0x562fea5fa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:00.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.365+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5fd100 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0059d0 tx=0x7fa1ac009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4848d114284260a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.365+0000 7fa197fff700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:00.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.366+0000 7fa197fff700 1 -- 172.21.15.111:0/3534645802 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac024850 con 0x562fea5f0b60 2023-10-11T16:33:00.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.366+0000 7fa197fff700 1 -- 172.21.15.111:0/3534645802 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa1ac0249b0 con 0x562fea5f0b60 2023-10-11T16:33:00.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.366+0000 7fa1d261fb80 1 -- 172.21.15.111:0/3534645802 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562fea5e3b20 con 0x562fea5f0b60 2023-10-11T16:33:00.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.366+0000 7fa197fff700 1 -- 172.21.15.111:0/3534645802 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1ac024b70 con 0x562fea5f0b60 2023-10-11T16:33:00.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.366+0000 7fa1d261fb80 10 client.25676 fetch_fsmap learned FSMap version 27 2023-10-11T16:33:00.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.366+0000 7fa1d261fb80 1 -- 172.21.15.111:0/3534645802 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562fea508190 con 0x562fea5f0b60 2023-10-11T16:33:00.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.366+0000 7fa197fff700 1 -- 172.21.15.111:0/3534645802 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562fea5e3b20 con 0x562fea5f0b60 2023-10-11T16:33:00.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.367+0000 7fa197fff700 1 -- 172.21.15.111:0/3534645802 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fa1ac017cf0 con 0x562fea5f0b60 2023-10-11T16:33:00.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.367+0000 7fa1d261fb80 10 client.25676 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:33:00.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.367+0000 7fa1d261fb80 10 client.25676 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:33:00.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.367+0000 7fa1d261fb80 20 client.25676 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:00.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.367+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x562fea60a7f0 0x562fea60cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.367+0000 7fa1d261fb80 4 client.25676 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:00.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.367+0000 7fa1d261fb80 1 -- 172.21.15.111:0/3534645802 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562fea5040a0 con 0x562fea60a7f0 2023-10-11T16:33:00.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.368+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x562fea60a7f0 0x562fea60cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.368+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x562fea60a7f0 0x562fea60cc00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=16939ba20f4f3ba3 server_cookie=ebb501aa42303d0d in_seq=0 out_seq=0 2023-10-11T16:33:00.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.369+0000 7fa197fff700 10 client.25676 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:33:00.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.369+0000 7fa197fff700 1 -- 172.21.15.111:0/3534645802 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562fea5040a0 con 0x562fea60a7f0 2023-10-11T16:33:00.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.369+0000 7fa197fff700 10 client.25676 handle_command_reply: tid=0 2023-10-11T16:33:00.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 client.25676 shutdown 2023-10-11T16:33:00.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 -- 172.21.15.111:0/3534645802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 msgr2=0x562fea5fd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5fd100 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0059d0 tx=0x7fa1ac009310 comp rx=0 tx=0).stop 2023-10-11T16:33:00.390 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:00 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/737701936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:33:00.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 -- 172.21.15.111:0/3534645802 shutdown_connections 2023-10-11T16:33:00.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x562fea60a7f0 0x562fea60cc00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea5faa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f0b60 0x562fea5fd100 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/3534645802 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f1470 0x562fea5fa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa1d261fb80 1 -- 172.21.15.111:0/3534645802 >> 172.21.15.111:0/3534645802 conn(0x562fea5e6360 msgr2=0x562fea5f6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.370+0000 7fa197fff700 0 client.25676 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:33:00.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.371+0000 7fa1d261fb80 1 -- 172.21.15.111:0/3534645802 shutdown_connections 2023-10-11T16:33:00.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.371+0000 7fa1d261fb80 1 -- 172.21.15.111:0/3534645802 wait complete. 2023-10-11T16:33:00.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.372+0000 7fa1d261fb80 20 client.25676 trim_cache size 0 max 16384 2023-10-11T16:33:00.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.401+0000 7fa1d261fb80 1 Processor -- start 2023-10-11T16:33:00.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.401+0000 7fa1d261fb80 1 -- start start 2023-10-11T16:33:00.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.401+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea5fcc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.401+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 0x562fea5f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.401+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f58b0 0x562fea5f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.401+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea5f44a0 con 0x562fea5f0b60 2023-10-11T16:33:00.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.401+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea5f3270 con 0x562fea5f58b0 2023-10-11T16:33:00.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.401+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea5f3430 con 0x562fea5f1470 2023-10-11T16:33:00.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bd39c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f58b0 0x562fea5f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bd39c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f58b0 0x562fea5f81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46170/0 (socket says 172.21.15.111:46170) 2023-10-11T16:33:00.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bd39c700 1 -- 172.21.15.111:0/2143763694 learned_addr learned my addr 172.21.15.111:0/2143763694 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea5fcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 0x562fea5f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bd39c700 1 -- 172.21.15.111:0/2143763694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 msgr2=0x562fea5f7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 0x562fea5f7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bd39c700 1 -- 172.21.15.111:0/2143763694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 msgr2=0x562fea5fcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea5fcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bd39c700 1 -- 172.21.15.111:0/2143763694 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fea507ee0 con 0x562fea5f58b0 2023-10-11T16:33:00.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.402+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea5fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:00.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.403+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f58b0 0x562fea5f81d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0099390 tx=0x7fa1b00b9040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5718e7206b79d0d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.403+0000 7fa1a37fe700 1 -- 172.21.15.111:0/2143763694 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0098ff0 con 0x562fea5f58b0 2023-10-11T16:33:00.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.403+0000 7fa1a37fe700 1 -- 172.21.15.111:0/2143763694 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1b0082e90 con 0x562fea5f58b0 2023-10-11T16:33:00.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.403+0000 7fa1a37fe700 1 -- 172.21.15.111:0/2143763694 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00ca2d0 con 0x562fea5f58b0 2023-10-11T16:33:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.404+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2143763694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f58b0 msgr2=0x562fea5f81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.404+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f58b0 0x562fea5f81d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0099390 tx=0x7fa1b00b9040 comp rx=0 tx=0).stop 2023-10-11T16:33:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.404+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2143763694 shutdown_connections 2023-10-11T16:33:00.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.404+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f58b0 0x562fea5f81d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.404+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea5fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.404+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2143763694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f1470 0x562fea5f7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.404+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2143763694 >> 172.21.15.111:0/2143763694 conn(0x562fea5e6360 msgr2=0x562fea5f3930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.405+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2143763694 shutdown_connections 2023-10-11T16:33:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.405+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2143763694 wait complete. 2023-10-11T16:33:00.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.405+0000 7fa1d261fb80 1 -- start start 2023-10-11T16:33:00.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.406+0000 7fa1d261fb80 1 Processor -- start 2023-10-11T16:33:00.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.406+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea600bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.407+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f1470 0x562fea601110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.407+0000 7fa1d261fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea65d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.407+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea601960 con 0x562fea5f0b60 2023-10-11T16:33:00.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.407+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea65db90 con 0x562fea5f1470 2023-10-11T16:33:00.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.407+0000 7fa1d261fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fea65dcd0 con 0x562fea5f58b0 2023-10-11T16:33:00.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.407+0000 7fa1d261fb80 10 client.0 _ll_register_callbacks cb 0x7fa1ac00bf40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.408+0000 7fa1b7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f1470 0x562fea601110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.408+0000 7fa1b7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f1470 0x562fea601110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46180/0 (socket says 172.21.15.111:46180) 2023-10-11T16:33:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.408+0000 7fa1b7fff700 1 -- 172.21.15.111:0/2440422139 learned_addr learned my addr 172.21.15.111:0/2440422139 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.408+0000 7fa1bd39c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea65d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.408+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea600bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.409+0000 7fa1b7fff700 1 -- 172.21.15.111:0/2440422139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 msgr2=0x562fea65d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.409+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea65d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.409+0000 7fa1b7fff700 1 -- 172.21.15.111:0/2440422139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 msgr2=0x562fea600bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.409+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea600bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.409+0000 7fa1b7fff700 1 -- 172.21.15.111:0/2440422139 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b8101580 con 0x562fea5f1470 2023-10-11T16:33:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.409+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea600bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.409+0000 7fa1bd39c700 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea65d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.410+0000 7fa1b7fff700 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f1470 0x562fea601110 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa1a401c000 tx=0x7fa1a4005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359d3422711f73c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.410+0000 7fa197fff700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:00.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.411+0000 7fa197fff700 1 -- 172.21.15.111:0/2440422139 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4024040 con 0x562fea5f1470 2023-10-11T16:33:00.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.411+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2440422139 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1a4017070 con 0x562fea5f1470 2023-10-11T16:33:00.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.411+0000 7fa197fff700 1 -- 172.21.15.111:0/2440422139 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa1a4005140 con 0x562fea5f1470 2023-10-11T16:33:00.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.411+0000 7fa197fff700 1 -- 172.21.15.111:0/2440422139 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1a4005300 con 0x562fea5f1470 2023-10-11T16:33:00.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.411+0000 7fa1d261fb80 10 client.25682 fetch_fsmap learned FSMap version 27 2023-10-11T16:33:00.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.412+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2440422139 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562fea358c60 con 0x562fea5f1470 2023-10-11T16:33:00.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.412+0000 7fa197fff700 1 -- 172.21.15.111:0/2440422139 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1a4017070 con 0x562fea5f1470 2023-10-11T16:33:00.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.412+0000 7fa197fff700 1 -- 172.21.15.111:0/2440422139 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fa1a4004710 con 0x562fea5f1470 2023-10-11T16:33:00.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.412+0000 7fa1d261fb80 10 client.25682 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:33:00.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.413+0000 7fa1d261fb80 10 client.25682 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:33:00.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.413+0000 7fa1d261fb80 20 client.25682 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:00.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.413+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x562fea670340 0x562fea672750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.413+0000 7fa1d261fb80 4 client.25682 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:33:00.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.413+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2440422139 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562fea510660 con 0x562fea670340 2023-10-11T16:33:00.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.413+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x562fea670340 0x562fea672750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.414+0000 7fa1bcb9b700 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x562fea670340 0x562fea672750 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ed0945da4386507 server_cookie=ef9b66f5c89f6d06 in_seq=0 out_seq=0 2023-10-11T16:33:00.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.414+0000 7fa197fff700 10 client.25682 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:33:00.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.414+0000 7fa197fff700 1 -- 172.21.15.111:0/2440422139 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562fea510660 con 0x562fea670340 2023-10-11T16:33:00.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.414+0000 7fa197fff700 10 client.25682 handle_command_reply: tid=0 2023-10-11T16:33:00.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.414+0000 7fa1d261fb80 1 client.25682 shutdown 2023-10-11T16:33:00.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.415+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2440422139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f1470 msgr2=0x562fea601110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.415+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f1470 0x562fea601110 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa1a401c000 tx=0x7fa1a4005dc0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.415+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2440422139 shutdown_connections 2023-10-11T16:33:00.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.415+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x562fea670340 0x562fea672750 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.415+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562fea5f58b0 0x562fea65d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.415+0000 7fa197fff700 0 client.25682 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:33:00.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.415+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562fea5f0b60 0x562fea600bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.415+0000 7fa1d261fb80 1 --2- 172.21.15.111:0/2440422139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562fea5f1470 0x562fea601110 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.416+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2440422139 >> 172.21.15.111:0/2440422139 conn(0x562fea5e6360 msgr2=0x562fea5f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.416+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2440422139 shutdown_connections 2023-10-11T16:33:00.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.416+0000 7fa1d261fb80 1 -- 172.21.15.111:0/2440422139 wait complete. 2023-10-11T16:33:00.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.416+0000 7fa1d261fb80 20 client.25682 trim_cache size 0 max 16384 2023-10-11T16:33:00.432 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:33:00.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.418+0000 7fa1bedff700 1 -- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1a80a1410 msgr2=0x7fa1a80a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.418+0000 7fa1bedff700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1a80a1410 0x7fa1a80a38c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00b2e0 tx=0x7fa1ac005fd0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 -- 172.21.15.111:0/1807678589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b810cec0 msgr2=0x7fa1b810afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b810cec0 0x7fa1b810afc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0007ee0 tx=0x7fa1b000d3b0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 -- 172.21.15.111:0/1807678589 shutdown_connections 2023-10-11T16:33:00.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1a80a1410 0x7fa1a80a38c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1b810cec0 0x7fa1b810afc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1b80746a0 0x7fa1b810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 --2- 172.21.15.111:0/1807678589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1b8074b00 0x7fa1b810a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 -- 172.21.15.111:0/1807678589 >> 172.21.15.111:0/1807678589 conn(0x7fa1b8077590 msgr2=0x7fa1b8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 -- 172.21.15.111:0/1807678589 shutdown_connections 2023-10-11T16:33:00.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.419+0000 7fa1bedff700 1 -- 172.21.15.111:0/1807678589 wait complete. 2023-10-11T16:33:00.451 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:33:00.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:00 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/737701936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:33:00.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:00 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/737701936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:33:00.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:33:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:33:00.367+0000 7f1c3f2cb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:33:00.636 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:33:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:33:00.367+0000 7f1c3f2cb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:33:00.636 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:33:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:33:00.367+0000 7f1c3f2cb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:33:00.636 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:33:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:33:00.367+0000 7f1c3f2cb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:33:00.636 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:33:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x[98777]: 2023-10-11T16:33:00.367+0000 7f1c3f2cb700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-11T16:33:00.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 -- 172.21.15.111:0/3937191306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1000ffdf0 msgr2=0x7fb100103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 --2- 172.21.15.111:0/3937191306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1000ffdf0 0x7fb100103db0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40099c0 tx=0x7fb0f4009cd0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 -- 172.21.15.111:0/3937191306 shutdown_connections 2023-10-11T16:33:00.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 --2- 172.21.15.111:0/3937191306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb100104490 0x7fb10010c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 --2- 172.21.15.111:0/3937191306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1000ff450 0x7fb1000ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 --2- 172.21.15.111:0/3937191306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1000ffdf0 0x7fb100103db0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 -- 172.21.15.111:0/3937191306 >> 172.21.15.111:0/3937191306 conn(0x7fb1000770b0 msgr2=0x7fb1000774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 -- 172.21.15.111:0/3937191306 shutdown_connections 2023-10-11T16:33:00.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.796+0000 7fb104dff700 1 -- 172.21.15.111:0/3937191306 wait complete. 2023-10-11T16:33:00.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.798+0000 7fb104dff700 1 Processor -- start 2023-10-11T16:33:00.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.798+0000 7fb104dff700 1 -- start start 2023-10-11T16:33:00.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.798+0000 7fb104dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1000ff450 0x7fb10019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.798+0000 7fb104dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb100104490 0x7fb10019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.798+0000 7fb104dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1001986f0 0x7fb100198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb104dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb100199160 con 0x7fb100104490 2023-10-11T16:33:00.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb104dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1001992a0 con 0x7fb1001986f0 2023-10-11T16:33:00.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb104dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb100199460 con 0x7fb1000ff450 2023-10-11T16:33:00.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fe59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1000ff450 0x7fb10019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fe59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1000ff450 0x7fb10019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40930/0 (socket says 172.21.15.111:40930) 2023-10-11T16:33:00.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fe59c700 1 -- 172.21.15.111:0/100497284 learned_addr learned my addr 172.21.15.111:0/100497284 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb100104490 0x7fb10019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fed9d700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1001986f0 0x7fb100198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/100497284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1000ff450 msgr2=0x7fb10019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1000ff450 0x7fb10019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/100497284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1001986f0 msgr2=0x7fb100198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1001986f0 0x7fb100198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.799+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/100497284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f40095f0 con 0x7fb100104490 2023-10-11T16:33:00.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.800+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb100104490 0x7fb10019ebc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb1001002a0 tx=0x7fb0f400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.800+0000 7fb0ef7fe700 1 -- 172.21.15.111:0/100497284 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f401d070 con 0x7fb100104490 2023-10-11T16:33:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.800+0000 7fb0ef7fe700 1 -- 172.21.15.111:0/100497284 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0f4005030 con 0x7fb100104490 2023-10-11T16:33:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.800+0000 7fb104dff700 1 -- 172.21.15.111:0/100497284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb100110400 con 0x7fb100104490 2023-10-11T16:33:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.800+0000 7fb0ef7fe700 1 -- 172.21.15.111:0/100497284 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4004970 con 0x7fb100104490 2023-10-11T16:33:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.801+0000 7fb104dff700 1 -- 172.21.15.111:0/100497284 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb100110270 con 0x7fb100104490 2023-10-11T16:33:00.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.807+0000 7fb0ef7fe700 1 -- 172.21.15.111:0/100497284 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 23) v1 ==== 135892+0+0 (secure 0 0 0) 0x7fb0f40053a0 con 0x7fb100104490 2023-10-11T16:33:00.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.808+0000 7fb11869eb80 1 Processor -- start 2023-10-11T16:33:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.809+0000 7fb11869eb80 1 -- start start 2023-10-11T16:33:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.809+0000 7fb0ef7fe700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb0e80a13d0 0x7fb0e80a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.809+0000 7fb0ef7fe700 1 -- 172.21.15.111:0/100497284 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb0f40cdfe0 con 0x7fb100104490 2023-10-11T16:33:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c5ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5f280 0x564444c71280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444c73ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb11869eb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444b64060 con 0x564444c717c0 2023-10-11T16:33:00.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb11869eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444bb68a0 con 0x564444c5f280 2023-10-11T16:33:00.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb11869eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444bafd30 con 0x564444c5e970 2023-10-11T16:33:00.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb0fed9d700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb0e80a13d0 0x7fb0e80a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb0fe59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c5ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb0fe59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c5ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40938/0 (socket says 172.21.15.111:40938) 2023-10-11T16:33:00.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.810+0000 7fb0fe59c700 1 -- 172.21.15.111:0/870309102 learned_addr learned my addr 172.21.15.111:0/870309102 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.811+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5f280 0x564444c71280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.811+0000 7fb0fed9d700 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444c73ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.811+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/870309102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 msgr2=0x564444c5ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.811+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c5ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.811+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/870309102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 msgr2=0x564444c73ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.811+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444c73ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.811+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/870309102 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564444a31630 con 0x564444c5f280 2023-10-11T16:33:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.811+0000 7fb0fed9d700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb0e80a13d0 0x7fb0e80a3880 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009c10 tx=0x7fb0f8004040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.812+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5f280 0x564444c71280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40cb2c0 tx=0x7fb0f409aa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e33a21886e0bb65 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.812+0000 7fb0ecff9700 1 -- 172.21.15.111:0/870309102 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f409ac20 con 0x564444c5f280 2023-10-11T16:33:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.812+0000 7fb0ecff9700 1 -- 172.21.15.111:0/870309102 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0f409ec50 con 0x564444c5f280 2023-10-11T16:33:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.812+0000 7fb0ecff9700 1 -- 172.21.15.111:0/870309102 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f409ae80 con 0x564444c5f280 2023-10-11T16:33:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 -- 172.21.15.111:0/870309102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5f280 msgr2=0x564444c71280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5f280 0x564444c71280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40cb2c0 tx=0x7fb0f409aa70 comp rx=0 tx=0).stop 2023-10-11T16:33:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 -- 172.21.15.111:0/870309102 shutdown_connections 2023-10-11T16:33:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444c73ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c5ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 --2- 172.21.15.111:0/870309102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5f280 0x564444c71280 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 -- 172.21.15.111:0/870309102 >> 172.21.15.111:0/870309102 conn(0x564444c54360 msgr2=0x564444c59a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 -- 172.21.15.111:0/870309102 shutdown_connections 2023-10-11T16:33:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.813+0000 7fb11869eb80 1 -- 172.21.15.111:0/870309102 wait complete. 2023-10-11T16:33:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.814+0000 7fb11869eb80 1 -- start start 2023-10-11T16:33:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.814+0000 7fb11869eb80 1 Processor -- start 2023-10-11T16:33:00.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.815+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c71b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.815+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.815+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c717c0 0x564444c72610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.815+0000 7fb11869eb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444b518e0 con 0x564444c5f280 2023-10-11T16:33:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.815+0000 7fb11869eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444b51580 con 0x564444c717c0 2023-10-11T16:33:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.815+0000 7fb11869eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444a33920 con 0x564444c5e970 2023-10-11T16:33:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.815+0000 7fb11869eb80 10 client.0 _ll_register_callbacks cb 0x564444b6f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fdd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fdd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33772/0 (socket says 172.21.15.111:33772) 2023-10-11T16:33:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/1343320041 learned_addr learned my addr 172.21.15.111:0/1343320041 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c71b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fed9d700 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c717c0 0x564444c72610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/1343320041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 msgr2=0x564444c71b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c71b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/1343320041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c717c0 msgr2=0x564444c72610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c717c0 0x564444c72610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.816+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/1343320041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb100110270 con 0x564444c5f280 2023-10-11T16:33:00.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.817+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c71b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:00.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.817+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c720d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40cb980 tx=0x7fb0f4020e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941fb13b08dccf89 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.817+0000 7fb0e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.817+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/1343320041 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f40b3380 con 0x564444c5f280 2023-10-11T16:33:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.817+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/1343320041 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb0f409ac20 con 0x564444c5f280 2023-10-11T16:33:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.818+0000 7fb11869eb80 1 -- 172.21.15.111:0/1343320041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564444c51b20 con 0x564444c5f280 2023-10-11T16:33:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.818+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/1343320041 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0f40a2fe0 con 0x564444c5f280 2023-10-11T16:33:00.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.818+0000 7fb11869eb80 10 client.15654 fetch_fsmap learned FSMap version 27 2023-10-11T16:33:00.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.818+0000 7fb11869eb80 1 -- 172.21.15.111:0/1343320041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564444b76190 con 0x564444c5f280 2023-10-11T16:33:00.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.818+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/1343320041 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564444c51b20 con 0x564444c5f280 2023-10-11T16:33:00.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.819+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/1343320041 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fb0f40a2b50 con 0x564444c5f280 2023-10-11T16:33:00.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.819+0000 7fb11869eb80 10 client.15654 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:33:00.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.819+0000 7fb11869eb80 10 client.15654 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:00.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.819+0000 7fb11869eb80 20 client.15654 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:00.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.819+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564444c80850 0x564444c82c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.819+0000 7fb11869eb80 4 client.15654 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:00.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.819+0000 7fb11869eb80 1 -- 172.21.15.111:0/1343320041 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564444b720a0 con 0x564444c80850 2023-10-11T16:33:00.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.819+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564444c80850 0x564444c82c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.820+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564444c80850 0x564444c82c30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f34692a36f94293 server_cookie=cf1f65a5f01ab8ee in_seq=0 out_seq=0 2023-10-11T16:33:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.820+0000 7fb0e5ffb700 10 client.15654 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/1343320041 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564444b720a0 con 0x564444c80850 2023-10-11T16:33:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb0e5ffb700 10 client.15654 handle_command_reply: tid=0 2023-10-11T16:33:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 client.15654 shutdown 2023-10-11T16:33:00.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 -- 172.21.15.111:0/1343320041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 msgr2=0x564444c720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c720d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40cb980 tx=0x7fb0f4020e40 comp rx=0 tx=0).stop 2023-10-11T16:33:00.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 -- 172.21.15.111:0/1343320041 shutdown_connections 2023-10-11T16:33:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564444c80850 0x564444c82c30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c717c0 0x564444c72610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5e970 0x564444c71b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb0e5ffb700 0 client.15654 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:00.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1343320041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c720d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.821+0000 7fb11869eb80 1 -- 172.21.15.111:0/1343320041 >> 172.21.15.111:0/1343320041 conn(0x564444c54360 msgr2=0x564444c58de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.822+0000 7fb11869eb80 1 -- 172.21.15.111:0/1343320041 shutdown_connections 2023-10-11T16:33:00.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.822+0000 7fb11869eb80 1 -- 172.21.15.111:0/1343320041 wait complete. 2023-10-11T16:33:00.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.822+0000 7fb11869eb80 20 client.15654 trim_cache size 0 max 16384 2023-10-11T16:33:00.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.848+0000 7fb11869eb80 1 Processor -- start 2023-10-11T16:33:00.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.848+0000 7fb11869eb80 1 -- start start 2023-10-11T16:33:00.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.848+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c7d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.848+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c7c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.848+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c717c0 0x564444c59560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.848+0000 7fb11869eb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444c5b4a0 con 0x564444c5f280 2023-10-11T16:33:00.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.848+0000 7fb11869eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444c5fb90 con 0x564444c5e970 2023-10-11T16:33:00.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.848+0000 7fb11869eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444c70930 con 0x564444c717c0 2023-10-11T16:33:00.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fe59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c7d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fdd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c7c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fe59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c7d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46236/0 (socket says 172.21.15.111:46236) 2023-10-11T16:33:00.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fe59c700 1 -- 172.21.15.111:0/1840862320 learned_addr learned my addr 172.21.15.111:0/1840862320 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fed9d700 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c717c0 0x564444c59560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fe59c700 1 -- 172.21.15.111:0/1840862320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c717c0 msgr2=0x564444c59560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c717c0 0x564444c59560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fe59c700 1 -- 172.21.15.111:0/1840862320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 msgr2=0x564444c7c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c7c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fe59c700 1 -- 172.21.15.111:0/1840862320 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564444b75ee0 con 0x564444c5e970 2023-10-11T16:33:00.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.849+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c7c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:00.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.850+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c7d090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001fcb0 tx=0x7fb0f0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d9dea40fe85abe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.850+0000 7fb0ed7fa700 1 -- 172.21.15.111:0/1840862320 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0011c30 con 0x564444c5e970 2023-10-11T16:33:00.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.850+0000 7fb0ed7fa700 1 -- 172.21.15.111:0/1840862320 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0f0023070 con 0x564444c5e970 2023-10-11T16:33:00.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.850+0000 7fb0ed7fa700 1 -- 172.21.15.111:0/1840862320 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0004830 con 0x564444c5e970 2023-10-11T16:33:00.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 -- 172.21.15.111:0/1840862320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 msgr2=0x564444c7d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c7d090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001fcb0 tx=0x7fb0f0009260 comp rx=0 tx=0).stop 2023-10-11T16:33:00.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 -- 172.21.15.111:0/1840862320 shutdown_connections 2023-10-11T16:33:00.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c717c0 0x564444c59560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c7d090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 --2- 172.21.15.111:0/1840862320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c5f280 0x564444c7c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 -- 172.21.15.111:0/1840862320 >> 172.21.15.111:0/1840862320 conn(0x564444c54360 msgr2=0x564444c72420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 -- 172.21.15.111:0/1840862320 shutdown_connections 2023-10-11T16:33:00.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.851+0000 7fb11869eb80 1 -- 172.21.15.111:0/1840862320 wait complete. 2023-10-11T16:33:00.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.852+0000 7fb11869eb80 1 -- start start 2023-10-11T16:33:00.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.852+0000 7fb11869eb80 1 Processor -- start 2023-10-11T16:33:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.853+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c74950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.853+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5f280 0x564444c74e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.853+0000 7fb11869eb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444cd3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.853+0000 7fb11869eb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444cd3b40 con 0x564444c717c0 2023-10-11T16:33:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.853+0000 7fb11869eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444cd3c80 con 0x564444c5e970 2023-10-11T16:33:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.853+0000 7fb11869eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564444c60a10 con 0x564444c5f280 2023-10-11T16:33:00.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.853+0000 7fb11869eb80 10 client.0 _ll_register_callbacks cb 0x7fb0f8009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:00.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.853+0000 7fb0fe59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c74950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.854+0000 7fb0fe59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c74950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46246/0 (socket says 172.21.15.111:46246) 2023-10-11T16:33:00.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.854+0000 7fb0fed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444cd3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.854+0000 7fb0fe59c700 1 -- 172.21.15.111:0/3955282315 learned_addr learned my addr 172.21.15.111:0/3955282315 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.854+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5f280 0x564444c74e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.854+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/3955282315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 msgr2=0x564444c74950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.854+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c74950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.854+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/3955282315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 msgr2=0x564444cd3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.854+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444cd3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.855+0000 7fb0fdd9b700 1 -- 172.21.15.111:0/3955282315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564444b76190 con 0x564444c5f280 2023-10-11T16:33:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.855+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c74950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.855+0000 7fb0fed9d700 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444cd3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.855+0000 7fb0fdd9b700 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5f280 0x564444c74e90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb0f409a920 tx=0x7fb0f409edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eebba0f7059eb36a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.855+0000 7fb0e5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:33:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.855+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/3955282315 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4004e30 con 0x564444c5f280 2023-10-11T16:33:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.855+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/3955282315 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb0f409ac20 con 0x564444c5f280 2023-10-11T16:33:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.855+0000 7fb11869eb80 1 -- 172.21.15.111:0/3955282315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0f409ec50 con 0x564444c5f280 2023-10-11T16:33:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.856+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/3955282315 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0f4020eb0 con 0x564444c5f280 2023-10-11T16:33:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.856+0000 7fb11869eb80 10 client.25744 fetch_fsmap learned FSMap version 27 2023-10-11T16:33:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.856+0000 7fb11869eb80 1 -- 172.21.15.111:0/3955282315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fb0f001fa40 con 0x564444c5f280 2023-10-11T16:33:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.856+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/3955282315 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0f4083030 con 0x564444c5f280 2023-10-11T16:33:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.857+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/3955282315 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fb0f409a110 con 0x564444c5f280 2023-10-11T16:33:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.857+0000 7fb11869eb80 10 client.25744 fetch_fsmap finished waiting for FSMap version 27 2023-10-11T16:33:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.857+0000 7fb11869eb80 10 client.25744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:00.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.857+0000 7fb11869eb80 20 client.25744 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:00.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.857+0000 7fb11869eb80 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564444c61c10 0x564444c63ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:00.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.857+0000 7fb11869eb80 4 client.25744 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:33:00.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.857+0000 7fb11869eb80 1 -- 172.21.15.111:0/3955282315 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564444b7e660 con 0x564444c61c10 2023-10-11T16:33:00.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.858+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564444c61c10 0x564444c63ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:00.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb0fe59c700 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564444c61c10 0x564444c63ff0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c64342cfbaa7ac8 server_cookie=93a9d4c496fa714f in_seq=0 out_seq=0 2023-10-11T16:33:00.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb0e5ffb700 10 client.25744 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:00.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb0e5ffb700 1 -- 172.21.15.111:0/3955282315 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564444b7e660 con 0x564444c61c10 2023-10-11T16:33:00.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb0e5ffb700 10 client.25744 handle_command_reply: tid=0 2023-10-11T16:33:00.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 client.25744 shutdown 2023-10-11T16:33:00.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 -- 172.21.15.111:0/3955282315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5f280 msgr2=0x564444c74e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5f280 0x564444c74e90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb0f409a920 tx=0x7fb0f409edc0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 -- 172.21.15.111:0/3955282315 shutdown_connections 2023-10-11T16:33:00.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564444c61c10 0x564444c63ff0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564444c717c0 0x564444cd3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564444c5e970 0x564444c74950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 --2- 172.21.15.111:0/3955282315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564444c5f280 0x564444c74e90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb11869eb80 1 -- 172.21.15.111:0/3955282315 >> 172.21.15.111:0/3955282315 conn(0x564444c54360 msgr2=0x564444c72420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.859+0000 7fb0e5ffb700 0 client.25744 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:00.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.860+0000 7fb11869eb80 1 -- 172.21.15.111:0/3955282315 shutdown_connections 2023-10-11T16:33:00.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.860+0000 7fb11869eb80 1 -- 172.21.15.111:0/3955282315 wait complete. 2023-10-11T16:33:00.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.860+0000 7fb11869eb80 20 client.25744 trim_cache size 0 max 16384 2023-10-11T16:33:00.871 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:33:00.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 -- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb0e80a13d0 msgr2=0x7fb0e80a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb0e80a13d0 0x7fb0e80a3880 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009c10 tx=0x7fb0f8004040 comp rx=0 tx=0).stop 2023-10-11T16:33:00.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 -- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb100104490 msgr2=0x7fb10019ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:00.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb100104490 0x7fb10019ebc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb1001002a0 tx=0x7fb0f400f690 comp rx=0 tx=0).stop 2023-10-11T16:33:00.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 -- 172.21.15.111:0/100497284 shutdown_connections 2023-10-11T16:33:00.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb0e80a13d0 0x7fb0e80a3880 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1001986f0 0x7fb100198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1000ff450 0x7fb10019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 --2- 172.21.15.111:0/100497284 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb100104490 0x7fb10019ebc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:00.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 -- 172.21.15.111:0/100497284 >> 172.21.15.111:0/100497284 conn(0x7fb1000770b0 msgr2=0x7fb1000fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:00.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.861+0000 7fb104dff700 1 -- 172.21.15.111:0/100497284 shutdown_connections 2023-10-11T16:33:00.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:00.862+0000 7fb104dff700 1 -- 172.21.15.111:0/100497284 wait complete. 2023-10-11T16:33:01.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:01 smithi111 ceph-mon[98527]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2023-10-11T16:33:01.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:01 smithi150 ceph-mon[132217]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2023-10-11T16:33:01.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:01 smithi130 ceph-mon[131859]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2023-10-11T16:33:01.891 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:33:01.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:33:02.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 -- 172.21.15.111:0/1858718074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101390 msgr2=0x7f56c8101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/1858718074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101390 0x7f56c8101760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009990 tx=0x7f56b8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 -- 172.21.15.111:0/1858718074 shutdown_connections 2023-10-11T16:33:02.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/1858718074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c810a8b0 0x7f56c810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/1858718074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101390 0x7f56c8101760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/1858718074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56c8101ca0 0x7f56c810a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 -- 172.21.15.111:0/1858718074 >> 172.21.15.111:0/1858718074 conn(0x7f56c80770b0 msgr2=0x7f56c80774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 -- 172.21.15.111:0/1858718074 shutdown_connections 2023-10-11T16:33:02.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.226+0000 7f56cd7ff700 1 -- 172.21.15.111:0/1858718074 wait complete. 2023-10-11T16:33:02.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.227+0000 7f56cd7ff700 1 Processor -- start 2023-10-11T16:33:02.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.227+0000 7f56cd7ff700 1 -- start start 2023-10-11T16:33:02.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.227+0000 7f56cd7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8101390 0x7f56c8198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.227+0000 7f56cd7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101ca0 0x7f56c8198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.227+0000 7f56cd7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56c810a8b0 0x7f56c819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56cd7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c819d640 con 0x7f56c8101390 2023-10-11T16:33:02.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56cd7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c819d7b0 con 0x7f56c810a8b0 2023-10-11T16:33:02.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56cd7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c819d970 con 0x7f56c8101ca0 2023-10-11T16:33:02.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8101390 0x7f56c8198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8101390 0x7f56c8198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33804/0 (socket says 172.21.15.111:33804) 2023-10-11T16:33:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101ca0 0x7f56c8198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c6ffd700 1 -- 172.21.15.111:0/3523740806 learned_addr learned my addr 172.21.15.111:0/3523740806 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c77fe700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56c810a8b0 0x7f56c819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c67fc700 1 -- 172.21.15.111:0/3523740806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56c810a8b0 msgr2=0x7f56c819d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c67fc700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56c810a8b0 0x7f56c819d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c67fc700 1 -- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8101390 msgr2=0x7f56c8198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c67fc700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8101390 0x7f56c8198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c67fc700 1 -- 172.21.15.111:0/3523740806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b80095f0 con 0x7f56c8101ca0 2023-10-11T16:33:02.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.228+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8101390 0x7f56c8198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:02.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.229+0000 7f56c77fe700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56c810a8b0 0x7f56c819d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:02.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.229+0000 7f56c67fc700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101ca0 0x7f56c8198ba0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00e9f0 tx=0x7f56bc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.229+0000 7f56affff700 1 -- 172.21.15.111:0/3523740806 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00c9c0 con 0x7f56c8101ca0 2023-10-11T16:33:02.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.229+0000 7f56cd7ff700 1 -- 172.21.15.111:0/3523740806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c8110380 con 0x7f56c8101ca0 2023-10-11T16:33:02.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.229+0000 7f56cd7ff700 1 -- 172.21.15.111:0/3523740806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c81101f0 con 0x7f56c8101ca0 2023-10-11T16:33:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.230+0000 7f56affff700 1 -- 172.21.15.111:0/3523740806 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56bc010430 con 0x7f56c8101ca0 2023-10-11T16:33:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.230+0000 7f56affff700 1 -- 172.21.15.111:0/3523740806 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc018680 con 0x7f56c8101ca0 2023-10-11T16:33:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.233+0000 7f56affff700 1 -- 172.21.15.111:0/3523740806 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f56bc00cb20 con 0x7f56c8101ca0 2023-10-11T16:33:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.234+0000 7f56affff700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f56b00a1460 0x7f56b00a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.235+0000 7f56affff700 1 -- 172.21.15.111:0/3523740806 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f56bc014070 con 0x7f56c8101ca0 2023-10-11T16:33:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.235+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f56b00a1460 0x7f56b00a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.236+0000 7f56e113cb80 1 Processor -- start 2023-10-11T16:33:02.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.236+0000 7f56e113cb80 1 -- start start 2023-10-11T16:33:02.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.237+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f56b00a1460 0x7f56b00a3910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f56b8006010 tx=0x7f56b800b260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.237+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x556515948fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.237+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x55651594d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.237+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 0x55651594dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.237+0000 7f56e113cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55651584e0e0 con 0x556515948be0 2023-10-11T16:33:02.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.237+0000 7f56e113cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565158a0920 con 0x5565159494f0 2023-10-11T16:33:02.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.237+0000 7f56e113cb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556515899db0 con 0x55651594d930 2023-10-11T16:33:02.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x55651594d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x55651594d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46262/0 (socket says 172.21.15.111:46262) 2023-10-11T16:33:02.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 -- 172.21.15.111:0/504266549 learned_addr learned my addr 172.21.15.111:0/504266549 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c77fe700 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 0x55651594dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x556515948fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 -- 172.21.15.111:0/504266549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 msgr2=0x55651594dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 0x55651594dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 -- 172.21.15.111:0/504266549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 msgr2=0x556515948fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x556515948fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 -- 172.21.15.111:0/504266549 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55651571b6b0 con 0x5565159494f0 2023-10-11T16:33:02.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x556515948fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:02.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.238+0000 7f56c67fc700 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x55651594d3f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00e9c0 tx=0x7f56bc099ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b22418ddf72b8e17 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.239+0000 7f56ad7fa700 1 -- 172.21.15.111:0/504266549 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc09dea0 con 0x5565159494f0 2023-10-11T16:33:02.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.239+0000 7f56ad7fa700 1 -- 172.21.15.111:0/504266549 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56bc0cae90 con 0x5565159494f0 2023-10-11T16:33:02.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.239+0000 7f56ad7fa700 1 -- 172.21.15.111:0/504266549 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0a1e30 con 0x5565159494f0 2023-10-11T16:33:02.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.240+0000 7f56e113cb80 1 -- 172.21.15.111:0/504266549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 msgr2=0x55651594d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.240+0000 7f56e113cb80 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x55651594d3f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00e9c0 tx=0x7f56bc099ab0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.240+0000 7f56e113cb80 1 -- 172.21.15.111:0/504266549 shutdown_connections 2023-10-11T16:33:02.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.241+0000 7f56e113cb80 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 0x55651594dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.241+0000 7f56e113cb80 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x556515948fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.241+0000 7f56e113cb80 1 --2- 172.21.15.111:0/504266549 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x55651594d3f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.241+0000 7f56e113cb80 1 -- 172.21.15.111:0/504266549 >> 172.21.15.111:0/504266549 conn(0x55651593e3e0 msgr2=0x556515943bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.241+0000 7f56e113cb80 1 -- 172.21.15.111:0/504266549 shutdown_connections 2023-10-11T16:33:02.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.241+0000 7f56e113cb80 1 -- 172.21.15.111:0/504266549 wait complete. 2023-10-11T16:33:02.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.242+0000 7f56e113cb80 1 -- start start 2023-10-11T16:33:02.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.242+0000 7f56e113cb80 1 Processor -- start 2023-10-11T16:33:02.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556515948be0 0x5565159550a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565159494f0 0x5565159522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 0x5565159528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56e113cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55651583b960 con 0x5565159494f0 2023-10-11T16:33:02.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56e113cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55651583b600 con 0x556515948be0 2023-10-11T16:33:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56e113cb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55651571d9a0 con 0x55651594d930 2023-10-11T16:33:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56e113cb80 10 client.0 _ll_register_callbacks cb 0x556515859800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56c6ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556515948be0 0x5565159550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56c67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565159494f0 0x5565159522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56c6ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556515948be0 0x5565159550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46272/0 (socket says 172.21.15.111:46272) 2023-10-11T16:33:02.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56c6ffd700 1 -- 172.21.15.111:0/1493984152 learned_addr learned my addr 172.21.15.111:0/1493984152 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.243+0000 7f56c77fe700 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 0x5565159528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56c67fc700 1 -- 172.21.15.111:0/1493984152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 msgr2=0x5565159528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56c67fc700 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 0x5565159528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56c67fc700 1 -- 172.21.15.111:0/1493984152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556515948be0 msgr2=0x5565159550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56c67fc700 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556515948be0 0x5565159550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56c67fc700 1 -- 172.21.15.111:0/1493984152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c81101f0 con 0x5565159494f0 2023-10-11T16:33:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56c67fc700 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565159494f0 0x5565159522b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x5565159556a0 tx=0x7f56bc099fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb15427d3563f74 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56aaff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56aaff5700 1 -- 172.21.15.111:0/1493984152 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0b82c0 con 0x5565159494f0 2023-10-11T16:33:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.244+0000 7f56aaff5700 1 -- 172.21.15.111:0/1493984152 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f56bc097fb0 con 0x5565159494f0 2023-10-11T16:33:02.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.245+0000 7f56e113cb80 1 -- 172.21.15.111:0/1493984152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55651593bba0 con 0x5565159494f0 2023-10-11T16:33:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.245+0000 7f56aaff5700 1 -- 172.21.15.111:0/1493984152 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56bc09dea0 con 0x5565159494f0 2023-10-11T16:33:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.245+0000 7f56e113cb80 10 client.15669 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.245+0000 7f56e113cb80 1 -- 172.21.15.111:0/1493984152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556515860210 con 0x5565159494f0 2023-10-11T16:33:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.245+0000 7f56aaff5700 1 -- 172.21.15.111:0/1493984152 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55651593bba0 con 0x5565159494f0 2023-10-11T16:33:02.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.246+0000 7f56aaff5700 1 -- 172.21.15.111:0/1493984152 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56bc0ca8a0 con 0x5565159494f0 2023-10-11T16:33:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.246+0000 7f56e113cb80 10 client.15669 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.246+0000 7f56e113cb80 10 client.15669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.246+0000 7f56e113cb80 20 client.15669 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.246+0000 7f56e113cb80 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55651596ad60 0x55651596d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.246+0000 7f56e113cb80 4 client.15669 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.246+0000 7f56e113cb80 1 -- 172.21.15.111:0/1493984152 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55651585c120 con 0x55651596ad60 2023-10-11T16:33:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.247+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55651596ad60 0x55651596d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.247+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55651596ad60 0x55651596d170 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75511403e6af95bd server_cookie=d077014d28142d94 in_seq=0 out_seq=0 2023-10-11T16:33:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.247+0000 7f56aaff5700 10 client.15669 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56aaff5700 1 -- 172.21.15.111:0/1493984152 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55651585c120 con 0x55651596ad60 2023-10-11T16:33:02.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56aaff5700 10 client.15669 handle_command_reply: tid=0 2023-10-11T16:33:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 client.15669 shutdown 2023-10-11T16:33:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 -- 172.21.15.111:0/1493984152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565159494f0 msgr2=0x5565159522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565159494f0 0x5565159522b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x5565159556a0 tx=0x7f56bc099fb0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 -- 172.21.15.111:0/1493984152 shutdown_connections 2023-10-11T16:33:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55651596ad60 0x55651596d170 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55651594d930 0x5565159528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556515948be0 0x5565159550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 --2- 172.21.15.111:0/1493984152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5565159494f0 0x5565159522b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56e113cb80 1 -- 172.21.15.111:0/1493984152 >> 172.21.15.111:0/1493984152 conn(0x55651593e3e0 msgr2=0x55651594ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.248+0000 7f56aaff5700 0 client.15669 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:02.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.249+0000 7f56e113cb80 1 -- 172.21.15.111:0/1493984152 shutdown_connections 2023-10-11T16:33:02.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.249+0000 7f56e113cb80 1 -- 172.21.15.111:0/1493984152 wait complete. 2023-10-11T16:33:02.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.249+0000 7f56e113cb80 20 client.15669 trim_cache size 0 max 16384 2023-10-11T16:33:02.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.282+0000 7f56e113cb80 1 Processor -- start 2023-10-11T16:33:02.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.282+0000 7f56e113cb80 1 -- start start 2023-10-11T16:33:02.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.283+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556515948be0 0x556515943d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.283+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x556515954bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.283+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55651594d930 0x556515951a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.283+0000 7f56e113cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55651594c440 con 0x55651594d930 2023-10-11T16:33:02.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.283+0000 7f56e113cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565159556d0 con 0x5565159494f0 2023-10-11T16:33:02.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.283+0000 7f56e113cb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556515966a00 con 0x556515948be0 2023-10-11T16:33:02.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.283+0000 7f56c67fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x556515954bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.283+0000 7f56c6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556515948be0 0x556515943d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556515948be0 0x556515943d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41032/0 (socket says 172.21.15.111:41032) 2023-10-11T16:33:02.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c6ffd700 1 -- 172.21.15.111:0/2035296933 learned_addr learned my addr 172.21.15.111:0/2035296933 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c77fe700 1 --2- 172.21.15.111:0/2035296933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55651594d930 0x556515951a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c6ffd700 1 -- 172.21.15.111:0/2035296933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 msgr2=0x556515954bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/2035296933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x556515954bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c6ffd700 1 -- 172.21.15.111:0/2035296933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55651594d930 msgr2=0x556515951a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/2035296933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55651594d930 0x556515951a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c6ffd700 1 -- 172.21.15.111:0/2035296933 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55651585ff60 con 0x556515948be0 2023-10-11T16:33:02.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/2035296933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556515948be0 0x556515943d20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56b8000c00 tx=0x7f56b8023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a3c32c42362ffee server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.284+0000 7f56adffb700 1 -- 172.21.15.111:0/2035296933 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b800fbc0 con 0x556515948be0 2023-10-11T16:33:02.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.285+0000 7f56adffb700 1 -- 172.21.15.111:0/2035296933 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56b802c040 con 0x556515948be0 2023-10-11T16:33:02.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.285+0000 7f56adffb700 1 -- 172.21.15.111:0/2035296933 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b80249d0 con 0x556515948be0 2023-10-11T16:33:02.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 -- 172.21.15.111:0/2035296933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556515948be0 msgr2=0x556515943d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 --2- 172.21.15.111:0/2035296933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556515948be0 0x556515943d20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56b8000c00 tx=0x7f56b8023000 comp rx=0 tx=0).stop 2023-10-11T16:33:02.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 -- 172.21.15.111:0/2035296933 shutdown_connections 2023-10-11T16:33:02.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 --2- 172.21.15.111:0/2035296933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55651594d930 0x556515951a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 --2- 172.21.15.111:0/2035296933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556515948be0 0x556515943d20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 --2- 172.21.15.111:0/2035296933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5565159494f0 0x556515954bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 -- 172.21.15.111:0/2035296933 >> 172.21.15.111:0/2035296933 conn(0x55651593e3e0 msgr2=0x556515948270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 -- 172.21.15.111:0/2035296933 shutdown_connections 2023-10-11T16:33:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.286+0000 7f56e113cb80 1 -- 172.21.15.111:0/2035296933 wait complete. 2023-10-11T16:33:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.287+0000 7f56e113cb80 1 -- start start 2023-10-11T16:33:02.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.287+0000 7f56e113cb80 1 Processor -- start 2023-10-11T16:33:02.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x5565159b9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5565159494f0 0x5565159b9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56e113cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55651594d930 0x556515961790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56e113cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x556515961da0 con 0x556515948be0 2023-10-11T16:33:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56e113cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556515961f10 con 0x55651594d930 2023-10-11T16:33:02.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56e113cb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565159620d0 con 0x5565159494f0 2023-10-11T16:33:02.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56e113cb80 10 client.0 _ll_register_callbacks cb 0x7f56c000d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:02.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56c67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5565159494f0 0x5565159b9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56c67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5565159494f0 0x5565159b9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41048/0 (socket says 172.21.15.111:41048) 2023-10-11T16:33:02.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56c67fc700 1 -- 172.21.15.111:0/523027052 learned_addr learned my addr 172.21.15.111:0/523027052 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.289+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x5565159b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.290+0000 7f56c6ffd700 1 -- 172.21.15.111:0/523027052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5565159494f0 msgr2=0x5565159b9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.290+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5565159494f0 0x5565159b9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.290+0000 7f56c6ffd700 1 -- 172.21.15.111:0/523027052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55651594d930 msgr2=0x556515961790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.290+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55651594d930 0x556515961790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.290+0000 7f56c6ffd700 1 -- 172.21.15.111:0/523027052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c810d990 con 0x556515948be0 2023-10-11T16:33:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.291+0000 7f56c6ffd700 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x5565159b9590 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f56b800b6f0 tx=0x7f56b8009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b2d84d226554d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.291+0000 7f56aaff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.291+0000 7f56aaff5700 1 -- 172.21.15.111:0/523027052 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b80352f0 con 0x556515948be0 2023-10-11T16:33:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.291+0000 7f56aaff5700 1 -- 172.21.15.111:0/523027052 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f56b8035450 con 0x556515948be0 2023-10-11T16:33:02.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.291+0000 7f56e113cb80 1 -- 172.21.15.111:0/523027052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56bc0cae90 con 0x556515948be0 2023-10-11T16:33:02.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56aaff5700 1 -- 172.21.15.111:0/523027052 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56b80352f0 con 0x556515948be0 2023-10-11T16:33:02.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56e113cb80 10 client.15675 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:02.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56e113cb80 1 -- 172.21.15.111:0/523027052 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5565156b0ce0 con 0x556515948be0 2023-10-11T16:33:02.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56aaff5700 1 -- 172.21.15.111:0/523027052 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56bc0cae90 con 0x556515948be0 2023-10-11T16:33:02.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56aaff5700 1 -- 172.21.15.111:0/523027052 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56b8033420 con 0x556515948be0 2023-10-11T16:33:02.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56e113cb80 10 client.15675 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:02.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56e113cb80 10 client.15675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:02.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56e113cb80 20 client.15675 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:02.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56e113cb80 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5565159c1050 0x5565159c3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56e113cb80 4 client.15675 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:33:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.292+0000 7f56e113cb80 1 -- 172.21.15.111:0/523027052 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5565158686e0 con 0x5565159c1050 2023-10-11T16:33:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.293+0000 7f56c67fc700 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5565159c1050 0x5565159c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.293+0000 7f56c67fc700 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5565159c1050 0x5565159c3460 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef2c9fcff8b16998 server_cookie=38e59b2470c343b6 in_seq=0 out_seq=0 2023-10-11T16:33:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.293+0000 7f56aaff5700 10 client.15675 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:02.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56aaff5700 1 -- 172.21.15.111:0/523027052 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5565158686e0 con 0x5565159c1050 2023-10-11T16:33:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56aaff5700 10 client.15675 handle_command_reply: tid=0 2023-10-11T16:33:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 client.15675 shutdown 2023-10-11T16:33:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 -- 172.21.15.111:0/523027052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 msgr2=0x5565159b9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x5565159b9590 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f56b800b6f0 tx=0x7f56b8009290 comp rx=0 tx=0).stop 2023-10-11T16:33:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 -- 172.21.15.111:0/523027052 shutdown_connections 2023-10-11T16:33:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5565159c1050 0x5565159c3460 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55651594d930 0x556515961790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556515948be0 0x5565159b9590 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 --2- 172.21.15.111:0/523027052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5565159494f0 0x5565159b9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56e113cb80 1 -- 172.21.15.111:0/523027052 >> 172.21.15.111:0/523027052 conn(0x55651593e3e0 msgr2=0x55651594c850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.295+0000 7f56aaff5700 0 client.15675 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.296+0000 7f56e113cb80 1 -- 172.21.15.111:0/523027052 shutdown_connections 2023-10-11T16:33:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.296+0000 7f56e113cb80 1 -- 172.21.15.111:0/523027052 wait complete. 2023-10-11T16:33:02.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.296+0000 7f56e113cb80 20 client.15675 trim_cache size 0 max 16384 2023-10-11T16:33:02.305 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:33:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:33:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "b79602ef-3327-41f1-bed2-8ffc1590ee87", 2023-10-11T16:33:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:33:02.305 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:33:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 -- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f56b00a1460 msgr2=0x7f56b00a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f56b00a1460 0x7f56b00a3910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f56b8006010 tx=0x7f56b800b260 comp rx=0 tx=0).stop 2023-10-11T16:33:02.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 -- 172.21.15.111:0/3523740806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101ca0 msgr2=0x7f56c8198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101ca0 0x7f56c8198ba0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f56bc00e9f0 tx=0x7f56bc00ed00 comp rx=0 tx=0).stop 2023-10-11T16:33:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 -- 172.21.15.111:0/3523740806 shutdown_connections 2023-10-11T16:33:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f56b00a1460 0x7f56b00a3910 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f56c810a8b0 0x7f56c819d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56c8101390 0x7f56c8198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 --2- 172.21.15.111:0/3523740806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c8101ca0 0x7f56c8198ba0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 -- 172.21.15.111:0/3523740806 >> 172.21.15.111:0/3523740806 conn(0x7f56c80770b0 msgr2=0x7f56c80fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 -- 172.21.15.111:0/3523740806 shutdown_connections 2023-10-11T16:33:02.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.297+0000 7f56cd7ff700 1 -- 172.21.15.111:0/3523740806 wait complete. 2023-10-11T16:33:02.315 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:02 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:33:02.318 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:33:02.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:02 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:33:02.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.550+0000 7f7b7cdff700 1 -- 172.21.15.111:0/3561415006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101ca0 msgr2=0x7f7b7810a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.550+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/3561415006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101ca0 0x7f7b7810a370 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009990 tx=0x7f7b6c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.550+0000 7f7b7cdff700 1 -- 172.21.15.111:0/3561415006 shutdown_connections 2023-10-11T16:33:02.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.550+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/3561415006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7810a8b0 0x7f7b7810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.550+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/3561415006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b78101390 0x7f7b78101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.550+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/3561415006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101ca0 0x7f7b7810a370 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.550+0000 7f7b7cdff700 1 -- 172.21.15.111:0/3561415006 >> 172.21.15.111:0/3561415006 conn(0x7f7b780770b0 msgr2=0x7f7b780774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.551+0000 7f7b7cdff700 1 -- 172.21.15.111:0/3561415006 shutdown_connections 2023-10-11T16:33:02.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.551+0000 7f7b7cdff700 1 -- 172.21.15.111:0/3561415006 wait complete. 2023-10-11T16:33:02.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.552+0000 7f7b7cdff700 1 Processor -- start 2023-10-11T16:33:02.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.553+0000 7f7b7cdff700 1 -- start start 2023-10-11T16:33:02.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.553+0000 7f7b7cdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101390 0x7f7b78198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.553+0000 7f7b7cdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b78101ca0 0x7f7b78198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.553+0000 7f7b7cdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7810a8b0 0x7f7b7819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.553+0000 7f7b7cdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7819d610 con 0x7f7b78101ca0 2023-10-11T16:33:02.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.553+0000 7f7b7cdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7819d780 con 0x7f7b78101390 2023-10-11T16:33:02.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.553+0000 7f7b7cdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7819d940 con 0x7f7b7810a8b0 2023-10-11T16:33:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.553+0000 7f7b75d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b78101ca0 0x7f7b78198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b75d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b78101ca0 0x7f7b78198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33862/0 (socket says 172.21.15.111:33862) 2023-10-11T16:33:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b75d9b700 1 -- 172.21.15.111:0/2268676685 learned_addr learned my addr 172.21.15.111:0/2268676685 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b7659c700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101390 0x7f7b78198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b7659c700 1 -- 172.21.15.111:0/2268676685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7810a8b0 msgr2=0x7f7b7819cfd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-11T16:33:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b7659c700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7810a8b0 0x7f7b7819cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b7659c700 1 -- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b78101ca0 msgr2=0x7f7b78198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b7659c700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b78101ca0 0x7f7b78198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b7659c700 1 -- 172.21.15.111:0/2268676685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c0095f0 con 0x7f7b78101390 2023-10-11T16:33:02.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b78101ca0 0x7f7b78198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.554+0000 7f7b7659c700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101390 0x7f7b78198630 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800e9f0 tx=0x7f7b6800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.555+0000 7f7b677fe700 1 -- 172.21.15.111:0/2268676685 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6800eee0 con 0x7f7b78101390 2023-10-11T16:33:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.555+0000 7f7b677fe700 1 -- 172.21.15.111:0/2268676685 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b6800cbd0 con 0x7f7b78101390 2023-10-11T16:33:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.555+0000 7f7b677fe700 1 -- 172.21.15.111:0/2268676685 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68018680 con 0x7f7b78101390 2023-10-11T16:33:02.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.555+0000 7f7b7cdff700 1 -- 172.21.15.111:0/2268676685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b78110390 con 0x7f7b78101390 2023-10-11T16:33:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.555+0000 7f7b7cdff700 1 -- 172.21.15.111:0/2268676685 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b78110200 con 0x7f7b78101390 2023-10-11T16:33:02.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.558+0000 7f7b677fe700 1 -- 172.21.15.111:0/2268676685 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f7b6800cd40 con 0x7f7b78101390 2023-10-11T16:33:02.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.560+0000 7f7b677fe700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b600a13e0 0x7f7b600a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.560+0000 7f7b677fe700 1 -- 172.21.15.111:0/2268676685 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f7b68014070 con 0x7f7b78101390 2023-10-11T16:33:02.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.561+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b600a13e0 0x7f7b600a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.561+0000 7f7b90762b80 1 Processor -- start 2023-10-11T16:33:02.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.561+0000 7f7b90762b80 1 -- start start 2023-10-11T16:33:02.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x561515403740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403c80 0x561515407b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615154080c0 0x561515408510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b90762b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561515300060 con 0x561515403c80 2023-10-11T16:33:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b90762b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615153528a0 con 0x5615154080c0 2023-10-11T16:33:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b90762b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56151534bd30 con 0x561515403370 2023-10-11T16:33:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b7659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x561515403740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b600a13e0 0x7f7b600a3890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7b78199c40 tx=0x7f7b6c00b260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b7659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x561515403740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41076/0 (socket says 172.21.15.111:41076) 2023-10-11T16:33:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b7659c700 1 -- 172.21.15.111:0/323022790 learned_addr learned my addr 172.21.15.111:0/323022790 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403c80 0x561515407b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.562+0000 7f7b76d9d700 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615154080c0 0x561515408510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.563+0000 7f7b7659c700 1 -- 172.21.15.111:0/323022790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615154080c0 msgr2=0x561515408510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.563+0000 7f7b7659c700 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615154080c0 0x561515408510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.563+0000 7f7b7659c700 1 -- 172.21.15.111:0/323022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403c80 msgr2=0x561515407b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.563+0000 7f7b7659c700 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403c80 0x561515407b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.563+0000 7f7b7659c700 1 -- 172.21.15.111:0/323022790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615151cd630 con 0x561515403370 2023-10-11T16:33:02.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.563+0000 7f7b7659c700 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x561515403740 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b68099d50 tx=0x7f7b68099a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef3632d5e1b44a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.564+0000 7f7b64ff9700 1 -- 172.21.15.111:0/323022790 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6809e080 con 0x561515403370 2023-10-11T16:33:02.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.564+0000 7f7b64ff9700 1 -- 172.21.15.111:0/323022790 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b680a24a0 con 0x561515403370 2023-10-11T16:33:02.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.564+0000 7f7b64ff9700 1 -- 172.21.15.111:0/323022790 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b680a2010 con 0x561515403370 2023-10-11T16:33:02.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.564+0000 7f7b90762b80 1 -- 172.21.15.111:0/323022790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 msgr2=0x561515403740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.564+0000 7f7b90762b80 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x561515403740 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b68099d50 tx=0x7f7b68099a30 comp rx=0 tx=0).stop 2023-10-11T16:33:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.565+0000 7f7b90762b80 1 -- 172.21.15.111:0/323022790 shutdown_connections 2023-10-11T16:33:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.565+0000 7f7b90762b80 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615154080c0 0x561515408510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.565+0000 7f7b90762b80 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x561515403740 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.565+0000 7f7b90762b80 1 --2- 172.21.15.111:0/323022790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403c80 0x561515407b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.565+0000 7f7b90762b80 1 -- 172.21.15.111:0/323022790 >> 172.21.15.111:0/323022790 conn(0x5615153f0360 msgr2=0x5615153f5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.565+0000 7f7b90762b80 1 -- 172.21.15.111:0/323022790 shutdown_connections 2023-10-11T16:33:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.565+0000 7f7b90762b80 1 -- 172.21.15.111:0/323022790 wait complete. 2023-10-11T16:33:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.566+0000 7f7b90762b80 1 -- start start 2023-10-11T16:33:02.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.566+0000 7f7b90762b80 1 Processor -- start 2023-10-11T16:33:02.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x56151540f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x56151540cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x56151540d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b90762b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615152ed8e0 con 0x5615154080c0 2023-10-11T16:33:02.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b90762b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615152ed580 con 0x561515403c80 2023-10-11T16:33:02.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b90762b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615151cf920 con 0x561515403370 2023-10-11T16:33:02.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b90762b80 10 client.0 _ll_register_callbacks cb 0x56151530b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:02.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b7659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x56151540f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b7659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x56151540f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41084/0 (socket says 172.21.15.111:41084) 2023-10-11T16:33:02.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x56151540cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b76d9d700 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x56151540d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.567+0000 7f7b7659c700 1 -- 172.21.15.111:0/1199763113 learned_addr learned my addr 172.21.15.111:0/1199763113 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b75d9b700 1 -- 172.21.15.111:0/1199763113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 msgr2=0x56151540f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x56151540f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b75d9b700 1 -- 172.21.15.111:0/1199763113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 msgr2=0x56151540d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x56151540d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b75d9b700 1 -- 172.21.15.111:0/1199763113 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b78110200 con 0x561515403c80 2023-10-11T16:33:02.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b76d9d700 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x56151540d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:02.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x56151540cad0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00b6c0 tx=0x7f7b6c01a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a0a1daa541c4627 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b5dffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.568+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1199763113 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c00f870 con 0x561515403c80 2023-10-11T16:33:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.569+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1199763113 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f7b6c00f9d0 con 0x561515403c80 2023-10-11T16:33:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.569+0000 7f7b90762b80 1 -- 172.21.15.111:0/1199763113 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615153edb20 con 0x561515403c80 2023-10-11T16:33:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.569+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1199763113 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b6c00fb90 con 0x561515403c80 2023-10-11T16:33:02.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.569+0000 7f7b90762b80 10 client.25727 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.569+0000 7f7b90762b80 1 -- 172.21.15.111:0/1199763113 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561515312190 con 0x561515403c80 2023-10-11T16:33:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.569+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1199763113 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5615153edb20 con 0x561515403c80 2023-10-11T16:33:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.570+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1199763113 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7b6c02f410 con 0x561515403c80 2023-10-11T16:33:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.571+0000 7f7b90762b80 10 client.25727 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:02.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.571+0000 7f7b90762b80 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.571+0000 7f7b90762b80 20 client.25727 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.571+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5615153fb820 0x5615153fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.571+0000 7f7b90762b80 4 client.25727 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.571+0000 7f7b90762b80 1 -- 172.21.15.111:0/1199763113 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56151530e0a0 con 0x5615153fb820 2023-10-11T16:33:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.571+0000 7f7b7659c700 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5615153fb820 0x5615153fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.572+0000 7f7b7659c700 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5615153fb820 0x5615153fdc30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ef4b2960a1fb8f2 server_cookie=a1aa0e39c0723c23 in_seq=0 out_seq=0 2023-10-11T16:33:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.572+0000 7f7b5dffb700 10 client.25727 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.572+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1199763113 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56151530e0a0 con 0x5615153fb820 2023-10-11T16:33:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.572+0000 7f7b5dffb700 10 client.25727 handle_command_reply: tid=0 2023-10-11T16:33:02.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.572+0000 7f7b90762b80 1 client.25727 shutdown 2023-10-11T16:33:02.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 -- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 msgr2=0x56151540cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x56151540cad0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00b6c0 tx=0x7f7b6c01a050 comp rx=0 tx=0).stop 2023-10-11T16:33:02.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 -- 172.21.15.111:0/1199763113 shutdown_connections 2023-10-11T16:33:02.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5615153fb820 0x5615153fdc30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x56151540d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403370 0x56151540f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1199763113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x56151540cad0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 -- 172.21.15.111:0/1199763113 >> 172.21.15.111:0/1199763113 conn(0x5615153f0360 msgr2=0x561515409570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b5dffb700 0 client.25727 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:02.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 -- 172.21.15.111:0/1199763113 shutdown_connections 2023-10-11T16:33:02.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 1 -- 172.21.15.111:0/1199763113 wait complete. 2023-10-11T16:33:02.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.573+0000 7f7b90762b80 20 client.25727 trim_cache size 0 max 16384 2023-10-11T16:33:02.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:02 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:33:02.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.606+0000 7f7b90762b80 1 Processor -- start 2023-10-11T16:33:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.606+0000 7f7b90762b80 1 -- start start 2023-10-11T16:33:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403370 0x56151540c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403c80 0x5615153f5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x5615153f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b90762b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56151540ffa0 con 0x5615154080c0 2023-10-11T16:33:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b90762b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56151530e0d0 con 0x561515403370 2023-10-11T16:33:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b90762b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615153f7440 con 0x561515403c80 2023-10-11T16:33:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b7659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403370 0x56151540c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b7659c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403370 0x56151540c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46322/0 (socket says 172.21.15.111:46322) 2023-10-11T16:33:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b7659c700 1 -- 172.21.15.111:0/2243842133 learned_addr learned my addr 172.21.15.111:0/2243842133 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403c80 0x5615153f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.607+0000 7f7b76d9d700 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x5615153f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b7659c700 1 -- 172.21.15.111:0/2243842133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403c80 msgr2=0x5615153f5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b7659c700 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403c80 0x5615153f5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b7659c700 1 -- 172.21.15.111:0/2243842133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 msgr2=0x5615153f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b7659c700 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x5615153f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b7659c700 1 -- 172.21.15.111:0/2243842133 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615154089b0 con 0x561515403370 2023-10-11T16:33:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b7659c700 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403370 0x56151540c290 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7b68096ac0 tx=0x7f7b6801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8a23ab862f53401 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b657fa700 1 -- 172.21.15.111:0/2243842133 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b680c9070 con 0x561515403370 2023-10-11T16:33:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b657fa700 1 -- 172.21.15.111:0/2243842133 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b680ca6e0 con 0x561515403370 2023-10-11T16:33:02.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.608+0000 7f7b657fa700 1 -- 172.21.15.111:0/2243842133 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68099540 con 0x561515403370 2023-10-11T16:33:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 -- 172.21.15.111:0/2243842133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403370 msgr2=0x56151540c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403370 0x56151540c290 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7b68096ac0 tx=0x7f7b6801f000 comp rx=0 tx=0).stop 2023-10-11T16:33:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 -- 172.21.15.111:0/2243842133 shutdown_connections 2023-10-11T16:33:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5615154080c0 0x5615153f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403370 0x56151540c290 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 --2- 172.21.15.111:0/2243842133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561515403c80 0x5615153f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 -- 172.21.15.111:0/2243842133 >> 172.21.15.111:0/2243842133 conn(0x5615153f0360 msgr2=0x561515406180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 -- 172.21.15.111:0/2243842133 shutdown_connections 2023-10-11T16:33:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.609+0000 7f7b90762b80 1 -- 172.21.15.111:0/2243842133 wait complete. 2023-10-11T16:33:02.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.610+0000 7f7b90762b80 1 -- start start 2023-10-11T16:33:02.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.610+0000 7f7b90762b80 1 Processor -- start 2023-10-11T16:33:02.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403370 0x561515400e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x561515401340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b90762b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615154080c0 0x561515402160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b90762b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561515402760 con 0x561515403370 2023-10-11T16:33:02.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b90762b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615154028d0 con 0x561515403c80 2023-10-11T16:33:02.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b90762b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561515402a90 con 0x5615154080c0 2023-10-11T16:33:02.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b90762b80 10 client.0 _ll_register_callbacks cb 0x7f7b6c005730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:02.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b7659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403370 0x561515400e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b7659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403370 0x561515400e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33902/0 (socket says 172.21.15.111:33902) 2023-10-11T16:33:02.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.611+0000 7f7b7659c700 1 -- 172.21.15.111:0/1226252556 learned_addr learned my addr 172.21.15.111:0/1226252556 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:02.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x561515401340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b7659c700 1 -- 172.21.15.111:0/1226252556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615154080c0 msgr2=0x561515402160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b7659c700 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615154080c0 0x561515402160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b7659c700 1 -- 172.21.15.111:0/1226252556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 msgr2=0x561515401340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:02.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b7659c700 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x561515401340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b7659c700 1 -- 172.21.15.111:0/1226252556 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5615151cd630 con 0x561515403370 2023-10-11T16:33:02.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b7659c700 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403370 0x561515400e00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7b6809a6b0 tx=0x7f7b680b82c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc6db9b7ddfd37c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:02.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b5dffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:02.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.612+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1226252556 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b680c9070 con 0x561515403370 2023-10-11T16:33:02.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.613+0000 7f7b90762b80 1 -- 172.21.15.111:0/1226252556 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b680a24a0 con 0x561515403370 2023-10-11T16:33:02.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.613+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1226252556 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f7b680cd180 con 0x561515403370 2023-10-11T16:33:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.613+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1226252556 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b680cd380 con 0x561515403370 2023-10-11T16:33:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.613+0000 7f7b90762b80 10 client.15693 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.613+0000 7f7b90762b80 1 -- 172.21.15.111:0/1226252556 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561515179de0 con 0x561515403370 2023-10-11T16:33:02.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.613+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1226252556 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b680a24a0 con 0x561515403370 2023-10-11T16:33:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.613+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1226252556 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7b68095e40 con 0x561515403370 2023-10-11T16:33:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.614+0000 7f7b90762b80 10 client.15693 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.614+0000 7f7b90762b80 10 client.15693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:02.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.614+0000 7f7b90762b80 20 client.15693 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.614+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56151547a160 0x56151547c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.614+0000 7f7b90762b80 4 client.15693 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:33:02.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.614+0000 7f7b90762b80 1 -- 172.21.15.111:0/1226252556 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56151531a660 con 0x56151547a160 2023-10-11T16:33:02.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.614+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56151547a160 0x56151547c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:02.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b75d9b700 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56151547a160 0x56151547c540 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24f24d212b82d7bc server_cookie=42d56a3d070f37c9 in_seq=0 out_seq=0 2023-10-11T16:33:02.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b5dffb700 10 client.15693 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:02.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b5dffb700 1 -- 172.21.15.111:0/1226252556 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56151531a660 con 0x56151547a160 2023-10-11T16:33:02.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b5dffb700 10 client.15693 handle_command_reply: tid=0 2023-10-11T16:33:02.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 client.15693 shutdown 2023-10-11T16:33:02.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 -- 172.21.15.111:0/1226252556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403370 msgr2=0x561515400e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403370 0x561515400e00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7b6809a6b0 tx=0x7f7b680b82c0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 -- 172.21.15.111:0/1226252556 shutdown_connections 2023-10-11T16:33:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56151547a160 0x56151547c540 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5615154080c0 0x561515402160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561515403370 0x561515400e00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b5dffb700 0 client.15693 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 --2- 172.21.15.111:0/1226252556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561515403c80 0x561515401340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.615+0000 7f7b90762b80 1 -- 172.21.15.111:0/1226252556 >> 172.21.15.111:0/1226252556 conn(0x5615153f0360 msgr2=0x5615153f5600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.616+0000 7f7b90762b80 1 -- 172.21.15.111:0/1226252556 shutdown_connections 2023-10-11T16:33:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.616+0000 7f7b90762b80 1 -- 172.21.15.111:0/1226252556 wait complete. 2023-10-11T16:33:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.616+0000 7f7b90762b80 20 client.15693 trim_cache size 0 max 16384 2023-10-11T16:33:02.633 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:33:02.633 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:33:02.633 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:33:02.633 INFO:teuthology.orchestra.run.smithi111.stdout: "b79602ef-3327-41f1-bed2-8ffc1590ee87": { 2023-10-11T16:33:02.634 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:33:02.634 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "b79602ef-3327-41f1-bed2-8ffc1590ee87", 2023-10-11T16:33:02.634 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:33:02.634 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:33:02.635 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:33:02.635 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:33:02.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 -- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b600a13e0 msgr2=0x7f7b600a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b600a13e0 0x7f7b600a3890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7b78199c40 tx=0x7f7b6c00b260 comp rx=0 tx=0).stop 2023-10-11T16:33:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 -- 172.21.15.111:0/2268676685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101390 msgr2=0x7f7b78198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101390 0x7f7b78198630 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b6800e9f0 tx=0x7f7b6800ed00 comp rx=0 tx=0).stop 2023-10-11T16:33:02.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 -- 172.21.15.111:0/2268676685 shutdown_connections 2023-10-11T16:33:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b600a13e0 0x7f7b600a3890 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b7810a8b0 0x7f7b7819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b78101390 0x7f7b78198630 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 --2- 172.21.15.111:0/2268676685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b78101ca0 0x7f7b78198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 -- 172.21.15.111:0/2268676685 >> 172.21.15.111:0/2268676685 conn(0x7f7b780770b0 msgr2=0x7f7b780ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:02.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 -- 172.21.15.111:0/2268676685 shutdown_connections 2023-10-11T16:33:02.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:02.617+0000 7f7b7cdff700 1 -- 172.21.15.111:0/2268676685 wait complete. 2023-10-11T16:33:02.645 INFO:tasks.cephfs.filesystem:scrub status for tag:b79602ef-3327-41f1-bed2-8ffc1590ee87 - {'path': '/', 'tag': 'b79602ef-3327-41f1-bed2-8ffc1590ee87', 'options': 'recursive,force'} 2023-10-11T16:33:03.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:03 smithi150 ceph-mon[132217]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-10-11T16:33:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:03 smithi150 ceph-mon[132217]: mgrmap e24: x(active, since 24s), standbys: y, z 2023-10-11T16:33:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:03 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:33:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:03 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:03.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:03 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:33:03.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:03 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:03.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:03 smithi130 ceph-mon[131859]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-10-11T16:33:03.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:03 smithi130 ceph-mon[131859]: mgrmap e24: x(active, since 24s), standbys: y, z 2023-10-11T16:33:03.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:03 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:33:03.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:03 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:03.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:03 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:33:03.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:03 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:03.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:03 smithi111 ceph-mon[98527]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-10-11T16:33:03.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:03 smithi111 ceph-mon[98527]: mgrmap e24: x(active, since 24s), standbys: y, z 2023-10-11T16:33:03.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:03 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:33:03.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:03 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:03.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:03 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:33:03.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:03 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:04.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:04 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:33:04.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:04 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:04.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:04 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:33:04.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:04 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:04.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:04 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:33:04.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:04 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:05.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:05 smithi150 ceph-mon[132217]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-10-11T16:33:05.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:05 smithi130 ceph-mon[131859]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-10-11T16:33:05.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:05 smithi111 ceph-mon[98527]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-10-11T16:33:07.026 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.025+0000 7f84affff700 1 -- 172.21.15.111:0/1340887394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b0100c80 msgr2=0x7f84b0104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.025+0000 7f84affff700 1 --2- 172.21.15.111:0/1340887394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b0100c80 0x7f84b0104c40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009990 tx=0x7f84a4009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.025+0000 7f84affff700 1 -- 172.21.15.111:0/1340887394 shutdown_connections 2023-10-11T16:33:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.025+0000 7f84affff700 1 --2- 172.21.15.111:0/1340887394 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b0105320 0x7f84b010cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.027 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.025+0000 7f84affff700 1 --2- 172.21.15.111:0/1340887394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84b01002e0 0x7f84b01006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.025+0000 7f84affff700 1 --2- 172.21.15.111:0/1340887394 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b0100c80 0x7f84b0104c40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.025+0000 7f84affff700 1 -- 172.21.15.111:0/1340887394 >> 172.21.15.111:0/1340887394 conn(0x7f84b00fbb70 msgr2=0x7f84b00fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.026+0000 7f84affff700 1 -- 172.21.15.111:0/1340887394 shutdown_connections 2023-10-11T16:33:07.028 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.026+0000 7f84affff700 1 -- 172.21.15.111:0/1340887394 wait complete. 2023-10-11T16:33:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.027+0000 7f84affff700 1 Processor -- start 2023-10-11T16:33:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.027+0000 7f84affff700 1 -- start start 2023-10-11T16:33:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84affff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84b01002e0 0x7f84b0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84affff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b0100c80 0x7f84b0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84affff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b0105320 0x7f84b0073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84affff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b019e2c0 con 0x7f84b0105320 2023-10-11T16:33:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84affff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b01a8c60 con 0x7f84b0100c80 2023-10-11T16:33:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84affff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b01a8e20 con 0x7f84b01002e0 2023-10-11T16:33:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84aeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84b01002e0 0x7f84b0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84aeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84b01002e0 0x7f84b0072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:47410/0 (socket says 172.21.15.111:47410) 2023-10-11T16:33:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84aeffd700 1 -- 172.21.15.111:0/1701137662 learned_addr learned my addr 172.21.15.111:0/1701137662 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b0100c80 0x7f84b0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84af7fe700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b0105320 0x7f84b0073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84aeffd700 1 -- 172.21.15.111:0/1701137662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b0100c80 msgr2=0x7f84b0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84aeffd700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b0100c80 0x7f84b0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84aeffd700 1 -- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b0105320 msgr2=0x7f84b0073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.028+0000 7f84aeffd700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b0105320 0x7f84b0073870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.029+0000 7f84aeffd700 1 -- 172.21.15.111:0/1701137662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a8009c40 con 0x7f84b01002e0 2023-10-11T16:33:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.029+0000 7f84aeffd700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84b01002e0 0x7f84b0072df0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f84a000e9a0 tx=0x7f84a000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.029+0000 7f84b49fd700 1 -- 172.21.15.111:0/1701137662 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0009790 con 0x7f84b01002e0 2023-10-11T16:33:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.029+0000 7f84affff700 1 -- 172.21.15.111:0/1701137662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a40095f0 con 0x7f84b01002e0 2023-10-11T16:33:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.029+0000 7f84b49fd700 1 -- 172.21.15.111:0/1701137662 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84a000cbd0 con 0x7f84b01002e0 2023-10-11T16:33:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.029+0000 7f84b49fd700 1 -- 172.21.15.111:0/1701137662 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0018680 con 0x7f84b01002e0 2023-10-11T16:33:07.036 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.029+0000 7f84affff700 1 -- 172.21.15.111:0/1701137662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b01100f0 con 0x7f84b01002e0 2023-10-11T16:33:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.034+0000 7f84c93c2b80 1 Processor -- start 2023-10-11T16:33:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.034+0000 7f84b49fd700 1 -- 172.21.15.111:0/1701137662 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f84a000cd40 con 0x7f84b01002e0 2023-10-11T16:33:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.034+0000 7f84c93c2b80 1 -- start start 2023-10-11T16:33:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b898306b0 0x561b899da450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 0x561b899d47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899d5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b898da000 con 0x561b899d4cf0 2023-10-11T16:33:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b8992c840 con 0x561b899da990 2023-10-11T16:33:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b89925cd0 con 0x561b898306b0 2023-10-11T16:33:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84af7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b898306b0 0x561b899da450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84af7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b898306b0 0x561b899da450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:47424/0 (socket says 172.21.15.111:47424) 2023-10-11T16:33:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.035+0000 7f84af7fe700 1 -- 172.21.15.111:0/1352098280 learned_addr learned my addr 172.21.15.111:0/1352098280 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899d5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84b49fd700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84980a13e0 0x7f84980a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84aeffd700 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 0x561b899d47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84af7fe700 1 -- 172.21.15.111:0/1352098280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 msgr2=0x561b899d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84af7fe700 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 0x561b899d47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84af7fe700 1 -- 172.21.15.111:0/1352098280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 msgr2=0x561b899d5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84af7fe700 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899d5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84af7fe700 1 -- 172.21.15.111:0/1352098280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b897a75d0 con 0x561b898306b0 2023-10-11T16:33:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84b49fd700 1 -- 172.21.15.111:0/1701137662 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f84a0014070 con 0x7f84b01002e0 2023-10-11T16:33:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.036+0000 7f84af7fe700 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b898306b0 0x561b899da450 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009480 tx=0x7f84a8009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f94f72b5bb4a23 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.037+0000 7f84af7fe700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84980a13e0 0x7f84980a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.037+0000 7f8495ffb700 1 -- 172.21.15.111:0/1352098280 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8015690 con 0x561b898306b0 2023-10-11T16:33:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.037+0000 7f8495ffb700 1 -- 172.21.15.111:0/1352098280 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84a8011040 con 0x561b898306b0 2023-10-11T16:33:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.037+0000 7f8495ffb700 1 -- 172.21.15.111:0/1352098280 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8014590 con 0x561b898306b0 2023-10-11T16:33:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 -- 172.21.15.111:0/1352098280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b898306b0 msgr2=0x561b899da450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84af7fe700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84980a13e0 0x7f84980a3890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009550 tx=0x7f84a8014a90 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b898306b0 0x561b899da450 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009480 tx=0x7f84a8009920 comp rx=0 tx=0).stop 2023-10-11T16:33:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 -- 172.21.15.111:0/1352098280 shutdown_connections 2023-10-11T16:33:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b898306b0 0x561b899da450 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/1352098280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 0x561b899d47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 -- 172.21.15.111:0/1352098280 >> 172.21.15.111:0/1352098280 conn(0x561b899ca1a0 msgr2=0x561b899cf3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 -- 172.21.15.111:0/1352098280 shutdown_connections 2023-10-11T16:33:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 -- 172.21.15.111:0/1352098280 wait complete. 2023-10-11T16:33:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.038+0000 7f84c93c2b80 1 -- start start 2023-10-11T16:33:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.039+0000 7f84c93c2b80 1 Processor -- start 2023-10-11T16:33:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.040+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b898306b0 0x561b899e0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.040+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899ddea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.040+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899da990 0x561b899de5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.040+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b898c7880 con 0x561b899d4cf0 2023-10-11T16:33:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.040+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b898c7520 con 0x561b898306b0 2023-10-11T16:33:07.049 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.040+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b897a98c0 con 0x561b899da990 2023-10-11T16:33:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.040+0000 7f84c93c2b80 10 client.0 _ll_register_callbacks cb 0x561b898e5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.041+0000 7f84aeffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b898306b0 0x561b899e0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.041+0000 7f84aeffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b898306b0 0x561b899e0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:53146/0 (socket says 172.21.15.111:53146) 2023-10-11T16:33:07.050 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.041+0000 7f84aeffd700 1 -- 172.21.15.111:0/4171910113 learned_addr learned my addr 172.21.15.111:0/4171910113 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.041+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899ddea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.041+0000 7f84af7fe700 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899da990 0x561b899de5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f84aeffd700 1 -- 172.21.15.111:0/4171910113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899da990 msgr2=0x561b899de5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f84aeffd700 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899da990 0x561b899de5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f84aeffd700 1 -- 172.21.15.111:0/4171910113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 msgr2=0x561b899ddea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f84aeffd700 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899ddea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f84aeffd700 1 -- 172.21.15.111:0/4171910113 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a8009130 con 0x561b898306b0 2023-10-11T16:33:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f84af7fe700 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899da990 0x561b899de5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:33:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f84aeffd700 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b898306b0 0x561b899e0c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f84a00096b0 tx=0x7f84a009a670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39232ead850fc99 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f848effd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.042+0000 7f848effd700 1 -- 172.21.15.111:0/4171910113 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a00cd480 con 0x561b898306b0 2023-10-11T16:33:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 1 -- 172.21.15.111:0/4171910113 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b899c7960 con 0x561b898306b0 2023-10-11T16:33:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f848effd700 1 -- 172.21.15.111:0/4171910113 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f84a009e160 con 0x561b898306b0 2023-10-11T16:33:07.054 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f848effd700 1 -- 172.21.15.111:0/4171910113 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84a00a2580 con 0x561b898306b0 2023-10-11T16:33:07.054 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 10 client.25742 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:07.054 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 1 -- 172.21.15.111:0/4171910113 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561b898ec130 con 0x561b898306b0 2023-10-11T16:33:07.055 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f848effd700 1 -- 172.21.15.111:0/4171910113 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561b899c7960 con 0x561b898306b0 2023-10-11T16:33:07.055 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f848effd700 1 -- 172.21.15.111:0/4171910113 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f84a0099090 con 0x561b898306b0 2023-10-11T16:33:07.055 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 10 client.25742 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:07.055 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 10 client.25742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:07.055 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 20 client.25742 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:07.056 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x561b899ee0f0 0x561b899f0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.056 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 4 client.25742 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:07.056 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.043+0000 7f84c93c2b80 1 -- 172.21.15.111:0/4171910113 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b898e8040 con 0x561b899ee0f0 2023-10-11T16:33:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.044+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x561b899ee0f0 0x561b899f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.045+0000 7f848effd700 10 client.25742 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.045+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x561b899ee0f0 0x561b899f0500 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6b1c32f264630c3 server_cookie=4fd25a723717d36d in_seq=0 out_seq=0 2023-10-11T16:33:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.045+0000 7f848effd700 1 -- 172.21.15.111:0/4171910113 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561b898e8040 con 0x561b899ee0f0 2023-10-11T16:33:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.045+0000 7f848effd700 10 client.25742 handle_command_reply: tid=0 2023-10-11T16:33:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.045+0000 7f84c93c2b80 1 client.25742 shutdown 2023-10-11T16:33:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 -- 172.21.15.111:0/4171910113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b898306b0 msgr2=0x561b899e0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b898306b0 0x561b899e0c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f84a00096b0 tx=0x7f84a009a670 comp rx=0 tx=0).stop 2023-10-11T16:33:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 -- 172.21.15.111:0/4171910113 shutdown_connections 2023-10-11T16:33:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x561b899ee0f0 0x561b899f0500 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899da990 0x561b899de5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b898306b0 0x561b899e0c90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f848effd700 0 client.25742 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/4171910113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899d4cf0 0x561b899ddea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 -- 172.21.15.111:0/4171910113 >> 172.21.15.111:0/4171910113 conn(0x561b899ca1a0 msgr2=0x561b899cea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 -- 172.21.15.111:0/4171910113 shutdown_connections 2023-10-11T16:33:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 1 -- 172.21.15.111:0/4171910113 wait complete. 2023-10-11T16:33:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.046+0000 7f84c93c2b80 20 client.25742 trim_cache size 0 max 16384 2023-10-11T16:33:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.075+0000 7f84c93c2b80 1 Processor -- start 2023-10-11T16:33:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.075+0000 7f84c93c2b80 1 -- start start 2023-10-11T16:33:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b898306b0 0x561b899d47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899d4cf0 0x561b899e5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 0x561b899cf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b899d12f0 con 0x561b898306b0 2023-10-11T16:33:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b899debe0 con 0x561b899da990 2023-10-11T16:33:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b899dee80 con 0x561b899d4cf0 2023-10-11T16:33:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84aeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b898306b0 0x561b899d47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84aeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b898306b0 0x561b899d47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52526/0 (socket says 172.21.15.111:52526) 2023-10-11T16:33:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84aeffd700 1 -- 172.21.15.111:0/74772410 learned_addr learned my addr 172.21.15.111:0/74772410 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.076+0000 7f84af7fe700 1 --2- 172.21.15.111:0/74772410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 0x561b899cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84aeffd700 1 -- 172.21.15.111:0/74772410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899d4cf0 msgr2=0x561b899e5250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:33:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84aeffd700 1 --2- 172.21.15.111:0/74772410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899d4cf0 0x561b899e5250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84aeffd700 1 -- 172.21.15.111:0/74772410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 msgr2=0x561b899cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84aeffd700 1 --2- 172.21.15.111:0/74772410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 0x561b899cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84aeffd700 1 -- 172.21.15.111:0/74772410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b899cb940 con 0x561b898306b0 2023-10-11T16:33:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84aeffd700 1 --2- 172.21.15.111:0/74772410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b898306b0 0x561b899d47b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f84a000c5b0 tx=0x7f84a0096470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7c6630e2435e5c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84967fc700 1 -- 172.21.15.111:0/74772410 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0099460 con 0x561b898306b0 2023-10-11T16:33:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84967fc700 1 -- 172.21.15.111:0/74772410 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84a0099a90 con 0x561b898306b0 2023-10-11T16:33:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.077+0000 7f84967fc700 1 -- 172.21.15.111:0/74772410 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a0096250 con 0x561b898306b0 2023-10-11T16:33:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.078+0000 7f84c93c2b80 1 -- 172.21.15.111:0/74772410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b898306b0 msgr2=0x561b899d47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.078+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/74772410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b898306b0 0x561b899d47b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f84a000c5b0 tx=0x7f84a0096470 comp rx=0 tx=0).stop 2023-10-11T16:33:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.078+0000 7f84c93c2b80 1 -- 172.21.15.111:0/74772410 shutdown_connections 2023-10-11T16:33:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.078+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/74772410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899da990 0x561b899cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.078+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/74772410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b898306b0 0x561b899d47b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.078+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/74772410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899d4cf0 0x561b899e5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.078+0000 7f84c93c2b80 1 -- 172.21.15.111:0/74772410 >> 172.21.15.111:0/74772410 conn(0x561b899ca1a0 msgr2=0x561b899e9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.079+0000 7f84c93c2b80 1 -- 172.21.15.111:0/74772410 shutdown_connections 2023-10-11T16:33:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.079+0000 7f84c93c2b80 1 -- 172.21.15.111:0/74772410 wait complete. 2023-10-11T16:33:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.079+0000 7f84c93c2b80 1 -- start start 2023-10-11T16:33:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.080+0000 7f84c93c2b80 1 Processor -- start 2023-10-11T16:33:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899d4cf0 0x561b899e17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899da990 0x561b899e1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84c93c2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899e23a0 0x561b89a40fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b89a41660 con 0x561b899da990 2023-10-11T16:33:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b89a417d0 con 0x561b899d4cf0 2023-10-11T16:33:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84c93c2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b89a41990 con 0x561b899e23a0 2023-10-11T16:33:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84c93c2b80 10 client.0 _ll_register_callbacks cb 0x7f84a0099f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84ae7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899da990 0x561b899e1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84ae7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899da990 0x561b899e1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52536/0 (socket says 172.21.15.111:52536) 2023-10-11T16:33:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84ae7fc700 1 -- 172.21.15.111:0/2467246602 learned_addr learned my addr 172.21.15.111:0/2467246602 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84aeffd700 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899d4cf0 0x561b899e17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84af7fe700 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899e23a0 0x561b89a40fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84ae7fc700 1 -- 172.21.15.111:0/2467246602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899e23a0 msgr2=0x561b89a40fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899e23a0 0x561b89a40fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84ae7fc700 1 -- 172.21.15.111:0/2467246602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899d4cf0 msgr2=0x561b899e17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.081+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899d4cf0 0x561b899e17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.082+0000 7f84ae7fc700 1 -- 172.21.15.111:0/2467246602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b898ec130 con 0x561b899da990 2023-10-11T16:33:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.082+0000 7f84ae7fc700 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899da990 0x561b899e1d00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009990 tx=0x7f84a400bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14052dcaefee73de server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.082+0000 7f848effd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.082+0000 7f848effd700 1 -- 172.21.15.111:0/2467246602 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a401d040 con 0x561b899da990 2023-10-11T16:33:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.082+0000 7f848effd700 1 -- 172.21.15.111:0/2467246602 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f84a4028ce0 con 0x561b899da990 2023-10-11T16:33:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.082+0000 7f84c93c2b80 1 -- 172.21.15.111:0/2467246602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84a8011040 con 0x561b899da990 2023-10-11T16:33:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f848effd700 1 -- 172.21.15.111:0/2467246602 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84a4028ea0 con 0x561b899da990 2023-10-11T16:33:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f84c93c2b80 10 client.15708 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f84c93c2b80 1 -- 172.21.15.111:0/2467246602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561b89753d80 con 0x561b899da990 2023-10-11T16:33:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f848effd700 1 -- 172.21.15.111:0/2467246602 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84a4036030 con 0x561b899da990 2023-10-11T16:33:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f848effd700 1 -- 172.21.15.111:0/2467246602 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f84a4005440 con 0x561b899da990 2023-10-11T16:33:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f84c93c2b80 10 client.15708 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f84c93c2b80 10 client.15708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f84c93c2b80 20 client.15708 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x561b89a430f0 0x561b89a45500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f84c93c2b80 4 client.15708 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:33:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.083+0000 7f84c93c2b80 1 -- 172.21.15.111:0/2467246602 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561b898f4600 con 0x561b89a430f0 2023-10-11T16:33:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.084+0000 7f84aeffd700 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x561b89a430f0 0x561b89a45500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.085+0000 7f848effd700 10 client.15708 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.085+0000 7f84aeffd700 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x561b89a430f0 0x561b89a45500 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96f2163003576040 server_cookie=72359be8f24909bf in_seq=0 out_seq=0 2023-10-11T16:33:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.093+0000 7f848effd700 1 -- 172.21.15.111:0/2467246602 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561b898f4600 con 0x561b89a430f0 2023-10-11T16:33:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.093+0000 7f848effd700 10 client.15708 handle_command_reply: tid=0 2023-10-11T16:33:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.093+0000 7f84c93c2b80 1 client.15708 shutdown 2023-10-11T16:33:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 -- 172.21.15.111:0/2467246602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899da990 msgr2=0x561b899e1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899da990 0x561b899e1d00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f84a4009990 tx=0x7f84a400bbb0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 -- 172.21.15.111:0/2467246602 shutdown_connections 2023-10-11T16:33:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x561b89a430f0 0x561b89a45500 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561b899e23a0 0x561b89a40fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f848effd700 0 client.15708 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561b899d4cf0 0x561b899e17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 --2- 172.21.15.111:0/2467246602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561b899da990 0x561b899e1d00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 -- 172.21.15.111:0/2467246602 >> 172.21.15.111:0/2467246602 conn(0x561b899ca1a0 msgr2=0x561b899db110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 -- 172.21.15.111:0/2467246602 shutdown_connections 2023-10-11T16:33:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 1 -- 172.21.15.111:0/2467246602 wait complete. 2023-10-11T16:33:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.094+0000 7f84c93c2b80 20 client.15708 trim_cache size 0 max 16384 2023-10-11T16:33:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:33:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:33:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:33:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:33:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 -- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84980a13e0 msgr2=0x7f84980a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84980a13e0 0x7f84980a3890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f84a8009550 tx=0x7f84a8014a90 comp rx=0 tx=0).stop 2023-10-11T16:33:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 -- 172.21.15.111:0/1701137662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84b01002e0 msgr2=0x7f84b0072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84b01002e0 0x7f84b0072df0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f84a000e9a0 tx=0x7f84a000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 -- 172.21.15.111:0/1701137662 shutdown_connections 2023-10-11T16:33:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84980a13e0 0x7f84980a3890 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b0105320 0x7f84b0073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84b01002e0 0x7f84b0072df0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 --2- 172.21.15.111:0/1701137662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b0100c80 0x7f84b0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 -- 172.21.15.111:0/1701137662 >> 172.21.15.111:0/1701137662 conn(0x7f84b00fbb70 msgr2=0x7f84b00fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 -- 172.21.15.111:0/1701137662 shutdown_connections 2023-10-11T16:33:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:07.095+0000 7f84affff700 1 -- 172.21.15.111:0/1701137662 wait complete. 2023-10-11T16:33:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:07 smithi150 ceph-mon[132217]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 24 op/s 2023-10-11T16:33:07.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:07 smithi130 ceph-mon[131859]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 24 op/s 2023-10-11T16:33:07.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:07 smithi111 ceph-mon[98527]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 24 op/s 2023-10-11T16:33:09.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:09 smithi150 ceph-mon[132217]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 21 op/s 2023-10-11T16:33:09.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:09 smithi130 ceph-mon[131859]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 21 op/s 2023-10-11T16:33:09.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:09 smithi111 ceph-mon[98527]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 21 op/s 2023-10-11T16:33:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:11 smithi150 ceph-mon[132217]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 25 op/s 2023-10-11T16:33:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:11 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:33:11.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:11 smithi130 ceph-mon[131859]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 25 op/s 2023-10-11T16:33:11.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:11 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:33:11.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:11 smithi111 ceph-mon[98527]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 25 op/s 2023-10-11T16:33:11.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:11 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:33:12.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:12 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:33:12.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:12 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:33:12.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:12 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:33:13.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:13 smithi150 ceph-mon[132217]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 20 op/s 2023-10-11T16:33:13.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:13 smithi130 ceph-mon[131859]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 20 op/s 2023-10-11T16:33:13.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:13 smithi111 ceph-mon[98527]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 20 op/s 2023-10-11T16:33:15.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:15 smithi150 ceph-mon[132217]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 21 op/s 2023-10-11T16:33:15.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:15 smithi130 ceph-mon[131859]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 21 op/s 2023-10-11T16:33:15.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:15 smithi111 ceph-mon[98527]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 21 op/s 2023-10-11T16:33:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:17 smithi150 ceph-mon[132217]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 23 op/s 2023-10-11T16:33:17.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:17 smithi130 ceph-mon[131859]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 23 op/s 2023-10-11T16:33:17.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:17 smithi111 ceph-mon[98527]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 23 op/s 2023-10-11T16:33:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:19 smithi150 ceph-mon[132217]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-10-11T16:33:19.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:19 smithi130 ceph-mon[131859]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-10-11T16:33:19.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:19 smithi111 ceph-mon[98527]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-10-11T16:33:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 -- 172.21.15.111:0/47315450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 msgr2=0x7f7b3c073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 --2- 172.21.15.111:0/47315450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 0x7f7b3c073260 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009990 tx=0x7f7b2c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 -- 172.21.15.111:0/47315450 shutdown_connections 2023-10-11T16:33:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 --2- 172.21.15.111:0/47315450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b3c10ce50 0x7f7b3c10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 --2- 172.21.15.111:0/47315450 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b3c0746a0 0x7f7b3c072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 --2- 172.21.15.111:0/47315450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 0x7f7b3c073260 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 -- 172.21.15.111:0/47315450 >> 172.21.15.111:0/47315450 conn(0x7f7b3c077550 msgr2=0x7f7b3c077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 -- 172.21.15.111:0/47315450 shutdown_connections 2023-10-11T16:33:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.247+0000 7f7b439ff700 1 -- 172.21.15.111:0/47315450 wait complete. 2023-10-11T16:33:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.249+0000 7f7b439ff700 1 Processor -- start 2023-10-11T16:33:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.249+0000 7f7b439ff700 1 -- start start 2023-10-11T16:33:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b439ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b3c0746a0 0x7f7b3c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b439ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 0x7f7b3c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b439ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b3c10ce50 0x7f7b3c19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b439ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c19d5c0 con 0x7f7b3c074b00 2023-10-11T16:33:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b439ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c19d730 con 0x7f7b3c0746a0 2023-10-11T16:33:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b439ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c19d8f0 con 0x7f7b3c10ce50 2023-10-11T16:33:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b4179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b3c0746a0 0x7f7b3c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b40f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 0x7f7b3c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b40f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 0x7f7b3c198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60982/0 (socket says 172.21.15.111:60982) 2023-10-11T16:33:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b41f9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b3c10ce50 0x7f7b3c19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b40f9a700 1 -- 172.21.15.111:0/185644947 learned_addr learned my addr 172.21.15.111:0/185644947 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b40f9a700 1 -- 172.21.15.111:0/185644947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b3c10ce50 msgr2=0x7f7b3c19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b3c10ce50 0x7f7b3c19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b40f9a700 1 -- 172.21.15.111:0/185644947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b3c0746a0 msgr2=0x7f7b3c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b3c0746a0 0x7f7b3c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.250+0000 7f7b40f9a700 1 -- 172.21.15.111:0/185644947 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0095f0 con 0x7f7b3c074b00 2023-10-11T16:33:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.251+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 0x7f7b3c198b20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0095c0 tx=0x7f7b2c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.251+0000 7f7b327fc700 1 -- 172.21.15.111:0/185644947 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c01d070 con 0x7f7b3c074b00 2023-10-11T16:33:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.251+0000 7f7b327fc700 1 -- 172.21.15.111:0/185644947 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b2c005050 con 0x7f7b3c074b00 2023-10-11T16:33:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.251+0000 7f7b439ff700 1 -- 172.21.15.111:0/185644947 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c112570 con 0x7f7b3c074b00 2023-10-11T16:33:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.251+0000 7f7b327fc700 1 -- 172.21.15.111:0/185644947 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c004a90 con 0x7f7b3c074b00 2023-10-11T16:33:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.251+0000 7f7b439ff700 1 -- 172.21.15.111:0/185644947 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c1123e0 con 0x7f7b3c074b00 2023-10-11T16:33:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.254+0000 7f7b327fc700 1 -- 172.21.15.111:0/185644947 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f7b2c005430 con 0x7f7b3c074b00 2023-10-11T16:33:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.255+0000 7f7b5723ab80 1 Processor -- start 2023-10-11T16:33:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b5723ab80 1 -- start start 2023-10-11T16:33:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6ded6b0 0x55fea6f971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6f97720 0x55fea6f91560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97b90 0x55fea6f91b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6e97000 con 0x55fea6f97b90 2023-10-11T16:33:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6ee9840 con 0x55fea6f97720 2023-10-11T16:33:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6ee2cd0 con 0x55fea6ded6b0 2023-10-11T16:33:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b40f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97b90 0x55fea6f91b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b4179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6f97720 0x55fea6f91560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b327fc700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b280a1460 0x7f7b280a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b4179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6f97720 0x55fea6f91560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45272/0 (socket says 172.21.15.111:45272) 2023-10-11T16:33:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b41f9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6ded6b0 0x55fea6f971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.256+0000 7f7b4179b700 1 -- 172.21.15.111:0/1018000024 learned_addr learned my addr 172.21.15.111:0/1018000024 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b4179b700 1 -- 172.21.15.111:0/1018000024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6ded6b0 msgr2=0x55fea6f971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b4179b700 1 --2- 172.21.15.111:0/1018000024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6ded6b0 0x55fea6f971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b4179b700 1 -- 172.21.15.111:0/1018000024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97b90 msgr2=0x55fea6f91b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b327fc700 1 -- 172.21.15.111:0/185644947 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f7b2c0ce180 con 0x7f7b3c074b00 2023-10-11T16:33:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b4179b700 1 --2- 172.21.15.111:0/1018000024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97b90 0x55fea6f91b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b4179b700 1 -- 172.21.15.111:0/1018000024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fea6d645d0 con 0x55fea6f97720 2023-10-11T16:33:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b41f9c700 1 --2- 172.21.15.111:0/1018000024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6ded6b0 0x55fea6f971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b41f9c700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b280a1460 0x7f7b280a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b4179b700 1 --2- 172.21.15.111:0/1018000024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6f97720 0x55fea6f91560 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800bd70 tx=0x7f7b380106b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dceca5767516eadb server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.257+0000 7f7b277fe700 1 -- 172.21.15.111:0/1018000024 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38010e10 con 0x55fea6f97720 2023-10-11T16:33:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.258+0000 7f7b277fe700 1 -- 172.21.15.111:0/1018000024 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b38015070 con 0x55fea6f97720 2023-10-11T16:33:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.258+0000 7f7b277fe700 1 -- 172.21.15.111:0/1018000024 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38011730 con 0x55fea6f97720 2023-10-11T16:33:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.258+0000 7f7b41f9c700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b280a1460 0x7f7b280a3910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009710 tx=0x7f7b3400b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.258+0000 7f7b5723ab80 1 -- 172.21.15.111:0/1018000024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6f97720 msgr2=0x55fea6f91560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.258+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/1018000024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6f97720 0x55fea6f91560 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800bd70 tx=0x7f7b380106b0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 -- 172.21.15.111:0/1018000024 shutdown_connections 2023-10-11T16:33:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/1018000024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97b90 0x55fea6f91b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/1018000024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6ded6b0 0x55fea6f971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/1018000024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6f97720 0x55fea6f91560 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 -- 172.21.15.111:0/1018000024 >> 172.21.15.111:0/1018000024 conn(0x55fea6f86f50 msgr2=0x55fea6f8c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 -- 172.21.15.111:0/1018000024 shutdown_connections 2023-10-11T16:33:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 -- 172.21.15.111:0/1018000024 wait complete. 2023-10-11T16:33:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 -- start start 2023-10-11T16:33:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.259+0000 7f7b5723ab80 1 Processor -- start 2023-10-11T16:33:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6e84880 con 0x55fea6f97720 2023-10-11T16:33:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6e84520 con 0x55fea6ded6b0 2023-10-11T16:33:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6d668c0 con 0x55fea6f97b90 2023-10-11T16:33:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b5723ab80 10 client.0 _ll_register_callbacks cb 0x55fea6ea2720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b4179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b40f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b4179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45284/0 (socket says 172.21.15.111:45284) 2023-10-11T16:33:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b4179b700 1 -- 172.21.15.111:0/705713933 learned_addr learned my addr 172.21.15.111:0/705713933 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.260+0000 7f7b41f9c700 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b4179b700 1 -- 172.21.15.111:0/705713933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 msgr2=0x55fea6f9b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b4179b700 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b4179b700 1 -- 172.21.15.111:0/705713933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 msgr2=0x55fea6f9ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b4179b700 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b4179b700 1 -- 172.21.15.111:0/705713933 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3800b9a0 con 0x55fea6ded6b0 2023-10-11T16:33:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b4179b700 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9daf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7b38000c00 tx=0x7f7b38010e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31110c61e8446ff5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b24ff9700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b24ff9700 1 -- 172.21.15.111:0/705713933 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3801c890 con 0x55fea6ded6b0 2023-10-11T16:33:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b5723ab80 1 -- 172.21.15.111:0/705713933 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fea6f84710 con 0x55fea6ded6b0 2023-10-11T16:33:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.261+0000 7f7b24ff9700 1 -- 172.21.15.111:0/705713933 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f7b38017070 con 0x55fea6ded6b0 2023-10-11T16:33:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b24ff9700 1 -- 172.21.15.111:0/705713933 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b3800bec0 con 0x55fea6ded6b0 2023-10-11T16:33:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b5723ab80 10 client.25760 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b5723ab80 1 -- 172.21.15.111:0/705713933 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fea6ea9130 con 0x55fea6ded6b0 2023-10-11T16:33:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b24ff9700 1 -- 172.21.15.111:0/705713933 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fea6f84710 con 0x55fea6ded6b0 2023-10-11T16:33:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b24ff9700 1 -- 172.21.15.111:0/705713933 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7b38003be0 con 0x55fea6ded6b0 2023-10-11T16:33:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b5723ab80 10 client.25760 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b5723ab80 10 client.25760 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b5723ab80 20 client.25760 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fea6faae20 0x55fea6fad230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b5723ab80 4 client.25760 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.262+0000 7f7b5723ab80 1 -- 172.21.15.111:0/705713933 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fea6ea5040 con 0x55fea6faae20 2023-10-11T16:33:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.263+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fea6faae20 0x55fea6fad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.263+0000 7f7b24ff9700 10 client.25760 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.263+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fea6faae20 0x55fea6fad230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1212111e7832c35 server_cookie=2fd7fc6dda9bf62c in_seq=0 out_seq=0 2023-10-11T16:33:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b24ff9700 1 -- 172.21.15.111:0/705713933 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55fea6ea5040 con 0x55fea6faae20 2023-10-11T16:33:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b24ff9700 10 client.25760 handle_command_reply: tid=0 2023-10-11T16:33:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 client.25760 shutdown 2023-10-11T16:33:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 -- 172.21.15.111:0/705713933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 msgr2=0x55fea6f9daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9daf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7b38000c00 tx=0x7f7b38010e40 comp rx=0 tx=0).stop 2023-10-11T16:33:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 -- 172.21.15.111:0/705713933 shutdown_connections 2023-10-11T16:33:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fea6faae20 0x55fea6fad230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b24ff9700 0 client.25760 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9daf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/705713933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 -- 172.21.15.111:0/705713933 >> 172.21.15.111:0/705713933 conn(0x55fea6f86f50 msgr2=0x55fea6f8b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 -- 172.21.15.111:0/705713933 shutdown_connections 2023-10-11T16:33:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 1 -- 172.21.15.111:0/705713933 wait complete. 2023-10-11T16:33:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.264+0000 7f7b5723ab80 20 client.25760 trim_cache size 0 max 16384 2023-10-11T16:33:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.294+0000 7f7b5723ab80 1 Processor -- start 2023-10-11T16:33:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.294+0000 7f7b5723ab80 1 -- start start 2023-10-11T16:33:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.294+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6fa2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f8bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6f95990 con 0x55fea6f97720 2023-10-11T16:33:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6f8cf40 con 0x55fea6ded6b0 2023-10-11T16:33:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6f91e10 con 0x55fea6f97b90 2023-10-11T16:33:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b40f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f8bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b40f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f8bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32780/0 (socket says 172.21.15.111:32780) 2023-10-11T16:33:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b40f9a700 1 -- 172.21.15.111:0/2075293345 learned_addr learned my addr 172.21.15.111:0/2075293345 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b4179b700 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6fa2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b41f9c700 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b4179b700 1 -- 172.21.15.111:0/2075293345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 msgr2=0x55fea6f9d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b4179b700 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b4179b700 1 -- 172.21.15.111:0/2075293345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 msgr2=0x55fea6f8bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:20.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b4179b700 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f8bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b4179b700 1 -- 172.21.15.111:0/2075293345 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fea6ea8e80 con 0x55fea6ded6b0 2023-10-11T16:33:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.295+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f8bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.296+0000 7f7b4179b700 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6fa2090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7b380106f0 tx=0x7f7b38010b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82ee56b3cf453c3b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.296+0000 7f7b27fff700 1 -- 172.21.15.111:0/2075293345 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3800bec0 con 0x55fea6ded6b0 2023-10-11T16:33:20.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.296+0000 7f7b27fff700 1 -- 172.21.15.111:0/2075293345 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b3801de90 con 0x55fea6ded6b0 2023-10-11T16:33:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.296+0000 7f7b27fff700 1 -- 172.21.15.111:0/2075293345 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3801c8f0 con 0x55fea6ded6b0 2023-10-11T16:33:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 -- 172.21.15.111:0/2075293345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 msgr2=0x55fea6fa2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6fa2090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7b380106f0 tx=0x7f7b38010b30 comp rx=0 tx=0).stop 2023-10-11T16:33:20.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 -- 172.21.15.111:0/2075293345 shutdown_connections 2023-10-11T16:33:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6fa2090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/2075293345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f8bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 -- 172.21.15.111:0/2075293345 >> 172.21.15.111:0/2075293345 conn(0x55fea6f86f50 msgr2=0x55fea6f91700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 -- 172.21.15.111:0/2075293345 shutdown_connections 2023-10-11T16:33:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 -- 172.21.15.111:0/2075293345 wait complete. 2023-10-11T16:33:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.297+0000 7f7b5723ab80 1 -- start start 2023-10-11T16:33:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.298+0000 7f7b5723ab80 1 Processor -- start 2023-10-11T16:33:20.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.298+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.298+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.298+0000 7f7b5723ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.298+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6ffdcd0 con 0x55fea6f97720 2023-10-11T16:33:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.298+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6ffde10 con 0x55fea6ded6b0 2023-10-11T16:33:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.298+0000 7f7b5723ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea6ffdfd0 con 0x55fea6f97b90 2023-10-11T16:33:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.298+0000 7f7b5723ab80 10 client.0 _ll_register_callbacks cb 0x7f7b2c09aa50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:32782/0 (socket says 172.21.15.111:32782) 2023-10-11T16:33:20.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 -- 172.21.15.111:0/711068954 learned_addr learned my addr 172.21.15.111:0/711068954 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b41f9c700 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b4179b700 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.310 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 -- 172.21.15.111:0/711068954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 msgr2=0x55fea6f9f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 -- 172.21.15.111:0/711068954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 msgr2=0x55fea6f9e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 -- 172.21.15.111:0/711068954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c1123e0 con 0x55fea6f97720 2023-10-11T16:33:20.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b40f9a700 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9eb60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c097180 tx=0x7f7b2c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3765d9ed252dc server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.299+0000 7f7b24ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b24ff9700 1 -- 172.21.15.111:0/711068954 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c0a3040 con 0x55fea6f97720 2023-10-11T16:33:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b24ff9700 1 -- 172.21.15.111:0/711068954 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f7b2c09aee0 con 0x55fea6f97720 2023-10-11T16:33:20.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 1 -- 172.21.15.111:0/711068954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b38015070 con 0x55fea6f97720 2023-10-11T16:33:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b24ff9700 1 -- 172.21.15.111:0/711068954 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b2c09b0a0 con 0x55fea6f97720 2023-10-11T16:33:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 10 client.15726 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 1 -- 172.21.15.111:0/711068954 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fea6ea9130 con 0x55fea6f97720 2023-10-11T16:33:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b24ff9700 1 -- 172.21.15.111:0/711068954 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b38015070 con 0x55fea6f97720 2023-10-11T16:33:20.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b24ff9700 1 -- 172.21.15.111:0/711068954 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7b2c09a200 con 0x55fea6f97720 2023-10-11T16:33:20.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 10 client.15726 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:20.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 10 client.15726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:20.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 20 client.15726 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:20.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fea6fff310 0x55fea7001720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:20.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 4 client.15726 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:33:20.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.300+0000 7f7b5723ab80 1 -- 172.21.15.111:0/711068954 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fea6eb1600 con 0x55fea6fff310 2023-10-11T16:33:20.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.301+0000 7f7b4179b700 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fea6fff310 0x55fea7001720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.301+0000 7f7b4179b700 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fea6fff310 0x55fea7001720 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6900bae3b9da2436 server_cookie=d7dddcb86e17a4bf in_seq=0 out_seq=0 2023-10-11T16:33:20.316 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.301+0000 7f7b24ff9700 10 client.15726 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:20.317 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.305+0000 7f7b24ff9700 1 -- 172.21.15.111:0/711068954 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fea6eb1600 con 0x55fea6fff310 2023-10-11T16:33:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.305+0000 7f7b24ff9700 10 client.15726 handle_command_reply: tid=0 2023-10-11T16:33:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.305+0000 7f7b5723ab80 1 client.15726 shutdown 2023-10-11T16:33:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 -- 172.21.15.111:0/711068954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 msgr2=0x55fea6f9eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9eb60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c097180 tx=0x7f7b2c009310 comp rx=0 tx=0).stop 2023-10-11T16:33:20.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 -- 172.21.15.111:0/711068954 shutdown_connections 2023-10-11T16:33:20.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fea6fff310 0x55fea7001720 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fea6f97b90 0x55fea6f9f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fea6ded6b0 0x55fea6f9e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 --2- 172.21.15.111:0/711068954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fea6f97720 0x55fea6f9eb60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b24ff9700 0 client.15726 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:20.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 -- 172.21.15.111:0/711068954 >> 172.21.15.111:0/711068954 conn(0x55fea6f86f50 msgr2=0x55fea6f94e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:20.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 -- 172.21.15.111:0/711068954 shutdown_connections 2023-10-11T16:33:20.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 1 -- 172.21.15.111:0/711068954 wait complete. 2023-10-11T16:33:20.321 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.306+0000 7f7b5723ab80 20 client.15726 trim_cache size 0 max 16384 2023-10-11T16:33:20.321 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:33:20.321 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:33:20.322 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:33:20.322 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:33:20.322 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 -- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b280a1460 msgr2=0x7f7b280a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b280a1460 0x7f7b280a3910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009710 tx=0x7f7b3400b040 comp rx=0 tx=0).stop 2023-10-11T16:33:20.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 -- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 msgr2=0x7f7b3c198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:20.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 0x7f7b3c198b20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0095c0 tx=0x7f7b2c00f690 comp rx=0 tx=0).stop 2023-10-11T16:33:20.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 -- 172.21.15.111:0/185644947 shutdown_connections 2023-10-11T16:33:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7b280a1460 0x7f7b280a3910 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b3c10ce50 0x7f7b3c19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b3c0746a0 0x7f7b3c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 --2- 172.21.15.111:0/185644947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b3c074b00 0x7f7b3c198b20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:20.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 -- 172.21.15.111:0/185644947 >> 172.21.15.111:0/185644947 conn(0x7f7b3c077550 msgr2=0x7f7b3c104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 -- 172.21.15.111:0/185644947 shutdown_connections 2023-10-11T16:33:20.325 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:20.307+0000 7f7b439ff700 1 -- 172.21.15.111:0/185644947 wait complete. 2023-10-11T16:33:21.429 INFO:tasks.workunit.client.0.smithi111.stderr:Updating files: 73% (8704/11778) Updating files: 74% (8716/11778) Updating files: 75% (8834/11778) Updating files: 76% (8952/11778) Updating files: 77% (9070/11778) Updating files: 78% (9187/11778) Updating files: 79% (9305/11778) Updating files: 80% (9423/11778) Updating files: 81% (9541/11778) Updating files: 82% (9658/11778) Updating files: 83% (9776/11778) Updating files: 84% (9894/11778) Updating files: 85% (10012/11778) Updating files: 86% (10130/11778) Updating files: 87% (10247/11778) Updating files: 88% (10365/11778) Updating files: 89% (10483/11778) Updating files: 90% (10601/11778) Updating files: 91% (10718/11778) Updating files: 92% (10836/11778) Updating files: 93% (10954/11778) Updating files: 94% (11072/11778) Updating files: 95% (11190/11778) Updating files: 96% (11307/11778) Updating files: 97% (11425/11778) Updating files: 98% (11543/11778) Updating files: 99% (11661/11778) Updating files: 100% (11778/11778) Updating files: 100% (11778/11778), done. 2023-10-11T16:33:21.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:21 smithi150 ceph-mon[132217]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2023-10-11T16:33:21.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:21 smithi130 ceph-mon[131859]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2023-10-11T16:33:21.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:21 smithi111 ceph-mon[98527]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2023-10-11T16:33:21.766 INFO:tasks.workunit.client.0.smithi111.stderr:Note: switching to '3f7a9d091654926539d989dd4c83dfee517abc9e'. 2023-10-11T16:33:21.767 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-11T16:33:21.767 INFO:tasks.workunit.client.0.smithi111.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-11T16:33:21.767 INFO:tasks.workunit.client.0.smithi111.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-11T16:33:21.768 INFO:tasks.workunit.client.0.smithi111.stderr:state without impacting any branches by switching back to a branch. 2023-10-11T16:33:21.768 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-11T16:33:21.768 INFO:tasks.workunit.client.0.smithi111.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-11T16:33:21.768 INFO:tasks.workunit.client.0.smithi111.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-11T16:33:21.768 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-11T16:33:21.769 INFO:tasks.workunit.client.0.smithi111.stderr: git switch -c 2023-10-11T16:33:21.769 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-11T16:33:21.769 INFO:tasks.workunit.client.0.smithi111.stderr:Or undo this operation with: 2023-10-11T16:33:21.769 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-11T16:33:21.769 INFO:tasks.workunit.client.0.smithi111.stderr: git switch - 2023-10-11T16:33:21.770 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-11T16:33:21.770 INFO:tasks.workunit.client.0.smithi111.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-11T16:33:21.770 INFO:tasks.workunit.client.0.smithi111.stderr: 2023-10-11T16:33:21.770 INFO:tasks.workunit.client.0.smithi111.stderr:HEAD is now at 3f7a9d09165 doc/cephfs: Update docs to match remove functionality and respective vxattrs 2023-10-11T16:33:21.772 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-11T16:33:21.825 INFO:tasks.workunit.client.0.smithi111.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-11T16:33:21.830 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-11T16:33:21.831 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-11T16:33:22.388 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-11T16:33:22.470 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-11T16:33:22.535 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-11T16:33:22.539 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-11T16:33:22.539 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-11T16:33:22.602 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-11T16:33:22.609 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:33:22.609 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-11T16:33:22.654 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2023-10-11T16:33:22.657 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2023-10-11T16:33:22.658 DEBUG:teuthology.orchestra.run.smithi111:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2023-10-11T16:33:22.751 INFO:tasks.workunit.client.0.smithi111.stderr:+ set -e 2023-10-11T16:33:22.752 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir -p testdir 2023-10-11T16:33:22.755 INFO:tasks.workunit.client.0.smithi111.stderr:+ cd testdir 2023-10-11T16:33:22.755 INFO:tasks.workunit.client.0.smithi111.stderr:+ set +e 2023-10-11T16:33:22.756 INFO:tasks.workunit.client.0.smithi111.stderr:+ setfacl -d -m u:nobody:rw . 2023-10-11T16:33:22.795 INFO:tasks.workunit.client.0.smithi111.stderr:+ test 0 '!=' 0 2023-10-11T16:33:22.796 INFO:tasks.workunit.client.0.smithi111.stderr:+ set -e 2023-10-11T16:33:22.796 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=0 2023-10-11T16:33:22.796 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 0 -lt 100 ']' 2023-10-11T16:33:22.797 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 0 + 1 2023-10-11T16:33:22.798 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=1 2023-10-11T16:33:22.798 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:22.803 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:22.803 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:22.806 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:22.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:22.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:23.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:23 smithi150 ceph-mon[132217]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-10-11T16:33:23.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:23 smithi130 ceph-mon[131859]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-10-11T16:33:23.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:23 smithi111 ceph-mon[98527]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-10-11T16:33:25.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:25 smithi150 ceph-mon[132217]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2023-10-11T16:33:25.497 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:25.497 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:25.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:25 smithi130 ceph-mon[131859]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2023-10-11T16:33:25.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:25 smithi111 ceph-mon[98527]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2023-10-11T16:33:25.688 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:25.688 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:25.723 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:25.724 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:25.724 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 1 -lt 100 ']' 2023-10-11T16:33:25.724 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 1 + 1 2023-10-11T16:33:25.773 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=2 2023-10-11T16:33:25.773 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:25.852 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:25.853 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:25.855 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:25.855 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:25.856 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:26.338 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:26.339 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:26.488 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:26.488 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:26.498 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:26.498 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:26.498 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -lt 100 ']' 2023-10-11T16:33:26.499 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 2 + 1 2023-10-11T16:33:26.534 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=3 2023-10-11T16:33:26.534 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:26.577 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:26.578 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:26.581 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:26.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:26.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:26.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:26.988 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:27.129 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:27.130 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:27.139 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:27.140 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:27.140 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 3 -lt 100 ']' 2023-10-11T16:33:27.141 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 3 + 1 2023-10-11T16:33:27.167 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=4 2023-10-11T16:33:27.168 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:27.219 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:27.220 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:27.222 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:27.222 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:27.223 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:27.449 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:27 smithi111 ceph-mon[98527]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2023-10-11T16:33:27.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:27 smithi150 ceph-mon[132217]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2023-10-11T16:33:27.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:27 smithi130 ceph-mon[131859]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2023-10-11T16:33:27.614 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:27.614 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:27.829 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:27.829 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:27.848 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:27.849 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:27.849 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 4 -lt 100 ']' 2023-10-11T16:33:27.849 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 4 + 1 2023-10-11T16:33:27.939 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=5 2023-10-11T16:33:27.940 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:27.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:27.978 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:27.980 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:27.981 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:27.981 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:28.429 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:28.430 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:28.571 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:28.571 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:28.581 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:28.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:28.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 5 -lt 100 ']' 2023-10-11T16:33:28.582 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 5 + 1 2023-10-11T16:33:28.609 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=6 2023-10-11T16:33:28.609 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:28.647 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:28.647 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:28.649 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:28.650 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:28.650 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:29.054 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:29.055 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:29.212 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:29.213 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:29.222 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:29.222 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:29.223 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 6 -lt 100 ']' 2023-10-11T16:33:29.223 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 6 + 1 2023-10-11T16:33:29.250 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=7 2023-10-11T16:33:29.250 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:29.286 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:29.286 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:29.288 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:29.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:29.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:29.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:29 smithi150 ceph-mon[132217]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2023-10-11T16:33:29.516 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:29 smithi111 ceph-mon[98527]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2023-10-11T16:33:29.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:29 smithi130 ceph-mon[131859]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2023-10-11T16:33:29.680 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:29.680 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:29.803 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:29.803 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:29.814 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:29.815 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:29.815 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 7 -lt 100 ']' 2023-10-11T16:33:29.815 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 7 + 1 2023-10-11T16:33:29.850 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=8 2023-10-11T16:33:29.850 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:29.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:29.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:29.888 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:29.889 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:29.889 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:30.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:30.371 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:30.512 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:30.512 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:30.522 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:30.522 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:30.523 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 8 -lt 100 ']' 2023-10-11T16:33:30.523 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 8 + 1 2023-10-11T16:33:30.550 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=9 2023-10-11T16:33:30.550 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:30.586 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:30.586 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:30.588 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:30.589 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:30.589 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:31.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:31.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:31.437 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:31.437 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:31.447 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:31.447 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:31.448 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 9 -lt 100 ']' 2023-10-11T16:33:31.448 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 9 + 1 2023-10-11T16:33:31.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=10 2023-10-11T16:33:31.481 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:31.519 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:31.519 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:31.521 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:31.522 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:31.522 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:31.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:31 smithi130 ceph-mon[131859]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2023-10-11T16:33:31.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:31 smithi111 ceph-mon[98527]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2023-10-11T16:33:31.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:31 smithi150 ceph-mon[132217]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2023-10-11T16:33:31.929 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:31.929 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:32.062 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:32.062 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:32.081 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:32.081 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:32.082 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 10 -lt 100 ']' 2023-10-11T16:33:32.082 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 10 + 1 2023-10-11T16:33:32.141 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=11 2023-10-11T16:33:32.142 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:32.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:32.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:32.214 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:32.215 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:32.215 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:32.637 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:32.638 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:32.646 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:33:32.793 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:32.794 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:32.806 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:32.806 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:32.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 11 -lt 100 ']' 2023-10-11T16:33:32.807 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 11 + 1 2023-10-11T16:33:32.830 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=12 2023-10-11T16:33:32.831 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:32.843 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:32.843 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:32.845 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:32.845 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:32.846 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:33.353 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:33.354 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:33.445 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:33.445 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:33.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:33.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:33.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 12 -lt 100 ']' 2023-10-11T16:33:33.481 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 12 + 1 2023-10-11T16:33:33.513 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=13 2023-10-11T16:33:33.513 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:33.542 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:33.542 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:33.544 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:33.545 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:33.545 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:33.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:33 smithi130 ceph-mon[131859]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.6 KiB/s wr, 6 op/s 2023-10-11T16:33:33.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:33 smithi111 ceph-mon[98527]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.6 KiB/s wr, 6 op/s 2023-10-11T16:33:33.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:33 smithi150 ceph-mon[132217]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.6 KiB/s wr, 6 op/s 2023-10-11T16:33:34.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:34.217 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:34.461 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:34.462 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:34.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:34.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:34.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 13 -lt 100 ']' 2023-10-11T16:33:34.481 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 13 + 1 2023-10-11T16:33:34.524 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=14 2023-10-11T16:33:34.525 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:34.594 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:34.594 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:34.597 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:34.597 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:34.598 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:35.262 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:35.262 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:35.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:35 smithi111 ceph-mon[98527]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2023-10-11T16:33:35.553 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:35.553 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:35.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:35 smithi130 ceph-mon[131859]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2023-10-11T16:33:35.598 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:35.598 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:35.598 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 14 -lt 100 ']' 2023-10-11T16:33:35.599 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 14 + 1 2023-10-11T16:33:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 -- 172.21.15.111:0/2864207973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60281063f0 msgr2=0x7f60281067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 --2- 172.21.15.111:0/2864207973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60281063f0 0x7f60281067c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f60180099c0 tx=0x7f6018009cd0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 -- 172.21.15.111:0/2864207973 shutdown_connections 2023-10-11T16:33:35.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 --2- 172.21.15.111:0/2864207973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6028101400 0x7f602810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 --2- 172.21.15.111:0/2864207973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60281063f0 0x7f60281067c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 --2- 172.21.15.111:0/2864207973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6028100a50 0x7f6028100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 -- 172.21.15.111:0/2864207973 >> 172.21.15.111:0/2864207973 conn(0x7f6028077590 msgr2=0x7f6028077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 -- 172.21.15.111:0/2864207973 shutdown_connections 2023-10-11T16:33:35.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f602cbff700 1 -- 172.21.15.111:0/2864207973 wait complete. 2023-10-11T16:33:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f74689ff700 1 -- 172.21.15.111:0/2195684195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7464101d50 msgr2=0x7f746410a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.605 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f74689ff700 1 --2- 172.21.15.111:0/2195684195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7464101d50 0x7f746410a240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7458009990 tx=0x7f7458009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f74689ff700 1 -- 172.21.15.111:0/2195684195 shutdown_connections 2023-10-11T16:33:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f74689ff700 1 --2- 172.21.15.111:0/2195684195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f746410a810 0x7f746410cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f74689ff700 1 --2- 172.21.15.111:0/2195684195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7464101440 0x7f7464101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f74689ff700 1 --2- 172.21.15.111:0/2195684195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7464101d50 0x7f746410a240 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f74689ff700 1 -- 172.21.15.111:0/2195684195 >> 172.21.15.111:0/2195684195 conn(0x7f74640763a0 msgr2=0x7f74640767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.601+0000 7f74689ff700 1 -- 172.21.15.111:0/2195684195 shutdown_connections 2023-10-11T16:33:35.607 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.602+0000 7f74689ff700 1 -- 172.21.15.111:0/2195684195 wait complete. 2023-10-11T16:33:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.602+0000 7f602cbff700 1 Processor -- start 2023-10-11T16:33:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.602+0000 7f602cbff700 1 -- start start 2023-10-11T16:33:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602cbff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6028100a50 0x7f6028196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602cbff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6028101400 0x7f6028196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602cbff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60281063f0 0x7f602819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602cbff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602819b400 con 0x7f6028100a50 2023-10-11T16:33:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602cbff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602819b570 con 0x7f60281063f0 2023-10-11T16:33:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602cbff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602819b730 con 0x7f6028101400 2023-10-11T16:33:35.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6028100a50 0x7f6028196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6028100a50 0x7f6028196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45372/0 (socket says 172.21.15.111:45372) 2023-10-11T16:33:35.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602659c700 1 -- 172.21.15.111:0/205175713 learned_addr learned my addr 172.21.15.111:0/205175713 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f74689ff700 1 Processor -- start 2023-10-11T16:33:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f6025d9b700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6028101400 0x7f6028196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f6026d9d700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60281063f0 0x7f602819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f74689ff700 1 -- start start 2023-10-11T16:33:35.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602659c700 1 -- 172.21.15.111:0/205175713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6028101400 msgr2=0x7f6028196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602659c700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6028101400 0x7f6028196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602659c700 1 -- 172.21.15.111:0/205175713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60281063f0 msgr2=0x7f602819adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602659c700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60281063f0 0x7f602819adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.603+0000 7f602659c700 1 -- 172.21.15.111:0/205175713 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60180095f0 con 0x7f6028100a50 2023-10-11T16:33:35.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f602659c700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6028100a50 0x7f6028196420 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6018000c00 tx=0x7f601800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f74689ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7464101440 0x7f7464196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f74689ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7464101d50 0x7f74641968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f74689ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f746410a810 0x7f746419ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.614 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f74689ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746419b360 con 0x7f7464101440 2023-10-11T16:33:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f74689ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746419b4d0 con 0x7f746410a810 2023-10-11T16:33:35.615 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f74689ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746419b690 con 0x7f7464101d50 2023-10-11T16:33:35.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f60177fe700 1 -- 172.21.15.111:0/205175713 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601801d070 con 0x7f6028100a50 2023-10-11T16:33:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f60177fe700 1 -- 172.21.15.111:0/205175713 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6018005050 con 0x7f6028100a50 2023-10-11T16:33:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f60177fe700 1 -- 172.21.15.111:0/205175713 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6018004ab0 con 0x7f6028100a50 2023-10-11T16:33:35.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f602cbff700 1 -- 172.21.15.111:0/205175713 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6028110380 con 0x7f6028100a50 2023-10-11T16:33:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f746259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7464101440 0x7f7464196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f746259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7464101440 0x7f7464196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45382/0 (socket says 172.21.15.111:45382) 2023-10-11T16:33:35.617 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f746259c700 1 -- 172.21.15.111:0/2769950301 learned_addr learned my addr 172.21.15.111:0/2769950301 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f602cbff700 1 -- 172.21.15.111:0/205175713 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60281101f0 con 0x7f6028100a50 2023-10-11T16:33:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f7461d9b700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7464101d50 0x7f74641968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.604+0000 7f7462d9d700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f746410a810 0x7f746419ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.605+0000 7f746259c700 1 -- 172.21.15.111:0/2769950301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7464101d50 msgr2=0x7f74641968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.605+0000 7f746259c700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7464101d50 0x7f74641968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.605+0000 7f746259c700 1 -- 172.21.15.111:0/2769950301 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f746410a810 msgr2=0x7f746419ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.605+0000 7f746259c700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f746410a810 0x7f746419ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.605+0000 7f746259c700 1 -- 172.21.15.111:0/2769950301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74580095f0 con 0x7f7464101440 2023-10-11T16:33:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.605+0000 7f746259c700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7464101440 0x7f7464196380 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f745400e9a0 tx=0x7f745400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.606+0000 7f74537fe700 1 -- 172.21.15.111:0/2769950301 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454009790 con 0x7f7464101440 2023-10-11T16:33:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.607+0000 7f74537fe700 1 -- 172.21.15.111:0/2769950301 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7454010430 con 0x7f7464101440 2023-10-11T16:33:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.607+0000 7f74689ff700 1 -- 172.21.15.111:0/2769950301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74641102d0 con 0x7f7464101440 2023-10-11T16:33:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.607+0000 7f74537fe700 1 -- 172.21.15.111:0/2769950301 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454018680 con 0x7f7464101440 2023-10-11T16:33:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.607+0000 7f74689ff700 1 -- 172.21.15.111:0/2769950301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7464110140 con 0x7f7464101440 2023-10-11T16:33:35.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.608+0000 7f60177fe700 1 -- 172.21.15.111:0/205175713 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f60180051c0 con 0x7f6028100a50 2023-10-11T16:33:35.622 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.610+0000 7f74537fe700 1 -- 172.21.15.111:0/2769950301 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f74540105a0 con 0x7f7464101440 2023-10-11T16:33:35.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.610+0000 7f60177fe700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60100a13e0 0x7f60100a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.610+0000 7f60177fe700 1 -- 172.21.15.111:0/205175713 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f60180cdc10 con 0x7f6028100a50 2023-10-11T16:33:35.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.610+0000 7f6025d9b700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60100a13e0 0x7f60100a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.611+0000 7f6025d9b700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60100a13e0 0x7f60100a3890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6028197a30 tx=0x7f601c009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.611+0000 7f74537fe700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f744c0a1460 0x7f744c0a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.611+0000 7f74537fe700 1 -- 172.21.15.111:0/2769950301 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f7454014070 con 0x7f7464101440 2023-10-11T16:33:35.624 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.611+0000 7f7461d9b700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f744c0a1460 0x7f744c0a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.625 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.612+0000 7f7461d9b700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f744c0a1460 0x7f744c0a3910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7464197990 tx=0x7f7458005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.638 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=15 2023-10-11T16:33:35.639 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.665+0000 7f747c35bb80 1 Processor -- start 2023-10-11T16:33:35.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.665+0000 7f6040619b80 1 Processor -- start 2023-10-11T16:33:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f747c35bb80 1 -- start start 2023-10-11T16:33:35.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6040619b80 1 -- start start 2023-10-11T16:33:35.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6040619b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec2399f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e40950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6040619b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec239e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6040619b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec239ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6040619b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec229f060 con 0x562ec239e8b0 2023-10-11T16:33:35.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6040619b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec22f18a0 con 0x562ec239a470 2023-10-11T16:33:35.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6040619b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec22ead30 con 0x562ec2399b60 2023-10-11T16:33:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e33560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e33ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f747c35bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9d39000 con 0x5640a9e33aa0 2023-10-11T16:33:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f747c35bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9d8b840 con 0x5640a9c8f6b0 2023-10-11T16:33:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f747c35bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9d84cd0 con 0x5640a9e40e90 2023-10-11T16:33:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f746259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e40950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6025d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec239e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f746259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e40950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54110/0 (socket says 172.21.15.111:54110) 2023-10-11T16:33:35.674 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f746259c700 1 -- 172.21.15.111:0/2848469975 learned_addr learned my addr 172.21.15.111:0/2848469975 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6025d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec239e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54122/0 (socket says 172.21.15.111:54122) 2023-10-11T16:33:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f6025d9b700 1 -- 172.21.15.111:0/688875601 learned_addr learned my addr 172.21.15.111:0/688875601 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f602659c700 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec2399f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.666+0000 7f7461d9b700 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e33560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.675 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f7462d9d700 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e33ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f6026d9d700 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec239ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f6025d9b700 1 -- 172.21.15.111:0/688875601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 msgr2=0x562ec2399f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f6025d9b700 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec2399f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f6025d9b700 1 -- 172.21.15.111:0/688875601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 msgr2=0x562ec239ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.677 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f746259c700 1 -- 172.21.15.111:0/2848469975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 msgr2=0x5640a9e33560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.677 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f746259c700 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e33560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.678 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f746259c700 1 -- 172.21.15.111:0/2848469975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 msgr2=0x5640a9e33ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.678 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f746259c700 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e33ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.678 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f746259c700 1 -- 172.21.15.111:0/2848469975 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640a9c065d0 con 0x5640a9c8f6b0 2023-10-11T16:33:35.679 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f7462d9d700 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e33ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:35.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f6025d9b700 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec239ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f6025d9b700 1 -- 172.21.15.111:0/688875601 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ec216c630 con 0x562ec239a470 2023-10-11T16:33:35.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f6026d9d700 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec239ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:35.680 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.667+0000 7f746259c700 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e40950 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f74540964a0 tx=0x7f7454099bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99fc88abbcf1ef37 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.681 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f7450ff9700 1 -- 172.21.15.111:0/2848469975 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7454099fa0 con 0x5640a9c8f6b0 2023-10-11T16:33:35.681 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f7450ff9700 1 -- 172.21.15.111:0/2848469975 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f745409dfd0 con 0x5640a9c8f6b0 2023-10-11T16:33:35.681 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f7450ff9700 1 -- 172.21.15.111:0/2848469975 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74540996f0 con 0x5640a9c8f6b0 2023-10-11T16:33:35.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f6025d9b700 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec239e370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f601c011d50 tx=0x7f601c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d18f754afc85d12 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f6014ff9700 1 -- 172.21.15.111:0/688875601 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c00eb20 con 0x562ec239a470 2023-10-11T16:33:35.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f6014ff9700 1 -- 172.21.15.111:0/688875601 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f601c021070 con 0x562ec239a470 2023-10-11T16:33:35.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f6014ff9700 1 -- 172.21.15.111:0/688875601 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c01d610 con 0x562ec239a470 2023-10-11T16:33:35.683 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f747c35bb80 1 -- 172.21.15.111:0/2848469975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 msgr2=0x5640a9e40950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.683 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.668+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e40950 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f74540964a0 tx=0x7f7454099bb0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f747c35bb80 1 -- 172.21.15.111:0/2848469975 shutdown_connections 2023-10-11T16:33:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e33ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e40950 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2848469975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e33560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f747c35bb80 1 -- 172.21.15.111:0/2848469975 >> 172.21.15.111:0/2848469975 conn(0x5640a9e28f50 msgr2=0x5640a9e2e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 -- 172.21.15.111:0/688875601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 msgr2=0x562ec239e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec239e370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f601c011d50 tx=0x7f601c014000 comp rx=0 tx=0).stop 2023-10-11T16:33:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f747c35bb80 1 -- 172.21.15.111:0/2848469975 shutdown_connections 2023-10-11T16:33:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f747c35bb80 1 -- 172.21.15.111:0/2848469975 wait complete. 2023-10-11T16:33:35.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 -- 172.21.15.111:0/688875601 shutdown_connections 2023-10-11T16:33:35.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec239ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec2399f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 --2- 172.21.15.111:0/688875601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec239e370 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 -- 172.21.15.111:0/688875601 >> 172.21.15.111:0/688875601 conn(0x562ec238f360 msgr2=0x562ec2394b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 -- 172.21.15.111:0/688875601 shutdown_connections 2023-10-11T16:33:35.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 -- 172.21.15.111:0/688875601 wait complete. 2023-10-11T16:33:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f747c35bb80 1 -- start start 2023-10-11T16:33:35.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.669+0000 7f6040619b80 1 -- start start 2023-10-11T16:33:35.689 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f747c35bb80 1 Processor -- start 2023-10-11T16:33:35.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6040619b80 1 Processor -- start 2023-10-11T16:33:35.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6040619b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6040619b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 0x562ec23a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6040619b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6040619b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec228c8e0 con 0x562ec239e8b0 2023-10-11T16:33:35.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6040619b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec228c580 con 0x562ec2399b60 2023-10-11T16:33:35.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6040619b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec216e920 con 0x562ec239a470 2023-10-11T16:33:35.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6040619b80 10 client.0 _ll_register_callbacks cb 0x562ec22aa780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e482f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e45500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e45c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f747c35bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9d26880 con 0x5640a9c8f6b0 2023-10-11T16:33:35.693 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f747c35bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9d26520 con 0x5640a9e33aa0 2023-10-11T16:33:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f747c35bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9c088c0 con 0x5640a9e40e90 2023-10-11T16:33:35.694 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f747c35bb80 10 client.0 _ll_register_callbacks cb 0x5640a9d44720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:35.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6026d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6026d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45404/0 (socket says 172.21.15.111:45404) 2023-10-11T16:33:35.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f6026d9d700 1 -- 172.21.15.111:0/3583751499 learned_addr learned my addr 172.21.15.111:0/3583751499 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.670+0000 7f602659c700 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f6025d9b700 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 0x562ec23a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7461d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e45500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.696 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7461d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e45500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54128/0 (socket says 172.21.15.111:54128) 2023-10-11T16:33:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7461d9b700 1 -- 172.21.15.111:0/1431727393 learned_addr learned my addr 172.21.15.111:0/1431727393 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7462d9d700 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e45c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f746259c700 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e482f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7462d9d700 1 -- 172.21.15.111:0/1431727393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 msgr2=0x5640a9e45500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7462d9d700 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e45500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7462d9d700 1 -- 172.21.15.111:0/1431727393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 msgr2=0x5640a9e482f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f602659c700 1 -- 172.21.15.111:0/3583751499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 msgr2=0x562ec23a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f602659c700 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 0x562ec23a32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f602659c700 1 -- 172.21.15.111:0/3583751499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 msgr2=0x562ec23a3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f602659c700 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f602659c700 1 -- 172.21.15.111:0/3583751499 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601c017070 con 0x562ec2399b60 2023-10-11T16:33:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7462d9d700 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e482f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f7462d9d700 1 -- 172.21.15.111:0/1431727393 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7464110140 con 0x5640a9e40e90 2023-10-11T16:33:35.700 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f746259c700 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:35.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f602659c700 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a6060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f60180cadd0 tx=0x7f6018098cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21d205b53fa1eb4f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.671+0000 7f600dffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f7462d9d700 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e45c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f745c007c00 tx=0x7f745c007c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da7638d025428e59 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.701 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f7449ffb700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:33:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f600dffb700 1 -- 172.21.15.111:0/3583751499 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6018099e00 con 0x562ec2399b60 2023-10-11T16:33:35.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f600dffb700 1 -- 172.21.15.111:0/3583751499 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f601809e740 con 0x562ec2399b60 2023-10-11T16:33:35.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 1 -- 172.21.15.111:0/3583751499 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ec238cb20 con 0x562ec2399b60 2023-10-11T16:33:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f7449ffb700 1 -- 172.21.15.111:0/1431727393 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c00e040 con 0x5640a9e40e90 2023-10-11T16:33:35.703 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f747c35bb80 1 -- 172.21.15.111:0/1431727393 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5640a9e26710 con 0x5640a9e40e90 2023-10-11T16:33:35.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f600dffb700 1 -- 172.21.15.111:0/3583751499 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60180a2ad0 con 0x562ec2399b60 2023-10-11T16:33:35.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 10 client.25787 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:35.704 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f7449ffb700 1 -- 172.21.15.111:0/1431727393 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f745c004db0 con 0x5640a9e40e90 2023-10-11T16:33:35.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 1 -- 172.21.15.111:0/3583751499 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562ec22b1190 con 0x562ec2399b60 2023-10-11T16:33:35.705 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f7449ffb700 1 -- 172.21.15.111:0/1431727393 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f745c003a00 con 0x5640a9e40e90 2023-10-11T16:33:35.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f600dffb700 1 -- 172.21.15.111:0/3583751499 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562ec238cb20 con 0x562ec2399b60 2023-10-11T16:33:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f747c35bb80 10 client.25822 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:35.706 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f747c35bb80 1 -- 172.21.15.111:0/1431727393 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5640a9d4b130 con 0x5640a9e40e90 2023-10-11T16:33:35.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:35 smithi150 ceph-mon[132217]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 9 op/s 2023-10-11T16:33:35.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f600dffb700 1 -- 172.21.15.111:0/3583751499 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f60180cb0b0 con 0x562ec2399b60 2023-10-11T16:33:35.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 10 client.25787 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:35.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 20 client.25787 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562ec23b3480 0x562ec23b5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 4 client.25787 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:35.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f6040619b80 1 -- 172.21.15.111:0/3583751499 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ec22ad0a0 con 0x562ec23b3480 2023-10-11T16:33:35.710 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.672+0000 7f7449ffb700 1 -- 172.21.15.111:0/1431727393 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5640a9e26710 con 0x5640a9e40e90 2023-10-11T16:33:35.710 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f7449ffb700 1 -- 172.21.15.111:0/1431727393 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f745c01e020 con 0x5640a9e40e90 2023-10-11T16:33:35.710 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f747c35bb80 10 client.25822 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:35.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f6025d9b700 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562ec23b3480 0x562ec23b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f747c35bb80 10 client.25822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f747c35bb80 20 client.25822 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5640a9e55a20 0x5640a9e57e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f747c35bb80 4 client.25822 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f747c35bb80 1 -- 172.21.15.111:0/1431727393 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640a9d47040 con 0x5640a9e55a20 2023-10-11T16:33:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f746259c700 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5640a9e55a20 0x5640a9e57e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f6025d9b700 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562ec23b3480 0x562ec23b5890 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30d8399ec3fdba5d server_cookie=fe7ea2c65dea7810 in_seq=0 out_seq=0 2023-10-11T16:33:35.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.673+0000 7f600dffb700 10 client.25787 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f7449ffb700 10 client.25822 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f746259c700 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5640a9e55a20 0x5640a9e57e30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=119455abfcb81b67 server_cookie=ff90449832d1b5dd in_seq=0 out_seq=0 2023-10-11T16:33:35.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f600dffb700 1 -- 172.21.15.111:0/3583751499 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562ec22ad0a0 con 0x562ec23b3480 2023-10-11T16:33:35.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f600dffb700 10 client.25787 handle_command_reply: tid=0 2023-10-11T16:33:35.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 client.25787 shutdown 2023-10-11T16:33:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f7449ffb700 1 -- 172.21.15.111:0/1431727393 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5640a9d47040 con 0x5640a9e55a20 2023-10-11T16:33:35.715 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f7449ffb700 10 client.25822 handle_command_reply: tid=0 2023-10-11T16:33:35.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 -- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 msgr2=0x562ec23a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a6060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f60180cadd0 tx=0x7f6018098cc0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.717 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f747c35bb80 1 client.25822 shutdown 2023-10-11T16:33:35.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 -- 172.21.15.111:0/3583751499 shutdown_connections 2023-10-11T16:33:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562ec23b3480 0x562ec23b5890 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a6060 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 --2- 172.21.15.111:0/3583751499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 0x562ec23a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f6040619b80 1 -- 172.21.15.111:0/3583751499 >> 172.21.15.111:0/3583751499 conn(0x562ec238f360 msgr2=0x562ec2398960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.674+0000 7f600dffb700 0 client.25787 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:35.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f6040619b80 1 -- 172.21.15.111:0/3583751499 shutdown_connections 2023-10-11T16:33:35.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 -- 172.21.15.111:0/1431727393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 msgr2=0x5640a9e45c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e45c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f745c007c00 tx=0x7f745c007c30 comp rx=0 tx=0).stop 2023-10-11T16:33:35.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f6040619b80 1 -- 172.21.15.111:0/3583751499 wait complete. 2023-10-11T16:33:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 -- 172.21.15.111:0/1431727393 shutdown_connections 2023-10-11T16:33:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5640a9e55a20 0x5640a9e57e30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e45c40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e482f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1431727393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e45500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f7449ffb700 0 client.25822 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 -- 172.21.15.111:0/1431727393 >> 172.21.15.111:0/1431727393 conn(0x5640a9e28f50 msgr2=0x5640a9e3f600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 -- 172.21.15.111:0/1431727393 shutdown_connections 2023-10-11T16:33:35.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 1 -- 172.21.15.111:0/1431727393 wait complete. 2023-10-11T16:33:35.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f6040619b80 20 client.25787 trim_cache size 0 max 16384 2023-10-11T16:33:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.675+0000 7f747c35bb80 20 client.25822 trim_cache size 0 max 16384 2023-10-11T16:33:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.690+0000 7f747c35bb80 1 Processor -- start 2023-10-11T16:33:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f747c35bb80 1 -- start start 2023-10-11T16:33:35.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e33e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e47e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f747c35bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9e53010 con 0x5640a9c8f6b0 2023-10-11T16:33:35.727 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f747c35bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9e321c0 con 0x5640a9e33aa0 2023-10-11T16:33:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f747c35bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9e46220 con 0x5640a9e40e90 2023-10-11T16:33:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f746259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f746259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45434/0 (socket says 172.21.15.111:45434) 2023-10-11T16:33:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f746259c700 1 -- 172.21.15.111:0/1753214227 learned_addr learned my addr 172.21.15.111:0/1753214227 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.728 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f7462d9d700 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e47e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.691+0000 7f7461d9b700 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e33e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f746259c700 1 -- 172.21.15.111:0/1753214227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 msgr2=0x5640a9e47e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.729 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f746259c700 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e47e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f746259c700 1 -- 172.21.15.111:0/1753214227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 msgr2=0x5640a9e33e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f746259c700 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e33e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f746259c700 1 -- 172.21.15.111:0/1753214227 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640a9d4ae80 con 0x5640a9c8f6b0 2023-10-11T16:33:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f7462d9d700 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e47e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:35.730 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f746259c700 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4c950 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7454099ca0 tx=0x7f7454096410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6daff524c08ec3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f74517fa700 1 -- 172.21.15.111:0/1753214227 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74540a23f0 con 0x5640a9c8f6b0 2023-10-11T16:33:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f74517fa700 1 -- 172.21.15.111:0/1753214227 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74540cd1b0 con 0x5640a9c8f6b0 2023-10-11T16:33:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.692+0000 7f74517fa700 1 -- 172.21.15.111:0/1753214227 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74540a21c0 con 0x5640a9c8f6b0 2023-10-11T16:33:35.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f747c35bb80 1 -- 172.21.15.111:0/1753214227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 msgr2=0x5640a9e4c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4c950 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7454099ca0 tx=0x7f7454096410 comp rx=0 tx=0).stop 2023-10-11T16:33:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f747c35bb80 1 -- 172.21.15.111:0/1753214227 shutdown_connections 2023-10-11T16:33:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e47e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4c950 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f747c35bb80 1 --2- 172.21.15.111:0/1753214227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9e33aa0 0x5640a9e33e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f747c35bb80 1 -- 172.21.15.111:0/1753214227 >> 172.21.15.111:0/1753214227 conn(0x5640a9e28f50 msgr2=0x5640a9e31ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.733 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f747c35bb80 1 -- 172.21.15.111:0/1753214227 shutdown_connections 2023-10-11T16:33:35.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f6040619b80 1 Processor -- start 2023-10-11T16:33:35.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.693+0000 7f6040619b80 1 -- start start 2023-10-11T16:33:35.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6040619b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6040619b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 0x562ec23aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6040619b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6040619b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec2395a10 con 0x562ec239e8b0 2023-10-11T16:33:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6040619b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec239a8b0 con 0x562ec2399b60 2023-10-11T16:33:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6040619b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec238ef10 con 0x562ec239a470 2023-10-11T16:33:35.736 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f747c35bb80 1 -- 172.21.15.111:0/1753214227 wait complete. 2023-10-11T16:33:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6026d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6026d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45444/0 (socket says 172.21.15.111:45444) 2023-10-11T16:33:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6026d9d700 1 -- 172.21.15.111:0/3937434281 learned_addr learned my addr 172.21.15.111:0/3937434281 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6025d9b700 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 0x562ec23aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.738 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f747c35bb80 1 -- start start 2023-10-11T16:33:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f602659c700 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6026d9d700 1 -- 172.21.15.111:0/3937434281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 msgr2=0x562ec23aa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6026d9d700 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 0x562ec23aa9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6026d9d700 1 -- 172.21.15.111:0/3937434281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 msgr2=0x562ec23a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6026d9d700 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f6026d9d700 1 -- 172.21.15.111:0/3937434281 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ec22b0ee0 con 0x562ec239e8b0 2023-10-11T16:33:35.740 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.694+0000 7f747c35bb80 1 Processor -- start 2023-10-11T16:33:35.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f6026d9d700 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a5c00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6020003b90 tx=0x7f6020009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77a5d4f07211ed1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.741 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:35.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f60157fa700 1 -- 172.21.15.111:0/3937434281 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602000e040 con 0x562ec239e8b0 2023-10-11T16:33:35.742 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f60157fa700 1 -- 172.21.15.111:0/3937434281 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f602000f040 con 0x562ec239e8b0 2023-10-11T16:33:35.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f60157fa700 1 -- 172.21.15.111:0/3937434281 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020020470 con 0x562ec239e8b0 2023-10-11T16:33:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e4ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f747c35bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e4b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f747c35bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9e4cef0 con 0x5640a9e33aa0 2023-10-11T16:33:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f747c35bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9ea88e0 con 0x5640a9c8f6b0 2023-10-11T16:33:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f747c35bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640a9ea8a20 con 0x5640a9e40e90 2023-10-11T16:33:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.695+0000 7f747c35bb80 10 client.0 _ll_register_callbacks cb 0x7f745c0093f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e4ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e4ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45460/0 (socket says 172.21.15.111:45460) 2023-10-11T16:33:35.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 -- 172.21.15.111:0/2916369734 learned_addr learned my addr 172.21.15.111:0/2916369734 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7462d9d700 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e4b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f746259c700 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 -- 172.21.15.111:0/2916369734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 msgr2=0x5640a9e4b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e4b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 -- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 msgr2=0x5640a9e4a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 -- 172.21.15.111:0/2916369734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640a9d4b130 con 0x5640a9e33aa0 2023-10-11T16:33:35.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f6040619b80 1 -- 172.21.15.111:0/3937434281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 msgr2=0x562ec23a5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f6040619b80 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a5c00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6020003b90 tx=0x7f6020009920 comp rx=0 tx=0).stop 2023-10-11T16:33:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f746259c700 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:35.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f6040619b80 1 -- 172.21.15.111:0/3937434281 shutdown_connections 2023-10-11T16:33:35.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f6040619b80 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec23a5c00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f6040619b80 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec2399b60 0x562ec23a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f6040619b80 1 --2- 172.21.15.111:0/3937434281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec239a470 0x562ec23aa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f6040619b80 1 -- 172.21.15.111:0/3937434281 >> 172.21.15.111:0/3937434281 conn(0x562ec238f360 msgr2=0x562ec23b5490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7461d9b700 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e4ae80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7458009990 tx=0x7f7458009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6292e31a28ada0df server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.696+0000 7f7449ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:35.752 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:35.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f6040619b80 1 -- 172.21.15.111:0/3937434281 shutdown_connections 2023-10-11T16:33:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f7449ffb700 1 -- 172.21.15.111:0/2916369734 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745802d030 con 0x5640a9e33aa0 2023-10-11T16:33:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f747c35bb80 1 -- 172.21.15.111:0/2916369734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f745409dfd0 con 0x5640a9e33aa0 2023-10-11T16:33:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f7449ffb700 1 -- 172.21.15.111:0/2916369734 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f7458034680 con 0x5640a9e33aa0 2023-10-11T16:33:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f7449ffb700 1 -- 172.21.15.111:0/2916369734 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7458004e80 con 0x5640a9e33aa0 2023-10-11T16:33:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f747c35bb80 10 client.15759 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f747c35bb80 1 -- 172.21.15.111:0/2916369734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5640a9c065d0 con 0x5640a9e33aa0 2023-10-11T16:33:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f7449ffb700 1 -- 172.21.15.111:0/2916369734 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f745409dfd0 con 0x5640a9e33aa0 2023-10-11T16:33:35.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f6040619b80 1 -- 172.21.15.111:0/3937434281 wait complete. 2023-10-11T16:33:35.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f6040619b80 1 -- start start 2023-10-11T16:33:35.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.697+0000 7f6040619b80 1 Processor -- start 2023-10-11T16:33:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f7449ffb700 1 -- 172.21.15.111:0/2916369734 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f745800f990 con 0x5640a9e33aa0 2023-10-11T16:33:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f747c35bb80 10 client.15759 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f747c35bb80 10 client.15759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f747c35bb80 20 client.15759 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5640a9eaa1c0 0x5640a9eac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f747c35bb80 4 client.15759 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:33:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f747c35bb80 1 -- 172.21.15.111:0/2916369734 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5640a9d53600 con 0x5640a9eaa1c0 2023-10-11T16:33:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f746259c700 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5640a9eaa1c0 0x5640a9eac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f6040619b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec23a9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f6040619b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec23aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f6040619b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec2406300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f6040619b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec23aa810 con 0x562ec239e8b0 2023-10-11T16:33:35.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f6040619b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec23aa980 con 0x562ec239a470 2023-10-11T16:33:35.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f6040619b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec2406840 con 0x562ec2399b60 2023-10-11T16:33:35.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.698+0000 7f6040619b80 10 client.0 _ll_register_callbacks cb 0x562ec22aa780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f7449ffb700 10 client.15759 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f746259c700 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5640a9eaa1c0 0x5640a9eac5d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66b1a95e9932941 server_cookie=3b1b80b428d0519b in_seq=0 out_seq=0 2023-10-11T16:33:35.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f602659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec23a9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec23aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec23aa0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54158/0 (socket says 172.21.15.111:54158) 2023-10-11T16:33:35.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 -- 172.21.15.111:0/2636115081 learned_addr learned my addr 172.21.15.111:0/2636115081 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:35.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f602659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec23a9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56188/0 (socket says 172.21.15.111:56188) 2023-10-11T16:33:35.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6026d9d700 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec2406300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 -- 172.21.15.111:0/2636115081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 msgr2=0x562ec23a9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec23a9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 -- 172.21.15.111:0/2636115081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 msgr2=0x562ec2406300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:35.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec2406300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 -- 172.21.15.111:0/2636115081 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60200093c0 con 0x562ec239a470 2023-10-11T16:33:35.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6026d9d700 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec2406300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:33:35.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f602659c700 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec23a9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:35.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f6025d9b700 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec23aa0a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f601c011d20 tx=0x7f601c01ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9149c73ac049eda server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:35.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.699+0000 7f600dffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:35.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.700+0000 7f600dffb700 1 -- 172.21.15.111:0/2636115081 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c00ebe0 con 0x562ec239a470 2023-10-11T16:33:35.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.700+0000 7f600dffb700 1 -- 172.21.15.111:0/2636115081 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f601c00ed40 con 0x562ec239a470 2023-10-11T16:33:35.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.700+0000 7f6040619b80 1 -- 172.21.15.111:0/2636115081 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f601c021070 con 0x562ec239a470 2023-10-11T16:33:35.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.700+0000 7f600dffb700 1 -- 172.21.15.111:0/2636115081 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f601c033ae0 con 0x562ec239a470 2023-10-11T16:33:35.766 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:35.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.700+0000 7f6040619b80 10 client.25802 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:35.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.700+0000 7f6040619b80 1 -- 172.21.15.111:0/2636115081 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562ec2101c60 con 0x562ec239a470 2023-10-11T16:33:35.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f600dffb700 1 -- 172.21.15.111:0/2636115081 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f601c021070 con 0x562ec239a470 2023-10-11T16:33:35.767 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:35.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f600dffb700 1 -- 172.21.15.111:0/2636115081 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f601c01c5b0 con 0x562ec239a470 2023-10-11T16:33:35.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f6040619b80 10 client.25802 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:35.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f6040619b80 10 client.25802 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:35.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f6040619b80 20 client.25802 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:35.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f6040619b80 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562ec2407f40 0x562ec240a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:35.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f6040619b80 4 client.25802 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:33:35.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f6040619b80 1 -- 172.21.15.111:0/2636115081 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562ec22b9660 con 0x562ec2407f40 2023-10-11T16:33:35.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:35.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.701+0000 7f602659c700 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562ec2407f40 0x562ec240a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:35.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.702+0000 7f602659c700 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562ec2407f40 0x562ec240a350 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=569d3313fa955f39 server_cookie=9d713ebe75da0b9b in_seq=0 out_seq=0 2023-10-11T16:33:35.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.702+0000 7f600dffb700 10 client.25802 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:35.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.706+0000 7f7449ffb700 1 -- 172.21.15.111:0/2916369734 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5640a9d53600 con 0x5640a9eaa1c0 2023-10-11T16:33:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.706+0000 7f7449ffb700 10 client.15759 handle_command_reply: tid=0 2023-10-11T16:33:35.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.706+0000 7f600dffb700 1 -- 172.21.15.111:0/2636115081 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562ec22b9660 con 0x562ec2407f40 2023-10-11T16:33:35.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.706+0000 7f600dffb700 10 client.25802 handle_command_reply: tid=0 2023-10-11T16:33:35.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.706+0000 7f6040619b80 1 client.25802 shutdown 2023-10-11T16:33:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.706+0000 7f747c35bb80 1 client.15759 shutdown 2023-10-11T16:33:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 -- 172.21.15.111:0/2916369734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 msgr2=0x5640a9e4ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e4ae80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7458009990 tx=0x7f7458009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 -- 172.21.15.111:0/2636115081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 msgr2=0x562ec23aa0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec23aa0a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f601c011d20 tx=0x7f601c01ccb0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 -- 172.21.15.111:0/2636115081 shutdown_connections 2023-10-11T16:33:35.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562ec2407f40 0x562ec240a350 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562ec239e8b0 0x562ec2406300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562ec2399b60 0x562ec23a9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 --2- 172.21.15.111:0/2636115081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562ec239a470 0x562ec23aa0a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 -- 172.21.15.111:0/2636115081 >> 172.21.15.111:0/2636115081 conn(0x562ec238f360 msgr2=0x562ec2398960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f600dffb700 0 client.25802 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 -- 172.21.15.111:0/2916369734 shutdown_connections 2023-10-11T16:33:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5640a9eaa1c0 0x5640a9eac5d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5640a9e40e90 0x5640a9e4b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5640a9c8f6b0 0x5640a9e4a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f7449ffb700 0 client.15759 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 --2- 172.21.15.111:0/2916369734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5640a9e33aa0 0x5640a9e4ae80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 -- 172.21.15.111:0/2916369734 >> 172.21.15.111:0/2916369734 conn(0x5640a9e28f50 msgr2=0x5640a9e33e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 -- 172.21.15.111:0/2636115081 shutdown_connections 2023-10-11T16:33:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 -- 172.21.15.111:0/2916369734 shutdown_connections 2023-10-11T16:33:35.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 1 -- 172.21.15.111:0/2636115081 wait complete. 2023-10-11T16:33:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 1 -- 172.21.15.111:0/2916369734 wait complete. 2023-10-11T16:33:35.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f6040619b80 20 client.25802 trim_cache size 0 max 16384 2023-10-11T16:33:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.707+0000 7f747c35bb80 20 client.15759 trim_cache size 0 max 16384 2023-10-11T16:33:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:33:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:33:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:33:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:33:35.782 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:33:35.782 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:33:35.782 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:33:35.782 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:33:35.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 -- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f744c0a1460 msgr2=0x7f744c0a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f744c0a1460 0x7f744c0a3910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7464197990 tx=0x7f7458005fb0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.783 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 -- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7464101440 msgr2=0x7f7464196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.784 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7464101440 0x7f7464196380 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f745400e9a0 tx=0x7f745400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 -- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60100a13e0 msgr2=0x7f60100a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60100a13e0 0x7f60100a3890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6028197a30 tx=0x7f601c009260 comp rx=0 tx=0).stop 2023-10-11T16:33:35.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 -- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6028100a50 msgr2=0x7f6028196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:35.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6028100a50 0x7f6028196420 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6018000c00 tx=0x7f601800bfd0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.785 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 -- 172.21.15.111:0/2769950301 shutdown_connections 2023-10-11T16:33:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f744c0a1460 0x7f744c0a3910 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f746410a810 0x7f746419ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7464101440 0x7f7464196380 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 --2- 172.21.15.111:0/2769950301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7464101d50 0x7f74641968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 -- 172.21.15.111:0/2769950301 >> 172.21.15.111:0/2769950301 conn(0x7f74640763a0 msgr2=0x7f74640fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 -- 172.21.15.111:0/205175713 shutdown_connections 2023-10-11T16:33:35.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60100a13e0 0x7f60100a3890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60281063f0 0x7f602819adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6028100a50 0x7f6028196420 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 --2- 172.21.15.111:0/205175713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6028101400 0x7f6028196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:35.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f602cbff700 1 -- 172.21.15.111:0/205175713 >> 172.21.15.111:0/205175713 conn(0x7f6028077590 msgr2=0x7f60280ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 -- 172.21.15.111:0/2769950301 shutdown_connections 2023-10-11T16:33:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:35.708+0000 7f74689ff700 1 -- 172.21.15.111:0/2769950301 wait complete. 2023-10-11T16:33:35.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.709+0000 7f602cbff700 1 -- 172.21.15.111:0/205175713 shutdown_connections 2023-10-11T16:33:35.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:35.709+0000 7f602cbff700 1 -- 172.21.15.111:0/205175713 wait complete. 2023-10-11T16:33:35.790 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:33:35.791 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:33:36.561 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:36.562 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:36.705 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:36.705 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:36.730 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:36.730 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:36.731 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 15 -lt 100 ']' 2023-10-11T16:33:36.731 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 15 + 1 2023-10-11T16:33:36.748 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=16 2023-10-11T16:33:36.749 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:36.823 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:36.824 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:36.826 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:36.827 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:36.827 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:37.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:37.370 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:37.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:37 smithi130 ceph-mon[131859]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 7 op/s 2023-10-11T16:33:37.594 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:37.595 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:37.604 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:37.604 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:37.604 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 16 -lt 100 ']' 2023-10-11T16:33:37.605 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 16 + 1 2023-10-11T16:33:37.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:37 smithi111 ceph-mon[98527]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 7 op/s 2023-10-11T16:33:37.663 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=17 2023-10-11T16:33:37.663 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:37 smithi150 ceph-mon[132217]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 7 op/s 2023-10-11T16:33:37.734 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:37.735 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:37.737 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:37.738 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:37.738 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:38.461 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:38.461 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:38.682 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:38.682 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:38.712 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:38.712 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:38.713 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 17 -lt 100 ']' 2023-10-11T16:33:38.713 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 17 + 1 2023-10-11T16:33:38.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 -- 172.21.15.111:0/3835873243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f8410a880 msgr2=0x7f4f8410cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/3835873243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f8410a880 0x7f4f8410cca0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00b0a0 tx=0x7f4f7c00b3b0 comp rx=0 tx=0).stop 2023-10-11T16:33:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 -- 172.21.15.111:0/3835873243 shutdown_connections 2023-10-11T16:33:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/3835873243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f8410a880 0x7f4f8410cca0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/3835873243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f84101360 0x7f4f84101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/3835873243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f84101c70 0x7f4f8410a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 -- 172.21.15.111:0/3835873243 >> 172.21.15.111:0/3835873243 conn(0x7f4f84076310 msgr2=0x7f4f84076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 -- 172.21.15.111:0/3835873243 shutdown_connections 2023-10-11T16:33:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.817+0000 7f4f8bfff700 1 -- 172.21.15.111:0/3835873243 wait complete. 2023-10-11T16:33:38.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.818+0000 7f4f8bfff700 1 Processor -- start 2023-10-11T16:33:38.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.818+0000 7f4f8bfff700 1 -- start start 2023-10-11T16:33:38.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.818+0000 7f4f8bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f84101360 0x7f4f840ff190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:38.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.818+0000 7f4f8bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f84101c70 0x7f4f840ff6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:38.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.818+0000 7f4f8bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8410a880 0x7f4f841a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:38.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.818+0000 7f4f8bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f840fff00 con 0x7f4f84101c70 2023-10-11T16:33:38.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.818+0000 7f4f8bfff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f841a4c00 con 0x7f4f84101360 2023-10-11T16:33:38.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.818+0000 7f4f8bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f841a4d40 con 0x7f4f8410a880 2023-10-11T16:33:38.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8959a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f84101c70 0x7f4f840ff6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:38.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8410a880 0x7f4f841a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:38.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8959a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f84101c70 0x7f4f840ff6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45482/0 (socket says 172.21.15.111:45482) 2023-10-11T16:33:38.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8959a700 1 -- 172.21.15.111:0/831258125 learned_addr learned my addr 172.21.15.111:0/831258125 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:38.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f89d9b700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f84101360 0x7f4f840ff190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:38.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8a59c700 1 -- 172.21.15.111:0/831258125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f84101360 msgr2=0x7f4f840ff190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:38.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8a59c700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f84101360 0x7f4f840ff190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:38.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8a59c700 1 -- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f84101c70 msgr2=0x7f4f840ff6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:38.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8a59c700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f84101c70 0x7f4f840ff6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:38.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8a59c700 1 -- 172.21.15.111:0/831258125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c009d00 con 0x7f4f8410a880 2023-10-11T16:33:38.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8959a700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f84101c70 0x7f4f840ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:38.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.819+0000 7f4f8a59c700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8410a880 0x7f4f841a46c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00b070 tx=0x7f4f7c007930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:38.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.820+0000 7f4f7affd700 1 -- 172.21.15.111:0/831258125 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c00e040 con 0x7f4f8410a880 2023-10-11T16:33:38.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.820+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f84110320 con 0x7f4f8410a880 2023-10-11T16:33:38.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.820+0000 7f4f7affd700 1 -- 172.21.15.111:0/831258125 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f7c01d470 con 0x7f4f8410a880 2023-10-11T16:33:38.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.820+0000 7f4f7affd700 1 -- 172.21.15.111:0/831258125 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0052a0 con 0x7f4f8410a880 2023-10-11T16:33:38.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.820+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f84110190 con 0x7f4f8410a880 2023-10-11T16:33:38.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.821+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f841104b0 con 0x7f4f8410a880 2023-10-11T16:33:38.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.823+0000 7f4f7affd700 1 -- 172.21.15.111:0/831258125 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f4f7c01d5e0 con 0x7f4f8410a880 2023-10-11T16:33:38.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.824+0000 7f4f7affd700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f700a13e0 0x7f4f700a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:38.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.825+0000 7f4f7affd700 1 -- 172.21.15.111:0/831258125 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f4f7c0ce810 con 0x7f4f8410a880 2023-10-11T16:33:38.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.825+0000 7f4f89d9b700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f700a13e0 0x7f4f700a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:38.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.825+0000 7f4f89d9b700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f700a13e0 0x7f4f700a3890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009c60 tx=0x7f4f80009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:38.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:38.836+0000 7f4f7affd700 1 -- 172.21.15.111:0/831258125 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4f7c097650 con 0x7f4f8410a880 2023-10-11T16:33:38.855 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=18 2023-10-11T16:33:38.855 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:38.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:38.887 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:38.889 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:38.889 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:38.890 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:39.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.090+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f84067b00 con 0x7f4f8410a880 2023-10-11T16:33:39.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.091+0000 7f4f7affd700 1 -- 172.21.15.111:0/831258125 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10820 (secure 0 0 0) 0x7f4f7c09ac70 con 0x7f4f8410a880 2023-10-11T16:33:39.093 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:33:39.094 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:33:01.126571+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:33:39.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f700a13e0 msgr2=0x7f4f700a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:39.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f700a13e0 0x7f4f700a3890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009c60 tx=0x7f4f80009260 comp rx=0 tx=0).stop 2023-10-11T16:33:39.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8410a880 msgr2=0x7f4f841a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:39.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8410a880 0x7f4f841a46c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00b070 tx=0x7f4f7c007930 comp rx=0 tx=0).stop 2023-10-11T16:33:39.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 shutdown_connections 2023-10-11T16:33:39.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f700a13e0 0x7f4f700a3890 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:39.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f8410a880 0x7f4f841a46c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f84101360 0x7f4f840ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:39.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 --2- 172.21.15.111:0/831258125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f84101c70 0x7f4f840ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:39.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 >> 172.21.15.111:0/831258125 conn(0x7f4f84076310 msgr2=0x7f4f84102260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:39.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 shutdown_connections 2023-10-11T16:33:39.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:39.094+0000 7f4f8bfff700 1 -- 172.21.15.111:0/831258125 wait complete. 2023-10-11T16:33:39.101 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 28 2023-10-11T16:33:39.143 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:33:39.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:39.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:39.451 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:39.451 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:39.463 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:39.463 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:39.463 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 18 -lt 100 ']' 2023-10-11T16:33:39.464 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 18 + 1 2023-10-11T16:33:39.504 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=19 2023-10-11T16:33:39.504 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:39.512 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:39.513 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:39.514 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:39.515 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:39.515 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:39.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:39 smithi130 ceph-mon[131859]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2023-10-11T16:33:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:39 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/831258125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:33:39.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:39 smithi111 ceph-mon[98527]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2023-10-11T16:33:39.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:39 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/831258125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:33:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:39 smithi150 ceph-mon[132217]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2023-10-11T16:33:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:39 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/831258125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:33:40.019 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:40.020 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:40.110 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:40.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:40.154 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:40.154 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:40.154 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 19 -lt 100 ']' 2023-10-11T16:33:40.155 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 19 + 1 2023-10-11T16:33:40.187 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=20 2023-10-11T16:33:40.188 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:40.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:40.215 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:40.217 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:40.218 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:40.220 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:40.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:40.886 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:41.094 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:41.094 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:41.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:41.112 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:41.112 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 20 -lt 100 ']' 2023-10-11T16:33:41.112 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 20 + 1 2023-10-11T16:33:41.162 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=21 2023-10-11T16:33:41.163 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:41.435 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:41.435 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:41.437 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:41.438 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:41.438 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:41.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:41 smithi130 ceph-mon[131859]: pgmap v34: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-10-11T16:33:41.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:41 smithi111 ceph-mon[98527]: pgmap v34: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-10-11T16:33:41.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:41 smithi150 ceph-mon[132217]: pgmap v34: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 11 op/s 2023-10-11T16:33:42.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:42.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:42.234 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:42.235 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:42.246 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:42.246 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:42.247 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 21 -lt 100 ']' 2023-10-11T16:33:42.247 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 21 + 1 2023-10-11T16:33:42.312 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=22 2023-10-11T16:33:42.312 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:42.401 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:42.402 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:42.404 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:42.404 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:42.405 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:42.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 -- 172.21.15.111:0/2915378667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d8074ac0 msgr2=0x7fe6d8073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 --2- 172.21.15.111:0/2915378667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d8074ac0 0x7fe6d8073260 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009990 tx=0x7fe6c4009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 -- 172.21.15.111:0/2915378667 shutdown_connections 2023-10-11T16:33:42.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 --2- 172.21.15.111:0/2915378667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d810cec0 0x7fe6d810f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 --2- 172.21.15.111:0/2915378667 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d80746f0 0x7fe6d8072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 --2- 172.21.15.111:0/2915378667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d8074ac0 0x7fe6d8073260 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 -- 172.21.15.111:0/2915378667 >> 172.21.15.111:0/2915378667 conn(0x7fe6d8077560 msgr2=0x7fe6d8077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:42.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 -- 172.21.15.111:0/2915378667 shutdown_connections 2023-10-11T16:33:42.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.474+0000 7fe6debff700 1 -- 172.21.15.111:0/2915378667 wait complete. 2023-10-11T16:33:42.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.475+0000 7fe6debff700 1 Processor -- start 2023-10-11T16:33:42.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.475+0000 7fe6debff700 1 -- start start 2023-10-11T16:33:42.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6debff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d80746f0 0x7fe6d819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6debff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d8074ac0 0x7fe6d819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6debff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d810cec0 0x7fe6d8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6debff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d8198d40 con 0x7fe6d810cec0 2023-10-11T16:33:42.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6debff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d8198eb0 con 0x7fe6d80746f0 2023-10-11T16:33:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6debff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d8199070 con 0x7fe6d8074ac0 2023-10-11T16:33:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dd19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d810cec0 0x7fe6d8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6d7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d8074ac0 0x7fe6d819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dc99b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d80746f0 0x7fe6d819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dd19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d810cec0 0x7fe6d8198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45510/0 (socket says 172.21.15.111:45510) 2023-10-11T16:33:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dd19c700 1 -- 172.21.15.111:0/3719968924 learned_addr learned my addr 172.21.15.111:0/3719968924 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6d7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d8074ac0 0x7fe6d819ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56216/0 (socket says 172.21.15.111:56216) 2023-10-11T16:33:42.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dd19c700 1 -- 172.21.15.111:0/3719968924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d8074ac0 msgr2=0x7fe6d819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d8074ac0 0x7fe6d819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dd19c700 1 -- 172.21.15.111:0/3719968924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d80746f0 msgr2=0x7fe6d819e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d80746f0 0x7fe6d819e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.476+0000 7fe6dd19c700 1 -- 172.21.15.111:0/3719968924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c40095f0 con 0x7fe6d810cec0 2023-10-11T16:33:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.477+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d810cec0 0x7fe6d8198570 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000e940 tx=0x7fe6d000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.477+0000 7fe6d5ffb700 1 -- 172.21.15.111:0/3719968924 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0009750 con 0x7fe6d810cec0 2023-10-11T16:33:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.477+0000 7fe6d5ffb700 1 -- 172.21.15.111:0/3719968924 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6d0005020 con 0x7fe6d810cec0 2023-10-11T16:33:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.477+0000 7fe6debff700 1 -- 172.21.15.111:0/3719968924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d81125f0 con 0x7fe6d810cec0 2023-10-11T16:33:42.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.477+0000 7fe6d5ffb700 1 -- 172.21.15.111:0/3719968924 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d000cbc0 con 0x7fe6d810cec0 2023-10-11T16:33:42.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.477+0000 7fe6debff700 1 -- 172.21.15.111:0/3719968924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d8112460 con 0x7fe6d810cec0 2023-10-11T16:33:42.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.480+0000 7fe6d5ffb700 1 -- 172.21.15.111:0/3719968924 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7fe6d0005400 con 0x7fe6d810cec0 2023-10-11T16:33:42.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.481+0000 7fe6d5ffb700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe6c80a13e0 0x7fe6c80a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.481+0000 7fe6d5ffb700 1 -- 172.21.15.111:0/3719968924 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fe6d0014070 con 0x7fe6d810cec0 2023-10-11T16:33:42.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.481+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe6c80a13e0 0x7fe6c80a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.482+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe6c80a13e0 0x7fe6c80a3890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00f540 tx=0x7fe6cc00b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:42.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.563+0000 7fe6f24d4b80 1 Processor -- start 2023-10-11T16:33:42.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.563+0000 7fe6f24d4b80 1 -- start start 2023-10-11T16:33:42.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a5ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a63370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 0x5630d4a63d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4964060 con 0x5630d4a5f470 2023-10-11T16:33:42.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d49b68a0 con 0x5630d4a638b0 2023-10-11T16:33:42.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d49afd30 con 0x5630d4a5eb60 2023-10-11T16:33:42.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a63370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6dc99b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a5ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a63370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45524/0 (socket says 172.21.15.111:45524) 2023-10-11T16:33:42.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6d7fff700 1 -- 172.21.15.111:0/1740797651 learned_addr learned my addr 172.21.15.111:0/1740797651 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:42.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/1740797651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 0x5630d4a63d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6d7fff700 1 -- 172.21.15.111:0/1740797651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 msgr2=0x5630d4a5ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:42.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.564+0000 7fe6d7fff700 1 --2- 172.21.15.111:0/1740797651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a5ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.565+0000 7fe6d7fff700 1 -- 172.21.15.111:0/1740797651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 msgr2=0x5630d4a63d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:42.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.565+0000 7fe6d7fff700 1 --2- 172.21.15.111:0/1740797651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 0x5630d4a63d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.565+0000 7fe6d7fff700 1 -- 172.21.15.111:0/1740797651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630d4831630 con 0x5630d4a5f470 2023-10-11T16:33:42.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.565+0000 7fe6d7fff700 1 --2- 172.21.15.111:0/1740797651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a63370 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009990 tx=0x7fe6c4005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948bc7e277b312e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:42.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.565+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1740797651 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c401d070 con 0x5630d4a5f470 2023-10-11T16:33:42.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.565+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1740797651 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6c401e040 con 0x5630d4a5f470 2023-10-11T16:33:42.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.565+0000 7fe6c2ffd700 1 -- 172.21.15.111:0/1740797651 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4004740 con 0x5630d4a5f470 2023-10-11T16:33:42.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/1740797651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 msgr2=0x5630d4a63370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/1740797651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a63370 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009990 tx=0x7fe6c4005ea0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/1740797651 shutdown_connections 2023-10-11T16:33:42.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/1740797651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 0x5630d4a63d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/1740797651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a5ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/1740797651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a63370 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/1740797651 >> 172.21.15.111:0/1740797651 conn(0x5630d4a54360 msgr2=0x5630d4a59b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:42.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/1740797651 shutdown_connections 2023-10-11T16:33:42.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/1740797651 wait complete. 2023-10-11T16:33:42.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.566+0000 7fe6f24d4b80 1 -- start start 2023-10-11T16:33:42.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.567+0000 7fe6f24d4b80 1 Processor -- start 2023-10-11T16:33:42.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a68260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a68870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d49518e0 con 0x5630d4a638b0 2023-10-11T16:33:42.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4951580 con 0x5630d4a5f470 2023-10-11T16:33:42.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4833920 con 0x5630d4a5eb60 2023-10-11T16:33:42.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6f24d4b80 10 client.0 _ll_register_callbacks cb 0x5630d496f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:42.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6d7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a68260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6dd19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a68870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6dc99b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6d7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a68260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54216/0 (socket says 172.21.15.111:54216) 2023-10-11T16:33:42.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6d7fff700 1 -- 172.21.15.111:0/3636540317 learned_addr learned my addr 172.21.15.111:0/3636540317 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:42.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6dd19c700 1 -- 172.21.15.111:0/3636540317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 msgr2=0x5630d4a6b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:42.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6dd19c700 1 -- 172.21.15.111:0/3636540317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 msgr2=0x5630d4a68260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:42.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a68260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6dd19c700 1 -- 172.21.15.111:0/3636540317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d810fbd0 con 0x5630d4a638b0 2023-10-11T16:33:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.568+0000 7fe6d7fff700 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a68260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a68870 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00ca6a0 tx=0x7fe6d0005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230e09f4d127a56e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6b7fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6b7fff700 1 -- 172.21.15.111:0/3636540317 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00a23a0 con 0x5630d4a638b0 2023-10-11T16:33:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6b7fff700 1 -- 172.21.15.111:0/3636540317 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fe6d00994d0 con 0x5630d4a638b0 2023-10-11T16:33:42.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/3636540317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630d4a51b20 con 0x5630d4a638b0 2023-10-11T16:33:42.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6b7fff700 1 -- 172.21.15.111:0/3636540317 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6d0099690 con 0x5630d4a638b0 2023-10-11T16:33:42.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6f24d4b80 10 client.15777 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:42.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/3636540317 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5630d4976190 con 0x5630d4a638b0 2023-10-11T16:33:42.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6b7fff700 1 -- 172.21.15.111:0/3636540317 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5630d4a51b20 con 0x5630d4a638b0 2023-10-11T16:33:42.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6b7fff700 1 -- 172.21.15.111:0/3636540317 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe6d00a2010 con 0x5630d4a638b0 2023-10-11T16:33:42.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6f24d4b80 10 client.15777 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:42.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6f24d4b80 10 client.15777 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:33:42.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.569+0000 7fe6f24d4b80 20 client.15777 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:42.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.570+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5630d4a78820 0x5630d4a7ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.570+0000 7fe6f24d4b80 4 client.15777 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:42.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.570+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/3636540317 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630d49720a0 con 0x5630d4a78820 2023-10-11T16:33:42.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.570+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5630d4a78820 0x5630d4a7ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.571+0000 7fe6b7fff700 10 client.15777 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:33:42.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.571+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5630d4a78820 0x5630d4a7ac30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9558f305b014b679 server_cookie=5e61552716ba90d in_seq=0 out_seq=0 2023-10-11T16:33:42.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.571+0000 7fe6b7fff700 1 -- 172.21.15.111:0/3636540317 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5630d49720a0 con 0x5630d4a78820 2023-10-11T16:33:42.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.571+0000 7fe6b7fff700 10 client.15777 handle_command_reply: tid=0 2023-10-11T16:33:42.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.571+0000 7fe6f24d4b80 1 client.15777 shutdown 2023-10-11T16:33:42.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.571+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/3636540317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 msgr2=0x5630d4a68870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.571+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a68870 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00ca6a0 tx=0x7fe6d0005280 comp rx=0 tx=0).stop 2023-10-11T16:33:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/3636540317 shutdown_connections 2023-10-11T16:33:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5630d4a78820 0x5630d4a7ac30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a68870 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6b7fff700 0 client.15777 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:33:42.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/3636540317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a68260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/3636540317 >> 172.21.15.111:0/3636540317 conn(0x5630d4a54360 msgr2=0x5630d4a64d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:42.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/3636540317 shutdown_connections 2023-10-11T16:33:42.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/3636540317 wait complete. 2023-10-11T16:33:42.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.572+0000 7fe6f24d4b80 20 client.15777 trim_cache size 0 max 16384 2023-10-11T16:33:42.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.585+0000 7fe6f24d4b80 1 Processor -- start 2023-10-11T16:33:42.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.585+0000 7fe6f24d4b80 1 -- start start 2023-10-11T16:33:42.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.585+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.585+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a58c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a59140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4a78600 con 0x5630d4a638b0 2023-10-11T16:33:42.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4a66800 con 0x5630d4a5f470 2023-10-11T16:33:42.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4a623f0 con 0x5630d4a5eb60 2023-10-11T16:33:42.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a59140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a59140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45538/0 (socket says 172.21.15.111:45538) 2023-10-11T16:33:42.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 -- 172.21.15.111:0/933200964 learned_addr learned my addr 172.21.15.111:0/933200964 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:42.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6d7fff700 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a58c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 -- 172.21.15.111:0/933200964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 msgr2=0x5630d4a6ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:42.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 -- 172.21.15.111:0/933200964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 msgr2=0x5630d4a58c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:42.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a58c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 -- 172.21.15.111:0/933200964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630d4a58820 con 0x5630d4a638b0 2023-10-11T16:33:42.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:42.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.586+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a59140 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0009230 tx=0x7fe6d009e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e0c98b501d5e5d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:42.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6c37fe700 1 -- 172.21.15.111:0/933200964 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00a23a0 con 0x5630d4a638b0 2023-10-11T16:33:42.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6c37fe700 1 -- 172.21.15.111:0/933200964 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6d00cd150 con 0x5630d4a638b0 2023-10-11T16:33:42.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6c37fe700 1 -- 172.21.15.111:0/933200964 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00b83a0 con 0x5630d4a638b0 2023-10-11T16:33:42.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/933200964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 msgr2=0x5630d4a59140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a59140 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0009230 tx=0x7fe6d009e0d0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/933200964 shutdown_connections 2023-10-11T16:33:42.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a638b0 0x5630d4a59140 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/933200964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a5f470 0x5630d4a58c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/933200964 >> 172.21.15.111:0/933200964 conn(0x5630d4a54360 msgr2=0x5630d4a65c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:42.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/933200964 shutdown_connections 2023-10-11T16:33:42.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.587+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/933200964 wait complete. 2023-10-11T16:33:42.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.588+0000 7fe6f24d4b80 1 -- start start 2023-10-11T16:33:42.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.588+0000 7fe6f24d4b80 1 Processor -- start 2023-10-11T16:33:42.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.588+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.588+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a6de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.588+0000 7fe6f24d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 0x5630d4a6e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4acb6e0 con 0x5630d4a5f470 2023-10-11T16:33:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4acb820 con 0x5630d4a638b0 2023-10-11T16:33:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6f24d4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630d4acb960 con 0x5630d4a5eb60 2023-10-11T16:33:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6f24d4b80 10 client.0 _ll_register_callbacks cb 0x7fe6d0099c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a6de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6dc99b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a6de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45550/0 (socket says 172.21.15.111:45550) 2023-10-11T16:33:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6d7fff700 1 -- 172.21.15.111:0/2323388037 learned_addr learned my addr 172.21.15.111:0/2323388037 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6dd19c700 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 0x5630d4a6e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6dc99b700 1 -- 172.21.15.111:0/2323388037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 msgr2=0x5630d4a6e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:42.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 0x5630d4a6e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6dc99b700 1 -- 172.21.15.111:0/2323388037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 msgr2=0x5630d4a6de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:42.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a6de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6dc99b700 1 -- 172.21.15.111:0/2323388037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630d4976190 con 0x5630d4a5eb60 2023-10-11T16:33:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6d7fff700 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a6de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6dc99b700 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6d930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x5630d4a7b4e0 tx=0x7fe6cc004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9606b060ca34d01 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.589+0000 7fe6b7fff700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:33:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.590+0000 7fe6b7fff700 1 -- 172.21.15.111:0/2323388037 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc02b070 con 0x5630d4a5eb60 2023-10-11T16:33:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.590+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/2323388037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6c401e040 con 0x5630d4a5eb60 2023-10-11T16:33:42.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.590+0000 7fe6b7fff700 1 -- 172.21.15.111:0/2323388037 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fe6cc02b3d0 con 0x5630d4a5eb60 2023-10-11T16:33:42.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.590+0000 7fe6b7fff700 1 -- 172.21.15.111:0/2323388037 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6cc00e040 con 0x5630d4a5eb60 2023-10-11T16:33:42.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.590+0000 7fe6f24d4b80 10 client.25858 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:42.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.590+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/2323388037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5630d47ddde0 con 0x5630d4a5eb60 2023-10-11T16:33:42.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.590+0000 7fe6b7fff700 1 -- 172.21.15.111:0/2323388037 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6c401e040 con 0x5630d4a5eb60 2023-10-11T16:33:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.590+0000 7fe6b7fff700 1 -- 172.21.15.111:0/2323388037 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe6cc0086a0 con 0x5630d4a5eb60 2023-10-11T16:33:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6f24d4b80 10 client.25858 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6f24d4b80 10 client.25858 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:33:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6f24d4b80 20 client.25858 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5630d4acd090 0x5630d4acf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6f24d4b80 4 client.25858 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:33:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/2323388037 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5630d497e660 con 0x5630d4acd090 2023-10-11T16:33:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6d7fff700 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5630d4acd090 0x5630d4acf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6b7fff700 10 client.25858 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:33:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.591+0000 7fe6d7fff700 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5630d4acd090 0x5630d4acf4a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=81d387dac5ac655f server_cookie=f925de3b65ae8166 in_seq=0 out_seq=0 2023-10-11T16:33:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6b7fff700 1 -- 172.21.15.111:0/2323388037 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5630d497e660 con 0x5630d4acd090 2023-10-11T16:33:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6b7fff700 10 client.25858 handle_command_reply: tid=0 2023-10-11T16:33:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 client.25858 shutdown 2023-10-11T16:33:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/2323388037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 msgr2=0x5630d4a6d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6d930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x5630d4a7b4e0 tx=0x7fe6cc004ad0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/2323388037 shutdown_connections 2023-10-11T16:33:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5630d4acd090 0x5630d4acf4a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5630d4a638b0 0x5630d4a6e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5630d4a5eb60 0x5630d4a6d930 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 --2- 172.21.15.111:0/2323388037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5630d4a5f470 0x5630d4a6de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6b7fff700 0 client.25858 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:33:42.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/2323388037 >> 172.21.15.111:0/2323388037 conn(0x5630d4a54360 msgr2=0x5630d4a58c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:42.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/2323388037 shutdown_connections 2023-10-11T16:33:42.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 1 -- 172.21.15.111:0/2323388037 wait complete. 2023-10-11T16:33:42.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.592+0000 7fe6f24d4b80 20 client.25858 trim_cache size 0 max 16384 2023-10-11T16:33:42.611 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:33:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 -- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe6c80a13e0 msgr2=0x7fe6c80a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe6c80a13e0 0x7fe6c80a3890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00f540 tx=0x7fe6cc00b040 comp rx=0 tx=0).stop 2023-10-11T16:33:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 -- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d810cec0 msgr2=0x7fe6d8198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d810cec0 0x7fe6d8198570 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000e940 tx=0x7fe6d000ec50 comp rx=0 tx=0).stop 2023-10-11T16:33:42.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 -- 172.21.15.111:0/3719968924 shutdown_connections 2023-10-11T16:33:42.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe6c80a13e0 0x7fe6c80a3890 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d810cec0 0x7fe6d8198570 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6d80746f0 0x7fe6d819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 --2- 172.21.15.111:0/3719968924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6d8074ac0 0x7fe6d819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:42.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 -- 172.21.15.111:0/3719968924 >> 172.21.15.111:0/3719968924 conn(0x7fe6d8077560 msgr2=0x7fe6d8104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:42.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 -- 172.21.15.111:0/3719968924 shutdown_connections 2023-10-11T16:33:42.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:42.593+0000 7fe6debff700 1 -- 172.21.15.111:0/3719968924 wait complete. 2023-10-11T16:33:42.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:33:43.069 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:43.069 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:43.167 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:43.167 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:43.211 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:43.212 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:43.212 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 22 -lt 100 ']' 2023-10-11T16:33:43.212 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 22 + 1 2023-10-11T16:33:43.245 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=23 2023-10-11T16:33:43.246 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:43.299 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:43.300 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:43.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:43.302 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:43.303 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:43.530 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:43 smithi111 ceph-mon[98527]: pgmap v35: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-10-11T16:33:43.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:43 smithi130 ceph-mon[131859]: pgmap v35: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-10-11T16:33:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:43 smithi150 ceph-mon[132217]: pgmap v35: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-10-11T16:33:43.727 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:43.727 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:43.842 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:43.843 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:43.869 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:43.870 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:43.870 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 23 -lt 100 ']' 2023-10-11T16:33:43.870 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 23 + 1 2023-10-11T16:33:43.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=24 2023-10-11T16:33:43.904 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:43.951 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:43.952 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:43.954 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:43.955 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:43.955 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:44.602 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:44.602 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:44.751 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:44.751 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:44.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:44.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:44.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 24 -lt 100 ']' 2023-10-11T16:33:44.771 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 24 + 1 2023-10-11T16:33:44.813 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=25 2023-10-11T16:33:44.814 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:44.866 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:44.867 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:44.869 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:44.869 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:44.870 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:45.452 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:45.452 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:45.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:45 smithi130 ceph-mon[131859]: pgmap v36: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 12 op/s 2023-10-11T16:33:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:45 smithi111 ceph-mon[98527]: pgmap v36: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 12 op/s 2023-10-11T16:33:45.651 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:45.652 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:45.694 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:45.695 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:45.695 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 25 -lt 100 ']' 2023-10-11T16:33:45.696 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 25 + 1 2023-10-11T16:33:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:45 smithi150 ceph-mon[132217]: pgmap v36: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 12 op/s 2023-10-11T16:33:45.744 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=26 2023-10-11T16:33:45.745 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:45.889 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:45.889 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:45.891 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:45.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:45.892 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 -- 172.21.15.111:0/1315561482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac101660 msgr2=0x7f16ac104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 --2- 172.21.15.111:0/1315561482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac101660 0x7f16ac104ef0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f16a00099c0 tx=0x7f16a0009cd0 comp rx=0 tx=0).stop 2023-10-11T16:33:45.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 -- 172.21.15.111:0/1315561482 shutdown_connections 2023-10-11T16:33:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 --2- 172.21.15.111:0/1315561482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ac105620 0x7f16ac10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 --2- 172.21.15.111:0/1315561482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ac100cc0 0x7f16ac101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:45.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 --2- 172.21.15.111:0/1315561482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac101660 0x7f16ac104ef0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:45.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 -- 172.21.15.111:0/1315561482 >> 172.21.15.111:0/1315561482 conn(0x7f16ac077560 msgr2=0x7f16ac077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:45.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 -- 172.21.15.111:0/1315561482 shutdown_connections 2023-10-11T16:33:45.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.974+0000 7f16b21ff700 1 -- 172.21.15.111:0/1315561482 wait complete. 2023-10-11T16:33:45.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.975+0000 7f16b21ff700 1 Processor -- start 2023-10-11T16:33:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.975+0000 7f16b21ff700 1 -- start start 2023-10-11T16:33:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.975+0000 7f16b21ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac100cc0 0x7f16ac072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.975+0000 7f16b21ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ac101660 0x7f16ac073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.975+0000 7f16b21ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ac105620 0x7f16ac1a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.975+0000 7f16b21ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac073b60 con 0x7f16ac100cc0 2023-10-11T16:33:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.975+0000 7f16b21ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1a4cd0 con 0x7f16ac101660 2023-10-11T16:33:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.975+0000 7f16b21ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1a4e10 con 0x7f16ac105620 2023-10-11T16:33:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac100cc0 0x7f16ac072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ac105620 0x7f16ac1a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16ab7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac100cc0 0x7f16ac072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55098/0 (socket says 172.21.15.111:55098) 2023-10-11T16:33:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16ab7fe700 1 -- 172.21.15.111:0/3690193139 learned_addr learned my addr 172.21.15.111:0/3690193139 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16aaffd700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ac101660 0x7f16ac073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16abfff700 1 -- 172.21.15.111:0/3690193139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ac101660 msgr2=0x7f16ac073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16abfff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ac101660 0x7f16ac073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16abfff700 1 -- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac100cc0 msgr2=0x7f16ac072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16abfff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac100cc0 0x7f16ac072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:45.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16abfff700 1 -- 172.21.15.111:0/3690193139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a00095f0 con 0x7f16ac105620 2023-10-11T16:33:45.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac100cc0 0x7f16ac072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:45.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.976+0000 7f16abfff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ac105620 0x7f16ac1a4790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f16a400efc0 tx=0x7f16a400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:45.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.977+0000 7f16a8ff9700 1 -- 172.21.15.111:0/3690193139 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4018070 con 0x7f16ac105620 2023-10-11T16:33:45.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.977+0000 7f16b21ff700 1 -- 172.21.15.111:0/3690193139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac110370 con 0x7f16ac105620 2023-10-11T16:33:45.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.977+0000 7f16a8ff9700 1 -- 172.21.15.111:0/3690193139 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16a400f040 con 0x7f16ac105620 2023-10-11T16:33:45.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.977+0000 7f16a8ff9700 1 -- 172.21.15.111:0/3690193139 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a400cd40 con 0x7f16ac105620 2023-10-11T16:33:45.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.977+0000 7f16b21ff700 1 -- 172.21.15.111:0/3690193139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac1101e0 con 0x7f16ac105620 2023-10-11T16:33:45.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.980+0000 7f16a8ff9700 1 -- 172.21.15.111:0/3690193139 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f16a4003a00 con 0x7f16ac105620 2023-10-11T16:33:45.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.981+0000 7f16a8ff9700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f16940a13e0 0x7f16940a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:45.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.982+0000 7f16a8ff9700 1 -- 172.21.15.111:0/3690193139 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f16a40cced0 con 0x7f16ac105620 2023-10-11T16:33:45.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.982+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f16940a13e0 0x7f16940a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:45.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:45.982+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f16940a13e0 0x7f16940a3890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f169c011550 tx=0x7f169c009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:46.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16c5ad9b80 1 Processor -- start 2023-10-11T16:33:46.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16c5ad9b80 1 -- start start 2023-10-11T16:33:46.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58005b60 0x558d58005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58006470 0x558d5800a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d57f0b060 con 0x558d5800a8b0 2023-10-11T16:33:46.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d57f5d8a0 con 0x558d58005b60 2023-10-11T16:33:46.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d57f56d30 con 0x558d58006470 2023-10-11T16:33:46.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55112/0 (socket says 172.21.15.111:55112) 2023-10-11T16:33:46.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.063+0000 7f16abfff700 1 -- 172.21.15.111:0/3487262352 learned_addr learned my addr 172.21.15.111:0/3487262352 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:46.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58005b60 0x558d58005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f16aaffd700 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58006470 0x558d5800a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f16abfff700 1 -- 172.21.15.111:0/3487262352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58006470 msgr2=0x558d5800a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:46.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f16abfff700 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58006470 0x558d5800a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f16abfff700 1 -- 172.21.15.111:0/3487262352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58005b60 msgr2=0x558d58005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:46.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f16abfff700 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58005b60 0x558d58005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f16abfff700 1 -- 172.21.15.111:0/3487262352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d57dd8630 con 0x558d5800a8b0 2023-10-11T16:33:46.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f16abfff700 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800ad00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f16a400ef90 tx=0x7f16a4095e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847c27aad4320a39 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:46.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f1691ffb700 1 -- 172.21.15.111:0/3487262352 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4004ad0 con 0x558d5800a8b0 2023-10-11T16:33:46.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f1691ffb700 1 -- 172.21.15.111:0/3487262352 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16a40999d0 con 0x558d5800a8b0 2023-10-11T16:33:46.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.064+0000 7f1691ffb700 1 -- 172.21.15.111:0/3487262352 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4098ec0 con 0x558d5800a8b0 2023-10-11T16:33:46.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/3487262352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 msgr2=0x558d5800ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:46.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800ad00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f16a400ef90 tx=0x7f16a4095e10 comp rx=0 tx=0).stop 2023-10-11T16:33:46.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/3487262352 shutdown_connections 2023-10-11T16:33:46.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800ad00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58005b60 0x558d58005f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/3487262352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58006470 0x558d5800a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/3487262352 >> 172.21.15.111:0/3487262352 conn(0x558d57ffb360 msgr2=0x558d58000b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:46.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/3487262352 shutdown_connections 2023-10-11T16:33:46.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.065+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/3487262352 wait complete. 2023-10-11T16:33:46.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.066+0000 7f16c5ad9b80 1 -- start start 2023-10-11T16:33:46.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.066+0000 7f16c5ad9b80 1 Processor -- start 2023-10-11T16:33:46.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.066+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.066+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d5800f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.066+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.066+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d57ef88e0 con 0x558d5800a8b0 2023-10-11T16:33:46.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.066+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d57ef8580 con 0x558d58006470 2023-10-11T16:33:46.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.066+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d57dda920 con 0x558d58005b60 2023-10-11T16:33:46.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16c5ad9b80 10 client.0 _ll_register_callbacks cb 0x558d57f16780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:46.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55128/0 (socket says 172.21.15.111:55128) 2023-10-11T16:33:46.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 -- 172.21.15.111:0/4234589126 learned_addr learned my addr 172.21.15.111:0/4234589126 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:46.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16aaffd700 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d5800f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 -- 172.21.15.111:0/4234589126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 msgr2=0x558d580120e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:46.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580120e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 -- 172.21.15.111:0/4234589126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 msgr2=0x558d5800f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:46.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d5800f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 -- 172.21.15.111:0/4234589126 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac1101e0 con 0x558d5800a8b0 2023-10-11T16:33:46.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16aaffd700 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d5800f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:46.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f16abfff700 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800fa30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f16a40090a0 tx=0x7f16a4004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2af7c36b350895 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:46.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f168affd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:46.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.067+0000 7f168affd700 1 -- 172.21.15.111:0/4234589126 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a409da00 con 0x558d5800a8b0 2023-10-11T16:33:46.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f168affd700 1 -- 172.21.15.111:0/4234589126 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f16a4004ad0 con 0x558d5800a8b0 2023-10-11T16:33:46.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/4234589126 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558d57ff8b20 con 0x558d5800a8b0 2023-10-11T16:33:46.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f168affd700 1 -- 172.21.15.111:0/4234589126 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16a40b2120 con 0x558d5800a8b0 2023-10-11T16:33:46.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 10 client.15795 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:46.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/4234589126 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558d57f1d190 con 0x558d5800a8b0 2023-10-11T16:33:46.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f168affd700 1 -- 172.21.15.111:0/4234589126 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558d57ff8b20 con 0x558d5800a8b0 2023-10-11T16:33:46.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f168affd700 1 -- 172.21.15.111:0/4234589126 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f16a4098b90 con 0x558d5800a8b0 2023-10-11T16:33:46.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 10 client.15795 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:46.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 10 client.15795 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:33:46.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 20 client.15795 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:46.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558d5801f480 0x558d58021860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 4 client.15795 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:46.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.068+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/4234589126 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558d57f190a0 con 0x558d5801f480 2023-10-11T16:33:46.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.069+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558d5801f480 0x558d58021860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.069+0000 7f168affd700 10 client.15795 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:33:46.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.069+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558d5801f480 0x558d58021860 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf3742a3ad1652cd server_cookie=69f954d1387d7b4 in_seq=0 out_seq=0 2023-10-11T16:33:46.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.070+0000 7f168affd700 1 -- 172.21.15.111:0/4234589126 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558d57f190a0 con 0x558d5801f480 2023-10-11T16:33:46.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.070+0000 7f168affd700 10 client.15795 handle_command_reply: tid=0 2023-10-11T16:33:46.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.070+0000 7f16c5ad9b80 1 client.15795 shutdown 2023-10-11T16:33:46.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.070+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/4234589126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 msgr2=0x558d5800fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:46.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.070+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800fa30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f16a40090a0 tx=0x7f16a4004cc0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.070+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/4234589126 shutdown_connections 2023-10-11T16:33:46.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558d5801f480 0x558d58021860 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d5800fa30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/4234589126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d5800f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f168affd700 0 client.15795 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:33:46.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/4234589126 >> 172.21.15.111:0/4234589126 conn(0x558d57ffb360 msgr2=0x558d5800bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:46.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/4234589126 shutdown_connections 2023-10-11T16:33:46.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/4234589126 wait complete. 2023-10-11T16:33:46.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.071+0000 7f16c5ad9b80 20 client.15795 trim_cache size 0 max 16384 2023-10-11T16:33:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.084+0000 7f16c5ad9b80 1 Processor -- start 2023-10-11T16:33:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.084+0000 7f16c5ad9b80 1 -- start start 2023-10-11T16:33:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580169a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d58011c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d580088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d58010780 con 0x558d5800a8b0 2023-10-11T16:33:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d58009420 con 0x558d58006470 2023-10-11T16:33:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d58016ee0 con 0x558d58005b60 2023-10-11T16:33:46.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16ab7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580169a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d580088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16ab7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580169a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:38032/0 (socket says 172.21.15.111:38032) 2023-10-11T16:33:46.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16ab7fe700 1 -- 172.21.15.111:0/1295266822 learned_addr learned my addr 172.21.15.111:0/1295266822 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:46.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16aaffd700 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d58011c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16ab7fe700 1 -- 172.21.15.111:0/1295266822 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 msgr2=0x558d58011c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:46.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d58011c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16ab7fe700 1 -- 172.21.15.111:0/1295266822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 msgr2=0x558d580088f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:46.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d580088f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16ab7fe700 1 -- 172.21.15.111:0/1295266822 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d57f1cee0 con 0x558d58005b60 2023-10-11T16:33:46.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.085+0000 7f16abfff700 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d580088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:46.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580169a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f169c013010 tx=0x7f169c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12927982d2857d92 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:46.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16927fc700 1 -- 172.21.15.111:0/1295266822 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c022040 con 0x558d58005b60 2023-10-11T16:33:46.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16927fc700 1 -- 172.21.15.111:0/1295266822 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f169c023040 con 0x558d58005b60 2023-10-11T16:33:46.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16927fc700 1 -- 172.21.15.111:0/1295266822 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c01da40 con 0x558d58005b60 2023-10-11T16:33:46.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/1295266822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 msgr2=0x558d580169a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:46.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580169a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f169c013010 tx=0x7f169c014000 comp rx=0 tx=0).stop 2023-10-11T16:33:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/1295266822 shutdown_connections 2023-10-11T16:33:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d5800a8b0 0x558d580088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d580169a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.086+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/1295266822 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d58006470 0x558d58011c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.087+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/1295266822 >> 172.21.15.111:0/1295266822 conn(0x558d57ffb360 msgr2=0x558d5800cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:46.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.087+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/1295266822 shutdown_connections 2023-10-11T16:33:46.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.087+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/1295266822 wait complete. 2023-10-11T16:33:46.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.087+0000 7f16c5ad9b80 1 -- start start 2023-10-11T16:33:46.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.087+0000 7f16c5ad9b80 1 Processor -- start 2023-10-11T16:33:46.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d58014990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d58006470 0x558d58014ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16c5ad9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d5800a8b0 0x558d58072310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d58015720 con 0x558d58006470 2023-10-11T16:33:46.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d58072850 con 0x558d5800a8b0 2023-10-11T16:33:46.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16c5ad9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d58072990 con 0x558d58005b60 2023-10-11T16:33:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16c5ad9b80 10 client.0 _ll_register_callbacks cb 0x7f169c009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16aaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d58006470 0x558d58014ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16ab7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d58014990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16aaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d58006470 0x558d58014ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55144/0 (socket says 172.21.15.111:55144) 2023-10-11T16:33:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16aaffd700 1 -- 172.21.15.111:0/2263851953 learned_addr learned my addr 172.21.15.111:0/2263851953 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:46.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.088+0000 7f16abfff700 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d5800a8b0 0x558d58072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f16ab7fe700 1 -- 172.21.15.111:0/2263851953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d5800a8b0 msgr2=0x558d58072310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d5800a8b0 0x558d58072310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f16ab7fe700 1 -- 172.21.15.111:0/2263851953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d58006470 msgr2=0x558d58014ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d58006470 0x558d58014ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f16ab7fe700 1 -- 172.21.15.111:0/2263851953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d57f1d190 con 0x558d58005b60 2023-10-11T16:33:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f16ab7fe700 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d58014990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f169c009cf0 tx=0x7f169c01d4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfd7551824ccf48e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f168affd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:33:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f168affd700 1 -- 172.21.15.111:0/2263851953 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c02b4c0 con 0x558d58005b60 2023-10-11T16:33:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/2263851953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16a40999d0 con 0x558d58005b60 2023-10-11T16:33:46.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.089+0000 7f168affd700 1 -- 172.21.15.111:0/2263851953 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f169c005040 con 0x558d58005b60 2023-10-11T16:33:46.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f168affd700 1 -- 172.21.15.111:0/2263851953 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f169c02b620 con 0x558d58005b60 2023-10-11T16:33:46.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16c5ad9b80 10 client.25876 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:46.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/2263851953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f169c017070 con 0x558d58005b60 2023-10-11T16:33:46.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f168affd700 1 -- 172.21.15.111:0/2263851953 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16a40999d0 con 0x558d58005b60 2023-10-11T16:33:46.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f168affd700 1 -- 172.21.15.111:0/2263851953 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f169c01d6c0 con 0x558d58005b60 2023-10-11T16:33:46.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16c5ad9b80 10 client.25876 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:46.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16c5ad9b80 10 client.25876 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:33:46.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16c5ad9b80 20 client.25876 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:46.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558d58073fc0 0x558d580763d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:46.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16c5ad9b80 4 client.25876 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:33:46.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/2263851953 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558d57f25660 con 0x558d58073fc0 2023-10-11T16:33:46.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.090+0000 7f16aaffd700 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558d58073fc0 0x558d580763d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:46.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16aaffd700 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558d58073fc0 0x558d580763d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ec116989261e8f6 server_cookie=24fd51c3d2115640 in_seq=0 out_seq=0 2023-10-11T16:33:46.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f168affd700 10 client.25876 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:33:46.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f168affd700 1 -- 172.21.15.111:0/2263851953 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558d57f25660 con 0x558d58073fc0 2023-10-11T16:33:46.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f168affd700 10 client.25876 handle_command_reply: tid=0 2023-10-11T16:33:46.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 client.25876 shutdown 2023-10-11T16:33:46.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/2263851953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 msgr2=0x558d58014990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:46.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d58014990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f169c009cf0 tx=0x7f169c01d4a0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/2263851953 shutdown_connections 2023-10-11T16:33:46.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558d58073fc0 0x558d580763d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558d5800a8b0 0x558d58072310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f168affd700 0 client.25876 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:33:46.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558d58005b60 0x558d58014990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 --2- 172.21.15.111:0/2263851953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558d58006470 0x558d58014ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.091+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/2263851953 >> 172.21.15.111:0/2263851953 conn(0x558d57ffb360 msgr2=0x558d580088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:46.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/2263851953 shutdown_connections 2023-10-11T16:33:46.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16c5ad9b80 1 -- 172.21.15.111:0/2263851953 wait complete. 2023-10-11T16:33:46.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16c5ad9b80 20 client.25876 trim_cache size 0 max 16384 2023-10-11T16:33:46.109 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:33:46.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 -- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f16940a13e0 msgr2=0x7f16940a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:46.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f16940a13e0 0x7f16940a3890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f169c011550 tx=0x7f169c009260 comp rx=0 tx=0).stop 2023-10-11T16:33:46.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 -- 172.21.15.111:0/3690193139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ac105620 msgr2=0x7f16ac1a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:46.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ac105620 0x7f16ac1a4790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f16a400efc0 tx=0x7f16a400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 -- 172.21.15.111:0/3690193139 shutdown_connections 2023-10-11T16:33:46.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f16940a13e0 0x7f16940a3890 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ac105620 0x7f16ac1a4790 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac100cc0 0x7f16ac072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 --2- 172.21.15.111:0/3690193139 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16ac101660 0x7f16ac073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:46.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.092+0000 7f16b21ff700 1 -- 172.21.15.111:0/3690193139 >> 172.21.15.111:0/3690193139 conn(0x7f16ac077560 msgr2=0x7f16ac0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:46.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.093+0000 7f16b21ff700 1 -- 172.21.15.111:0/3690193139 shutdown_connections 2023-10-11T16:33:46.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:46.093+0000 7f16b21ff700 1 -- 172.21.15.111:0/3690193139 wait complete. 2023-10-11T16:33:46.133 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:33:46.526 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:46.527 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:46.587 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:46.587 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:46.602 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:46.603 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:46.603 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 26 -lt 100 ']' 2023-10-11T16:33:46.603 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 26 + 1 2023-10-11T16:33:46.820 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=27 2023-10-11T16:33:46.820 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:46.832 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:46.832 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:46.834 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:46.835 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:46.836 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:47.285 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:47.285 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:47.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:47 smithi111 ceph-mon[98527]: pgmap v37: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-10-11T16:33:47.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:47 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:33:47.518 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:47.518 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:47.553 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:47.553 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:47.554 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 27 -lt 100 ']' 2023-10-11T16:33:47.554 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 27 + 1 2023-10-11T16:33:47.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:47 smithi130 ceph-mon[131859]: pgmap v37: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-10-11T16:33:47.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:47 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:33:47.586 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=28 2023-10-11T16:33:47.586 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:47 smithi150 ceph-mon[132217]: pgmap v37: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-10-11T16:33:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:47 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:33:47.761 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:47.761 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:47.763 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:47.764 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:47.764 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:48.735 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:48.735 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:48.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:48.935 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:48.953 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:48.953 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:48.954 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 28 -lt 100 ']' 2023-10-11T16:33:48.954 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 28 + 1 2023-10-11T16:33:48.988 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=29 2023-10-11T16:33:48.989 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:49.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:49.100 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:49.102 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:49.103 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:49.104 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:49.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:49 smithi130 ceph-mon[131859]: pgmap v38: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-10-11T16:33:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:49 smithi111 ceph-mon[98527]: pgmap v38: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-10-11T16:33:49.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:49 smithi150 ceph-mon[132217]: pgmap v38: 129 pgs: 129 active+clean; 1.5 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-10-11T16:33:50.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 -- 172.21.15.111:0/3945261529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508100a50 msgr2=0x7ff508100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3945261529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508100a50 0x7ff508100ec0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff4f00098f0 tx=0x7ff4f0009c00 comp rx=0 tx=0).stop 2023-10-11T16:33:50.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 -- 172.21.15.111:0/3945261529 shutdown_connections 2023-10-11T16:33:50.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3945261529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff508101400 0x7ff50810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3945261529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5081063f0 0x7ff5081067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3945261529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508100a50 0x7ff508100ec0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 -- 172.21.15.111:0/3945261529 >> 172.21.15.111:0/3945261529 conn(0x7ff508077590 msgr2=0x7ff508077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:50.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 -- 172.21.15.111:0/3945261529 shutdown_connections 2023-10-11T16:33:50.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.065+0000 7ff50ddff700 1 -- 172.21.15.111:0/3945261529 wait complete. 2023-10-11T16:33:50.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.074+0000 7ff50ddff700 1 Processor -- start 2023-10-11T16:33:50.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff50ddff700 1 -- start start 2023-10-11T16:33:50.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff50ddff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff508100a50 0x7ff508198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff50ddff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508101400 0x7ff508198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff50ddff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5081063f0 0x7ff50819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff50ddff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50819d640 con 0x7ff508101400 2023-10-11T16:33:50.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff50ddff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50819d7b0 con 0x7ff508100a50 2023-10-11T16:33:50.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff50ddff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50819d970 con 0x7ff5081063f0 2023-10-11T16:33:50.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff4fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508101400 0x7ff508198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff4fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508101400 0x7ff508198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55164/0 (socket says 172.21.15.111:55164) 2023-10-11T16:33:50.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff4fffff700 1 -- 172.21.15.111:0/3643099846 learned_addr learned my addr 172.21.15.111:0/3643099846 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:50.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff507fff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5081063f0 0x7ff50819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.075+0000 7ff5077fe700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff508100a50 0x7ff508198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff4fffff700 1 -- 172.21.15.111:0/3643099846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5081063f0 msgr2=0x7ff50819d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5081063f0 0x7ff50819d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff4fffff700 1 -- 172.21.15.111:0/3643099846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff508100a50 msgr2=0x7ff508198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff508100a50 0x7ff508198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff4fffff700 1 -- 172.21.15.111:0/3643099846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f0009520 con 0x7ff508101400 2023-10-11T16:33:50.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508101400 0x7ff508198ba0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009810 tx=0x7ff4f000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:50.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff5057fa700 1 -- 172.21.15.111:0/3643099846 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f001c070 con 0x7ff508101400 2023-10-11T16:33:50.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff5057fa700 1 -- 172.21.15.111:0/3643099846 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4f000bb20 con 0x7ff508101400 2023-10-11T16:33:50.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff50ddff700 1 -- 172.21.15.111:0/3643099846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff508110410 con 0x7ff508101400 2023-10-11T16:33:50.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff5057fa700 1 -- 172.21.15.111:0/3643099846 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0004930 con 0x7ff508101400 2023-10-11T16:33:50.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.076+0000 7ff50ddff700 1 -- 172.21.15.111:0/3643099846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff508110280 con 0x7ff508101400 2023-10-11T16:33:50.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.079+0000 7ff5057fa700 1 -- 172.21.15.111:0/3643099846 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7ff4f0020070 con 0x7ff508101400 2023-10-11T16:33:50.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.080+0000 7ff5057fa700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff4f40a1460 0x7ff4f40a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.081+0000 7ff5057fa700 1 -- 172.21.15.111:0/3643099846 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff4f00cdfc0 con 0x7ff508101400 2023-10-11T16:33:50.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.081+0000 7ff5077fe700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff4f40a1460 0x7ff4f40a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.081+0000 7ff5077fe700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff4f40a1460 0x7ff4f40a3910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009be0 tx=0x7ff4f8009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:50.276 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:50.277 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.371+0000 7ff5217e1b80 1 Processor -- start 2023-10-11T16:33:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.371+0000 7ff5217e1b80 1 -- start start 2023-10-11T16:33:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48808f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48809470 0x557d4880d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d4880dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4870e060 con 0x557d4880d8b0 2023-10-11T16:33:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d487608a0 con 0x557d48808b60 2023-10-11T16:33:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d48759d30 con 0x557d48809470 2023-10-11T16:33:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5077fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48808f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5077fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48808f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40484/0 (socket says 172.21.15.111:40484) 2023-10-11T16:33:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff5077fe700 1 -- 172.21.15.111:0/1954927963 learned_addr learned my addr 172.21.15.111:0/1954927963 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:50.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff507fff700 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d4880dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.372+0000 7ff4fffff700 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48809470 0x557d4880d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff5077fe700 1 -- 172.21.15.111:0/1954927963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48809470 msgr2=0x557d4880d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:50.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff5077fe700 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48809470 0x557d4880d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff5077fe700 1 -- 172.21.15.111:0/1954927963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 msgr2=0x557d4880dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff5077fe700 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d4880dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff5077fe700 1 -- 172.21.15.111:0/1954927963 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d485db630 con 0x557d48808b60 2023-10-11T16:33:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff5077fe700 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48808f30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8011d70 tx=0x7ff4f8014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef1e5039a1d46488 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff4fdffb700 1 -- 172.21.15.111:0/1954927963 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8011da0 con 0x557d48808b60 2023-10-11T16:33:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff4fdffb700 1 -- 172.21.15.111:0/1954927963 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4f8004020 con 0x557d48808b60 2023-10-11T16:33:50.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.373+0000 7ff4fdffb700 1 -- 172.21.15.111:0/1954927963 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8003bc0 con 0x557d48808b60 2023-10-11T16:33:50.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1954927963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 msgr2=0x557d48808f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48808f30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8011d70 tx=0x7ff4f8014000 comp rx=0 tx=0).stop 2023-10-11T16:33:50.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1954927963 shutdown_connections 2023-10-11T16:33:50.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d4880dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48808f30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1954927963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48809470 0x557d4880d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1954927963 >> 172.21.15.111:0/1954927963 conn(0x557d487fe360 msgr2=0x557d48803b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1954927963 shutdown_connections 2023-10-11T16:33:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.374+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1954927963 wait complete. 2023-10-11T16:33:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.375+0000 7ff5217e1b80 1 -- start start 2023-10-11T16:33:50.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.375+0000 7ff5217e1b80 1 Processor -- start 2023-10-11T16:33:50.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.375+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48815110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.375+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d48812390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.375+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 0x557d48812ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.375+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d486fb8e0 con 0x557d48809470 2023-10-11T16:33:50.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.375+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d486fb580 con 0x557d48808b60 2023-10-11T16:33:50.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.375+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d485dd920 con 0x557d4880d8b0 2023-10-11T16:33:50.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5217e1b80 10 client.0 _ll_register_callbacks cb 0x557d48719780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:50.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff4fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d48812390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48815110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff507fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 0x557d48812ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48815110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40496/0 (socket says 172.21.15.111:40496) 2023-10-11T16:33:50.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 -- 172.21.15.111:0/3902113219 learned_addr learned my addr 172.21.15.111:0/3902113219 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:50.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff4fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d48812390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55190/0 (socket says 172.21.15.111:55190) 2023-10-11T16:33:50.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 -- 172.21.15.111:0/3902113219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 msgr2=0x557d48812ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 0x557d48812ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 -- 172.21.15.111:0/3902113219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 msgr2=0x557d48812390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d48812390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 -- 172.21.15.111:0/3902113219 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f8017070 con 0x557d48808b60 2023-10-11T16:33:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d48812390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:50.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff5077fe700 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48815110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009210 tx=0x7ff4f8004570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9d9c6e069bbce58 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:50.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.376+0000 7ff4e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:50.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/3902113219 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80037c0 con 0x557d48808b60 2023-10-11T16:33:50.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff5217e1b80 1 -- 172.21.15.111:0/3902113219 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d487fbb20 con 0x557d48808b60 2023-10-11T16:33:50.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/3902113219 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff4f8003920 con 0x557d48808b60 2023-10-11T16:33:50.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/3902113219 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4f8003ae0 con 0x557d48808b60 2023-10-11T16:33:50.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff5217e1b80 10 client.25856 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:50.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff5217e1b80 1 -- 172.21.15.111:0/3902113219 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557d48720190 con 0x557d48808b60 2023-10-11T16:33:50.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/3902113219 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d487fbb20 con 0x557d48808b60 2023-10-11T16:33:50.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/3902113219 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff4f8024030 con 0x557d48808b60 2023-10-11T16:33:50.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff5217e1b80 10 client.25856 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:50.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff5217e1b80 10 client.25856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:50.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.377+0000 7ff5217e1b80 20 client.25856 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:50.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.378+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557d488227a0 0x557d48824b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.378+0000 7ff5217e1b80 4 client.25856 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:50.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.378+0000 7ff5217e1b80 1 -- 172.21.15.111:0/3902113219 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d4871c0a0 con 0x557d488227a0 2023-10-11T16:33:50.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.378+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557d488227a0 0x557d48824b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.378+0000 7ff4fffff700 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557d488227a0 0x557d48824b80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed084ff3c5a3fc server_cookie=5331fd82a12e6b28 in_seq=0 out_seq=0 2023-10-11T16:33:50.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.378+0000 7ff4e6ffd700 10 client.25856 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:50.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/3902113219 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557d4871c0a0 con 0x557d488227a0 2023-10-11T16:33:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff4e6ffd700 10 client.25856 handle_command_reply: tid=0 2023-10-11T16:33:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 client.25856 shutdown 2023-10-11T16:33:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 -- 172.21.15.111:0/3902113219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 msgr2=0x557d48815110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48815110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009210 tx=0x7ff4f8004570 comp rx=0 tx=0).stop 2023-10-11T16:33:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 -- 172.21.15.111:0/3902113219 shutdown_connections 2023-10-11T16:33:50.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557d488227a0 0x557d48824b80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 0x557d48812ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48815110 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/3902113219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d48812390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff4e6ffd700 0 client.25856 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:50.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.379+0000 7ff5217e1b80 1 -- 172.21.15.111:0/3902113219 >> 172.21.15.111:0/3902113219 conn(0x557d487fe360 msgr2=0x557d48802550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:50.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.380+0000 7ff5217e1b80 1 -- 172.21.15.111:0/3902113219 shutdown_connections 2023-10-11T16:33:50.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.380+0000 7ff5217e1b80 1 -- 172.21.15.111:0/3902113219 wait complete. 2023-10-11T16:33:50.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.380+0000 7ff5217e1b80 20 client.25856 trim_cache size 0 max 16384 2023-10-11T16:33:50.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5217e1b80 1 Processor -- start 2023-10-11T16:33:50.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5217e1b80 1 -- start start 2023-10-11T16:33:50.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48814d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d4880b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 0x557d4880bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4880c520 con 0x557d48809470 2023-10-11T16:33:50.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d48804710 con 0x557d48808b60 2023-10-11T16:33:50.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d48809210 con 0x557d4880d8b0 2023-10-11T16:33:50.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff4fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d4880b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5077fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48814d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff4fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d4880b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55198/0 (socket says 172.21.15.111:55198) 2023-10-11T16:33:50.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff5077fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48814d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40500/0 (socket says 172.21.15.111:40500) 2023-10-11T16:33:50.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.394+0000 7ff4fffff700 1 -- 172.21.15.111:0/2204963642 learned_addr learned my addr 172.21.15.111:0/2204963642 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:50.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff507fff700 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 0x557d4880bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fffff700 1 -- 172.21.15.111:0/2204963642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 msgr2=0x557d4880bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:50.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fffff700 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 0x557d4880bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fffff700 1 -- 172.21.15.111:0/2204963642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 msgr2=0x557d48814d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:50.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fffff700 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48814d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fffff700 1 -- 172.21.15.111:0/2204963642 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d487ffb00 con 0x557d48809470 2023-10-11T16:33:50.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff5077fe700 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48814d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:50.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fffff700 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d4880b990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff4f009aa50 tx=0x7ff4f0005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b51fff9fbc4d7b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:50.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fe7fc700 1 -- 172.21.15.111:0/2204963642 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f0098e60 con 0x557d48809470 2023-10-11T16:33:50.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fe7fc700 1 -- 172.21.15.111:0/2204963642 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4f009ad20 con 0x557d48809470 2023-10-11T16:33:50.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.395+0000 7ff4fe7fc700 1 -- 172.21.15.111:0/2204963642 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f00a2c30 con 0x557d48809470 2023-10-11T16:33:50.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 -- 172.21.15.111:0/2204963642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 msgr2=0x557d4880b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d4880b990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff4f009aa50 tx=0x7ff4f0005570 comp rx=0 tx=0).stop 2023-10-11T16:33:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 -- 172.21.15.111:0/2204963642 shutdown_connections 2023-10-11T16:33:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d4880d8b0 0x557d4880bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48808b60 0x557d48814d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/2204963642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d48809470 0x557d4880b990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 -- 172.21.15.111:0/2204963642 >> 172.21.15.111:0/2204963642 conn(0x557d487fe360 msgr2=0x557d4880fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 -- 172.21.15.111:0/2204963642 shutdown_connections 2023-10-11T16:33:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.396+0000 7ff5217e1b80 1 -- 172.21.15.111:0/2204963642 wait complete. 2023-10-11T16:33:50.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 1 -- start start 2023-10-11T16:33:50.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 1 Processor -- start 2023-10-11T16:33:50.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48808b60 0x557d488119c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48809470 0x557d48811f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d488190b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d488195f0 con 0x557d4880d8b0 2023-10-11T16:33:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d48819760 con 0x557d48809470 2023-10-11T16:33:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d48819920 con 0x557d48808b60 2023-10-11T16:33:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.397+0000 7ff5217e1b80 10 client.0 _ll_register_callbacks cb 0x7ff500009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d488190b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d488190b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55204/0 (socket says 172.21.15.111:55204) 2023-10-11T16:33:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 -- 172.21.15.111:0/1830744087 learned_addr learned my addr 172.21.15.111:0/1830744087 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff4fffff700 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48809470 0x557d48811f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff5077fe700 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48808b60 0x557d488119c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 -- 172.21.15.111:0/1830744087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48808b60 msgr2=0x557d488119c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48808b60 0x557d488119c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 -- 172.21.15.111:0/1830744087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48809470 msgr2=0x557d48811f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48809470 0x557d48811f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 -- 172.21.15.111:0/1830744087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d48720190 con 0x557d4880d8b0 2023-10-11T16:33:50.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff5077fe700 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48808b60 0x557d488119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:33:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff507fff700 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d488190b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff500007d70 tx=0x7ff500015720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b7654f7fdc4597 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff4e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/1830744087 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50001c070 con 0x557d4880d8b0 2023-10-11T16:33:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/1830744087 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff500015e40 con 0x557d4880d8b0 2023-10-11T16:33:50.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.398+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1830744087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4f8004020 con 0x557d4880d8b0 2023-10-11T16:33:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/1830744087 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff50000e040 con 0x557d4880d8b0 2023-10-11T16:33:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff5217e1b80 10 client.15819 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1830744087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7ff508110280 con 0x557d4880d8b0 2023-10-11T16:33:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/1830744087 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4f8004020 con 0x557d4880d8b0 2023-10-11T16:33:50.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/1830744087 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff500016520 con 0x557d4880d8b0 2023-10-11T16:33:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff5217e1b80 10 client.15819 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff5217e1b80 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff5217e1b80 20 client.15819 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557d488768c0 0x557d48878cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:50.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff5217e1b80 4 client.15819 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:33:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.399+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1830744087 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557d48728660 con 0x557d488768c0 2023-10-11T16:33:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.400+0000 7ff5077fe700 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557d488768c0 0x557d48878cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.400+0000 7ff4e6ffd700 10 client.15819 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.400+0000 7ff5077fe700 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557d488768c0 0x557d48878cd0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1897205e714011b1 server_cookie=84494b40beb41561 in_seq=0 out_seq=0 2023-10-11T16:33:50.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.400+0000 7ff4e6ffd700 1 -- 172.21.15.111:0/1830744087 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557d48728660 con 0x557d488768c0 2023-10-11T16:33:50.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.400+0000 7ff4e6ffd700 10 client.15819 handle_command_reply: tid=0 2023-10-11T16:33:50.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.400+0000 7ff5217e1b80 1 client.15819 shutdown 2023-10-11T16:33:50.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.400+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1830744087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 msgr2=0x557d488190b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.400+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d488190b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff500007d70 tx=0x7ff500015720 comp rx=0 tx=0).stop 2023-10-11T16:33:50.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1830744087 shutdown_connections 2023-10-11T16:33:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557d488768c0 0x557d48878cd0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557d4880d8b0 0x557d488190b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557d48808b60 0x557d488119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 1 --2- 172.21.15.111:0/1830744087 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557d48809470 0x557d48811f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff4e6ffd700 0 client.15819 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:50.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1830744087 >> 172.21.15.111:0/1830744087 conn(0x557d487fe360 msgr2=0x557d4880b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:50.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1830744087 shutdown_connections 2023-10-11T16:33:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 1 -- 172.21.15.111:0/1830744087 wait complete. 2023-10-11T16:33:50.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff5217e1b80 20 client.15819 trim_cache size 0 max 16384 2023-10-11T16:33:50.419 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:33:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff50ddff700 1 -- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff4f40a1460 msgr2=0x7ff4f40a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff4f40a1460 0x7ff4f40a3910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009be0 tx=0x7ff4f8009260 comp rx=0 tx=0).stop 2023-10-11T16:33:50.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff50ddff700 1 -- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508101400 msgr2=0x7ff508198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.401+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508101400 0x7ff508198ba0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009810 tx=0x7ff4f000fa00 comp rx=0 tx=0).stop 2023-10-11T16:33:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.402+0000 7ff50ddff700 1 -- 172.21.15.111:0/3643099846 shutdown_connections 2023-10-11T16:33:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.402+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff4f40a1460 0x7ff4f40a3910 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.402+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5081063f0 0x7ff50819d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.402+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff508100a50 0x7ff508198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.402+0000 7ff50ddff700 1 --2- 172.21.15.111:0/3643099846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff508101400 0x7ff508198ba0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.402+0000 7ff50ddff700 1 -- 172.21.15.111:0/3643099846 >> 172.21.15.111:0/3643099846 conn(0x7ff508077590 msgr2=0x7ff50810b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.402+0000 7ff50ddff700 1 -- 172.21.15.111:0/3643099846 shutdown_connections 2023-10-11T16:33:50.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:50.402+0000 7ff50ddff700 1 -- 172.21.15.111:0/3643099846 wait complete. 2023-10-11T16:33:50.567 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:50.568 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:50.577 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:50.578 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:50.578 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 29 -lt 100 ']' 2023-10-11T16:33:50.579 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 29 + 1 2023-10-11T16:33:50.619 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=30 2023-10-11T16:33:50.619 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:50.691 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:50.691 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:50.693 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:50.693 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:50.694 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:51.478 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:33:51.478 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:33:51.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:51 smithi130 ceph-mon[131859]: pgmap v39: 129 pgs: 129 active+clean; 1.6 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 11 op/s 2023-10-11T16:33:51.600 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:51 smithi111 ceph-mon[98527]: pgmap v39: 129 pgs: 129 active+clean; 1.6 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 11 op/s 2023-10-11T16:33:51.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:51 smithi150 ceph-mon[132217]: pgmap v39: 129 pgs: 129 active+clean; 1.6 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 11 op/s 2023-10-11T16:33:52.667 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:52.668 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:52.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.791+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/2111735895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b741015f0 msgr2=0x7f2b74104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:52.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.791+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/2111735895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b741015f0 0x7f2b74104e80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2b640098f0 tx=0x7f2b64009c00 comp rx=0 tx=0).stop 2023-10-11T16:33:52.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.791+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/2111735895 shutdown_connections 2023-10-11T16:33:52.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.791+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/2111735895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b74105560 0x7f2b7410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:52.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.791+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/2111735895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b74100c50 0x7f2b74101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.791+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/2111735895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b741015f0 0x7f2b74104e80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.791+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/2111735895 >> 172.21.15.111:0/2111735895 conn(0x7f2b74077550 msgr2=0x7f2b74077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:52.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.791+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/2111735895 shutdown_connections 2023-10-11T16:33:52.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.792+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/2111735895 wait complete. 2023-10-11T16:33:52.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.793+0000 7f2b7b3ff700 1 Processor -- start 2023-10-11T16:33:52.798 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.793+0000 7f2b7b3ff700 1 -- start start 2023-10-11T16:33:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.793+0000 7f2b7b3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b74100c50 0x7f2b74072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:52.804 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.793+0000 7f2b7b3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b741015f0 0x7f2b74073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:52.804 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.793+0000 7f2b7b3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b74105560 0x7f2b741a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:52.804 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.793+0000 7f2b7b3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b74073b60 con 0x7f2b74100c50 2023-10-11T16:33:52.805 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.793+0000 7f2b7b3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b741a4b60 con 0x7f2b74105560 2023-10-11T16:33:52.805 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.793+0000 7f2b7b3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b741a4ca0 con 0x7f2b741015f0 2023-10-11T16:33:52.805 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b74100c50 0x7f2b74072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:52.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b74100c50 0x7f2b74072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55212/0 (socket says 172.21.15.111:55212) 2023-10-11T16:33:52.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7919b700 1 -- 172.21.15.111:0/1618810425 learned_addr learned my addr 172.21.15.111:0/1618810425 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:52.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7899a700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b741015f0 0x7f2b74073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7999c700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b74105560 0x7f2b741a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7919b700 1 -- 172.21.15.111:0/1618810425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b741015f0 msgr2=0x7f2b74073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7919b700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b741015f0 0x7f2b74073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:52.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7919b700 1 -- 172.21.15.111:0/1618810425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b74105560 msgr2=0x7f2b741a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:52.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7919b700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b74105560 0x7f2b741a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:52.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7919b700 1 -- 172.21.15.111:0/1618810425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b64009520 con 0x7f2b74100c50 2023-10-11T16:33:52.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.794+0000 7f2b7899a700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b741015f0 0x7f2b74073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.795+0000 7f2b7919b700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b74100c50 0x7f2b74072df0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000e9c0 tx=0x7f2b7000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.795+0000 7f2b6a7fc700 1 -- 172.21.15.111:0/1618810425 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70004a30 con 0x7f2b74100c50 2023-10-11T16:33:52.813 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.795+0000 7f2b6a7fc700 1 -- 172.21.15.111:0/1618810425 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b7000cc80 con 0x7f2b74100c50 2023-10-11T16:33:52.813 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.795+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/1618810425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b74110370 con 0x7f2b74100c50 2023-10-11T16:33:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.795+0000 7f2b6a7fc700 1 -- 172.21.15.111:0/1618810425 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b700054e0 con 0x7f2b74100c50 2023-10-11T16:33:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.795+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/1618810425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b741101e0 con 0x7f2b74100c50 2023-10-11T16:33:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.798+0000 7f2b6a7fc700 1 -- 172.21.15.111:0/1618810425 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f2b70004b90 con 0x7f2b74100c50 2023-10-11T16:33:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.800+0000 7f2b6a7fc700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b600a13e0 0x7f2b600a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.800+0000 7f2b6a7fc700 1 -- 172.21.15.111:0/1618810425 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2b70014070 con 0x7f2b74100c50 2023-10-11T16:33:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.800+0000 7f2b7899a700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b600a13e0 0x7f2b600a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:52.801+0000 7f2b7899a700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b600a13e0 0x7f2b600a3890 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009810 tx=0x7f2b64019040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.388+0000 7f2b8ec6fb80 1 Processor -- start 2023-10-11T16:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.388+0000 7f2b8ec6fb80 1 -- start start 2023-10-11T16:33:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3f9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef3fe5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef3feb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef2f7000 con 0x55a9ef3fa4a0 2023-10-11T16:33:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef349840 con 0x55a9ef3fa910 2023-10-11T16:33:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef342cd0 con 0x55a9ef24d6b0 2023-10-11T16:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b7899a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef3fe5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b7999c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef3feb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b7899a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef3fe5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55224/0 (socket says 172.21.15.111:55224) 2023-10-11T16:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b7919b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3f9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b7899a700 1 -- 172.21.15.111:0/2950794824 learned_addr learned my addr 172.21.15.111:0/2950794824 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.389+0000 7f2b7919b700 1 -- 172.21.15.111:0/2950794824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 msgr2=0x55a9ef3feb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b7919b700 1 --2- 172.21.15.111:0/2950794824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef3feb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b7919b700 1 -- 172.21.15.111:0/2950794824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa4a0 msgr2=0x55a9ef3fe5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b7919b700 1 --2- 172.21.15.111:0/2950794824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef3fe5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b7919b700 1 -- 172.21.15.111:0/2950794824 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9ef1c45d0 con 0x55a9ef24d6b0 2023-10-11T16:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b7999c700 1 --2- 172.21.15.111:0/2950794824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef3feb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b7919b700 1 --2- 172.21.15.111:0/2950794824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3f9f60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2b700ca670 tx=0x7f2b70099e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a0cf62e29b95151 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b5f7fe700 1 -- 172.21.15.111:0/2950794824 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70099f20 con 0x55a9ef24d6b0 2023-10-11T16:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b5f7fe700 1 -- 172.21.15.111:0/2950794824 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b7009df50 con 0x55a9ef24d6b0 2023-10-11T16:33:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.390+0000 7f2b5f7fe700 1 -- 172.21.15.111:0/2950794824 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7009a3e0 con 0x55a9ef24d6b0 2023-10-11T16:33:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2950794824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 msgr2=0x55a9ef3f9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2950794824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3f9f60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2b700ca670 tx=0x7f2b70099e20 comp rx=0 tx=0).stop 2023-10-11T16:33:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2950794824 shutdown_connections 2023-10-11T16:33:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2950794824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef3feb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2950794824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3f9f60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2950794824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef3fe5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2950794824 >> 172.21.15.111:0/2950794824 conn(0x55a9ef3e6f50 msgr2=0x55a9ef3ec270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2950794824 shutdown_connections 2023-10-11T16:33:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.391+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2950794824 wait complete. 2023-10-11T16:33:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.392+0000 7f2b8ec6fb80 1 -- start start 2023-10-11T16:33:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.392+0000 7f2b8ec6fb80 1 Processor -- start 2023-10-11T16:33:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef406320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef4035a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef403c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef2e4880 con 0x55a9ef3fa910 2023-10-11T16:33:53.403 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef2e4520 con 0x55a9ef3fa4a0 2023-10-11T16:33:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef1c68c0 con 0x55a9ef24d6b0 2023-10-11T16:33:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b8ec6fb80 10 client.0 _ll_register_callbacks cb 0x55a9ef302720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b7899a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef4035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b7999c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef403c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b7919b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef406320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b7899a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef4035a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40560/0 (socket says 172.21.15.111:40560) 2023-10-11T16:33:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.393+0000 7f2b7899a700 1 -- 172.21.15.111:0/2929484471 learned_addr learned my addr 172.21.15.111:0/2929484471 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b7919b700 1 -- 172.21.15.111:0/2929484471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 msgr2=0x55a9ef4035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b7919b700 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef4035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b7919b700 1 -- 172.21.15.111:0/2929484471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 msgr2=0x55a9ef403c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b7919b700 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef403c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b7919b700 1 -- 172.21.15.111:0/2929484471 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b741101e0 con 0x55a9ef24d6b0 2023-10-11T16:33:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b7919b700 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef406320 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55a9ef3ff2b0 tx=0x7f2b700ca5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=613e3c3ff4d394d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b5cff9700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:33:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b5cff9700 1 -- 172.21.15.111:0/2929484471 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b700b8310 con 0x55a9ef24d6b0 2023-10-11T16:33:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b5cff9700 1 -- 172.21.15.111:0/2929484471 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2b7009a180 con 0x55a9ef24d6b0 2023-10-11T16:33:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2929484471 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9ef3e4710 con 0x55a9ef24d6b0 2023-10-11T16:33:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b5cff9700 1 -- 172.21.15.111:0/2929484471 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b7009a340 con 0x55a9ef24d6b0 2023-10-11T16:33:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b8ec6fb80 10 client.25906 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.394+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2929484471 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a9ef309130 con 0x55a9ef24d6b0 2023-10-11T16:33:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b5cff9700 1 -- 172.21.15.111:0/2929484471 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a9ef3e4710 con 0x55a9ef24d6b0 2023-10-11T16:33:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b5cff9700 1 -- 172.21.15.111:0/2929484471 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b700ca950 con 0x55a9ef24d6b0 2023-10-11T16:33:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b8ec6fb80 10 client.25906 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b8ec6fb80 10 client.25906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b8ec6fb80 20 client.25906 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a9ef4133e0 0x55a9ef4157c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b8ec6fb80 4 client.25906 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2929484471 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9ef305040 con 0x55a9ef4133e0 2023-10-11T16:33:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.395+0000 7f2b7899a700 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a9ef4133e0 0x55a9ef4157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.396+0000 7f2b7899a700 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a9ef4133e0 0x55a9ef4157c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d3f8e645a3ef2fb server_cookie=9243487fb58e94ed in_seq=0 out_seq=0 2023-10-11T16:33:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.396+0000 7f2b5cff9700 10 client.25906 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b5cff9700 1 -- 172.21.15.111:0/2929484471 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a9ef305040 con 0x55a9ef4133e0 2023-10-11T16:33:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b5cff9700 10 client.25906 handle_command_reply: tid=0 2023-10-11T16:33:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 client.25906 shutdown 2023-10-11T16:33:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2929484471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 msgr2=0x55a9ef406320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef406320 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55a9ef3ff2b0 tx=0x7f2b700ca5e0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2929484471 shutdown_connections 2023-10-11T16:33:53.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a9ef4133e0 0x55a9ef4157c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef403c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef406320 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2929484471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef4035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b5cff9700 0 client.25906 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2929484471 >> 172.21.15.111:0/2929484471 conn(0x55a9ef3e6f50 msgr2=0x55a9ef3eb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2929484471 shutdown_connections 2023-10-11T16:33:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.397+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2929484471 wait complete. 2023-10-11T16:33:53.413 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.398+0000 7f2b8ec6fb80 20 client.25906 trim_cache size 0 max 16384 2023-10-11T16:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.414+0000 7f2b8ec6fb80 1 Processor -- start 2023-10-11T16:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b8ec6fb80 1 -- start start 2023-10-11T16:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3ebf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef405e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef3fe2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef3ff590 con 0x55a9ef3fa910 2023-10-11T16:33:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef3e6b70 con 0x55a9ef3fa4a0 2023-10-11T16:33:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef3f9f60 con 0x55a9ef24d6b0 2023-10-11T16:33:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b7999c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef3fe2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b7999c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef3fe2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55238/0 (socket says 172.21.15.111:55238) 2023-10-11T16:33:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b7999c700 1 -- 172.21.15.111:0/2761801886 learned_addr learned my addr 172.21.15.111:0/2761801886 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b7899a700 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef405e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.415+0000 7f2b7919b700 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3ebf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b7899a700 1 -- 172.21.15.111:0/2761801886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 msgr2=0x55a9ef3ebf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:53.419 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b7899a700 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3ebf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b7899a700 1 -- 172.21.15.111:0/2761801886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 msgr2=0x55a9ef3fe2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b7899a700 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef3fe2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b7899a700 1 -- 172.21.15.111:0/2761801886 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9ef308e80 con 0x55a9ef3fa4a0 2023-10-11T16:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b7899a700 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef405e50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x55a9ef4160d0 tx=0x7f2b6401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3fc65936ef34112 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b5ffff700 1 -- 172.21.15.111:0/2761801886 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64005110 con 0x55a9ef3fa4a0 2023-10-11T16:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b5ffff700 1 -- 172.21.15.111:0/2761801886 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b6401d040 con 0x55a9ef3fa4a0 2023-10-11T16:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.416+0000 7f2b5ffff700 1 -- 172.21.15.111:0/2761801886 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64003e50 con 0x55a9ef3fa4a0 2023-10-11T16:33:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2761801886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 msgr2=0x55a9ef405e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef405e50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x55a9ef4160d0 tx=0x7f2b6401f000 comp rx=0 tx=0).stop 2023-10-11T16:33:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2761801886 shutdown_connections 2023-10-11T16:33:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef3fa910 0x55a9ef3fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef24d6b0 0x55a9ef3ebf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/2761801886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef405e50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2761801886 >> 172.21.15.111:0/2761801886 conn(0x55a9ef3e6f50 msgr2=0x55a9ef3fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2761801886 shutdown_connections 2023-10-11T16:33:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.417+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/2761801886 wait complete. 2023-10-11T16:33:53.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.418+0000 7f2b8ec6fb80 1 -- start start 2023-10-11T16:33:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.418+0000 7f2b8ec6fb80 1 Processor -- start 2023-10-11T16:33:53.427 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.418+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef24d6b0 0x55a9ef40a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.418+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef408a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b8ec6fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef408fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef4094f0 con 0x55a9ef24d6b0 2023-10-11T16:33:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef409660 con 0x55a9ef3fa910 2023-10-11T16:33:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b8ec6fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9ef409820 con 0x55a9ef3fa4a0 2023-10-11T16:33:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b8ec6fb80 10 client.0 _ll_register_callbacks cb 0x7f2b64026040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef24d6b0 0x55a9ef40a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef24d6b0 0x55a9ef40a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55240/0 (socket says 172.21.15.111:55240) 2023-10-11T16:33:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7919b700 1 -- 172.21.15.111:0/3432155364 learned_addr learned my addr 172.21.15.111:0/3432155364 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7899a700 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef408a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7999c700 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef408fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7919b700 1 -- 172.21.15.111:0/3432155364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef3fa4a0 msgr2=0x55a9ef408a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7919b700 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef408a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7919b700 1 -- 172.21.15.111:0/3432155364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 msgr2=0x55a9ef408fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7919b700 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef408fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.419+0000 7f2b7919b700 1 -- 172.21.15.111:0/3432155364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7410d9b0 con 0x55a9ef24d6b0 2023-10-11T16:33:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b7919b700 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef24d6b0 0x55a9ef40a740 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2b70096990 tx=0x7f2b700ca7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e73d0b3716e81 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b5cff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:33:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b5cff9700 1 -- 172.21.15.111:0/3432155364 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70096260 con 0x55a9ef24d6b0 2023-10-11T16:33:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b5cff9700 1 -- 172.21.15.111:0/3432155364 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2b700b8310 con 0x55a9ef24d6b0 2023-10-11T16:33:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/3432155364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b7009df50 con 0x55a9ef24d6b0 2023-10-11T16:33:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b5cff9700 1 -- 172.21.15.111:0/3432155364 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b70099880 con 0x55a9ef24d6b0 2023-10-11T16:33:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b8ec6fb80 10 client.15837 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/3432155364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a9ef159c00 con 0x55a9ef24d6b0 2023-10-11T16:33:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.420+0000 7f2b5cff9700 1 -- 172.21.15.111:0/3432155364 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b7007c020 con 0x55a9ef24d6b0 2023-10-11T16:33:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.421+0000 7f2b5cff9700 1 -- 172.21.15.111:0/3432155364 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b7009a180 con 0x55a9ef24d6b0 2023-10-11T16:33:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.421+0000 7f2b8ec6fb80 10 client.15837 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.421+0000 7f2b8ec6fb80 10 client.15837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.421+0000 7f2b8ec6fb80 20 client.15837 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.421+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a9ef468200 0x55a9ef46a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.421+0000 7f2b8ec6fb80 4 client.15837 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:33:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.421+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/3432155364 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a9ef311600 con 0x55a9ef468200 2023-10-11T16:33:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.421+0000 7f2b7899a700 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a9ef468200 0x55a9ef46a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.422+0000 7f2b5cff9700 10 client.15837 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:53.441 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.422+0000 7f2b7899a700 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a9ef468200 0x55a9ef46a5e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d739d41eceebc1 server_cookie=53e9d31f68a18247 in_seq=0 out_seq=0 2023-10-11T16:33:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.429+0000 7f2b5cff9700 1 -- 172.21.15.111:0/3432155364 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a9ef311600 con 0x55a9ef468200 2023-10-11T16:33:53.442 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.429+0000 7f2b5cff9700 10 client.15837 handle_command_reply: tid=0 2023-10-11T16:33:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.429+0000 7f2b8ec6fb80 1 client.15837 shutdown 2023-10-11T16:33:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.429+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/3432155364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef24d6b0 msgr2=0x55a9ef40a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.429+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef24d6b0 0x55a9ef40a740 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2b70096990 tx=0x7f2b700ca7b0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/3432155364 shutdown_connections 2023-10-11T16:33:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a9ef468200 0x55a9ef46a5e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a9ef3fa910 0x55a9ef408fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a9ef24d6b0 0x55a9ef40a740 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.444 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b5cff9700 0 client.15837 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 1 --2- 172.21.15.111:0/3432155364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9ef3fa4a0 0x55a9ef408a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/3432155364 >> 172.21.15.111:0/3432155364 conn(0x55a9ef3e6f50 msgr2=0x55a9ef3fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/3432155364 shutdown_connections 2023-10-11T16:33:53.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 1 -- 172.21.15.111:0/3432155364 wait complete. 2023-10-11T16:33:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.430+0000 7f2b8ec6fb80 20 client.15837 trim_cache size 0 max 16384 2023-10-11T16:33:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:33:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:33:53.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:33:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:33:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b600a13e0 msgr2=0x7f2b600a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b600a13e0 0x7f2b600a3890 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009810 tx=0x7f2b64019040 comp rx=0 tx=0).stop 2023-10-11T16:33:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b74100c50 msgr2=0x7f2b74072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b74100c50 0x7f2b74072df0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000e9c0 tx=0x7f2b7000ecd0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/1618810425 shutdown_connections 2023-10-11T16:33:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b600a13e0 0x7f2b600a3890 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b74105560 0x7f2b741a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b74100c50 0x7f2b74072df0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 --2- 172.21.15.111:0/1618810425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b741015f0 0x7f2b74073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:53.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/1618810425 >> 172.21.15.111:0/1618810425 conn(0x7f2b74077550 msgr2=0x7f2b740fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/1618810425 shutdown_connections 2023-10-11T16:33:53.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:33:53.431+0000 7f2b7b3ff700 1 -- 172.21.15.111:0/1618810425 wait complete. 2023-10-11T16:33:53.491 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:53.491 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:53.511 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:53.512 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:53.512 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 30 -lt 100 ']' 2023-10-11T16:33:53.512 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 30 + 1 2023-10-11T16:33:53.552 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=31 2023-10-11T16:33:53.552 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:53.563 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:53.564 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:53.566 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:53.567 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:53.567 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:53.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:53 smithi130 ceph-mon[131859]: pgmap v40: 129 pgs: 129 active+clean; 1.6 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-10-11T16:33:53.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:53 smithi111 ceph-mon[98527]: pgmap v40: 129 pgs: 129 active+clean; 1.6 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-10-11T16:33:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:53 smithi150 ceph-mon[132217]: pgmap v40: 129 pgs: 129 active+clean; 1.6 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-10-11T16:33:54.609 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:54.609 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:54.690 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:54.690 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:54.710 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:54.711 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:54.711 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 31 -lt 100 ']' 2023-10-11T16:33:54.711 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 31 + 1 2023-10-11T16:33:54.769 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=32 2023-10-11T16:33:54.769 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:54.800 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:54.801 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:54.803 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:54.803 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:54.804 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:55.170 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:54 smithi111 ceph-mon[98527]: pgmap v41: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 11 op/s 2023-10-11T16:33:55.171 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:54 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:33:55.171 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:54 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:33:55.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:54 smithi150 ceph-mon[132217]: pgmap v41: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 11 op/s 2023-10-11T16:33:55.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:54 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:33:55.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:54 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:33:55.331 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:54 smithi130 ceph-mon[131859]: pgmap v41: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 11 op/s 2023-10-11T16:33:55.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:54 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:33:55.332 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:54 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:33:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:55.350 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:55.608 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:55.609 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:55.626 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:55.627 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:55.627 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 32 -lt 100 ']' 2023-10-11T16:33:55.627 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 32 + 1 2023-10-11T16:33:55.685 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=33 2023-10-11T16:33:55.685 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:55.766 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:55.767 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:55.769 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:55.769 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:55.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:56.509 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:56.509 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:56.708 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:56.709 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:56.769 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:56.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:56.770 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 33 -lt 100 ']' 2023-10-11T16:33:56.771 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 33 + 1 2023-10-11T16:33:56.901 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=34 2023-10-11T16:33:56.902 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:57.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:57.031 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:57.033 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:57.034 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:57.034 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:57.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 -- 172.21.15.111:0/893664778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43541063f0 msgr2=0x7f43541067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 --2- 172.21.15.111:0/893664778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43541063f0 0x7f43541067c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4344009990 tx=0x7f4344009ca0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 -- 172.21.15.111:0/893664778 shutdown_connections 2023-10-11T16:33:57.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 --2- 172.21.15.111:0/893664778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4354101400 0x7f435410d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 --2- 172.21.15.111:0/893664778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43541063f0 0x7f43541067c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 --2- 172.21.15.111:0/893664778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4354100a50 0x7f4354100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 -- 172.21.15.111:0/893664778 >> 172.21.15.111:0/893664778 conn(0x7f4354077590 msgr2=0x7f4354077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:57.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 -- 172.21.15.111:0/893664778 shutdown_connections 2023-10-11T16:33:57.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.098+0000 7f4359dff700 1 -- 172.21.15.111:0/893664778 wait complete. 2023-10-11T16:33:57.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.100+0000 7f4359dff700 1 Processor -- start 2023-10-11T16:33:57.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.100+0000 7f4359dff700 1 -- start start 2023-10-11T16:33:57.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4359dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4354100a50 0x7f4354196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4359dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4354101400 0x7f43541969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4359dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43541063f0 0x7f435419ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4359dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435419b450 con 0x7f4354101400 2023-10-11T16:33:57.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4359dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435419b5c0 con 0x7f43541063f0 2023-10-11T16:33:57.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4359dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435419b780 con 0x7f4354100a50 2023-10-11T16:33:57.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4352ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4354101400 0x7f43541969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4352ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4354101400 0x7f43541969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35028/0 (socket says 172.21.15.111:35028) 2023-10-11T16:33:57.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4352ffd700 1 -- 172.21.15.111:0/8229587 learned_addr learned my addr 172.21.15.111:0/8229587 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:57.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f43537fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4354100a50 0x7f4354196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4353fff700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43541063f0 0x7f435419ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4352ffd700 1 -- 172.21.15.111:0/8229587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4354100a50 msgr2=0x7f4354196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4352ffd700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4354100a50 0x7f4354196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4352ffd700 1 -- 172.21.15.111:0/8229587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43541063f0 msgr2=0x7f435419ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4352ffd700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43541063f0 0x7f435419ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f4352ffd700 1 -- 172.21.15.111:0/8229587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43440095f0 con 0x7f4354101400 2023-10-11T16:33:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.101+0000 7f43537fe700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4354100a50 0x7f4354196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.102+0000 7f4352ffd700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4354101400 0x7f43541969b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f434800e9a0 tx=0x7f434800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.102+0000 7f4350ff9700 1 -- 172.21.15.111:0/8229587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348009790 con 0x7f4354101400 2023-10-11T16:33:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.102+0000 7f4350ff9700 1 -- 172.21.15.111:0/8229587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4348010430 con 0x7f4354101400 2023-10-11T16:33:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.102+0000 7f4359dff700 1 -- 172.21.15.111:0/8229587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43541103d0 con 0x7f4354101400 2023-10-11T16:33:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.102+0000 7f4350ff9700 1 -- 172.21.15.111:0/8229587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348018680 con 0x7f4354101400 2023-10-11T16:33:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.102+0000 7f4359dff700 1 -- 172.21.15.111:0/8229587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4354110240 con 0x7f4354101400 2023-10-11T16:33:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.105+0000 7f4350ff9700 1 -- 172.21.15.111:0/8229587 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f43480105a0 con 0x7f4354101400 2023-10-11T16:33:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.106+0000 7f4350ff9700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f433c0a13e0 0x7f433c0a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.106+0000 7f4350ff9700 1 -- 172.21.15.111:0/8229587 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f4348014070 con 0x7f4354101400 2023-10-11T16:33:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.106+0000 7f43537fe700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f433c0a13e0 0x7f433c0a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.107+0000 7f43537fe700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f433c0a13e0 0x7f433c0a3890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f434400b2e0 tx=0x7f4344005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:57.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.254+0000 7f436d74db80 1 Processor -- start 2023-10-11T16:33:57.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.254+0000 7f436d74db80 1 -- start start 2023-10-11T16:33:57.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.254+0000 7f436d74db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42cfbe0 0x563db42cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.254+0000 7f436d74db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d04f0 0x563db42d43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.254+0000 7f436d74db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.254+0000 7f436d74db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db41d50e0 con 0x563db42d04f0 2023-10-11T16:33:57.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.254+0000 7f436d74db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db4227920 con 0x563db42d4930 2023-10-11T16:33:57.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.254+0000 7f436d74db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db4220db0 con 0x563db42cfbe0 2023-10-11T16:33:57.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d04f0 0x563db42d43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d04f0 0x563db42d43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35032/0 (socket says 172.21.15.111:35032) 2023-10-11T16:33:57.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 -- 172.21.15.111:0/328778946 learned_addr learned my addr 172.21.15.111:0/328778946 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:57.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4353fff700 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f43537fe700 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42cfbe0 0x563db42cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 -- 172.21.15.111:0/328778946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42cfbe0 msgr2=0x563db42cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:57.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42cfbe0 0x563db42cffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 -- 172.21.15.111:0/328778946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 msgr2=0x563db42d4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:57.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 -- 172.21.15.111:0/328778946 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563db40a26b0 con 0x563db42d04f0 2023-10-11T16:33:57.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4353fff700 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:57.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.255+0000 7f4352ffd700 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d04f0 0x563db42d43f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f434800c2d0 tx=0x7f4348099c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e3158ad3ad72f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:57.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.256+0000 7f4339ffb700 1 -- 172.21.15.111:0/328778946 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43480a2450 con 0x563db42d04f0 2023-10-11T16:33:57.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.256+0000 7f4339ffb700 1 -- 172.21.15.111:0/328778946 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4348010880 con 0x563db42d04f0 2023-10-11T16:33:57.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.256+0000 7f4339ffb700 1 -- 172.21.15.111:0/328778946 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348099710 con 0x563db42d04f0 2023-10-11T16:33:57.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.256+0000 7f436d74db80 1 -- 172.21.15.111:0/328778946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d04f0 msgr2=0x563db42d43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.256+0000 7f436d74db80 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d04f0 0x563db42d43f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f434800c2d0 tx=0x7f4348099c50 comp rx=0 tx=0).stop 2023-10-11T16:33:57.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 -- 172.21.15.111:0/328778946 shutdown_connections 2023-10-11T16:33:57.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42cfbe0 0x563db42cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 --2- 172.21.15.111:0/328778946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d04f0 0x563db42d43f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 -- 172.21.15.111:0/328778946 >> 172.21.15.111:0/328778946 conn(0x563db42c53e0 msgr2=0x563db42cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:57.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 -- 172.21.15.111:0/328778946 shutdown_connections 2023-10-11T16:33:57.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 -- 172.21.15.111:0/328778946 wait complete. 2023-10-11T16:33:57.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 -- start start 2023-10-11T16:33:57.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.257+0000 7f436d74db80 1 Processor -- start 2023-10-11T16:33:57.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f436d74db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42cfbe0 0x563db42dc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f436d74db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42d93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f436d74db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d4930 0x563db42d9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f436d74db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db41c2960 con 0x563db42d4930 2023-10-11T16:33:57.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f436d74db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db41c2600 con 0x563db42cfbe0 2023-10-11T16:33:57.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f436d74db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db40a49a0 con 0x563db42d04f0 2023-10-11T16:33:57.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f436d74db80 10 client.0 _ll_register_callbacks cb 0x563db41e0800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:57.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f43537fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42cfbe0 0x563db42dc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f43537fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42cfbe0 0x563db42dc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:58506/0 (socket says 172.21.15.111:58506) 2023-10-11T16:33:57.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f4353fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d4930 0x563db42d9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f43537fe700 1 -- 172.21.15.111:0/1337507650 learned_addr learned my addr 172.21.15.111:0/1337507650 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:57.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.258+0000 7f4352ffd700 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42d93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f43537fe700 1 -- 172.21.15.111:0/1337507650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 msgr2=0x563db42d93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:57.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f43537fe700 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42d93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f43537fe700 1 -- 172.21.15.111:0/1337507650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d4930 msgr2=0x563db42d9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:57.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f43537fe700 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d4930 0x563db42d9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f43537fe700 1 -- 172.21.15.111:0/1337507650 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4354110240 con 0x563db42cfbe0 2023-10-11T16:33:57.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f4353fff700 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d4930 0x563db42d9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:57.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f43537fe700 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42cfbe0 0x563db42dc150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4344015000 tx=0x7f4344020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a01e748fde7926e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:57.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f4332ffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:33:57.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f4332ffd700 1 -- 172.21.15.111:0/1337507650 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434402d040 con 0x563db42cfbe0 2023-10-11T16:33:57.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f436d74db80 1 -- 172.21.15.111:0/1337507650 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563db42c2ba0 con 0x563db42cfbe0 2023-10-11T16:33:57.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f4332ffd700 1 -- 172.21.15.111:0/1337507650 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f4344004a30 con 0x563db42cfbe0 2023-10-11T16:33:57.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f4332ffd700 1 -- 172.21.15.111:0/1337507650 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4344004bf0 con 0x563db42cfbe0 2023-10-11T16:33:57.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f436d74db80 10 client.25892 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:57.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.259+0000 7f436d74db80 1 -- 172.21.15.111:0/1337507650 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563db41e7210 con 0x563db42cfbe0 2023-10-11T16:33:57.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f4332ffd700 1 -- 172.21.15.111:0/1337507650 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563db42c2ba0 con 0x563db42cfbe0 2023-10-11T16:33:57.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f4332ffd700 1 -- 172.21.15.111:0/1337507650 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4344005020 con 0x563db42cfbe0 2023-10-11T16:33:57.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f436d74db80 10 client.25892 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:57.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f436d74db80 10 client.25892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:57.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f436d74db80 20 client.25892 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f436d74db80 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563db42e9a20 0x563db42ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f436d74db80 4 client.25892 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:33:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f436d74db80 1 -- 172.21.15.111:0/1337507650 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563db41e3120 con 0x563db42e9a20 2023-10-11T16:33:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.260+0000 7f4352ffd700 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563db42e9a20 0x563db42ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.261+0000 7f4332ffd700 10 client.25892 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:57.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.261+0000 7f4352ffd700 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563db42e9a20 0x563db42ebe30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a6a05eae3ae5a45 server_cookie=b768c1561f39ae2e in_seq=0 out_seq=0 2023-10-11T16:33:57.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f4332ffd700 1 -- 172.21.15.111:0/1337507650 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563db41e3120 con 0x563db42e9a20 2023-10-11T16:33:57.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f4332ffd700 10 client.25892 handle_command_reply: tid=0 2023-10-11T16:33:57.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 client.25892 shutdown 2023-10-11T16:33:57.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 -- 172.21.15.111:0/1337507650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42cfbe0 msgr2=0x563db42dc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42cfbe0 0x563db42dc150 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4344015000 tx=0x7f4344020000 comp rx=0 tx=0).stop 2023-10-11T16:33:57.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 -- 172.21.15.111:0/1337507650 shutdown_connections 2023-10-11T16:33:57.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563db42e9a20 0x563db42ebe30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42d4930 0x563db42d9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42cfbe0 0x563db42dc150 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 --2- 172.21.15.111:0/1337507650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42d93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f4332ffd700 0 client.25892 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:57.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 -- 172.21.15.111:0/1337507650 >> 172.21.15.111:0/1337507650 conn(0x563db42c53e0 msgr2=0x563db42d5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:57.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 -- 172.21.15.111:0/1337507650 shutdown_connections 2023-10-11T16:33:57.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 1 -- 172.21.15.111:0/1337507650 wait complete. 2023-10-11T16:33:57.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.262+0000 7f436d74db80 20 client.25892 trim_cache size 0 max 16384 2023-10-11T16:33:57.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.280+0000 7f436d74db80 1 Processor -- start 2023-10-11T16:33:57.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.280+0000 7f436d74db80 1 -- start start 2023-10-11T16:33:57.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.280+0000 7f436d74db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42d4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.280+0000 7f436d74db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.280+0000 7f436d74db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.280+0000 7f436d74db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db42da990 con 0x563db42cfbe0 2023-10-11T16:33:57.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.280+0000 7f436d74db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db42d7980 con 0x563db42d4930 2023-10-11T16:33:57.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.280+0000 7f436d74db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db42d3500 con 0x563db42d04f0 2023-10-11T16:33:57.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42d4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42d4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35050/0 (socket says 172.21.15.111:35050) 2023-10-11T16:33:57.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 -- 172.21.15.111:0/3279925827 learned_addr learned my addr 172.21.15.111:0/3279925827 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:57.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f4352ffd700 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f4353fff700 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 -- 172.21.15.111:0/3279925827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 msgr2=0x563db42dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:57.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 -- 172.21.15.111:0/3279925827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 msgr2=0x563db42d8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:57.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 -- 172.21.15.111:0/3279925827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563db41e6f60 con 0x563db42cfbe0 2023-10-11T16:33:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f43537fe700 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42d4020 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4344005a40 tx=0x7f43440262e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdc1787cdace7bc server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f433a7fc700 1 -- 172.21.15.111:0/3279925827 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434402d040 con 0x563db42cfbe0 2023-10-11T16:33:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f433a7fc700 1 -- 172.21.15.111:0/3279925827 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f434401e040 con 0x563db42cfbe0 2023-10-11T16:33:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.281+0000 7f433a7fc700 1 -- 172.21.15.111:0/3279925827 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344004bb0 con 0x563db42cfbe0 2023-10-11T16:33:57.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 -- 172.21.15.111:0/3279925827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 msgr2=0x563db42d4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42d4020 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4344005a40 tx=0x7f43440262e0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 -- 172.21.15.111:0/3279925827 shutdown_connections 2023-10-11T16:33:57.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42d8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42d4020 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 --2- 172.21.15.111:0/3279925827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 -- 172.21.15.111:0/3279925827 >> 172.21.15.111:0/3279925827 conn(0x563db42c53e0 msgr2=0x563db42d3910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:57.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 -- 172.21.15.111:0/3279925827 shutdown_connections 2023-10-11T16:33:57.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.282+0000 7f436d74db80 1 -- 172.21.15.111:0/3279925827 wait complete. 2023-10-11T16:33:57.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 1 -- start start 2023-10-11T16:33:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 1 Processor -- start 2023-10-11T16:33:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42e0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42df150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db42df690 con 0x563db42cfbe0 2023-10-11T16:33:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db42df7d0 con 0x563db42d4930 2023-10-11T16:33:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563db42df990 con 0x563db42d04f0 2023-10-11T16:33:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.283+0000 7f436d74db80 10 client.0 _ll_register_callbacks cb 0x7f434400f770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:33:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f43537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42e0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4353fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42df150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55772/0 (socket says 172.21.15.111:55772) 2023-10-11T16:33:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 -- 172.21.15.111:0/804034258 learned_addr learned my addr 172.21.15.111:0/804034258 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:33:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 -- 172.21.15.111:0/804034258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 msgr2=0x563db42df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:57.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42df150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 -- 172.21.15.111:0/804034258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 msgr2=0x563db42e0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:33:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42e0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 -- 172.21.15.111:0/804034258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563db41e7210 con 0x563db42d04f0 2023-10-11T16:33:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f43537fe700 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42e0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:33:57.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4352ffd700 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dec10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f434800e970 tx=0x7f4348099a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a482c8f9faa8bc34 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:33:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4332ffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:33:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.284+0000 7f4332ffd700 1 -- 172.21.15.111:0/804034258 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348096340 con 0x563db42d04f0 2023-10-11T16:33:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 1 -- 172.21.15.111:0/804034258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4348010880 con 0x563db42d04f0 2023-10-11T16:33:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f4332ffd700 1 -- 172.21.15.111:0/804034258 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f43480b7070 con 0x563db42d04f0 2023-10-11T16:33:57.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f4332ffd700 1 -- 172.21.15.111:0/804034258 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43480a2450 con 0x563db42d04f0 2023-10-11T16:33:57.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 10 client.25930 fetch_fsmap learned FSMap version 28 2023-10-11T16:33:57.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 1 -- 172.21.15.111:0/804034258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563db4037ce0 con 0x563db42d04f0 2023-10-11T16:33:57.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f4332ffd700 1 -- 172.21.15.111:0/804034258 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4348010880 con 0x563db42d04f0 2023-10-11T16:33:57.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f4332ffd700 1 -- 172.21.15.111:0/804034258 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f43480a1fc0 con 0x563db42d04f0 2023-10-11T16:33:57.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 10 client.25930 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:33:57.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 10 client.25930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:33:57.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 20 client.25930 populate_metadata read hostname 'smithi111' 2023-10-11T16:33:57.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563db433e9e0 0x563db4340df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:33:57.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 4 client.25930 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:33:57.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.285+0000 7f436d74db80 1 -- 172.21.15.111:0/804034258 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563db41ef6e0 con 0x563db433e9e0 2023-10-11T16:33:57.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.286+0000 7f43537fe700 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563db433e9e0 0x563db4340df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:33:57.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.286+0000 7f4332ffd700 10 client.25930 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:57.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.286+0000 7f43537fe700 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563db433e9e0 0x563db4340df0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cf33ec0d76e4cbb server_cookie=d15a6971c81b3830 in_seq=0 out_seq=0 2023-10-11T16:33:57.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f4332ffd700 1 -- 172.21.15.111:0/804034258 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563db41ef6e0 con 0x563db433e9e0 2023-10-11T16:33:57.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f4332ffd700 10 client.25930 handle_command_reply: tid=0 2023-10-11T16:33:57.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 client.25930 shutdown 2023-10-11T16:33:57.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 -- 172.21.15.111:0/804034258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 msgr2=0x563db42dec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dec10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f434800e970 tx=0x7f4348099a10 comp rx=0 tx=0).stop 2023-10-11T16:33:57.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 -- 172.21.15.111:0/804034258 shutdown_connections 2023-10-11T16:33:57.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563db433e9e0 0x563db4340df0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f4332ffd700 0 client.25930 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:33:57.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563db42d4930 0x563db42df150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563db42cfbe0 0x563db42e0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 --2- 172.21.15.111:0/804034258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563db42d04f0 0x563db42dec10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 -- 172.21.15.111:0/804034258 >> 172.21.15.111:0/804034258 conn(0x563db42c53e0 msgr2=0x563db42cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:57.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 -- 172.21.15.111:0/804034258 shutdown_connections 2023-10-11T16:33:57.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 1 -- 172.21.15.111:0/804034258 wait complete. 2023-10-11T16:33:57.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.288+0000 7f436d74db80 20 client.25930 trim_cache size 0 max 16384 2023-10-11T16:33:57.303 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:33:57.303 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:33:57.303 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "db36a076-41b2-441c-8798-7113b912baa8", 2023-10-11T16:33:57.303 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:33:57.303 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:33:57.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 -- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f433c0a13e0 msgr2=0x7f433c0a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f433c0a13e0 0x7f433c0a3890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f434400b2e0 tx=0x7f4344005fb0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 -- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4354101400 msgr2=0x7f43541969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:33:57.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4354101400 0x7f43541969b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f434800e9a0 tx=0x7f434800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 -- 172.21.15.111:0/8229587 shutdown_connections 2023-10-11T16:33:57.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f433c0a13e0 0x7f433c0a3890 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43541063f0 0x7f435419ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4354100a50 0x7f4354196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 --2- 172.21.15.111:0/8229587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4354101400 0x7f43541969b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:33:57.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 -- 172.21.15.111:0/8229587 >> 172.21.15.111:0/8229587 conn(0x7f4354077590 msgr2=0x7f43540ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:33:57.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 -- 172.21.15.111:0/8229587 shutdown_connections 2023-10-11T16:33:57.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:33:57.289+0000 7f4359dff700 1 -- 172.21.15.111:0/8229587 wait complete. 2023-10-11T16:33:57.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:57 smithi111 ceph-mon[98527]: pgmap v42: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:33:57.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:57 smithi150 ceph-mon[132217]: pgmap v42: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:33:57.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:33:57.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:57 smithi130 ceph-mon[131859]: pgmap v42: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:33:58.066 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:58.067 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:58.158 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:58.158 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:58.169 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:58.169 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:58.170 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 34 -lt 100 ']' 2023-10-11T16:33:58.170 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 34 + 1 2023-10-11T16:33:58.209 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=35 2023-10-11T16:33:58.210 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:58.296 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:58.296 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:58.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:58.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:58.300 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:33:58.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:58 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:33:58.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:58 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:58.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:58 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:33:58.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:58 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:58.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:58 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:33:58.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:58 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:58 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:33:58.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:58 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:58.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:58 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:33:58.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:58 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:58.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:58 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:33:58.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:58 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:59.343 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:59 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:33:59.343 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:59 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:59.343 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:33:59 smithi111 ceph-mon[98527]: pgmap v43: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:33:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:59 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:33:59.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:59 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:59.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:33:59 smithi150 ceph-mon[132217]: pgmap v43: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:33:59.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:59 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:33:59.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:59 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:33:59.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:33:59 smithi130 ceph-mon[131859]: pgmap v43: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:33:59.583 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:59.583 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:59.732 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:33:59.733 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:33:59.751 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:59.751 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:33:59.751 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 35 -lt 100 ']' 2023-10-11T16:33:59.752 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 35 + 1 2023-10-11T16:33:59.792 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=36 2023-10-11T16:33:59.793 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:33:59.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:33:59.848 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:33:59.850 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:33:59.851 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:33:59.851 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:00.816 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:00.816 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:00.906 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:00.906 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:00.951 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:00.951 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:00.951 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 36 -lt 100 ']' 2023-10-11T16:34:00.952 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 36 + 1 2023-10-11T16:34:01.029 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=37 2023-10-11T16:34:01.029 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:01.046 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:01.047 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:01.048 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:01.049 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:01.050 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:01.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:01 smithi111 ceph-mon[98527]: pgmap v44: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-10-11T16:34:01.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:01 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:34:01.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:01 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:34:01.386 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:01 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:34:01.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:01 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:34:01.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:01 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:34:01.387 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:01 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:34:01.388 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:01 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:34:01.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:01 smithi150 ceph-mon[132217]: pgmap v44: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-10-11T16:34:01.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:01 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:34:01.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:01 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:34:01.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:01 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:34:01.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:01 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:34:01.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:01 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:34:01.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:01 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:34:01.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:01 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:34:01.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 -- 172.21.15.111:0/2824831302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45281063f0 msgr2=0x7f45281067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 --2- 172.21.15.111:0/2824831302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45281063f0 0x7f45281067c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4518009990 tx=0x7f4518009ca0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 -- 172.21.15.111:0/2824831302 shutdown_connections 2023-10-11T16:34:01.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 --2- 172.21.15.111:0/2824831302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4528101400 0x7f452810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 --2- 172.21.15.111:0/2824831302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45281063f0 0x7f45281067c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 --2- 172.21.15.111:0/2824831302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4528100a50 0x7f4528100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 -- 172.21.15.111:0/2824831302 >> 172.21.15.111:0/2824831302 conn(0x7f4528077590 msgr2=0x7f4528077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:01.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 -- 172.21.15.111:0/2824831302 shutdown_connections 2023-10-11T16:34:01.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.506+0000 7f452ddff700 1 -- 172.21.15.111:0/2824831302 wait complete. 2023-10-11T16:34:01.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.507+0000 7f452ddff700 1 Processor -- start 2023-10-11T16:34:01.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f452ddff700 1 -- start start 2023-10-11T16:34:01.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f452ddff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4528100a50 0x7f4528196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f452ddff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4528101400 0x7f4528196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f452ddff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45281063f0 0x7f452819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f452ddff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452819b410 con 0x7f4528100a50 2023-10-11T16:34:01.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f452ddff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452819b580 con 0x7f4528101400 2023-10-11T16:34:01.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f452ddff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452819b740 con 0x7f45281063f0 2023-10-11T16:34:01.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f45277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4528100a50 0x7f4528196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.508+0000 7f45277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4528100a50 0x7f4528196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35078/0 (socket says 172.21.15.111:35078) 2023-10-11T16:34:01.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45277fe700 1 -- 172.21.15.111:0/2301896147 learned_addr learned my addr 172.21.15.111:0/2301896147 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:01.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f4527fff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45281063f0 0x7f452819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f451ffff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4528101400 0x7f4528196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45277fe700 1 -- 172.21.15.111:0/2301896147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45281063f0 msgr2=0x7f452819add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45277fe700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45281063f0 0x7f452819add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45277fe700 1 -- 172.21.15.111:0/2301896147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4528101400 msgr2=0x7f4528196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45277fe700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4528101400 0x7f4528196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45277fe700 1 -- 172.21.15.111:0/2301896147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45180095f0 con 0x7f4528100a50 2023-10-11T16:34:01.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f451ffff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4528101400 0x7f4528196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:01.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45277fe700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4528100a50 0x7f4528196430 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4518005910 tx=0x7f4518004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:01.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45257fa700 1 -- 172.21.15.111:0/2301896147 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451801d070 con 0x7f4528100a50 2023-10-11T16:34:01.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f452ddff700 1 -- 172.21.15.111:0/2301896147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45281103d0 con 0x7f4528100a50 2023-10-11T16:34:01.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.509+0000 7f45257fa700 1 -- 172.21.15.111:0/2301896147 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4518022470 con 0x7f4528100a50 2023-10-11T16:34:01.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.510+0000 7f452ddff700 1 -- 172.21.15.111:0/2301896147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4528110240 con 0x7f4528100a50 2023-10-11T16:34:01.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.510+0000 7f45257fa700 1 -- 172.21.15.111:0/2301896147 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45180175f0 con 0x7f4528100a50 2023-10-11T16:34:01.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.513+0000 7f45257fa700 1 -- 172.21.15.111:0/2301896147 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f451800fad0 con 0x7f4528100a50 2023-10-11T16:34:01.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.514+0000 7f45257fa700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f45140a13e0 0x7f45140a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.514+0000 7f45257fa700 1 -- 172.21.15.111:0/2301896147 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f45180ce7e0 con 0x7f4528100a50 2023-10-11T16:34:01.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.514+0000 7f451ffff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f45140a13e0 0x7f45140a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.515+0000 7f451ffff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f45140a13e0 0x7f45140a3890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4528197a40 tx=0x7f451000b5f0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:01.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45416f0b80 1 Processor -- start 2023-10-11T16:34:01.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45416f0b80 1 -- start start 2023-10-11T16:34:01.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03603b60 0x55dc03603f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03604470 0x55dc03608370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc03608d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45416f0b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03509060 con 0x55dc03603b60 2023-10-11T16:34:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45416f0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0355b8a0 con 0x55dc03604470 2023-10-11T16:34:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45416f0b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03554d30 con 0x55dc036088b0 2023-10-11T16:34:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03603b60 0x55dc03603f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03603b60 0x55dc03603f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35090/0 (socket says 172.21.15.111:35090) 2023-10-11T16:34:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f45277fe700 1 -- 172.21.15.111:0/2178002342 learned_addr learned my addr 172.21.15.111:0/2178002342 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.578+0000 7f451ffff700 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03604470 0x55dc03608370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f4527fff700 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc03608d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f45277fe700 1 -- 172.21.15.111:0/2178002342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 msgr2=0x55dc03608d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:01.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f45277fe700 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc03608d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f45277fe700 1 -- 172.21.15.111:0/2178002342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03604470 msgr2=0x55dc03608370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:01.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f45277fe700 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03604470 0x55dc03608370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f45277fe700 1 -- 172.21.15.111:0/2178002342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc033d6630 con 0x55dc03603b60 2023-10-11T16:34:01.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f45277fe700 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03603b60 0x55dc03603f30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f451809b3f0 tx=0x7f4518009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e50ce4edceed30e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:01.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f451dffb700 1 -- 172.21.15.111:0/2178002342 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451809b460 con 0x55dc03603b60 2023-10-11T16:34:01.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f451dffb700 1 -- 172.21.15.111:0/2178002342 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f451809f490 con 0x55dc03603b60 2023-10-11T16:34:01.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.579+0000 7f451dffb700 1 -- 172.21.15.111:0/2178002342 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45180ce3f0 con 0x55dc03603b60 2023-10-11T16:34:01.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 -- 172.21.15.111:0/2178002342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03603b60 msgr2=0x55dc03603f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03603b60 0x55dc03603f30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f451809b3f0 tx=0x7f4518009310 comp rx=0 tx=0).stop 2023-10-11T16:34:01.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 -- 172.21.15.111:0/2178002342 shutdown_connections 2023-10-11T16:34:01.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc03608d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03603b60 0x55dc03603f30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 --2- 172.21.15.111:0/2178002342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03604470 0x55dc03608370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 -- 172.21.15.111:0/2178002342 >> 172.21.15.111:0/2178002342 conn(0x55dc035f9360 msgr2=0x55dc035feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:01.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 -- 172.21.15.111:0/2178002342 shutdown_connections 2023-10-11T16:34:01.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.580+0000 7f45416f0b80 1 -- 172.21.15.111:0/2178002342 wait complete. 2023-10-11T16:34:01.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:01 smithi130 ceph-mon[131859]: pgmap v44: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-10-11T16:34:01.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:01 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:34:01.591 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:01 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:34:01.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:01 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:34:01.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:01 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:34:01.592 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:01 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:34:01.593 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:01 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:34:01.593 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:01 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:34:01.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 1 -- start start 2023-10-11T16:34:01.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 1 Processor -- start 2023-10-11T16:34:01.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc03603b60 0x55dc03617450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03617990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc036088b0 0x55dc03611710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc034f68e0 con 0x55dc03604470 2023-10-11T16:34:01.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc034f6580 con 0x55dc036088b0 2023-10-11T16:34:01.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc033d8920 con 0x55dc03603b60 2023-10-11T16:34:01.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.581+0000 7f45416f0b80 10 client.0 _ll_register_callbacks cb 0x55dc03514780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:01.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03617990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03617990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35104/0 (socket says 172.21.15.111:35104) 2023-10-11T16:34:01.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 -- 172.21.15.111:0/485917289 learned_addr learned my addr 172.21.15.111:0/485917289 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:01.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f4527fff700 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc036088b0 0x55dc03611710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f45277fe700 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc03603b60 0x55dc03617450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 -- 172.21.15.111:0/485917289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc03603b60 msgr2=0x55dc03617450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc03603b60 0x55dc03617450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 -- 172.21.15.111:0/485917289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc036088b0 msgr2=0x55dc03611710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc036088b0 0x55dc03611710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 -- 172.21.15.111:0/485917289 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4528110240 con 0x55dc03604470 2023-10-11T16:34:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f451ffff700 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03617990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f451000ff90 tx=0x7f4510004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77fc8dc6ebcb53f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f4506ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.582+0000 7f4506ffd700 1 -- 172.21.15.111:0/485917289 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510028470 con 0x55dc03604470 2023-10-11T16:34:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f4506ffd700 1 -- 172.21.15.111:0/485917289 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f45100285d0 con 0x55dc03604470 2023-10-11T16:34:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 1 -- 172.21.15.111:0/485917289 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dc035f6b20 con 0x55dc03604470 2023-10-11T16:34:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f4506ffd700 1 -- 172.21.15.111:0/485917289 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4510028790 con 0x55dc03604470 2023-10-11T16:34:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 10 client.15867 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 1 -- 172.21.15.111:0/485917289 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dc0351b190 con 0x55dc03604470 2023-10-11T16:34:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f4506ffd700 1 -- 172.21.15.111:0/485917289 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dc035f6b20 con 0x55dc03604470 2023-10-11T16:34:01.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f4506ffd700 1 -- 172.21.15.111:0/485917289 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f451001e550 con 0x55dc03604470 2023-10-11T16:34:01.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 10 client.15867 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:01.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:01.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 20 client.15867 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:01.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55dc03621990 0x55dc03623da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 4 client.15867 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:01.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.583+0000 7f45416f0b80 1 -- 172.21.15.111:0/485917289 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dc035170a0 con 0x55dc03621990 2023-10-11T16:34:01.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.584+0000 7f45277fe700 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55dc03621990 0x55dc03623da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.585+0000 7f4506ffd700 10 client.15867 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:01.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.585+0000 7f45277fe700 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55dc03621990 0x55dc03623da0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=434b4f69fd697e44 server_cookie=16c092f01edba25d in_seq=0 out_seq=0 2023-10-11T16:34:01.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.585+0000 7f4506ffd700 1 -- 172.21.15.111:0/485917289 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55dc035170a0 con 0x55dc03621990 2023-10-11T16:34:01.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.585+0000 7f4506ffd700 10 client.15867 handle_command_reply: tid=0 2023-10-11T16:34:01.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.585+0000 7f45416f0b80 1 client.15867 shutdown 2023-10-11T16:34:01.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 -- 172.21.15.111:0/485917289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 msgr2=0x55dc03617990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03617990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f451000ff90 tx=0x7f4510004750 comp rx=0 tx=0).stop 2023-10-11T16:34:01.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 -- 172.21.15.111:0/485917289 shutdown_connections 2023-10-11T16:34:01.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55dc03621990 0x55dc03623da0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc036088b0 0x55dc03611710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc03603b60 0x55dc03617450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 --2- 172.21.15.111:0/485917289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03617990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 -- 172.21.15.111:0/485917289 >> 172.21.15.111:0/485917289 conn(0x55dc035f9360 msgr2=0x55dc035fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:01.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f4506ffd700 0 client.15867 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:01.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 -- 172.21.15.111:0/485917289 shutdown_connections 2023-10-11T16:34:01.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 1 -- 172.21.15.111:0/485917289 wait complete. 2023-10-11T16:34:01.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.586+0000 7f45416f0b80 20 client.15867 trim_cache size 0 max 16384 2023-10-11T16:34:01.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.603+0000 7f45416f0b80 1 Processor -- start 2023-10-11T16:34:01.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.603+0000 7f45416f0b80 1 -- start start 2023-10-11T16:34:01.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03612040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03610790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0360f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f45416f0b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc035f6710 con 0x55dc03604470 2023-10-11T16:34:01.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f45416f0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03600480 con 0x55dc03603b60 2023-10-11T16:34:01.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f45416f0b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc03604bc0 con 0x55dc036088b0 2023-10-11T16:34:01.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f451ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03610790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f4527fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0360f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f451ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03610790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35112/0 (socket says 172.21.15.111:35112) 2023-10-11T16:34:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f451ffff700 1 -- 172.21.15.111:0/2585079464 learned_addr learned my addr 172.21.15.111:0/2585079464 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.604+0000 7f45277fe700 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03612040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f4527fff700 1 -- 172.21.15.111:0/2585079464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 msgr2=0x55dc03612040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f4527fff700 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03612040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f4527fff700 1 -- 172.21.15.111:0/2585079464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 msgr2=0x55dc03610790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:01.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f4527fff700 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03610790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f4527fff700 1 -- 172.21.15.111:0/2585079464 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dc0351aee0 con 0x55dc036088b0 2023-10-11T16:34:01.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f451ffff700 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03610790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:01.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f45277fe700 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03612040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:01.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f4527fff700 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0360f8f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4520011900 tx=0x7f4520009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a36336657f865ea0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:01.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f451e7fc700 1 -- 172.21.15.111:0/2585079464 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452000e040 con 0x55dc036088b0 2023-10-11T16:34:01.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f451e7fc700 1 -- 172.21.15.111:0/2585079464 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f452000f040 con 0x55dc036088b0 2023-10-11T16:34:01.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.605+0000 7f451e7fc700 1 -- 172.21.15.111:0/2585079464 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45200046b0 con 0x55dc036088b0 2023-10-11T16:34:01.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 -- 172.21.15.111:0/2585079464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 msgr2=0x55dc0360f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0360f8f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4520011900 tx=0x7f4520009920 comp rx=0 tx=0).stop 2023-10-11T16:34:01.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 -- 172.21.15.111:0/2585079464 shutdown_connections 2023-10-11T16:34:01.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0360f8f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03612040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 --2- 172.21.15.111:0/2585079464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03610790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 -- 172.21.15.111:0/2585079464 >> 172.21.15.111:0/2585079464 conn(0x55dc035f9360 msgr2=0x55dc036031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:01.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 -- 172.21.15.111:0/2585079464 shutdown_connections 2023-10-11T16:34:01.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 -- 172.21.15.111:0/2585079464 wait complete. 2023-10-11T16:34:01.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.606+0000 7f45416f0b80 1 -- start start 2023-10-11T16:34:01.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45416f0b80 1 Processor -- start 2023-10-11T16:34:01.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03669990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03669ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45416f0b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0366e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45416f0b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0366eda0 con 0x55dc03604470 2023-10-11T16:34:01.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45416f0b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0366ef10 con 0x55dc03603b60 2023-10-11T16:34:01.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45416f0b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dc0366f0d0 con 0x55dc036088b0 2023-10-11T16:34:01.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45416f0b80 10 client.0 _ll_register_callbacks cb 0x7f451000fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:01.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f451ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03669ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f45277fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03669990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f451ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03669ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35120/0 (socket says 172.21.15.111:35120) 2023-10-11T16:34:01.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.607+0000 7f451ffff700 1 -- 172.21.15.111:0/3831697424 learned_addr learned my addr 172.21.15.111:0/3831697424 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:01.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f4527fff700 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0366e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f45277fe700 1 -- 172.21.15.111:0/3831697424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 msgr2=0x55dc0366e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:01.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f45277fe700 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0366e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f45277fe700 1 -- 172.21.15.111:0/3831697424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 msgr2=0x55dc03669ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:01.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f45277fe700 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03669ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f45277fe700 1 -- 172.21.15.111:0/3831697424 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dc033d6630 con 0x55dc03603b60 2023-10-11T16:34:01.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f451ffff700 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03669ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:01.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f45277fe700 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03669990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f451800ffd0 tx=0x7f4518022c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c985106c57e1626a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:01.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f4506ffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:34:01.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f4506ffd700 1 -- 172.21.15.111:0/3831697424 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45180b97f0 con 0x55dc03603b60 2023-10-11T16:34:01.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f45416f0b80 1 -- 172.21.15.111:0/3831697424 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f451809f490 con 0x55dc03603b60 2023-10-11T16:34:01.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.608+0000 7f4506ffd700 1 -- 172.21.15.111:0/3831697424 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f451809b460 con 0x55dc03603b60 2023-10-11T16:34:01.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f4506ffd700 1 -- 172.21.15.111:0/3831697424 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45180a3820 con 0x55dc03603b60 2023-10-11T16:34:01.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f45416f0b80 10 client.25916 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:01.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f45416f0b80 1 -- 172.21.15.111:0/3831697424 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dc0351b190 con 0x55dc03603b60 2023-10-11T16:34:01.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f4506ffd700 1 -- 172.21.15.111:0/3831697424 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f451809f490 con 0x55dc03603b60 2023-10-11T16:34:01.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f4506ffd700 1 -- 172.21.15.111:0/3831697424 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f45180ce3f0 con 0x55dc03603b60 2023-10-11T16:34:01.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f45416f0b80 10 client.25916 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:01.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f45416f0b80 10 client.25916 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:01.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f45416f0b80 20 client.25916 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:01.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f45416f0b80 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55dc036763e0 0x55dc036787f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:01.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f45416f0b80 4 client.25916 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:34:01.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f45416f0b80 1 -- 172.21.15.111:0/3831697424 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dc03523660 con 0x55dc036763e0 2023-10-11T16:34:01.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.609+0000 7f451ffff700 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55dc036763e0 0x55dc036787f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:01.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f451ffff700 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55dc036763e0 0x55dc036787f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ef92415317d3934 server_cookie=f712b814ff9fc4a4 in_seq=0 out_seq=0 2023-10-11T16:34:01.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f4506ffd700 10 client.25916 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:01.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f4506ffd700 1 -- 172.21.15.111:0/3831697424 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55dc03523660 con 0x55dc036763e0 2023-10-11T16:34:01.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f4506ffd700 10 client.25916 handle_command_reply: tid=0 2023-10-11T16:34:01.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 client.25916 shutdown 2023-10-11T16:34:01.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 -- 172.21.15.111:0/3831697424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 msgr2=0x55dc03669990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03669990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f451800ffd0 tx=0x7f4518022c60 comp rx=0 tx=0).stop 2023-10-11T16:34:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 -- 172.21.15.111:0/3831697424 shutdown_connections 2023-10-11T16:34:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55dc036763e0 0x55dc036787f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55dc036088b0 0x55dc0366e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc03603b60 0x55dc03669990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f4506ffd700 0 client.25916 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:01.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 --2- 172.21.15.111:0/3831697424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55dc03604470 0x55dc03669ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.610+0000 7f45416f0b80 1 -- 172.21.15.111:0/3831697424 >> 172.21.15.111:0/3831697424 conn(0x55dc035f9360 msgr2=0x55dc03610790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:01.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f45416f0b80 1 -- 172.21.15.111:0/3831697424 shutdown_connections 2023-10-11T16:34:01.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f45416f0b80 1 -- 172.21.15.111:0/3831697424 wait complete. 2023-10-11T16:34:01.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f45416f0b80 20 client.25916 trim_cache size 0 max 16384 2023-10-11T16:34:01.639 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:34:01.639 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:34:01.639 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:34:01.640 INFO:teuthology.orchestra.run.smithi111.stdout: "db36a076-41b2-441c-8798-7113b912baa8": { 2023-10-11T16:34:01.640 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:34:01.640 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "db36a076-41b2-441c-8798-7113b912baa8", 2023-10-11T16:34:01.640 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:34:01.641 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:34:01.641 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:34:01.641 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:34:01.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 -- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f45140a13e0 msgr2=0x7f45140a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f45140a13e0 0x7f45140a3890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4528197a40 tx=0x7f451000b5f0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 -- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4528100a50 msgr2=0x7f4528196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:01.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4528100a50 0x7f4528196430 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4518005910 tx=0x7f4518004730 comp rx=0 tx=0).stop 2023-10-11T16:34:01.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 -- 172.21.15.111:0/2301896147 shutdown_connections 2023-10-11T16:34:01.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f45140a13e0 0x7f45140a3890 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45281063f0 0x7f452819add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4528100a50 0x7f4528196430 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 --2- 172.21.15.111:0/2301896147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4528101400 0x7f4528196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:01.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.611+0000 7f452ddff700 1 -- 172.21.15.111:0/2301896147 >> 172.21.15.111:0/2301896147 conn(0x7f4528077590 msgr2=0x7f45280ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:01.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.612+0000 7f452ddff700 1 -- 172.21.15.111:0/2301896147 shutdown_connections 2023-10-11T16:34:01.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:01.612+0000 7f452ddff700 1 -- 172.21.15.111:0/2301896147 wait complete. 2023-10-11T16:34:01.791 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:01.791 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:01.805 INFO:tasks.cephfs.filesystem:scrub status for tag:db36a076-41b2-441c-8798-7113b912baa8 - {'path': '/', 'tag': 'db36a076-41b2-441c-8798-7113b912baa8', 'options': 'recursive,force'} 2023-10-11T16:34:01.990 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:01.991 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:02.002 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:02.090 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:02.090 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 37 -lt 100 ']' 2023-10-11T16:34:02.092 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 37 + 1 2023-10-11T16:34:02.092 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=38 2023-10-11T16:34:02.092 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:02.092 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:02.093 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:02.093 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:02.093 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:02.093 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:02.574 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:02.575 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:02.715 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:02.716 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:02.725 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:02.725 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:02.725 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 38 -lt 100 ']' 2023-10-11T16:34:02.726 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 38 + 1 2023-10-11T16:34:02.759 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=39 2023-10-11T16:34:02.759 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:02.796 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:02.797 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:02.799 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:02.801 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:02.801 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:03.182 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:03.183 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:03.332 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:03.332 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:03.342 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:03.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:03.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 39 -lt 100 ']' 2023-10-11T16:34:03.343 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 39 + 1 2023-10-11T16:34:03.370 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=40 2023-10-11T16:34:03.370 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:03.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:03 smithi111 ceph-mon[98527]: pgmap v45: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:34:03.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:03.398 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:03.400 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:03.401 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:03.401 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:03.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:03 smithi150 ceph-mon[132217]: pgmap v45: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:34:03.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:03 smithi130 ceph-mon[131859]: pgmap v45: 129 pgs: 129 active+clean; 1.7 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-10-11T16:34:03.766 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:03.766 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:03.873 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:03.873 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:03.897 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:03.897 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:03.897 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 40 -lt 100 ']' 2023-10-11T16:34:03.898 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 40 + 1 2023-10-11T16:34:03.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=41 2023-10-11T16:34:03.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:03.968 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:03.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:03.971 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:03.971 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:03.972 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:04.324 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:04.324 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:04.490 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:04.490 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:04.509 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:04.510 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:04.510 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 41 -lt 100 ']' 2023-10-11T16:34:04.510 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 41 + 1 2023-10-11T16:34:04.544 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=42 2023-10-11T16:34:04.545 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:04.570 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:04.571 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:04.572 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:04.573 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:04.573 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:04.940 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:04.941 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:05.082 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:05.082 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:05.092 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:05.093 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:05.093 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 42 -lt 100 ']' 2023-10-11T16:34:05.093 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 42 + 1 2023-10-11T16:34:05.119 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=43 2023-10-11T16:34:05.120 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:05.162 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:05.163 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:05.165 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:05.165 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:05.166 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:05.410 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:05 smithi111 ceph-mon[98527]: pgmap v46: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 KiB/s wr, 14 op/s 2023-10-11T16:34:05.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:05 smithi150 ceph-mon[132217]: pgmap v46: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 KiB/s wr, 14 op/s 2023-10-11T16:34:05.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:05 smithi130 ceph-mon[131859]: pgmap v46: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 28 KiB/s wr, 14 op/s 2023-10-11T16:34:05.623 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:05.624 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:05.765 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:05.765 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:05.775 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:05.775 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:05.776 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 43 -lt 100 ']' 2023-10-11T16:34:05.776 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 43 + 1 2023-10-11T16:34:05.808 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=44 2023-10-11T16:34:05.809 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:05.846 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:05.847 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:05.849 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:05.850 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:05.850 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:06.232 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:06.233 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:06.373 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:06.374 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:06.383 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:06.384 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:06.384 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 44 -lt 100 ']' 2023-10-11T16:34:06.384 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 44 + 1 2023-10-11T16:34:06.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:06 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:34:06.411 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=45 2023-10-11T16:34:06.411 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:06.446 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:06.447 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:06.448 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:06.449 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:06.449 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:06.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:06 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:34:06.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:06 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:34:06.965 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:06.966 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:07.131 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:07.132 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:07.142 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:07.142 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:07.142 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 45 -lt 100 ']' 2023-10-11T16:34:07.143 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 45 + 1 2023-10-11T16:34:07.175 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=46 2023-10-11T16:34:07.175 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:07.213 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:07.213 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:07.215 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:07.216 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:07.216 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:07 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:34:07.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:07 smithi150 ceph-mon[132217]: pgmap v47: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 20 op/s 2023-10-11T16:34:07.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:07 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:34:07.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:07 smithi130 ceph-mon[131859]: pgmap v47: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 20 op/s 2023-10-11T16:34:07.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:07 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:34:07.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:07 smithi111 ceph-mon[98527]: pgmap v47: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 20 op/s 2023-10-11T16:34:07.957 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:07.957 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:08.181 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:08.182 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:08.193 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:08.193 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:08.193 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 46 -lt 100 ']' 2023-10-11T16:34:08.194 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 46 + 1 2023-10-11T16:34:08.250 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=47 2023-10-11T16:34:08.250 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:08.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:08.307 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:08.309 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:08.310 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:08.310 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.629+0000 7f8a58dff700 1 -- 172.21.15.111:0/3965533825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a54100540 msgr2=0x7f8a54100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.629+0000 7f8a58dff700 1 --2- 172.21.15.111:0/3965533825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a54100540 0x7f8a54100910 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8a440099c0 tx=0x7f8a44009cd0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.630+0000 7f8a58dff700 1 -- 172.21.15.111:0/3965533825 shutdown_connections 2023-10-11T16:34:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.630+0000 7f8a58dff700 1 --2- 172.21.15.111:0/3965533825 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a54105530 0x7f8a5410d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.630+0000 7f8a58dff700 1 --2- 172.21.15.111:0/3965533825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a54100540 0x7f8a54100910 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.630+0000 7f8a58dff700 1 --2- 172.21.15.111:0/3965533825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a54100ee0 0x7f8a54104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.630+0000 7f8a58dff700 1 -- 172.21.15.111:0/3965533825 >> 172.21.15.111:0/3965533825 conn(0x7f8a54078210 msgr2=0x7f8a54078610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.630+0000 7f8a58dff700 1 -- 172.21.15.111:0/3965533825 shutdown_connections 2023-10-11T16:34:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.630+0000 7f8a58dff700 1 -- 172.21.15.111:0/3965533825 wait complete. 2023-10-11T16:34:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.631+0000 7f8a58dff700 1 Processor -- start 2023-10-11T16:34:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.631+0000 7f8a58dff700 1 -- start start 2023-10-11T16:34:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.631+0000 7f8a58dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a54100540 0x7f8a54198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.631+0000 7f8a58dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a54100ee0 0x7f8a54198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.631+0000 7f8a58dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a54105530 0x7f8a5419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.631+0000 7f8a58dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5419d640 con 0x7f8a54105530 2023-10-11T16:34:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.631+0000 7f8a58dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5419d7b0 con 0x7f8a54100540 2023-10-11T16:34:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.631+0000 7f8a58dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5419d970 con 0x7f8a54100ee0 2023-10-11T16:34:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a54100540 0x7f8a54198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.636 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a51d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a54100ee0 0x7f8a54198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a54100540 0x7f8a54198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:39696/0 (socket says 172.21.15.111:39696) 2023-10-11T16:34:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 -- 172.21.15.111:0/2009943769 learned_addr learned my addr 172.21.15.111:0/2009943769 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a54100ee0 0x7f8a54198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41844/0 (socket says 172.21.15.111:41844) 2023-10-11T16:34:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a54105530 0x7f8a5419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 -- 172.21.15.111:0/2009943769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a54100ee0 msgr2=0x7f8a54198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.638 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a54100ee0 0x7f8a54198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.638 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 -- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a54105530 msgr2=0x7f8a5419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.638 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a54105530 0x7f8a5419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 -- 172.21.15.111:0/2009943769 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a440095f0 con 0x7f8a54100540 2023-10-11T16:34:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a54100ee0 0x7f8a54198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.632+0000 7f8a5259c700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a54100540 0x7f8a54198660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400b580 tx=0x7f8a4400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:08.639 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.633+0000 7f8a437fe700 1 -- 172.21.15.111:0/2009943769 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4401d070 con 0x7f8a54100540 2023-10-11T16:34:08.640 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.633+0000 7f8a58dff700 1 -- 172.21.15.111:0/2009943769 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a54110340 con 0x7f8a54100540 2023-10-11T16:34:08.640 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.633+0000 7f8a58dff700 1 -- 172.21.15.111:0/2009943769 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a541101b0 con 0x7f8a54100540 2023-10-11T16:34:08.640 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.633+0000 7f8a437fe700 1 -- 172.21.15.111:0/2009943769 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a44005050 con 0x7f8a54100540 2023-10-11T16:34:08.640 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.633+0000 7f8a437fe700 1 -- 172.21.15.111:0/2009943769 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440049e0 con 0x7f8a54100540 2023-10-11T16:34:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.636+0000 7f8a437fe700 1 -- 172.21.15.111:0/2009943769 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 24) v1 ==== 135949+0+0 (secure 0 0 0) 0x7f8a440051c0 con 0x7f8a54100540 2023-10-11T16:34:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.637+0000 7f8a437fe700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8a3c0a1460 0x7f8a3c0a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.637+0000 7f8a437fe700 1 -- 172.21.15.111:0/2009943769 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f8a440cdf90 con 0x7f8a54100540 2023-10-11T16:34:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.637+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8a3c0a1460 0x7f8a3c0a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.642 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.638+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8a3c0a1460 0x7f8a3c0a3910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8a54199c70 tx=0x7f8a48009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.743+0000 7f8a6c673b80 1 Processor -- start 2023-10-11T16:34:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a6c673b80 1 -- start start 2023-10-11T16:34:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14b206b0 0x55ab14cca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14cca720 0x55ab14cc4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14bca000 con 0x55ab14ccab90 2023-10-11T16:34:08.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14c1c840 con 0x55ab14cca720 2023-10-11T16:34:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14c15cd0 con 0x55ab14b206b0 2023-10-11T16:34:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a52d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a52d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35160/0 (socket says 172.21.15.111:35160) 2023-10-11T16:34:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a52d9d700 1 -- 172.21.15.111:0/3662886790 learned_addr learned my addr 172.21.15.111:0/3662886790 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14cca720 0x55ab14cc4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.744+0000 7f8a5259c700 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14b206b0 0x55ab14cca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a52d9d700 1 -- 172.21.15.111:0/3662886790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14b206b0 msgr2=0x55ab14cca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14b206b0 0x55ab14cca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a52d9d700 1 -- 172.21.15.111:0/3662886790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14cca720 msgr2=0x55ab14cc4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14cca720 0x55ab14cc4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a52d9d700 1 -- 172.21.15.111:0/3662886790 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab14a975d0 con 0x55ab14ccab90 2023-10-11T16:34:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc4b00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c011680 tx=0x7f8a4c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea1bdbc7ed9a53e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a40ff9700 1 -- 172.21.15.111:0/3662886790 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00e040 con 0x55ab14ccab90 2023-10-11T16:34:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a40ff9700 1 -- 172.21.15.111:0/3662886790 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a4c00f040 con 0x55ab14ccab90 2023-10-11T16:34:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.745+0000 7f8a40ff9700 1 -- 172.21.15.111:0/3662886790 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0048c0 con 0x55ab14ccab90 2023-10-11T16:34:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3662886790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 msgr2=0x55ab14cc4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc4b00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c011680 tx=0x7f8a4c009920 comp rx=0 tx=0).stop 2023-10-11T16:34:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3662886790 shutdown_connections 2023-10-11T16:34:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc4b00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14b206b0 0x55ab14cca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3662886790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14cca720 0x55ab14cc4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3662886790 >> 172.21.15.111:0/3662886790 conn(0x55ab14cb9f50 msgr2=0x55ab14cbf270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3662886790 shutdown_connections 2023-10-11T16:34:08.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.746+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3662886790 wait complete. 2023-10-11T16:34:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.747+0000 7f8a6c673b80 1 -- start start 2023-10-11T16:34:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.747+0000 7f8a6c673b80 1 Processor -- start 2023-10-11T16:34:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.747+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14b206b0 0x55ab14cd0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.747+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 0x55ab14ccdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14ccab90 0x55ab14cce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14bb7880 con 0x55ab14b206b0 2023-10-11T16:34:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14bb7520 con 0x55ab14ccab90 2023-10-11T16:34:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14a998c0 con 0x55ab14cca720 2023-10-11T16:34:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a6c673b80 10 client.0 _ll_register_callbacks cb 0x55ab14bd5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14b206b0 0x55ab14cd0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a51d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 0x55ab14ccdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14b206b0 0x55ab14cd0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35166/0 (socket says 172.21.15.111:35166) 2023-10-11T16:34:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 -- 172.21.15.111:0/3983721858 learned_addr learned my addr 172.21.15.111:0/3983721858 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14ccab90 0x55ab14cce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 -- 172.21.15.111:0/3983721858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 msgr2=0x55ab14ccdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 0x55ab14ccdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 -- 172.21.15.111:0/3983721858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14ccab90 msgr2=0x55ab14cce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14ccab90 0x55ab14cce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 -- 172.21.15.111:0/3983721858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c011330 con 0x55ab14b206b0 2023-10-11T16:34:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a5259c700 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14b206b0 0x55ab14cd0a90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8a44096f90 tx=0x7f8a4409ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc5d270a1e1c787 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.748+0000 7f8a39ffb700 1 -- 172.21.15.111:0/3983721858 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4409af00 con 0x55ab14b206b0 2023-10-11T16:34:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a39ffb700 1 -- 172.21.15.111:0/3983721858 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f8a44099f80 con 0x55ab14b206b0 2023-10-11T16:34:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3983721858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ab14cb7710 con 0x55ab14b206b0 2023-10-11T16:34:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a39ffb700 1 -- 172.21.15.111:0/3983721858 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a440b3150 con 0x55ab14b206b0 2023-10-11T16:34:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 10 client.15885 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3983721858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ab14bdc130 con 0x55ab14b206b0 2023-10-11T16:34:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a39ffb700 1 -- 172.21.15.111:0/3983721858 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ab14cb7710 con 0x55ab14b206b0 2023-10-11T16:34:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a39ffb700 1 -- 172.21.15.111:0/3983721858 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8a440cb4a0 con 0x55ab14b206b0 2023-10-11T16:34:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 10 client.15885 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 10 client.15885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 20 client.15885 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ab14cddbb0 0x55ab14cdffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 4 client.15885 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.749+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3983721858 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab14bd8040 con 0x55ab14cddbb0 2023-10-11T16:34:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.750+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ab14cddbb0 0x55ab14cdffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.750+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ab14cddbb0 0x55ab14cdffc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c02f9355e64316a server_cookie=1ecd6606829532a0 in_seq=0 out_seq=0 2023-10-11T16:34:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.750+0000 7f8a39ffb700 10 client.15885 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a39ffb700 1 -- 172.21.15.111:0/3983721858 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ab14bd8040 con 0x55ab14cddbb0 2023-10-11T16:34:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a39ffb700 10 client.15885 handle_command_reply: tid=0 2023-10-11T16:34:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 client.15885 shutdown 2023-10-11T16:34:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3983721858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14b206b0 msgr2=0x55ab14cd0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14b206b0 0x55ab14cd0a90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8a44096f90 tx=0x7f8a4409ec00 comp rx=0 tx=0).stop 2023-10-11T16:34:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3983721858 shutdown_connections 2023-10-11T16:34:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ab14cddbb0 0x55ab14cdffc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14ccab90 0x55ab14cce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14b206b0 0x55ab14cd0a90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a39ffb700 0 client.15885 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3983721858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 0x55ab14ccdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3983721858 >> 172.21.15.111:0/3983721858 conn(0x55ab14cb9f50 msgr2=0x55ab14cbe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3983721858 shutdown_connections 2023-10-11T16:34:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.751+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3983721858 wait complete. 2023-10-11T16:34:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.752+0000 7f8a6c673b80 20 client.15885 trim_cache size 0 max 16384 2023-10-11T16:34:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.767+0000 7f8a6c673b80 1 Processor -- start 2023-10-11T16:34:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.767+0000 7f8a6c673b80 1 -- start start 2023-10-11T16:34:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.767+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cd5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.779 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.767+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14cca720 0x55ab14cc9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.779 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.767+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14ccab90 0x55ab14cbef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.779 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.767+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14ccc2c0 con 0x55ab14cca720 2023-10-11T16:34:08.779 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.767+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14cd9f50 con 0x55ab14b206b0 2023-10-11T16:34:08.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.767+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14cc89e0 con 0x55ab14ccab90 2023-10-11T16:34:08.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14cca720 0x55ab14cc9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14cca720 0x55ab14cc9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35176/0 (socket says 172.21.15.111:35176) 2023-10-11T16:34:08.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 -- 172.21.15.111:0/3207123355 learned_addr learned my addr 172.21.15.111:0/3207123355 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:08.780 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14ccab90 0x55ab14cbef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a5259c700 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cd5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 -- 172.21.15.111:0/3207123355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14ccab90 msgr2=0x55ab14cbef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14ccab90 0x55ab14cbef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 -- 172.21.15.111:0/3207123355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 msgr2=0x55ab14cd5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cd5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 -- 172.21.15.111:0/3207123355 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab14cbb6f0 con 0x55ab14cca720 2023-10-11T16:34:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a5259c700 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cd5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.768+0000 7f8a51d9b700 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14cca720 0x55ab14cc9500 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x55ab14ccf350 tx=0x7f8a48014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd03ae57854b0175 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a417fa700 1 -- 172.21.15.111:0/3207123355 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48024040 con 0x55ab14cca720 2023-10-11T16:34:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a417fa700 1 -- 172.21.15.111:0/3207123355 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a48028da0 con 0x55ab14cca720 2023-10-11T16:34:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a417fa700 1 -- 172.21.15.111:0/3207123355 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48028780 con 0x55ab14cca720 2023-10-11T16:34:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3207123355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14cca720 msgr2=0x55ab14cc9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14cca720 0x55ab14cc9500 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x55ab14ccf350 tx=0x7f8a48014000 comp rx=0 tx=0).stop 2023-10-11T16:34:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3207123355 shutdown_connections 2023-10-11T16:34:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14ccab90 0x55ab14cbef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cd5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/3207123355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14cca720 0x55ab14cc9500 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3207123355 >> 172.21.15.111:0/3207123355 conn(0x55ab14cb9f50 msgr2=0x55ab14cc8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.769+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3207123355 shutdown_connections 2023-10-11T16:34:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.770+0000 7f8a6c673b80 1 -- 172.21.15.111:0/3207123355 wait complete. 2023-10-11T16:34:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.770+0000 7f8a6c673b80 1 -- start start 2023-10-11T16:34:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.770+0000 7f8a6c673b80 1 Processor -- start 2023-10-11T16:34:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cc5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 0x55ab14cc61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a6c673b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14cc6e30 con 0x55ab14ccab90 2023-10-11T16:34:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14cc6fa0 con 0x55ab14b206b0 2023-10-11T16:34:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a6c673b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab14cc7160 con 0x55ab14cca720 2023-10-11T16:34:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a6c673b80 10 client.0 _ll_register_callbacks cb 0x7f8a4409aa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a52d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a5259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cc5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a51d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 0x55ab14cc61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a52d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35190/0 (socket says 172.21.15.111:35190) 2023-10-11T16:34:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a52d9d700 1 -- 172.21.15.111:0/2996848691 learned_addr learned my addr 172.21.15.111:0/2996848691 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a52d9d700 1 -- 172.21.15.111:0/2996848691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 msgr2=0x55ab14cc61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 0x55ab14cc61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a52d9d700 1 -- 172.21.15.111:0/2996848691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 msgr2=0x55ab14cc5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cc5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a52d9d700 1 -- 172.21.15.111:0/2996848691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a48017070 con 0x55ab14ccab90 2023-10-11T16:34:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.771+0000 7f8a5259c700 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cc5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a52d9d700 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc6720 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009ec0 tx=0x7f8a4c004110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f056c7316107b179 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a39ffb700 1 -- 172.21.15.111:0/2996848691 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00e070 con 0x55ab14ccab90 2023-10-11T16:34:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a39ffb700 1 -- 172.21.15.111:0/2996848691 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f8a4c005220 con 0x55ab14ccab90 2023-10-11T16:34:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a6c673b80 1 -- 172.21.15.111:0/2996848691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a4c00f040 con 0x55ab14ccab90 2023-10-11T16:34:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a39ffb700 1 -- 172.21.15.111:0/2996848691 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a4c0093e0 con 0x55ab14ccab90 2023-10-11T16:34:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a6c673b80 10 client.15891 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a6c673b80 1 -- 172.21.15.111:0/2996848691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ab14a43d80 con 0x55ab14ccab90 2023-10-11T16:34:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a39ffb700 1 -- 172.21.15.111:0/2996848691 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a4c00f040 con 0x55ab14ccab90 2023-10-11T16:34:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.772+0000 7f8a39ffb700 1 -- 172.21.15.111:0/2996848691 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8a4c007b00 con 0x55ab14ccab90 2023-10-11T16:34:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a6c673b80 10 client.15891 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a6c673b80 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a6c673b80 20 client.15891 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ab14d32750 0x55ab14d34b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a6c673b80 4 client.15891 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:34:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a6c673b80 1 -- 172.21.15.111:0/2996848691 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ab14be4600 con 0x55ab14d32750 2023-10-11T16:34:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a5259c700 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ab14d32750 0x55ab14d34b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a39ffb700 10 client.15891 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.773+0000 7f8a5259c700 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ab14d32750 0x55ab14d34b60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d23aa642c8214b0 server_cookie=2ef6fb45f5faab1 in_seq=0 out_seq=0 2023-10-11T16:34:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.782+0000 7f8a39ffb700 1 -- 172.21.15.111:0/2996848691 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ab14be4600 con 0x55ab14d32750 2023-10-11T16:34:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.782+0000 7f8a39ffb700 10 client.15891 handle_command_reply: tid=0 2023-10-11T16:34:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.782+0000 7f8a6c673b80 1 client.15891 shutdown 2023-10-11T16:34:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.782+0000 7f8a6c673b80 1 -- 172.21.15.111:0/2996848691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 msgr2=0x55ab14cc6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.782+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc6720 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009ec0 tx=0x7f8a4c004110 comp rx=0 tx=0).stop 2023-10-11T16:34:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 1 -- 172.21.15.111:0/2996848691 shutdown_connections 2023-10-11T16:34:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ab14d32750 0x55ab14d34b60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ab14ccab90 0x55ab14cc6720 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ab14b206b0 0x55ab14cc5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 1 --2- 172.21.15.111:0/2996848691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ab14cca720 0x55ab14cc61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a39ffb700 0 client.15891 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 1 -- 172.21.15.111:0/2996848691 >> 172.21.15.111:0/2996848691 conn(0x55ab14cb9f50 msgr2=0x55ab14cc9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 1 -- 172.21.15.111:0/2996848691 shutdown_connections 2023-10-11T16:34:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 1 -- 172.21.15.111:0/2996848691 wait complete. 2023-10-11T16:34:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.783+0000 7f8a6c673b80 20 client.15891 trim_cache size 0 max 16384 2023-10-11T16:34:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:34:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:34:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:34:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:34:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 -- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8a3c0a1460 msgr2=0x7f8a3c0a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8a3c0a1460 0x7f8a3c0a3910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8a54199c70 tx=0x7f8a48009260 comp rx=0 tx=0).stop 2023-10-11T16:34:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 -- 172.21.15.111:0/2009943769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a54100540 msgr2=0x7f8a54198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a54100540 0x7f8a54198660 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400b580 tx=0x7f8a4400bfd0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 -- 172.21.15.111:0/2009943769 shutdown_connections 2023-10-11T16:34:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8a3c0a1460 0x7f8a3c0a3910 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a54105530 0x7f8a5419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a54100540 0x7f8a54198660 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 --2- 172.21.15.111:0/2009943769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a54100ee0 0x7f8a54198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 -- 172.21.15.111:0/2009943769 >> 172.21.15.111:0/2009943769 conn(0x7f8a54078210 msgr2=0x7f8a540fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 -- 172.21.15.111:0/2009943769 shutdown_connections 2023-10-11T16:34:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:08.784+0000 7f8a58dff700 1 -- 172.21.15.111:0/2009943769 wait complete. 2023-10-11T16:34:09.098 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:09.099 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:09.248 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:09.248 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:09.258 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:09.259 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:09.259 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 47 -lt 100 ']' 2023-10-11T16:34:09.259 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 47 + 1 2023-10-11T16:34:09.294 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=48 2023-10-11T16:34:09.294 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:09.329 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:09.329 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:09.331 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:09.332 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:09.332 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:09.453 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:09 smithi150 ceph-mon[132217]: pgmap v48: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 19 op/s 2023-10-11T16:34:09.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:09 smithi130 ceph-mon[131859]: pgmap v48: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 19 op/s 2023-10-11T16:34:09.609 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:09 smithi111 ceph-mon[98527]: pgmap v48: 129 pgs: 129 active+clean; 1.8 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 33 KiB/s wr, 19 op/s 2023-10-11T16:34:09.731 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:09.732 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:09.880 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:09.880 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:09.890 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:09.891 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:09.891 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 48 -lt 100 ']' 2023-10-11T16:34:09.891 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 48 + 1 2023-10-11T16:34:09.919 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=49 2023-10-11T16:34:09.919 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:09.962 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:09.963 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:09.965 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:09.966 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:09.966 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:10.314 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:10.315 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:10.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:10 smithi150 ceph-mon[132217]: mgrmap e25: x(active, since 91s), standbys: y, z 2023-10-11T16:34:10.556 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:10.556 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:10.566 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:10.567 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:10.567 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 49 -lt 100 ']' 2023-10-11T16:34:10.568 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 49 + 1 2023-10-11T16:34:10.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:10 smithi130 ceph-mon[131859]: mgrmap e25: x(active, since 91s), standbys: y, z 2023-10-11T16:34:10.602 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=50 2023-10-11T16:34:10.602 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:10.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:10 smithi111 ceph-mon[98527]: mgrmap e25: x(active, since 91s), standbys: y, z 2023-10-11T16:34:10.636 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:10.637 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:10.639 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:10.640 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:10.641 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:11.114 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:11.115 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:11.230 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:11.230 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:11.240 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:11.241 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:11.241 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 50 -lt 100 ']' 2023-10-11T16:34:11.242 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 50 + 1 2023-10-11T16:34:11.269 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=51 2023-10-11T16:34:11.269 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:11.312 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:11.313 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:11.315 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:11.315 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:11.316 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:11.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:11 smithi111 ceph-mon[98527]: pgmap v49: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 24 op/s 2023-10-11T16:34:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:11 smithi150 ceph-mon[132217]: pgmap v49: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 24 op/s 2023-10-11T16:34:11.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:11 smithi130 ceph-mon[131859]: pgmap v49: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 24 op/s 2023-10-11T16:34:11.764 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:11.765 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:11.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:11.923 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:11.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:11.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:11.933 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 51 -lt 100 ']' 2023-10-11T16:34:11.934 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 51 + 1 2023-10-11T16:34:11.960 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=52 2023-10-11T16:34:11.960 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:12.012 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:12.013 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:12.015 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:12.015 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:12.016 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:12.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:12.381 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:12.505 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:12.506 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:12.515 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:12.515 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:12.516 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 52 -lt 100 ']' 2023-10-11T16:34:12.516 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 52 + 1 2023-10-11T16:34:12.541 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=53 2023-10-11T16:34:12.541 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:12.578 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:12.579 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:12.581 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:12.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:12.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:13.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:13.023 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:13.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:13.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:13.182 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:13.183 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:13.183 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 53 -lt 100 ']' 2023-10-11T16:34:13.183 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 53 + 1 2023-10-11T16:34:13.210 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=54 2023-10-11T16:34:13.210 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:13.254 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:13.255 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:13.257 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:13.257 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:13.258 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:13.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:13 smithi150 ceph-mon[132217]: pgmap v50: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 20 op/s 2023-10-11T16:34:13.484 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:13 smithi111 ceph-mon[98527]: pgmap v50: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 20 op/s 2023-10-11T16:34:13.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:13 smithi130 ceph-mon[131859]: pgmap v50: 129 pgs: 129 active+clean; 1.9 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 20 op/s 2023-10-11T16:34:13.622 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:13.623 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:13.780 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:13.781 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:13.790 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:13.791 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:13.791 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 54 -lt 100 ']' 2023-10-11T16:34:13.791 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 54 + 1 2023-10-11T16:34:13.818 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=55 2023-10-11T16:34:13.819 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:13.854 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:13.854 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:13.856 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:13.857 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:13.857 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:14.339 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:14.339 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:14.472 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:14.472 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:14.482 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:14.483 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:14.483 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 55 -lt 100 ']' 2023-10-11T16:34:14.483 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 55 + 1 2023-10-11T16:34:14.509 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=56 2023-10-11T16:34:14.510 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:14.553 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:14.554 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:14.556 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:14.557 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:14.557 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:14.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:14.989 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:15.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:15.172 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:15.198 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:15.199 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:15.199 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 56 -lt 100 ']' 2023-10-11T16:34:15.200 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 56 + 1 2023-10-11T16:34:15.226 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=57 2023-10-11T16:34:15.227 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:15.271 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:15.272 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:15.274 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:15.275 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:15.275 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:15.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:15 smithi150 ceph-mon[132217]: pgmap v51: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 48 KiB/s wr, 26 op/s 2023-10-11T16:34:15.559 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:15 smithi111 ceph-mon[98527]: pgmap v51: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 48 KiB/s wr, 26 op/s 2023-10-11T16:34:15.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:15 smithi130 ceph-mon[131859]: pgmap v51: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 48 KiB/s wr, 26 op/s 2023-10-11T16:34:15.697 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:15.698 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:15.847 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:15.847 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:15.857 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:15.857 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:15.857 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 57 -lt 100 ']' 2023-10-11T16:34:15.858 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 57 + 1 2023-10-11T16:34:15.884 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=58 2023-10-11T16:34:15.885 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:15.928 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:15.928 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:15.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:15.931 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:15.932 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:16.288 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:16.289 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:16.438 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:16.439 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:16.448 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:16.448 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:16.448 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 58 -lt 100 ']' 2023-10-11T16:34:16.449 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 58 + 1 2023-10-11T16:34:16.485 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=59 2023-10-11T16:34:16.485 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:16.528 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:16.529 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:16.531 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:16.531 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:16.532 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:17.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:17.022 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:17.146 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:17.146 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:17.158 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:17.158 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:17.158 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 59 -lt 100 ']' 2023-10-11T16:34:17.159 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 59 + 1 2023-10-11T16:34:17.185 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=60 2023-10-11T16:34:17.186 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:17.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:17.228 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:17.230 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:17.231 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:17.231 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:17 smithi150 ceph-mon[132217]: pgmap v52: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 30 KiB/s wr, 20 op/s 2023-10-11T16:34:17.483 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:17 smithi111 ceph-mon[98527]: pgmap v52: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 30 KiB/s wr, 20 op/s 2023-10-11T16:34:17.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:17 smithi130 ceph-mon[131859]: pgmap v52: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 30 KiB/s wr, 20 op/s 2023-10-11T16:34:17.613 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:17.614 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:17.737 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:17.737 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:17.748 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:17.749 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:17.749 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 60 -lt 100 ']' 2023-10-11T16:34:17.749 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 60 + 1 2023-10-11T16:34:17.778 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=61 2023-10-11T16:34:17.778 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:17.802 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:17.802 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:17.804 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:17.805 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:17.806 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:18.172 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:18.172 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:18.288 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:18.288 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:18.298 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:18.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:18.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 61 -lt 100 ']' 2023-10-11T16:34:18.299 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 61 + 1 2023-10-11T16:34:18.331 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=62 2023-10-11T16:34:18.332 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:18.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:18.361 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:18.363 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:18.364 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:18.364 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:18.746 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:18.747 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:18.895 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:18.895 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:18.906 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:18.906 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:18.906 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 62 -lt 100 ']' 2023-10-11T16:34:18.907 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 62 + 1 2023-10-11T16:34:18.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=63 2023-10-11T16:34:18.934 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:18.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:18.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:18.971 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:18.972 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:18.972 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:19.388 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:19.388 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:19.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:19 smithi150 ceph-mon[132217]: pgmap v53: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-10-11T16:34:19.546 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:19.546 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:19.556 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:19.557 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:19.557 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 63 -lt 100 ']' 2023-10-11T16:34:19.557 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 63 + 1 2023-10-11T16:34:19.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:19 smithi130 ceph-mon[131859]: pgmap v53: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-10-11T16:34:19.584 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=64 2023-10-11T16:34:19.584 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:19.619 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:19.619 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:19.621 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:19.622 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:19.623 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:19.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:19 smithi111 ceph-mon[98527]: pgmap v53: 129 pgs: 129 active+clean; 2.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-10-11T16:34:20.071 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:20.072 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:20.204 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:20.204 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:20.214 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:20.215 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:20.215 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 64 -lt 100 ']' 2023-10-11T16:34:20.216 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 64 + 1 2023-10-11T16:34:20.248 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=65 2023-10-11T16:34:20.248 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:20.287 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:20.287 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:20.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:20.290 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:20.290 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:20.688 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:20.688 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:20.812 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:20.812 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:20.827 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:20.828 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:20.828 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 65 -lt 100 ']' 2023-10-11T16:34:20.828 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 65 + 1 2023-10-11T16:34:20.881 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=66 2023-10-11T16:34:20.881 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:20.910 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:20.910 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:20.912 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:20.913 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:20.913 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:21.321 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:21.322 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:21.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:21 smithi150 ceph-mon[132217]: pgmap v54: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-10-11T16:34:21.496 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:21.497 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:21.506 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:21.506 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:21.507 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 66 -lt 100 ']' 2023-10-11T16:34:21.507 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 66 + 1 2023-10-11T16:34:21.533 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=67 2023-10-11T16:34:21.534 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:21.578 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:21.578 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:21.580 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:21.581 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:21.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:21 smithi130 ceph-mon[131859]: pgmap v54: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-10-11T16:34:21.582 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:21.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:21 smithi111 ceph-mon[98527]: pgmap v54: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 13 op/s 2023-10-11T16:34:21.996 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:21.997 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:22.187 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:22.187 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:22.197 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:22.198 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:22.198 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 67 -lt 100 ']' 2023-10-11T16:34:22.198 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 67 + 1 2023-10-11T16:34:22.231 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=68 2023-10-11T16:34:22.231 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:22.262 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:22.263 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:22.265 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:22.266 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:22.266 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:22.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:22.897 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:23.046 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:23.046 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:23.064 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:23.064 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:23.065 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 68 -lt 100 ']' 2023-10-11T16:34:23.065 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 68 + 1 2023-10-11T16:34:23.106 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=69 2023-10-11T16:34:23.106 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:23.152 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:23.153 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:23.155 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:23.156 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:23.156 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:23.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:23 smithi150 ceph-mon[132217]: pgmap v55: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:23.574 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:23 smithi111 ceph-mon[98527]: pgmap v55: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:23.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:23 smithi130 ceph-mon[131859]: pgmap v55: 129 pgs: 129 active+clean; 2.1 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:23.821 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:23.821 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:24.037 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:24.038 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:24.066 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:24.066 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:24.067 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 69 -lt 100 ']' 2023-10-11T16:34:24.067 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 69 + 1 2023-10-11T16:34:24.156 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=70 2023-10-11T16:34:24.156 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:24.277 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:24.277 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:24.280 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:24.280 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:24.281 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 -- 172.21.15.111:0/302326806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 msgr2=0x7f65d8101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 --2- 172.21.15.111:0/302326806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 0x7f65d8101020 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f65d40099c0 tx=0x7f65d4009cd0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 -- 172.21.15.111:0/302326806 shutdown_connections 2023-10-11T16:34:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 --2- 172.21.15.111:0/302326806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65d8105560 0x7f65d810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 --2- 172.21.15.111:0/302326806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 0x7f65d8101020 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 --2- 172.21.15.111:0/302326806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65d81015f0 0x7f65d8104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 -- 172.21.15.111:0/302326806 >> 172.21.15.111:0/302326806 conn(0x7f65d8077550 msgr2=0x7f65d8077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 -- 172.21.15.111:0/302326806 shutdown_connections 2023-10-11T16:34:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.285+0000 7f65df5ff700 1 -- 172.21.15.111:0/302326806 wait complete. 2023-10-11T16:34:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.286+0000 7f65df5ff700 1 Processor -- start 2023-10-11T16:34:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.286+0000 7f65df5ff700 1 -- start start 2023-10-11T16:34:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65df5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 0x7f65d8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65df5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65d81015f0 0x7f65d8198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65df5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65d8105560 0x7f65d819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65df5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d819d600 con 0x7f65d8100c50 2023-10-11T16:34:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65df5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d819d770 con 0x7f65d8105560 2023-10-11T16:34:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65df5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d819d930 con 0x7f65d81015f0 2023-10-11T16:34:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65dd39b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 0x7f65d8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65dd39b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 0x7f65d8198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54430/0 (socket says 172.21.15.111:54430) 2023-10-11T16:34:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65dd39b700 1 -- 172.21.15.111:0/310557619 learned_addr learned my addr 172.21.15.111:0/310557619 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65d8105560 0x7f65d819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65d81015f0 0x7f65d8198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65dd39b700 1 -- 172.21.15.111:0/310557619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65d81015f0 msgr2=0x7f65d8198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.287+0000 7f65dd39b700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65d81015f0 0x7f65d8198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65dd39b700 1 -- 172.21.15.111:0/310557619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65d8105560 msgr2=0x7f65d819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65dd39b700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65d8105560 0x7f65d819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65dd39b700 1 -- 172.21.15.111:0/310557619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40095f0 con 0x7f65d8100c50 2023-10-11T16:34:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65d8105560 0x7f65d819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65dd39b700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 0x7f65d8198620 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f65d40098e0 tx=0x7f65d400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65ce7fc700 1 -- 172.21.15.111:0/310557619 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d401d070 con 0x7f65d8100c50 2023-10-11T16:34:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65ce7fc700 1 -- 172.21.15.111:0/310557619 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65d400fb80 con 0x7f65d8100c50 2023-10-11T16:34:24.299 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65df5ff700 1 -- 172.21.15.111:0/310557619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d81103b0 con 0x7f65d8100c50 2023-10-11T16:34:24.299 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65ce7fc700 1 -- 172.21.15.111:0/310557619 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d40173f0 con 0x7f65d8100c50 2023-10-11T16:34:24.300 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.288+0000 7f65df5ff700 1 -- 172.21.15.111:0/310557619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d8110220 con 0x7f65d8100c50 2023-10-11T16:34:24.301 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.292+0000 7f65ce7fc700 1 -- 172.21.15.111:0/310557619 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f65d400fcf0 con 0x7f65d8100c50 2023-10-11T16:34:24.301 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.293+0000 7f65ce7fc700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65c40a12c0 0x7f65c40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.302 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.293+0000 7f65ce7fc700 1 -- 172.21.15.111:0/310557619 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f65d40cdd50 con 0x7f65d8100c50 2023-10-11T16:34:24.302 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.293+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65c40a12c0 0x7f65c40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.294+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65c40a12c0 0x7f65c40a3770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65d8199c30 tx=0x7f65c8009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:24.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.399+0000 7f65f2ef1b80 1 Processor -- start 2023-10-11T16:34:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.399+0000 7f65f2ef1b80 1 -- start start 2023-10-11T16:34:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.399+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.399+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76720 0x55bd4ce70560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.399+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce70b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.399+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cd76000 con 0x55bd4cccc6b0 2023-10-11T16:34:24.414 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.399+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cdc8840 con 0x55bd4ce76b90 2023-10-11T16:34:24.414 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.399+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cdc1cd0 con 0x55bd4ce76720 2023-10-11T16:34:24.414 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce761e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54440/0 (socket says 172.21.15.111:54440) 2023-10-11T16:34:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 -- 172.21.15.111:0/2583915746 learned_addr learned my addr 172.21.15.111:0/2583915746 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76720 0x55bd4ce70560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce70b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 -- 172.21.15.111:0/2583915746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76720 msgr2=0x55bd4ce70560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76720 0x55bd4ce70560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 -- 172.21.15.111:0/2583915746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 msgr2=0x55bd4ce70b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce70b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 -- 172.21.15.111:0/2583915746 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd4cc435d0 con 0x55bd4cccc6b0 2023-10-11T16:34:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce70b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:24.419 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.400+0000 7f65dd39b700 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce761e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f65d4096fa0 tx=0x7f65d409a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a41d20dc5843ea2 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:24.419 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.401+0000 7f65c37fe700 1 -- 172.21.15.111:0/2583915746 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d409ead0 con 0x55bd4cccc6b0 2023-10-11T16:34:24.420 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.401+0000 7f65c37fe700 1 -- 172.21.15.111:0/2583915746 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65d40a2e60 con 0x55bd4cccc6b0 2023-10-11T16:34:24.420 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.401+0000 7f65c37fe700 1 -- 172.21.15.111:0/2583915746 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d40a29d0 con 0x55bd4cccc6b0 2023-10-11T16:34:24.420 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.401+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2583915746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 msgr2=0x55bd4ce761e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.401+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce761e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f65d4096fa0 tx=0x7f65d409a660 comp rx=0 tx=0).stop 2023-10-11T16:34:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.401+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2583915746 shutdown_connections 2023-10-11T16:34:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.402+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce70b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.402+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce761e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.402+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2583915746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76720 0x55bd4ce70560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.402+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2583915746 >> 172.21.15.111:0/2583915746 conn(0x55bd4ce65f50 msgr2=0x55bd4ce6b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.402+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2583915746 shutdown_connections 2023-10-11T16:34:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.402+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2583915746 wait complete. 2023-10-11T16:34:24.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.402+0000 7f65f2ef1b80 1 -- start start 2023-10-11T16:34:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.402+0000 7f65f2ef1b80 1 Processor -- start 2023-10-11T16:34:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76720 0x55bd4ce7fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce79a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cd63880 con 0x55bd4cccc6b0 2023-10-11T16:34:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cd63520 con 0x55bd4ce76720 2023-10-11T16:34:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cc458c0 con 0x55bd4ce76b90 2023-10-11T16:34:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65f2ef1b80 10 client.0 _ll_register_callbacks cb 0x55bd4cd81720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65dd39b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65dcb9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76720 0x55bd4ce7fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65dd39b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54442/0 (socket says 172.21.15.111:54442) 2023-10-11T16:34:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65dd39b700 1 -- 172.21.15.111:0/1056373805 learned_addr learned my addr 172.21.15.111:0/1056373805 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.403+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce79a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65dcb9a700 1 -- 172.21.15.111:0/1056373805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 msgr2=0x55bd4ce79a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce79a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65dcb9a700 1 -- 172.21.15.111:0/1056373805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 msgr2=0x55bd4ce7f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65dcb9a700 1 -- 172.21.15.111:0/1056373805 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d8110220 con 0x55bd4ce76720 2023-10-11T16:34:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65dd39b700 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76720 0x55bd4ce7fd30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009fd0 tx=0x7f65c80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4edcedc359891c81 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:34:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.404+0000 7f65c0ff9700 1 -- 172.21.15.111:0/1056373805 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80045b0 con 0x55bd4ce76720 2023-10-11T16:34:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.405+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/1056373805 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bd4ce63710 con 0x55bd4ce76720 2023-10-11T16:34:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.405+0000 7f65c0ff9700 1 -- 172.21.15.111:0/1056373805 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f65c801e420 con 0x55bd4ce76720 2023-10-11T16:34:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.405+0000 7f65c0ff9700 1 -- 172.21.15.111:0/1056373805 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65c801e5e0 con 0x55bd4ce76720 2023-10-11T16:34:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.405+0000 7f65f2ef1b80 10 client.25946 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.405+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/1056373805 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bd4cd88130 con 0x55bd4ce76720 2023-10-11T16:34:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.405+0000 7f65c0ff9700 1 -- 172.21.15.111:0/1056373805 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bd4ce63710 con 0x55bd4ce76720 2023-10-11T16:34:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.405+0000 7f65c0ff9700 1 -- 172.21.15.111:0/1056373805 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f65c80288e0 con 0x55bd4ce76720 2023-10-11T16:34:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.406+0000 7f65f2ef1b80 10 client.25946 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.406+0000 7f65f2ef1b80 10 client.25946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.406+0000 7f65f2ef1b80 20 client.25946 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.406+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bd4ce89cf0 0x55bd4ce8c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.406+0000 7f65f2ef1b80 4 client.25946 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.406+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/1056373805 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd4cd84040 con 0x55bd4ce89cf0 2023-10-11T16:34:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.406+0000 7f65dd39b700 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bd4ce89cf0 0x55bd4ce8c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.407+0000 7f65c0ff9700 10 client.25946 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.407+0000 7f65dd39b700 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bd4ce89cf0 0x55bd4ce8c100 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b57ca1cdbcd5f6e6 server_cookie=e842400909eeddfa in_seq=0 out_seq=0 2023-10-11T16:34:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.407+0000 7f65c0ff9700 1 -- 172.21.15.111:0/1056373805 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bd4cd84040 con 0x55bd4ce89cf0 2023-10-11T16:34:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.407+0000 7f65c0ff9700 10 client.25946 handle_command_reply: tid=0 2023-10-11T16:34:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 client.25946 shutdown 2023-10-11T16:34:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/1056373805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76720 msgr2=0x55bd4ce7fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76720 0x55bd4ce7fd30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009fd0 tx=0x7f65c80047c0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/1056373805 shutdown_connections 2023-10-11T16:34:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bd4ce89cf0 0x55bd4ce8c100 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce79a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65c0ff9700 0 client.25946 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/1056373805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76720 0x55bd4ce7fd30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/1056373805 >> 172.21.15.111:0/1056373805 conn(0x55bd4ce65f50 msgr2=0x55bd4ce6a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/1056373805 shutdown_connections 2023-10-11T16:34:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/1056373805 wait complete. 2023-10-11T16:34:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.408+0000 7f65f2ef1b80 20 client.25946 trim_cache size 0 max 16384 2023-10-11T16:34:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.425+0000 7f65f2ef1b80 1 Processor -- start 2023-10-11T16:34:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.425+0000 7f65f2ef1b80 1 -- start start 2023-10-11T16:34:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.425+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce79bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.425+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce73e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.425+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce6af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.425+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4ce6c1d0 con 0x55bd4ce76720 2023-10-11T16:34:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.425+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4ce6bf70 con 0x55bd4ce76b90 2023-10-11T16:34:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.425+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4ce6d010 con 0x55bd4cccc6b0 2023-10-11T16:34:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce73e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce73e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54444/0 (socket says 172.21.15.111:54444) 2023-10-11T16:34:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 -- 172.21.15.111:0/3761932381 learned_addr learned my addr 172.21.15.111:0/3761932381 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dd39b700 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce79bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce6af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 -- 172.21.15.111:0/3761932381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4cccc6b0 msgr2=0x55bd4ce79bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce79bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 -- 172.21.15.111:0/3761932381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 msgr2=0x55bd4ce6af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce6af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 -- 172.21.15.111:0/3761932381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd4cd87e80 con 0x55bd4ce76720 2023-10-11T16:34:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce6af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.426+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce73e60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009340 tx=0x7f65c8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf97e704bd4ed0d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65c3fff700 1 -- 172.21.15.111:0/3761932381 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8003e40 con 0x55bd4ce76720 2023-10-11T16:34:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65c3fff700 1 -- 172.21.15.111:0/3761932381 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65c8004cf0 con 0x55bd4ce76720 2023-10-11T16:34:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65c3fff700 1 -- 172.21.15.111:0/3761932381 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8028a60 con 0x55bd4ce76720 2023-10-11T16:34:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/3761932381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 msgr2=0x55bd4ce73e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce73e60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009340 tx=0x7f65c8011ba0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/3761932381 shutdown_connections 2023-10-11T16:34:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4ce76b90 0x55bd4ce6af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce79bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/3761932381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce73e60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.427+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/3761932381 >> 172.21.15.111:0/3761932381 conn(0x55bd4ce65f50 msgr2=0x55bd4ce7a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.428+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/3761932381 shutdown_connections 2023-10-11T16:34:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.428+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/3761932381 wait complete. 2023-10-11T16:34:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.428+0000 7f65f2ef1b80 1 -- start start 2023-10-11T16:34:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.428+0000 7f65f2ef1b80 1 Processor -- start 2023-10-11T16:34:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce7d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65f2ef1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce7dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cedcbf0 con 0x55bd4ce76720 2023-10-11T16:34:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cedcd60 con 0x55bd4cccc6b0 2023-10-11T16:34:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65f2ef1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd4cedcf20 con 0x55bd4ce76b90 2023-10-11T16:34:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65f2ef1b80 10 client.0 _ll_register_callbacks cb 0x7f65c80040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65dcb9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce7d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65dd39b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65dcb9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce7d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54452/0 (socket says 172.21.15.111:54452) 2023-10-11T16:34:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65dcb9a700 1 -- 172.21.15.111:0/2072848875 learned_addr learned my addr 172.21.15.111:0/2072848875 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65dd39b700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45430/0 (socket says 172.21.15.111:45430) 2023-10-11T16:34:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.429+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce7dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65dcb9a700 1 -- 172.21.15.111:0/2072848875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 msgr2=0x55bd4ce7dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce7dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65dcb9a700 1 -- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4cccc6b0 msgr2=0x55bd4ce7d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65dcb9a700 1 -- 172.21.15.111:0/2072848875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d0009280 con 0x55bd4ce76720 2023-10-11T16:34:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65dd39b700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65ddb9c700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce7dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65dcb9a700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce7d7e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f65c80097c0 tx=0x7f65c8003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b5b25a4acca644 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65c0ff9700 1 -- 172.21.15.111:0/2072848875 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80283f0 con 0x55bd4ce76720 2023-10-11T16:34:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65c0ff9700 1 -- 172.21.15.111:0/2072848875 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f65c8028550 con 0x55bd4ce76720 2023-10-11T16:34:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2072848875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65d40a2e60 con 0x55bd4ce76720 2023-10-11T16:34:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65c0ff9700 1 -- 172.21.15.111:0/2072848875 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65c8028770 con 0x55bd4ce76720 2023-10-11T16:34:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65f2ef1b80 10 client.15909 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.430+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2072848875 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bd4cd88130 con 0x55bd4ce76720 2023-10-11T16:34:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65c0ff9700 1 -- 172.21.15.111:0/2072848875 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65d40a2e60 con 0x55bd4ce76720 2023-10-11T16:34:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65c0ff9700 1 -- 172.21.15.111:0/2072848875 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f65c803b070 con 0x55bd4ce76720 2023-10-11T16:34:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65f2ef1b80 10 client.15909 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65f2ef1b80 10 client.15909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65f2ef1b80 20 client.15909 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bd4cede1e0 0x55bd4cee05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65f2ef1b80 4 client.15909 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:34:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2072848875 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bd4cd90600 con 0x55bd4cede1e0 2023-10-11T16:34:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.431+0000 7f65dd39b700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bd4cede1e0 0x55bd4cee05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.432+0000 7f65dd39b700 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bd4cede1e0 0x55bd4cee05f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97a26faff41dbff5 server_cookie=d9a3c7fdb8b23928 in_seq=0 out_seq=0 2023-10-11T16:34:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.432+0000 7f65c0ff9700 10 client.15909 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65c0ff9700 1 -- 172.21.15.111:0/2072848875 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bd4cd90600 con 0x55bd4cede1e0 2023-10-11T16:34:24.461 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65c0ff9700 10 client.15909 handle_command_reply: tid=0 2023-10-11T16:34:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 client.15909 shutdown 2023-10-11T16:34:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2072848875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 msgr2=0x55bd4ce7d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce7d7e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f65c80097c0 tx=0x7f65c8003ef0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.462 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2072848875 shutdown_connections 2023-10-11T16:34:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bd4cede1e0 0x55bd4cee05f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bd4ce76b90 0x55bd4ce7dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd4cccc6b0 0x55bd4ce7d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65c0ff9700 0 client.15909 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 --2- 172.21.15.111:0/2072848875 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bd4ce76720 0x55bd4ce7d7e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2072848875 >> 172.21.15.111:0/2072848875 conn(0x55bd4ce65f50 msgr2=0x55bd4ce73e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2072848875 shutdown_connections 2023-10-11T16:34:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 1 -- 172.21.15.111:0/2072848875 wait complete. 2023-10-11T16:34:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.440+0000 7f65f2ef1b80 20 client.15909 trim_cache size 0 max 16384 2023-10-11T16:34:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:34:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:34:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:34:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:34:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 -- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65c40a12c0 msgr2=0x7f65c40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65c40a12c0 0x7f65c40a3770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65d8199c30 tx=0x7f65c8009260 comp rx=0 tx=0).stop 2023-10-11T16:34:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 -- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 msgr2=0x7f65d8198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 0x7f65d8198620 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f65d40098e0 tx=0x7f65d400f690 comp rx=0 tx=0).stop 2023-10-11T16:34:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 -- 172.21.15.111:0/310557619 shutdown_connections 2023-10-11T16:34:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65c40a12c0 0x7f65c40a3770 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65d8105560 0x7f65d819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65d8100c50 0x7f65d8198620 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 --2- 172.21.15.111:0/310557619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65d81015f0 0x7f65d8198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 -- 172.21.15.111:0/310557619 >> 172.21.15.111:0/310557619 conn(0x7f65d8077550 msgr2=0x7f65d80fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 -- 172.21.15.111:0/310557619 shutdown_connections 2023-10-11T16:34:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:24.441+0000 7f65df5ff700 1 -- 172.21.15.111:0/310557619 wait complete. 2023-10-11T16:34:24.879 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:24.879 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:25.019 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:25.020 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:25.030 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:25.031 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:25.031 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 70 -lt 100 ']' 2023-10-11T16:34:25.032 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 70 + 1 2023-10-11T16:34:25.064 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=71 2023-10-11T16:34:25.064 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:25.102 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:25.103 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:25.105 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:25.106 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:25.106 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:25.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:25 smithi150 ceph-mon[132217]: pgmap v56: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 12 op/s 2023-10-11T16:34:25.479 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:25.480 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:25.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:25 smithi130 ceph-mon[131859]: pgmap v56: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 12 op/s 2023-10-11T16:34:25.619 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:25.619 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:25.631 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:25.631 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:25.631 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 71 -lt 100 ']' 2023-10-11T16:34:25.632 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 71 + 1 2023-10-11T16:34:25.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:25 smithi111 ceph-mon[98527]: pgmap v56: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 12 op/s 2023-10-11T16:34:25.689 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=72 2023-10-11T16:34:25.689 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:25.735 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:25.736 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:25.738 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:25.739 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:25.739 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:26.147 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:26.147 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:26.279 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:26.279 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:26.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:26.289 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:26.290 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 72 -lt 100 ']' 2023-10-11T16:34:26.290 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 72 + 1 2023-10-11T16:34:26.322 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=73 2023-10-11T16:34:26.322 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:26.360 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:26.360 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:26.363 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:26.364 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:26.364 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:26.729 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:26.729 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:26.862 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:26.862 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:26.872 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:26.872 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:26.873 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 73 -lt 100 ']' 2023-10-11T16:34:26.873 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 73 + 1 2023-10-11T16:34:26.914 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=74 2023-10-11T16:34:26.914 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:26.968 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:26.969 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:26.971 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:26.972 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:26.972 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:27.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:27.362 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:27.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:27 smithi150 ceph-mon[132217]: pgmap v57: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-10-11T16:34:27.511 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:27.512 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:27.522 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:27.522 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:27.522 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 74 -lt 100 ']' 2023-10-11T16:34:27.523 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 74 + 1 2023-10-11T16:34:27.546 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=75 2023-10-11T16:34:27.547 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:27.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:27 smithi130 ceph-mon[131859]: pgmap v57: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-10-11T16:34:27.594 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:27.594 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:27.596 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:27.597 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:27.598 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:27.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:27 smithi111 ceph-mon[98527]: pgmap v57: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-10-11T16:34:28.162 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:28.163 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:28.303 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:28.303 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:28.313 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:28.313 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:28.314 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 75 -lt 100 ']' 2023-10-11T16:34:28.314 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 75 + 1 2023-10-11T16:34:28.346 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=76 2023-10-11T16:34:28.347 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:28.384 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:28.384 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:28.387 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:28.388 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:28.388 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:28.812 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:28.812 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:28.952 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:28.953 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:28.963 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:28.964 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:28.964 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 76 -lt 100 ']' 2023-10-11T16:34:28.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 76 + 1 2023-10-11T16:34:28.997 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=77 2023-10-11T16:34:28.997 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:29.068 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:29.068 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:29.071 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:29.072 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:29.072 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:29.470 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:29.470 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:29.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:29 smithi130 ceph-mon[131859]: pgmap v58: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-10-11T16:34:29.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:29 smithi111 ceph-mon[98527]: pgmap v58: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-10-11T16:34:29.670 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:29.670 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:29.679 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:29.680 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:29.680 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 77 -lt 100 ']' 2023-10-11T16:34:29.680 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 77 + 1 2023-10-11T16:34:29.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:29 smithi150 ceph-mon[132217]: pgmap v58: 129 pgs: 129 active+clean; 2.2 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-10-11T16:34:29.707 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=78 2023-10-11T16:34:29.708 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:29.751 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:29.752 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:29.754 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:29.755 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:29.755 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:30.162 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:30.162 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:30.294 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:30.295 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:30.305 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:30.305 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:30.306 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 78 -lt 100 ']' 2023-10-11T16:34:30.306 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 78 + 1 2023-10-11T16:34:30.332 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=79 2023-10-11T16:34:30.332 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:30.367 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:30.368 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:30.370 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:30.370 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:30.371 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:30.720 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:30.720 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:30.853 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:30.853 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:30.862 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:30.863 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:30.863 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 79 -lt 100 ']' 2023-10-11T16:34:30.863 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 79 + 1 2023-10-11T16:34:30.891 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=80 2023-10-11T16:34:30.891 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:30.926 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:30.927 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:30.929 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:30.929 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:30.930 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:31.387 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:31.387 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:31.527 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:31.528 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:31.537 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:31.538 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:31.538 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 80 -lt 100 ']' 2023-10-11T16:34:31.538 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 80 + 1 2023-10-11T16:34:31.574 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=81 2023-10-11T16:34:31.574 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:31.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:31 smithi130 ceph-mon[131859]: pgmap v59: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 11 op/s 2023-10-11T16:34:31.609 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:31.609 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:31.612 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:31.612 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:31.613 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:31.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:31 smithi111 ceph-mon[98527]: pgmap v59: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 11 op/s 2023-10-11T16:34:31.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:31 smithi150 ceph-mon[132217]: pgmap v59: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 11 op/s 2023-10-11T16:34:31.807 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:34:32.128 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:32.129 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:32.176 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:32.176 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:32.182 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:32.182 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:32.182 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 81 -lt 100 ']' 2023-10-11T16:34:32.183 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 81 + 1 2023-10-11T16:34:32.229 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=82 2023-10-11T16:34:32.229 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:32.240 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:32.241 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:32.243 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:32.244 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:32.244 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:32.594 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:32.595 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:32.686 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:32.686 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:32.696 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:32.696 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:32.697 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 82 -lt 100 ']' 2023-10-11T16:34:32.697 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 82 + 1 2023-10-11T16:34:32.738 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=83 2023-10-11T16:34:32.741 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:32.759 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:32.759 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:32.761 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:32.762 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:32.763 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:33.328 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:33.328 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:33.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:33 smithi111 ceph-mon[98527]: pgmap v60: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-10-11T16:34:33.460 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:33.460 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:33.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:33.480 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:33.481 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 83 -lt 100 ']' 2023-10-11T16:34:33.481 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 83 + 1 2023-10-11T16:34:33.537 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=84 2023-10-11T16:34:33.538 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:33.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:33 smithi130 ceph-mon[131859]: pgmap v60: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-10-11T16:34:33.659 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:33.660 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:33.662 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:33.662 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:33.663 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:33.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:33 smithi150 ceph-mon[132217]: pgmap v60: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-10-11T16:34:34.419 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:34.420 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:34.577 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:34.578 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:34.605 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:34.606 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:34.606 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 84 -lt 100 ']' 2023-10-11T16:34:34.607 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 84 + 1 2023-10-11T16:34:34.654 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=85 2023-10-11T16:34:34.654 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:34.750 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:34.751 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:34.753 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:34.754 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:34.754 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:35.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.125+0000 7f49e1dff700 1 -- 172.21.15.111:0/2365593704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc074b00 msgr2=0x7f49dc073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.125+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2365593704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc074b00 0x7f49dc073260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009990 tx=0x7f49d0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.126+0000 7f49e1dff700 1 -- 172.21.15.111:0/2365593704 shutdown_connections 2023-10-11T16:34:35.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.126+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2365593704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49dc10cec0 0x7f49dc10f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.126+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2365593704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49dc0746a0 0x7f49dc072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.126+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2365593704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc074b00 0x7f49dc073260 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.126+0000 7f49e1dff700 1 -- 172.21.15.111:0/2365593704 >> 172.21.15.111:0/2365593704 conn(0x7f49dc077590 msgr2=0x7f49dc077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:35.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.126+0000 7f49e1dff700 1 -- 172.21.15.111:0/2365593704 shutdown_connections 2023-10-11T16:34:35.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.126+0000 7f49e1dff700 1 -- 172.21.15.111:0/2365593704 wait complete. 2023-10-11T16:34:35.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.127+0000 7f49e1dff700 1 Processor -- start 2023-10-11T16:34:35.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49e1dff700 1 -- start start 2023-10-11T16:34:35.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49e1dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc0746a0 0x7f49dc198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49e1dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49dc074b00 0x7f49dc198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49e1dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49dc10cec0 0x7f49dc19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49e1dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc19d630 con 0x7f49dc074b00 2023-10-11T16:34:35.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49e1dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc19d7a0 con 0x7f49dc0746a0 2023-10-11T16:34:35.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49e1dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc19d960 con 0x7f49dc10cec0 2023-10-11T16:34:35.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49db7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc0746a0 0x7f49dc198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49db7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc0746a0 0x7f49dc198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33504/0 (socket says 172.21.15.111:33504) 2023-10-11T16:34:35.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.128+0000 7f49db7fe700 1 -- 172.21.15.111:0/2920212979 learned_addr learned my addr 172.21.15.111:0/2920212979 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:35.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.129+0000 7f49dbfff700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49dc10cec0 0x7f49dc19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.129+0000 7f49daffd700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49dc074b00 0x7f49dc198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.129+0000 7f49db7fe700 1 -- 172.21.15.111:0/2920212979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49dc10cec0 msgr2=0x7f49dc19cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.129+0000 7f49db7fe700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49dc10cec0 0x7f49dc19cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.129+0000 7f49db7fe700 1 -- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49dc074b00 msgr2=0x7f49dc198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.129+0000 7f49db7fe700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49dc074b00 0x7f49dc198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.129+0000 7f49db7fe700 1 -- 172.21.15.111:0/2920212979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00095f0 con 0x7f49dc0746a0 2023-10-11T16:34:35.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.129+0000 7f49db7fe700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc0746a0 0x7f49dc198650 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00e9a0 tx=0x7f49cc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:35.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.130+0000 7f49d8ff9700 1 -- 172.21.15.111:0/2920212979 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc009790 con 0x7f49dc0746a0 2023-10-11T16:34:35.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.130+0000 7f49d8ff9700 1 -- 172.21.15.111:0/2920212979 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49cc010430 con 0x7f49dc0746a0 2023-10-11T16:34:35.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.130+0000 7f49e1dff700 1 -- 172.21.15.111:0/2920212979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49dc1125a0 con 0x7f49dc0746a0 2023-10-11T16:34:35.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.130+0000 7f49d8ff9700 1 -- 172.21.15.111:0/2920212979 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc018680 con 0x7f49dc0746a0 2023-10-11T16:34:35.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.130+0000 7f49e1dff700 1 -- 172.21.15.111:0/2920212979 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc112410 con 0x7f49dc0746a0 2023-10-11T16:34:35.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.136+0000 7f49d8ff9700 1 -- 172.21.15.111:0/2920212979 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f49cc0105a0 con 0x7f49dc0746a0 2023-10-11T16:34:35.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.137+0000 7f49d8ff9700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f49c40a1340 0x7f49c40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.137+0000 7f49d8ff9700 1 -- 172.21.15.111:0/2920212979 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f49cc014070 con 0x7f49dc0746a0 2023-10-11T16:34:35.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.138+0000 7f49daffd700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f49c40a1340 0x7f49c40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.138+0000 7f49daffd700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f49c40a1340 0x7f49c40a37f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f49dc199c60 tx=0x7f49d0005fd0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:35.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.205+0000 7f49f5789b80 1 Processor -- start 2023-10-11T16:34:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.205+0000 7f49f5789b80 1 -- start start 2023-10-11T16:34:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc043f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc048370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0488b0 0x55ecbc048d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49f5789b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbbf49060 con 0x55ecbc0488b0 2023-10-11T16:34:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49f5789b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbbf9b8a0 con 0x55ecbc044470 2023-10-11T16:34:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49f5789b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbbf94d30 con 0x55ecbc043b60 2023-10-11T16:34:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49dbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0488b0 0x55ecbc048d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49dbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0488b0 0x55ecbc048d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55542/0 (socket says 172.21.15.111:55542) 2023-10-11T16:34:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49dbfff700 1 -- 172.21.15.111:0/4044137240 learned_addr learned my addr 172.21.15.111:0/4044137240 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49db7fe700 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc043f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49daffd700 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc048370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49dbfff700 1 -- 172.21.15.111:0/4044137240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 msgr2=0x55ecbc043f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:35.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.206+0000 7f49dbfff700 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc043f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.207+0000 7f49dbfff700 1 -- 172.21.15.111:0/4044137240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 msgr2=0x55ecbc048370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.207+0000 7f49dbfff700 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc048370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.207+0000 7f49dbfff700 1 -- 172.21.15.111:0/4044137240 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecbbe16630 con 0x55ecbc0488b0 2023-10-11T16:34:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.207+0000 7f49db7fe700 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc043f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:35.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.207+0000 7f49dbfff700 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0488b0 0x55ecbc048d00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f49d4011600 tx=0x7f49d4009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bad5b02f5d278a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.207+0000 7f49c1ffb700 1 -- 172.21.15.111:0/4044137240 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d400e040 con 0x55ecbc0488b0 2023-10-11T16:34:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.207+0000 7f49c1ffb700 1 -- 172.21.15.111:0/4044137240 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49d400f040 con 0x55ecbc0488b0 2023-10-11T16:34:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.207+0000 7f49c1ffb700 1 -- 172.21.15.111:0/4044137240 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4004800 con 0x55ecbc0488b0 2023-10-11T16:34:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 -- 172.21.15.111:0/4044137240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0488b0 msgr2=0x55ecbc048d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0488b0 0x55ecbc048d00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f49d4011600 tx=0x7f49d4009920 comp rx=0 tx=0).stop 2023-10-11T16:34:35.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 -- 172.21.15.111:0/4044137240 shutdown_connections 2023-10-11T16:34:35.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0488b0 0x55ecbc048d00 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc043f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 --2- 172.21.15.111:0/4044137240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc048370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 -- 172.21.15.111:0/4044137240 >> 172.21.15.111:0/4044137240 conn(0x55ecbc039360 msgr2=0x55ecbc03eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:35.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 -- 172.21.15.111:0/4044137240 shutdown_connections 2023-10-11T16:34:35.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.208+0000 7f49f5789b80 1 -- 172.21.15.111:0/4044137240 wait complete. 2023-10-11T16:34:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.209+0000 7f49f5789b80 1 -- start start 2023-10-11T16:34:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.209+0000 7f49f5789b80 1 Processor -- start 2023-10-11T16:34:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc043b60 0x55ecbc04d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc04d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc0488b0 0x55ecbc04ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49f5789b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbbf368e0 con 0x55ecbc043b60 2023-10-11T16:34:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49f5789b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbbf36580 con 0x55ecbc044470 2023-10-11T16:34:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49f5789b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbbe18920 con 0x55ecbc0488b0 2023-10-11T16:34:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49f5789b80 10 client.0 _ll_register_callbacks cb 0x55ecbbf54780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:35.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc043b60 0x55ecbc04d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc043b60 0x55ecbc04d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55550/0 (socket says 172.21.15.111:55550) 2023-10-11T16:34:35.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.210+0000 7f49db7fe700 1 -- 172.21.15.111:0/1313687587 learned_addr learned my addr 172.21.15.111:0/1313687587 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:35.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49dbfff700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc0488b0 0x55ecbc04ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49daffd700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc04d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49db7fe700 1 -- 172.21.15.111:0/1313687587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc0488b0 msgr2=0x55ecbc04ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:35.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49db7fe700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc0488b0 0x55ecbc04ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49db7fe700 1 -- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 msgr2=0x55ecbc04d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:35.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49db7fe700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc04d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49db7fe700 1 -- 172.21.15.111:0/1313687587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d40112b0 con 0x55ecbc043b60 2023-10-11T16:34:35.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49daffd700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc04d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:35.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49dbfff700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc0488b0 0x55ecbc04ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49db7fe700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc043b60 0x55ecbc04d330 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00e970 tx=0x7f49cc09a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78611bea7d9ae3f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.211+0000 7f49baffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.212+0000 7f49baffd700 1 -- 172.21.15.111:0/1313687587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc099650 con 0x55ecbc043b60 2023-10-11T16:34:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.212+0000 7f49baffd700 1 -- 172.21.15.111:0/1313687587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f49cc0a23b0 con 0x55ecbc043b60 2023-10-11T16:34:35.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.212+0000 7f49f5789b80 1 -- 172.21.15.111:0/1313687587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ecbc036b20 con 0x55ecbc043b60 2023-10-11T16:34:35.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.217+0000 7f49baffd700 1 -- 172.21.15.111:0/1313687587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49cc0b82b0 con 0x55ecbc043b60 2023-10-11T16:34:35.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.217+0000 7f49f5789b80 10 client.15921 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:35.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.217+0000 7f49f5789b80 1 -- 172.21.15.111:0/1313687587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ecbbf5b190 con 0x55ecbc043b60 2023-10-11T16:34:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.218+0000 7f49baffd700 1 -- 172.21.15.111:0/1313687587 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49cc0ca080 con 0x55ecbc043b60 2023-10-11T16:34:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.218+0000 7f49baffd700 1 -- 172.21.15.111:0/1313687587 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f49cc098ec0 con 0x55ecbc043b60 2023-10-11T16:34:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.218+0000 7f49f5789b80 10 client.15921 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.218+0000 7f49f5789b80 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:35.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.218+0000 7f49f5789b80 20 client.15921 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.218+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecbc05d8a0 0x55ecbc05fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.218+0000 7f49f5789b80 4 client.15921 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.218+0000 7f49f5789b80 1 -- 172.21.15.111:0/1313687587 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecbbf570a0 con 0x55ecbc05d8a0 2023-10-11T16:34:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.219+0000 7f49daffd700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecbc05d8a0 0x55ecbc05fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.219+0000 7f49daffd700 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecbc05d8a0 0x55ecbc05fc80 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1baf64d4414c40fa server_cookie=a1ac417b614f9194 in_seq=0 out_seq=0 2023-10-11T16:34:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.219+0000 7f49baffd700 10 client.15921 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49baffd700 1 -- 172.21.15.111:0/1313687587 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ecbbf570a0 con 0x55ecbc05d8a0 2023-10-11T16:34:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49baffd700 10 client.15921 handle_command_reply: tid=0 2023-10-11T16:34:35.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 client.15921 shutdown 2023-10-11T16:34:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 -- 172.21.15.111:0/1313687587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc043b60 msgr2=0x55ecbc04d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc043b60 0x55ecbc04d330 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00e970 tx=0x7f49cc09a050 comp rx=0 tx=0).stop 2023-10-11T16:34:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 -- 172.21.15.111:0/1313687587 shutdown_connections 2023-10-11T16:34:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecbc05d8a0 0x55ecbc05fc80 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc0488b0 0x55ecbc04ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc043b60 0x55ecbc04d330 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49baffd700 0 client.15921 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1313687587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc044470 0x55ecbc04d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 -- 172.21.15.111:0/1313687587 >> 172.21.15.111:0/1313687587 conn(0x55ecbc039360 msgr2=0x55ecbc049d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:35.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.220+0000 7f49f5789b80 1 -- 172.21.15.111:0/1313687587 shutdown_connections 2023-10-11T16:34:35.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.221+0000 7f49f5789b80 1 -- 172.21.15.111:0/1313687587 wait complete. 2023-10-11T16:34:35.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.221+0000 7f49f5789b80 20 client.15921 trim_cache size 0 max 16384 2023-10-11T16:34:35.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49f5789b80 1 Processor -- start 2023-10-11T16:34:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49f5789b80 1 -- start start 2023-10-11T16:34:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc04dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc044470 0x55ecbc04ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 0x55ecbc04b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49f5789b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbc03e3b0 con 0x55ecbc044470 2023-10-11T16:34:35.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49f5789b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbc04b710 con 0x55ecbc0488b0 2023-10-11T16:34:35.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49f5789b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbc0474a0 con 0x55ecbc043b60 2023-10-11T16:34:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49daffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc044470 0x55ecbc04ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49daffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc044470 0x55ecbc04ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55564/0 (socket says 172.21.15.111:55564) 2023-10-11T16:34:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49daffd700 1 -- 172.21.15.111:0/1943727847 learned_addr learned my addr 172.21.15.111:0/1943727847 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49db7fe700 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc04dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.239+0000 7f49dbfff700 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 0x55ecbc04b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49daffd700 1 -- 172.21.15.111:0/1943727847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 msgr2=0x55ecbc04dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49daffd700 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc04dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49daffd700 1 -- 172.21.15.111:0/1943727847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 msgr2=0x55ecbc04b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:35.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49daffd700 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 0x55ecbc04b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49daffd700 1 -- 172.21.15.111:0/1943727847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecbbf5aee0 con 0x55ecbc044470 2023-10-11T16:34:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49daffd700 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc044470 0x55ecbc04ac90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f49d0015000 tx=0x7f49d0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf201fc0afb4468 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49c27fc700 1 -- 172.21.15.111:0/1943727847 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d002b040 con 0x55ecbc044470 2023-10-11T16:34:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49c27fc700 1 -- 172.21.15.111:0/1943727847 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49d002c040 con 0x55ecbc044470 2023-10-11T16:34:35.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.240+0000 7f49c27fc700 1 -- 172.21.15.111:0/1943727847 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d000f460 con 0x55ecbc044470 2023-10-11T16:34:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 -- 172.21.15.111:0/1943727847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc044470 msgr2=0x55ecbc04ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc044470 0x55ecbc04ac90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f49d0015000 tx=0x7f49d0020000 comp rx=0 tx=0).stop 2023-10-11T16:34:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 -- 172.21.15.111:0/1943727847 shutdown_connections 2023-10-11T16:34:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 0x55ecbc04b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc04dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 --2- 172.21.15.111:0/1943727847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc044470 0x55ecbc04ac90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 -- 172.21.15.111:0/1943727847 >> 172.21.15.111:0/1943727847 conn(0x55ecbc039360 msgr2=0x55ecbc059c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:35.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 -- 172.21.15.111:0/1943727847 shutdown_connections 2023-10-11T16:34:35.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.241+0000 7f49f5789b80 1 -- 172.21.15.111:0/1943727847 wait complete. 2023-10-11T16:34:35.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.242+0000 7f49f5789b80 1 -- start start 2023-10-11T16:34:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.242+0000 7f49f5789b80 1 Processor -- start 2023-10-11T16:34:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.242+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc052780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.242+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 0x55ecbc052cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.242+0000 7f49f5789b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0ae720 0x55ecbc0b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.242+0000 7f49f5789b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbc0b1040 con 0x55ecbc0ae720 2023-10-11T16:34:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.242+0000 7f49f5789b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbc0b1180 con 0x55ecbc0488b0 2023-10-11T16:34:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.242+0000 7f49f5789b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecbc0b1340 con 0x55ecbc043b60 2023-10-11T16:34:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49f5789b80 10 client.0 _ll_register_callbacks cb 0x7f49cc0998c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49db7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc052780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49dbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0ae720 0x55ecbc0b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49db7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc052780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:39160/0 (socket says 172.21.15.111:39160) 2023-10-11T16:34:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49db7fe700 1 -- 172.21.15.111:0/3919323621 learned_addr learned my addr 172.21.15.111:0/3919323621 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49daffd700 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 0x55ecbc052cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49db7fe700 1 -- 172.21.15.111:0/3919323621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 msgr2=0x55ecbc052cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:35.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49db7fe700 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 0x55ecbc052cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49db7fe700 1 -- 172.21.15.111:0/3919323621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0ae720 msgr2=0x55ecbc0b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49db7fe700 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0ae720 0x55ecbc0b0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49db7fe700 1 -- 172.21.15.111:0/3919323621 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc10fb90 con 0x55ecbc043b60 2023-10-11T16:34:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.243+0000 7f49dbfff700 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0ae720 0x55ecbc0b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:35.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49db7fe700 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc052780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f49cc099ef0 tx=0x7f49cc09a4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae0391c420a22e57 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49baffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:34:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49baffd700 1 -- 172.21.15.111:0/3919323621 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc099650 con 0x55ecbc043b60 2023-10-11T16:34:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49baffd700 1 -- 172.21.15.111:0/3919323621 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f49cc0a23b0 con 0x55ecbc043b60 2023-10-11T16:34:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49f5789b80 1 -- 172.21.15.111:0/3919323621 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49d400f040 con 0x55ecbc043b60 2023-10-11T16:34:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49baffd700 1 -- 172.21.15.111:0/3919323621 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49cc094b70 con 0x55ecbc043b60 2023-10-11T16:34:35.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49f5789b80 10 client.26002 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49f5789b80 1 -- 172.21.15.111:0/3919323621 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ecbbdabc60 con 0x55ecbc043b60 2023-10-11T16:34:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49baffd700 1 -- 172.21.15.111:0/3919323621 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49d400f040 con 0x55ecbc043b60 2023-10-11T16:34:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49baffd700 1 -- 172.21.15.111:0/3919323621 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f49cc0ca2e0 con 0x55ecbc043b60 2023-10-11T16:34:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49f5789b80 10 client.26002 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:35.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49f5789b80 10 client.26002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49f5789b80 20 client.26002 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.244+0000 7f49f5789b80 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecbc0b25f0 0x55ecbc0b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.245+0000 7f49f5789b80 4 client.26002 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:34:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.245+0000 7f49f5789b80 1 -- 172.21.15.111:0/3919323621 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ecbbf63660 con 0x55ecbc0b25f0 2023-10-11T16:34:35.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.245+0000 7f49daffd700 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecbc0b25f0 0x55ecbc0b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.245+0000 7f49baffd700 10 client.26002 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:35.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.245+0000 7f49daffd700 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecbc0b25f0 0x55ecbc0b49d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aea41b7a75a91a3 server_cookie=7ec1cd5cd01dbe26 in_seq=0 out_seq=0 2023-10-11T16:34:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49baffd700 1 -- 172.21.15.111:0/3919323621 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ecbbf63660 con 0x55ecbc0b25f0 2023-10-11T16:34:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49baffd700 10 client.26002 handle_command_reply: tid=0 2023-10-11T16:34:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 client.26002 shutdown 2023-10-11T16:34:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 -- 172.21.15.111:0/3919323621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 msgr2=0x55ecbc052780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc052780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f49cc099ef0 tx=0x7f49cc09a4a0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 -- 172.21.15.111:0/3919323621 shutdown_connections 2023-10-11T16:34:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecbc0b25f0 0x55ecbc0b49d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecbc0ae720 0x55ecbc0b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecbc043b60 0x55ecbc052780 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 --2- 172.21.15.111:0/3919323621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecbc0488b0 0x55ecbc052cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49baffd700 0 client.26002 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 -- 172.21.15.111:0/3919323621 >> 172.21.15.111:0/3919323621 conn(0x55ecbc039360 msgr2=0x55ecbc04ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 -- 172.21.15.111:0/3919323621 shutdown_connections 2023-10-11T16:34:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 1 -- 172.21.15.111:0/3919323621 wait complete. 2023-10-11T16:34:35.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.246+0000 7f49f5789b80 20 client.26002 trim_cache size 0 max 16384 2023-10-11T16:34:35.264 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:34:35.264 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:34:35.264 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:34:35.264 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:34:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 -- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f49c40a1340 msgr2=0x7f49c40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f49c40a1340 0x7f49c40a37f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f49dc199c60 tx=0x7f49d0005fd0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 -- 172.21.15.111:0/2920212979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc0746a0 msgr2=0x7f49dc198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc0746a0 0x7f49dc198650 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00e9a0 tx=0x7f49cc00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 -- 172.21.15.111:0/2920212979 shutdown_connections 2023-10-11T16:34:35.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f49c40a1340 0x7f49c40a37f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49dc10cec0 0x7f49dc19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49dc0746a0 0x7f49dc198650 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 --2- 172.21.15.111:0/2920212979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49dc074b00 0x7f49dc198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 -- 172.21.15.111:0/2920212979 >> 172.21.15.111:0/2920212979 conn(0x7f49dc077590 msgr2=0x7f49dc104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:35.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 -- 172.21.15.111:0/2920212979 shutdown_connections 2023-10-11T16:34:35.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:35.247+0000 7f49e1dff700 1 -- 172.21.15.111:0/2920212979 wait complete. 2023-10-11T16:34:35.292 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:34:35.293 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:34:35.492 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:35 smithi111 ceph-mon[98527]: pgmap v61: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-10-11T16:34:35.552 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:35.553 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:35.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:35 smithi130 ceph-mon[131859]: pgmap v61: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-10-11T16:34:35.634 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:35.634 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:35.654 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:35.655 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:35.655 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 85 -lt 100 ']' 2023-10-11T16:34:35.656 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 85 + 1 2023-10-11T16:34:35.695 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=86 2023-10-11T16:34:35.696 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:35.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:35 smithi150 ceph-mon[132217]: pgmap v61: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-10-11T16:34:35.712 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:35.712 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:35.714 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:35.715 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:35.715 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:36.085 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:36.086 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:36.198 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:36.198 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:36.229 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:36.230 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:36.230 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 86 -lt 100 ']' 2023-10-11T16:34:36.231 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 86 + 1 2023-10-11T16:34:36.298 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=87 2023-10-11T16:34:36.299 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:36.340 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:36.341 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:36.342 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:36.343 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:36.344 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:36.877 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:36.878 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:37.160 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:37.161 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:37.195 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:37.195 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:37.195 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 87 -lt 100 ']' 2023-10-11T16:34:37.196 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 87 + 1 2023-10-11T16:34:37.230 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=88 2023-10-11T16:34:37.231 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:37.302 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:37.303 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:37.305 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:37.305 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:37.306 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:37.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:37 smithi130 ceph-mon[131859]: pgmap v62: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:37.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:37 smithi111 ceph-mon[98527]: pgmap v62: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:37 smithi150 ceph-mon[132217]: pgmap v62: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:38.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 -- 172.21.15.111:0/1108116954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100550 msgr2=0x7f8134100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:38.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 --2- 172.21.15.111:0/1108116954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100550 0x7f8134100920 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f81280099c0 tx=0x7f8128009cd0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 -- 172.21.15.111:0/1108116954 shutdown_connections 2023-10-11T16:34:38.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 --2- 172.21.15.111:0/1108116954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8134105540 0x7f813410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 --2- 172.21.15.111:0/1108116954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100550 0x7f8134100920 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 --2- 172.21.15.111:0/1108116954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8134100ef0 0x7f8134104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 -- 172.21.15.111:0/1108116954 >> 172.21.15.111:0/1108116954 conn(0x7f8134077020 msgr2=0x7f8134077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:38.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 -- 172.21.15.111:0/1108116954 shutdown_connections 2023-10-11T16:34:38.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.708+0000 7f813afff700 1 -- 172.21.15.111:0/1108116954 wait complete. 2023-10-11T16:34:38.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.709+0000 7f813afff700 1 Processor -- start 2023-10-11T16:34:38.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f813afff700 1 -- start start 2023-10-11T16:34:38.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f813afff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8134100550 0x7f8134072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:38.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f813afff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100ef0 0x7f8134073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:38.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f813afff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8134105540 0x7f81341a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:38.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f813afff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8134073b60 con 0x7f8134100ef0 2023-10-11T16:34:38.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f813afff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81341a4b10 con 0x7f8134105540 2023-10-11T16:34:38.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f813afff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81341a4c50 con 0x7f8134100550 2023-10-11T16:34:38.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f8133fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100ef0 0x7f8134073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:38.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f8133fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100ef0 0x7f8134073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55588/0 (socket says 172.21.15.111:55588) 2023-10-11T16:34:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f8133fff700 1 -- 172.21.15.111:0/3821233615 learned_addr learned my addr 172.21.15.111:0/3821233615 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.710+0000 7f813959c700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8134105540 0x7f81341a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.711+0000 7f8138d9b700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8134100550 0x7f8134072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.711+0000 7f8133fff700 1 -- 172.21.15.111:0/3821233615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8134100550 msgr2=0x7f8134072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.711+0000 7f8133fff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8134100550 0x7f8134072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.711+0000 7f8133fff700 1 -- 172.21.15.111:0/3821233615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8134105540 msgr2=0x7f81341a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:38.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.711+0000 7f8133fff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8134105540 0x7f81341a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.711+0000 7f8133fff700 1 -- 172.21.15.111:0/3821233615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81280095f0 con 0x7f8134100ef0 2023-10-11T16:34:38.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.711+0000 7f813959c700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8134105540 0x7f81341a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:38.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.711+0000 7f8133fff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100ef0 0x7f8134073330 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f812000e940 tx=0x7f812000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:38.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.712+0000 7f8131ffb700 1 -- 172.21.15.111:0/3821233615 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8120009730 con 0x7f8134100ef0 2023-10-11T16:34:38.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.712+0000 7f8131ffb700 1 -- 172.21.15.111:0/3821233615 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8120010430 con 0x7f8134100ef0 2023-10-11T16:34:38.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.712+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8134110350 con 0x7f8134100ef0 2023-10-11T16:34:38.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.712+0000 7f8131ffb700 1 -- 172.21.15.111:0/3821233615 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812000cea0 con 0x7f8134100ef0 2023-10-11T16:34:38.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.712+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81341101c0 con 0x7f8134100ef0 2023-10-11T16:34:38.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.713+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81341104e0 con 0x7f8134100ef0 2023-10-11T16:34:38.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.716+0000 7f8131ffb700 1 -- 172.21.15.111:0/3821233615 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f8120010810 con 0x7f8134100ef0 2023-10-11T16:34:38.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.717+0000 7f8131ffb700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f81240a12c0 0x7f81240a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:38.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.717+0000 7f8131ffb700 1 -- 172.21.15.111:0/3821233615 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f8120014070 con 0x7f8134100ef0 2023-10-11T16:34:38.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.718+0000 7f8138d9b700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f81240a12c0 0x7f81240a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:38.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.718+0000 7f8138d9b700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f81240a12c0 0x7f81240a3770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8128009990 tx=0x7f8128005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:38.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.726+0000 7f8131ffb700 1 -- 172.21.15.111:0/3821233615 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8120096450 con 0x7f8134100ef0 2023-10-11T16:34:38.735 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:38.736 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:38.918 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:38.918 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:38.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:38.937 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:38.938 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 88 -lt 100 ']' 2023-10-11T16:34:38.938 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 88 + 1 2023-10-11T16:34:38.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.981+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8134067a60 con 0x7f8134100ef0 2023-10-11T16:34:38.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.982+0000 7f8131ffb700 1 -- 172.21.15.111:0/3821233615 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10820 (secure 0 0 0) 0x7f8120099a70 con 0x7f8134100ef0 2023-10-11T16:34:38.983 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:34:38.984 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:33:01.126571+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:34:38.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f81240a12c0 msgr2=0x7f81240a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:38.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f81240a12c0 0x7f81240a3770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8128009990 tx=0x7f8128005fb0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100ef0 msgr2=0x7f8134073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:38.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100ef0 0x7f8134073330 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f812000e940 tx=0x7f812000ec50 comp rx=0 tx=0).stop 2023-10-11T16:34:38.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 shutdown_connections 2023-10-11T16:34:38.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f81240a12c0 0x7f81240a3770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8134105540 0x7f81341a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8134100550 0x7f8134072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 --2- 172.21.15.111:0/3821233615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8134100ef0 0x7f8134073330 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:38.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.984+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 >> 172.21.15.111:0/3821233615 conn(0x7f8134077020 msgr2=0x7f81340fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:38.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.985+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 shutdown_connections 2023-10-11T16:34:38.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:38.985+0000 7f813afff700 1 -- 172.21.15.111:0/3821233615 wait complete. 2023-10-11T16:34:38.989 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=89 2023-10-11T16:34:38.989 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:38.989 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 28 2023-10-11T16:34:39.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:34:39.082 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:39.083 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:39.085 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:39.085 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:39.086 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:39.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:39 smithi111 ceph-mon[98527]: pgmap v63: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:39.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:39 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3821233615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:34:39.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:39 smithi130 ceph-mon[131859]: pgmap v63: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:39 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3821233615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:34:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:39 smithi150 ceph-mon[132217]: pgmap v63: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-10-11T16:34:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:39 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3821233615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:34:39.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:39.819 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:39.922 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:39.922 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:40.020 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:40.021 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:40.021 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 89 -lt 100 ']' 2023-10-11T16:34:40.021 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 89 + 1 2023-10-11T16:34:40.061 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=90 2023-10-11T16:34:40.062 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:40.109 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:40.109 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:40.111 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:40.112 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:40.112 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 -- 172.21.15.111:0/1584223344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f924c100c50 msgr2=0x7f924c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 --2- 172.21.15.111:0/1584223344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f924c100c50 0x7f924c101020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9248009990 tx=0x7f9248009ca0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 -- 172.21.15.111:0/1584223344 shutdown_connections 2023-10-11T16:34:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 --2- 172.21.15.111:0/1584223344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f924c105560 0x7f924c10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 --2- 172.21.15.111:0/1584223344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f924c100c50 0x7f924c101020 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 --2- 172.21.15.111:0/1584223344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c1015f0 0x7f924c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 -- 172.21.15.111:0/1584223344 >> 172.21.15.111:0/1584223344 conn(0x7f924c077550 msgr2=0x7f924c077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 -- 172.21.15.111:0/1584223344 shutdown_connections 2023-10-11T16:34:40.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.382+0000 7f92535ff700 1 -- 172.21.15.111:0/1584223344 wait complete. 2023-10-11T16:34:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.383+0000 7f92535ff700 1 Processor -- start 2023-10-11T16:34:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.383+0000 7f92535ff700 1 -- start start 2023-10-11T16:34:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f92535ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c100c50 0x7f924c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f92535ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f924c1015f0 0x7f924c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f92535ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f924c105560 0x7f924c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f92535ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c19d610 con 0x7f924c100c50 2023-10-11T16:34:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f92535ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c19d780 con 0x7f924c105560 2023-10-11T16:34:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f92535ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924c19d940 con 0x7f924c1015f0 2023-10-11T16:34:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c100c50 0x7f924c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c100c50 0x7f924c198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55602/0 (socket says 172.21.15.111:55602) 2023-10-11T16:34:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f925139b700 1 -- 172.21.15.111:0/3160161909 learned_addr learned my addr 172.21.15.111:0/3160161909 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f9250b9a700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f924c1015f0 0x7f924c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f9251b9c700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f924c105560 0x7f924c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f925139b700 1 -- 172.21.15.111:0/3160161909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f924c1015f0 msgr2=0x7f924c198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f925139b700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f924c1015f0 0x7f924c198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f925139b700 1 -- 172.21.15.111:0/3160161909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f924c105560 msgr2=0x7f924c19cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f925139b700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f924c105560 0x7f924c19cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.384+0000 7f925139b700 1 -- 172.21.15.111:0/3160161909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c009520 con 0x7f924c100c50 2023-10-11T16:34:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.385+0000 7f925139b700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c100c50 0x7f924c198630 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9248005910 tx=0x7f92480047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.385+0000 7f92427fc700 1 -- 172.21.15.111:0/3160161909 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924801d070 con 0x7f924c100c50 2023-10-11T16:34:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.385+0000 7f92535ff700 1 -- 172.21.15.111:0/3160161909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92480095f0 con 0x7f924c100c50 2023-10-11T16:34:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.385+0000 7f92427fc700 1 -- 172.21.15.111:0/3160161909 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9248022470 con 0x7f924c100c50 2023-10-11T16:34:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.385+0000 7f92427fc700 1 -- 172.21.15.111:0/3160161909 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248017650 con 0x7f924c100c50 2023-10-11T16:34:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.385+0000 7f92535ff700 1 -- 172.21.15.111:0/3160161909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c1103b0 con 0x7f924c100c50 2023-10-11T16:34:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.388+0000 7f92427fc700 1 -- 172.21.15.111:0/3160161909 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f92480225e0 con 0x7f924c100c50 2023-10-11T16:34:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.389+0000 7f92427fc700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f92380a12c0 0x7f92380a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.390+0000 7f92427fc700 1 -- 172.21.15.111:0/3160161909 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f92480ce470 con 0x7f924c100c50 2023-10-11T16:34:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.390+0000 7f9250b9a700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f92380a12c0 0x7f92380a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.390+0000 7f9250b9a700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f92380a12c0 0x7f92380a3770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f924c199c40 tx=0x7f923c009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.504+0000 7f9266fdcb80 1 Processor -- start 2023-10-11T16:34:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.504+0000 7f9266fdcb80 1 -- start start 2023-10-11T16:34:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.504+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7da1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.504+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7d4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.504+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.504+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f6da000 con 0x560a7f6306b0 2023-10-11T16:34:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.504+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f72c840 con 0x560a7f7da720 2023-10-11T16:34:40.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.504+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f725cd0 con 0x560a7f7dab90 2023-10-11T16:34:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7da1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f9251b9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f9250b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7d4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7da1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55614/0 (socket says 172.21.15.111:55614) 2023-10-11T16:34:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 -- 172.21.15.111:0/169249472 learned_addr learned my addr 172.21.15.111:0/169249472 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 -- 172.21.15.111:0/169249472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 msgr2=0x560a7f7d4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 --2- 172.21.15.111:0/169249472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7d4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 -- 172.21.15.111:0/169249472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 msgr2=0x560a7f7d4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 --2- 172.21.15.111:0/169249472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7d4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 -- 172.21.15.111:0/169249472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a7f5a75d0 con 0x560a7f6306b0 2023-10-11T16:34:40.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f9251b9c700 1 --2- 172.21.15.111:0/169249472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7d4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:40.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f925139b700 1 --2- 172.21.15.111:0/169249472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7da1e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9248024000 tx=0x7f924809ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b87a31ffdb03dcb server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:40.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f92377fe700 1 -- 172.21.15.111:0/169249472 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924809f1f0 con 0x560a7f6306b0 2023-10-11T16:34:40.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f92377fe700 1 -- 172.21.15.111:0/169249472 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92480a3580 con 0x560a7f6306b0 2023-10-11T16:34:40.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.505+0000 7f92377fe700 1 -- 172.21.15.111:0/169249472 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924809b680 con 0x560a7f6306b0 2023-10-11T16:34:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 -- 172.21.15.111:0/169249472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 msgr2=0x560a7f7da1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/169249472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7da1e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9248024000 tx=0x7f924809ad80 comp rx=0 tx=0).stop 2023-10-11T16:34:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 -- 172.21.15.111:0/169249472 shutdown_connections 2023-10-11T16:34:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/169249472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7d4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/169249472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7da1e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/169249472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7d4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 -- 172.21.15.111:0/169249472 >> 172.21.15.111:0/169249472 conn(0x560a7f7c9f50 msgr2=0x560a7f7cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 -- 172.21.15.111:0/169249472 shutdown_connections 2023-10-11T16:34:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.506+0000 7f9266fdcb80 1 -- 172.21.15.111:0/169249472 wait complete. 2023-10-11T16:34:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.507+0000 7f9266fdcb80 1 -- start start 2023-10-11T16:34:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.507+0000 7f9266fdcb80 1 Processor -- start 2023-10-11T16:34:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7dda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f6c7880 con 0x560a7f6306b0 2023-10-11T16:34:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f6c7520 con 0x560a7f7da720 2023-10-11T16:34:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f5a98c0 con 0x560a7f7dab90 2023-10-11T16:34:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9266fdcb80 10 client.0 _ll_register_callbacks cb 0x560a7f6e5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55630/0 (socket says 172.21.15.111:55630) 2023-10-11T16:34:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 -- 172.21.15.111:0/2670234320 learned_addr learned my addr 172.21.15.111:0/2670234320 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9250b9a700 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9251b9c700 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7dda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 -- 172.21.15.111:0/2670234320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 msgr2=0x560a7f7dda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7dda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 -- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 msgr2=0x560a7f7e3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 -- 172.21.15.111:0/2670234320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924c1103b0 con 0x560a7f6306b0 2023-10-11T16:34:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9250b9a700 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f925139b700 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e37a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f92480a3050 tx=0x7f9248097630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33d41c3285f9701 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.508+0000 7f9234ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:40.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9234ff9700 1 -- 172.21.15.111:0/2670234320 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924809f030 con 0x560a7f6306b0 2023-10-11T16:34:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9234ff9700 1 -- 172.21.15.111:0/2670234320 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f924809f1f0 con 0x560a7f6306b0 2023-10-11T16:34:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 1 -- 172.21.15.111:0/2670234320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a7f7c7710 con 0x560a7f6306b0 2023-10-11T16:34:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9234ff9700 1 -- 172.21.15.111:0/2670234320 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f924809b420 con 0x560a7f6306b0 2023-10-11T16:34:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 10 client.15945 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 1 -- 172.21.15.111:0/2670234320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560a7f6ec130 con 0x560a7f6306b0 2023-10-11T16:34:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9234ff9700 1 -- 172.21.15.111:0/2670234320 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560a7f7c7710 con 0x560a7f6306b0 2023-10-11T16:34:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9234ff9700 1 -- 172.21.15.111:0/2670234320 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9248096fb0 con 0x560a7f6306b0 2023-10-11T16:34:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 10 client.15945 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:40.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:40.525 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 20 client.15945 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:40.525 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a7f7ed7b0 0x560a7f7efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.525 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 4 client.15945 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:40.525 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.509+0000 7f9266fdcb80 1 -- 172.21.15.111:0/2670234320 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a7f6e8040 con 0x560a7f7ed7b0 2023-10-11T16:34:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.510+0000 7f9250b9a700 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a7f7ed7b0 0x560a7f7efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.510+0000 7f9234ff9700 10 client.15945 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.510+0000 7f9250b9a700 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a7f7ed7b0 0x560a7f7efbc0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=728ea2781a62eea server_cookie=d9093ffdf86ee889 in_seq=0 out_seq=0 2023-10-11T16:34:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9234ff9700 1 -- 172.21.15.111:0/2670234320 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560a7f6e8040 con 0x560a7f7ed7b0 2023-10-11T16:34:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9234ff9700 10 client.15945 handle_command_reply: tid=0 2023-10-11T16:34:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 client.15945 shutdown 2023-10-11T16:34:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 -- 172.21.15.111:0/2670234320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 msgr2=0x560a7f7e37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e37a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f92480a3050 tx=0x7f9248097630 comp rx=0 tx=0).stop 2023-10-11T16:34:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 -- 172.21.15.111:0/2670234320 shutdown_connections 2023-10-11T16:34:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a7f7ed7b0 0x560a7f7efbc0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7dda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e37a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9234ff9700 0 client.15945 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/2670234320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.511+0000 7f9266fdcb80 1 -- 172.21.15.111:0/2670234320 >> 172.21.15.111:0/2670234320 conn(0x560a7f7c9f50 msgr2=0x560a7f7ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.512+0000 7f9266fdcb80 1 -- 172.21.15.111:0/2670234320 shutdown_connections 2023-10-11T16:34:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.512+0000 7f9266fdcb80 1 -- 172.21.15.111:0/2670234320 wait complete. 2023-10-11T16:34:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.512+0000 7f9266fdcb80 20 client.15945 trim_cache size 0 max 16384 2023-10-11T16:34:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.527+0000 7f9266fdcb80 1 Processor -- start 2023-10-11T16:34:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.527+0000 7f9266fdcb80 1 -- start start 2023-10-11T16:34:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7ddb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7da720 0x560a7f7d7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7dab90 0x560a7f7cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f7deef0 con 0x560a7f6306b0 2023-10-11T16:34:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f7cffb0 con 0x560a7f7dab90 2023-10-11T16:34:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f7f0220 con 0x560a7f7da720 2023-10-11T16:34:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7ddb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7ddb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55642/0 (socket says 172.21.15.111:55642) 2023-10-11T16:34:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 -- 172.21.15.111:0/1310287804 learned_addr learned my addr 172.21.15.111:0/1310287804 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9250b9a700 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7da720 0x560a7f7d7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9251b9c700 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7dab90 0x560a7f7cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 -- 172.21.15.111:0/1310287804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7da720 msgr2=0x560a7f7d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7da720 0x560a7f7d7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 -- 172.21.15.111:0/1310287804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7dab90 msgr2=0x560a7f7cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7dab90 0x560a7f7cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 -- 172.21.15.111:0/1310287804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a7f6ebe80 con 0x560a7f6306b0 2023-10-11T16:34:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f9251b9c700 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7dab90 0x560a7f7cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.528+0000 7f925139b700 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7ddb60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f92480976c0 tx=0x7f9248009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd58ea0fbbb13a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9237fff700 1 -- 172.21.15.111:0/1310287804 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924809a120 con 0x560a7f6306b0 2023-10-11T16:34:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9237fff700 1 -- 172.21.15.111:0/1310287804 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f924809ab20 con 0x560a7f6306b0 2023-10-11T16:34:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9237fff700 1 -- 172.21.15.111:0/1310287804 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92480a30f0 con 0x560a7f6306b0 2023-10-11T16:34:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9266fdcb80 1 -- 172.21.15.111:0/1310287804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 msgr2=0x560a7f7ddb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7ddb60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f92480976c0 tx=0x7f9248009ca0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9266fdcb80 1 -- 172.21.15.111:0/1310287804 shutdown_connections 2023-10-11T16:34:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7dab90 0x560a7f7cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7ddb60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/1310287804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7da720 0x560a7f7d7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9266fdcb80 1 -- 172.21.15.111:0/1310287804 >> 172.21.15.111:0/1310287804 conn(0x560a7f7c9f50 msgr2=0x560a7f7de3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.529+0000 7f9266fdcb80 1 -- 172.21.15.111:0/1310287804 shutdown_connections 2023-10-11T16:34:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.530+0000 7f9266fdcb80 1 -- 172.21.15.111:0/1310287804 wait complete. 2023-10-11T16:34:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.530+0000 7f9266fdcb80 1 -- start start 2023-10-11T16:34:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.530+0000 7f9266fdcb80 1 Processor -- start 2023-10-11T16:34:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9266fdcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f840630 con 0x560a7f6306b0 2023-10-11T16:34:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f840770 con 0x560a7f7da720 2023-10-11T16:34:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9266fdcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a7f840900 con 0x560a7f7dab90 2023-10-11T16:34:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9266fdcb80 10 client.0 _ll_register_callbacks cb 0x7f923c009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55656/0 (socket says 172.21.15.111:55656) 2023-10-11T16:34:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 -- 172.21.15.111:0/3240970831 learned_addr learned my addr 172.21.15.111:0/3240970831 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9250b9a700 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9251b9c700 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 -- 172.21.15.111:0/3240970831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 msgr2=0x560a7f7e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7e1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 -- 172.21.15.111:0/3240970831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 msgr2=0x560a7f7e1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 -- 172.21.15.111:0/3240970831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a7f6ec130 con 0x560a7f6306b0 2023-10-11T16:34:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f925139b700 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e1150 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9248097340 tx=0x7f9248097430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb378175654e4fe0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.531+0000 7f9234ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9234ff9700 1 -- 172.21.15.111:0/3240970831 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92480a32f0 con 0x560a7f6306b0 2023-10-11T16:34:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9234ff9700 1 -- 172.21.15.111:0/3240970831 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f924809f1f0 con 0x560a7f6306b0 2023-10-11T16:34:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 1 -- 172.21.15.111:0/3240970831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92480a3580 con 0x560a7f6306b0 2023-10-11T16:34:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9234ff9700 1 -- 172.21.15.111:0/3240970831 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9248097500 con 0x560a7f6306b0 2023-10-11T16:34:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 10 client.15951 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:40.546 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 1 -- 172.21.15.111:0/3240970831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560a7f53cc00 con 0x560a7f6306b0 2023-10-11T16:34:40.546 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9234ff9700 1 -- 172.21.15.111:0/3240970831 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92480a3580 con 0x560a7f6306b0 2023-10-11T16:34:40.546 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9234ff9700 1 -- 172.21.15.111:0/3240970831 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f924800fb10 con 0x560a7f6306b0 2023-10-11T16:34:40.546 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 10 client.15951 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:40.546 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 10 client.15951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 20 client.15951 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a7f841b60 0x560a7f843f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 4 client.15951 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:34:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.532+0000 7f9266fdcb80 1 -- 172.21.15.111:0/3240970831 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560a7f6f4600 con 0x560a7f841b60 2023-10-11T16:34:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.533+0000 7f9250b9a700 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a7f841b60 0x560a7f843f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.533+0000 7f9250b9a700 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a7f841b60 0x560a7f843f40 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dde549e4792a624 server_cookie=93a8dd361804f69d in_seq=0 out_seq=0 2023-10-11T16:34:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.533+0000 7f9234ff9700 10 client.15951 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9234ff9700 1 -- 172.21.15.111:0/3240970831 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560a7f6f4600 con 0x560a7f841b60 2023-10-11T16:34:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9234ff9700 10 client.15951 handle_command_reply: tid=0 2023-10-11T16:34:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 client.15951 shutdown 2023-10-11T16:34:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 -- 172.21.15.111:0/3240970831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 msgr2=0x560a7f7e1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e1150 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9248097340 tx=0x7f9248097430 comp rx=0 tx=0).stop 2023-10-11T16:34:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 -- 172.21.15.111:0/3240970831 shutdown_connections 2023-10-11T16:34:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a7f841b60 0x560a7f843f40 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a7f7dab90 0x560a7f7e1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a7f6306b0 0x560a7f7e1150 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 --2- 172.21.15.111:0/3240970831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a7f7da720 0x560a7f7e1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 -- 172.21.15.111:0/3240970831 >> 172.21.15.111:0/3240970831 conn(0x560a7f7c9f50 msgr2=0x560a7f7d7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9234ff9700 0 client.15951 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 -- 172.21.15.111:0/3240970831 shutdown_connections 2023-10-11T16:34:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.541+0000 7f9266fdcb80 1 -- 172.21.15.111:0/3240970831 wait complete. 2023-10-11T16:34:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f9266fdcb80 20 client.15951 trim_cache size 0 max 16384 2023-10-11T16:34:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:34:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:34:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:34:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:34:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 -- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f92380a12c0 msgr2=0x7f92380a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f92380a12c0 0x7f92380a3770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f924c199c40 tx=0x7f923c009260 comp rx=0 tx=0).stop 2023-10-11T16:34:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 -- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c100c50 msgr2=0x7f924c198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c100c50 0x7f924c198630 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9248005910 tx=0x7f92480047c0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 -- 172.21.15.111:0/3160161909 shutdown_connections 2023-10-11T16:34:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f92380a12c0 0x7f92380a3770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f924c105560 0x7f924c19cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f924c100c50 0x7f924c198630 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 --2- 172.21.15.111:0/3160161909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f924c1015f0 0x7f924c198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 -- 172.21.15.111:0/3160161909 >> 172.21.15.111:0/3160161909 conn(0x7f924c077550 msgr2=0x7f924c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 -- 172.21.15.111:0/3160161909 shutdown_connections 2023-10-11T16:34:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:40.542+0000 7f92535ff700 1 -- 172.21.15.111:0/3160161909 wait complete. 2023-10-11T16:34:40.910 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:40.910 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:41.159 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:41.160 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:41.170 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:41.170 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:41.171 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 90 -lt 100 ']' 2023-10-11T16:34:41.171 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 90 + 1 2023-10-11T16:34:41.236 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=91 2023-10-11T16:34:41.237 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:41.291 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:41.291 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:41.293 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:41.293 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:41.294 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:41.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:41 smithi130 ceph-mon[131859]: pgmap v64: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2023-10-11T16:34:41.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:41 smithi111 ceph-mon[98527]: pgmap v64: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2023-10-11T16:34:41.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:41 smithi150 ceph-mon[132217]: pgmap v64: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 11 op/s 2023-10-11T16:34:41.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:41.818 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:42.018 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:42.018 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:42.044 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:42.044 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:42.044 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 91 -lt 100 ']' 2023-10-11T16:34:42.045 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 91 + 1 2023-10-11T16:34:42.094 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=92 2023-10-11T16:34:42.095 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:42.243 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:42.243 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:42.245 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:42.246 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:42.246 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:42.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 -- 172.21.15.111:0/513126220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94100cc0 msgr2=0x7f4e94101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 --2- 172.21.15.111:0/513126220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94100cc0 0x7f4e94101090 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4e840099c0 tx=0x7f4e84009cd0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 -- 172.21.15.111:0/513126220 shutdown_connections 2023-10-11T16:34:42.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 --2- 172.21.15.111:0/513126220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e94105620 0x7f4e9410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 --2- 172.21.15.111:0/513126220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94100cc0 0x7f4e94101090 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 --2- 172.21.15.111:0/513126220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e94101660 0x7f4e94104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 -- 172.21.15.111:0/513126220 >> 172.21.15.111:0/513126220 conn(0x7f4e94077560 msgr2=0x7f4e94077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:42.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 -- 172.21.15.111:0/513126220 shutdown_connections 2023-10-11T16:34:42.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.340+0000 7f4e993ff700 1 -- 172.21.15.111:0/513126220 wait complete. 2023-10-11T16:34:42.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.341+0000 7f4e993ff700 1 Processor -- start 2023-10-11T16:34:42.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.341+0000 7f4e993ff700 1 -- start start 2023-10-11T16:34:42.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e993ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e94100cc0 0x7f4e94072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e993ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94101660 0x7f4e94073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e993ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e94105620 0x7f4e941a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e993ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e94073b60 con 0x7f4e94101660 2023-10-11T16:34:42.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e993ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941a4c20 con 0x7f4e94100cc0 2023-10-11T16:34:42.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e993ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941a4d60 con 0x7f4e94105620 2023-10-11T16:34:42.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e927fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94101660 0x7f4e94073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e927fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94101660 0x7f4e94073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55674/0 (socket says 172.21.15.111:55674) 2023-10-11T16:34:42.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e927fc700 1 -- 172.21.15.111:0/2124659344 learned_addr learned my addr 172.21.15.111:0/2124659344 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:42.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e94100cc0 0x7f4e94072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.342+0000 7f4e937fe700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e94105620 0x7f4e941a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e927fc700 1 -- 172.21.15.111:0/2124659344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e94105620 msgr2=0x7f4e941a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e927fc700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e94105620 0x7f4e941a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e927fc700 1 -- 172.21.15.111:0/2124659344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e94100cc0 msgr2=0x7f4e94072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e927fc700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e94100cc0 0x7f4e94072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e927fc700 1 -- 172.21.15.111:0/2124659344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e840095f0 con 0x7f4e94101660 2023-10-11T16:34:42.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e94100cc0 0x7f4e94072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:42.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e927fc700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94101660 0x7f4e94073330 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800e910 tx=0x7f4e8800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:42.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e7bfff700 1 -- 172.21.15.111:0/2124659344 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88004a30 con 0x7f4e94101660 2023-10-11T16:34:42.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e7bfff700 1 -- 172.21.15.111:0/2124659344 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e880054e0 con 0x7f4e94101660 2023-10-11T16:34:42.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e993ff700 1 -- 172.21.15.111:0/2124659344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e941103d0 con 0x7f4e94101660 2023-10-11T16:34:42.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e7bfff700 1 -- 172.21.15.111:0/2124659344 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8800cc40 con 0x7f4e94101660 2023-10-11T16:34:42.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.343+0000 7f4e993ff700 1 -- 172.21.15.111:0/2124659344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e94110240 con 0x7f4e94101660 2023-10-11T16:34:42.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.346+0000 7f4e7bfff700 1 -- 172.21.15.111:0/2124659344 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f4e88005000 con 0x7f4e94101660 2023-10-11T16:34:42.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.348+0000 7f4e7bfff700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4e7c0a12c0 0x7f4e7c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.348+0000 7f4e7bfff700 1 -- 172.21.15.111:0/2124659344 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f4e88014070 con 0x7f4e94101660 2023-10-11T16:34:42.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.348+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4e7c0a12c0 0x7f4e7c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.349+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4e7c0a12c0 0x7f4e7c0a3770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009990 tx=0x7f4e8401a040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:42.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.420+0000 7f4eaccb1b80 1 Processor -- start 2023-10-11T16:34:42.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.420+0000 7f4eaccb1b80 1 -- start start 2023-10-11T16:34:42.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6bb60 0x561d78a6bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a70370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a708b0 0x561d78a70d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d78971060 con 0x561d78a6c470 2023-10-11T16:34:42.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d789c38a0 con 0x561d78a6bb60 2023-10-11T16:34:42.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d789bcd30 con 0x561d78a708b0 2023-10-11T16:34:42.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4e927fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a70370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4e937fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a708b0 0x561d78a70d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4e937fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a708b0 0x561d78a70d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:39242/0 (socket says 172.21.15.111:39242) 2023-10-11T16:34:42.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4e927fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a70370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55676/0 (socket says 172.21.15.111:55676) 2023-10-11T16:34:42.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4e937fe700 1 -- 172.21.15.111:0/3698013213 learned_addr learned my addr 172.21.15.111:0/3698013213 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:42.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6bb60 0x561d78a6bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.421+0000 7f4e937fe700 1 -- 172.21.15.111:0/3698013213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6bb60 msgr2=0x561d78a6bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:42.429 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e937fe700 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6bb60 0x561d78a6bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e937fe700 1 -- 172.21.15.111:0/3698013213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 msgr2=0x561d78a70370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e937fe700 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a70370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e937fe700 1 -- 172.21.15.111:0/3698013213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d7883e630 con 0x561d78a708b0 2023-10-11T16:34:42.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e927fc700 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a70370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e937fe700 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a708b0 0x561d78a70d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c011600 tx=0x7f4e8c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d33b6870f5a16cd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e797fa700 1 -- 172.21.15.111:0/3698013213 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c00e040 con 0x561d78a708b0 2023-10-11T16:34:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e797fa700 1 -- 172.21.15.111:0/3698013213 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e8c00f040 con 0x561d78a708b0 2023-10-11T16:34:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.422+0000 7f4e797fa700 1 -- 172.21.15.111:0/3698013213 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c004760 con 0x561d78a708b0 2023-10-11T16:34:42.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/3698013213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a708b0 msgr2=0x561d78a70d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a708b0 0x561d78a70d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c011600 tx=0x7f4e8c009920 comp rx=0 tx=0).stop 2023-10-11T16:34:42.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/3698013213 shutdown_connections 2023-10-11T16:34:42.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a708b0 0x561d78a70d00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6bb60 0x561d78a6bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/3698013213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a70370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/3698013213 >> 172.21.15.111:0/3698013213 conn(0x561d78a61360 msgr2=0x561d78a66b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:42.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/3698013213 shutdown_connections 2023-10-11T16:34:42.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.423+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/3698013213 wait complete. 2023-10-11T16:34:42.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 1 -- start start 2023-10-11T16:34:42.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 1 Processor -- start 2023-10-11T16:34:42.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a780d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a752e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a758f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d7895e8e0 con 0x561d78a708b0 2023-10-11T16:34:42.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d7895e580 con 0x561d78a6c470 2023-10-11T16:34:42.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d78840920 con 0x561d78a6bb60 2023-10-11T16:34:42.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.424+0000 7f4eaccb1b80 10 client.0 _ll_register_callbacks cb 0x561d7897c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:42.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a758f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a758f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55692/0 (socket says 172.21.15.111:55692) 2023-10-11T16:34:42.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 -- 172.21.15.111:0/4201069017 learned_addr learned my addr 172.21.15.111:0/4201069017 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:42.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e927fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a780d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 -- 172.21.15.111:0/4201069017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 msgr2=0x561d78a780d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:42.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a780d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 -- 172.21.15.111:0/4201069017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 msgr2=0x561d78a752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:42.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a752e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 -- 172.21.15.111:0/4201069017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c0112b0 con 0x561d78a708b0 2023-10-11T16:34:42.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e927fc700 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a752e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:42.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e937fe700 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a758f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c006080 tx=0x7f4e8c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266266dcf3157d8e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:42.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.425+0000 7f4e76ff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:42.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4e76ff5700 1 -- 172.21.15.111:0/4201069017 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c00e030 con 0x561d78a708b0 2023-10-11T16:34:42.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4e76ff5700 1 -- 172.21.15.111:0/4201069017 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f4e8c004ce0 con 0x561d78a708b0 2023-10-11T16:34:42.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/4201069017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561d78a5eb20 con 0x561d78a708b0 2023-10-11T16:34:42.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4e76ff5700 1 -- 172.21.15.111:0/4201069017 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e8c004ea0 con 0x561d78a708b0 2023-10-11T16:34:42.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4eaccb1b80 10 client.15963 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:42.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/4201069017 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561d78983190 con 0x561d78a708b0 2023-10-11T16:34:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4e76ff5700 1 -- 172.21.15.111:0/4201069017 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561d78a5eb20 con 0x561d78a708b0 2023-10-11T16:34:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4e76ff5700 1 -- 172.21.15.111:0/4201069017 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e8c024020 con 0x561d78a708b0 2023-10-11T16:34:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4eaccb1b80 10 client.15963 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4eaccb1b80 10 client.15963 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:34:42.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4eaccb1b80 20 client.15963 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x561d78a857c0 0x561d78a87ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.426+0000 7f4eaccb1b80 4 client.15963 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.427+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/4201069017 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d7897f0a0 con 0x561d78a857c0 2023-10-11T16:34:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.427+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x561d78a857c0 0x561d78a87ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.427+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x561d78a857c0 0x561d78a87ba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7c9d475400f86259 server_cookie=478eedcad300ac08 in_seq=0 out_seq=0 2023-10-11T16:34:42.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.427+0000 7f4e76ff5700 10 client.15963 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:34:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4e76ff5700 1 -- 172.21.15.111:0/4201069017 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561d7897f0a0 con 0x561d78a857c0 2023-10-11T16:34:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4e76ff5700 10 client.15963 handle_command_reply: tid=0 2023-10-11T16:34:42.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 client.15963 shutdown 2023-10-11T16:34:42.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/4201069017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 msgr2=0x561d78a758f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a758f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c006080 tx=0x7f4e8c0040c0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/4201069017 shutdown_connections 2023-10-11T16:34:42.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x561d78a857c0 0x561d78a87ba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a758f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a780d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/4201069017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a752e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4e76ff5700 0 client.15963 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:34:42.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.428+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/4201069017 >> 172.21.15.111:0/4201069017 conn(0x561d78a61360 msgr2=0x561d78a71d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:42.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.429+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/4201069017 shutdown_connections 2023-10-11T16:34:42.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.429+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/4201069017 wait complete. 2023-10-11T16:34:42.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.429+0000 7f4eaccb1b80 20 client.15963 trim_cache size 0 max 16384 2023-10-11T16:34:42.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.442+0000 7f4eaccb1b80 1 Processor -- start 2023-10-11T16:34:42.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.442+0000 7f4eaccb1b80 1 -- start start 2023-10-11T16:34:42.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a77c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a6e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d78a6cbc0 con 0x561d78a708b0 2023-10-11T16:34:42.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d78a75ed0 con 0x561d78a6c470 2023-10-11T16:34:42.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d78a73860 con 0x561d78a6bb60 2023-10-11T16:34:42.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e937fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a6e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e937fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a6e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55696/0 (socket says 172.21.15.111:55696) 2023-10-11T16:34:42.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e92ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e937fe700 1 -- 172.21.15.111:0/1856755746 learned_addr learned my addr 172.21.15.111:0/1856755746 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:42.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e927fc700 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a77c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e92ffd700 1 -- 172.21.15.111:0/1856755746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 msgr2=0x561d78a77c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:42.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a77c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e92ffd700 1 -- 172.21.15.111:0/1856755746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 msgr2=0x561d78a6e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:42.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a6e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e92ffd700 1 -- 172.21.15.111:0/1856755746 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d78a74e50 con 0x561d78a6bb60 2023-10-11T16:34:42.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e927fc700 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a77c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:42.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.443+0000 7f4e937fe700 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a6e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:42.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.444+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7c9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400b580 tx=0x7f4e8401f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ef78888be906e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:42.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.444+0000 7f4e79ffb700 1 -- 172.21.15.111:0/1856755746 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8402a070 con 0x561d78a6bb60 2023-10-11T16:34:42.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.444+0000 7f4e79ffb700 1 -- 172.21.15.111:0/1856755746 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e8402b040 con 0x561d78a6bb60 2023-10-11T16:34:42.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.444+0000 7f4e79ffb700 1 -- 172.21.15.111:0/1856755746 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84005510 con 0x561d78a6bb60 2023-10-11T16:34:42.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/1856755746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 msgr2=0x561d78a7c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7c9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400b580 tx=0x7f4e8401f000 comp rx=0 tx=0).stop 2023-10-11T16:34:42.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/1856755746 shutdown_connections 2023-10-11T16:34:42.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a708b0 0x561d78a6e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7c9a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/1856755746 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a6c470 0x561d78a77c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/1856755746 >> 172.21.15.111:0/1856755746 conn(0x561d78a61360 msgr2=0x561d78a72c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:42.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/1856755746 shutdown_connections 2023-10-11T16:34:42.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/1856755746 wait complete. 2023-10-11T16:34:42.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.445+0000 7f4eaccb1b80 1 -- start start 2023-10-11T16:34:42.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4eaccb1b80 1 Processor -- start 2023-10-11T16:34:42.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a7aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4eaccb1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a708b0 0x561d78a7b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d78ad8650 con 0x561d78a6c470 2023-10-11T16:34:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d78ad8790 con 0x561d78a708b0 2023-10-11T16:34:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4eaccb1b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d78ad8900 con 0x561d78a6bb60 2023-10-11T16:34:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4eaccb1b80 10 client.0 _ll_register_callbacks cb 0x7f4e88004e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4e927fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a7aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4e927fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a7aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55702/0 (socket says 172.21.15.111:55702) 2023-10-11T16:34:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4e927fc700 1 -- 172.21.15.111:0/295828881 learned_addr learned my addr 172.21.15.111:0/295828881 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:42.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4e937fe700 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a708b0 0x561d78a7b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.446+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e927fc700 1 -- 172.21.15.111:0/295828881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 msgr2=0x561d78a7a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:42.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e927fc700 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e927fc700 1 -- 172.21.15.111:0/295828881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a708b0 msgr2=0x561d78a7b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:42.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e927fc700 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a708b0 0x561d78a7b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e927fc700 1 -- 172.21.15.111:0/295828881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d78983190 con 0x561d78a6c470 2023-10-11T16:34:42.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e927fc700 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a7aed0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4e88004f10 tx=0x7f4e88005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd87247c30bef05 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:42.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e76ff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:42.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e76ff5700 1 -- 172.21.15.111:0/295828881 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880b6070 con 0x561d78a6c470 2023-10-11T16:34:42.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e76ff5700 1 -- 172.21.15.111:0/295828881 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f4e880a24f0 con 0x561d78a6c470 2023-10-11T16:34:42.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/295828881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e8c00f040 con 0x561d78a6c470 2023-10-11T16:34:42.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4e76ff5700 1 -- 172.21.15.111:0/295828881 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e8800cda0 con 0x561d78a6c470 2023-10-11T16:34:42.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4eaccb1b80 10 client.15969 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:42.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.447+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/295828881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f4e9410d9b0 con 0x561d78a6c470 2023-10-11T16:34:42.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4e76ff5700 1 -- 172.21.15.111:0/295828881 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e8c00f040 con 0x561d78a6c470 2023-10-11T16:34:42.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4e76ff5700 1 -- 172.21.15.111:0/295828881 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e880b82b0 con 0x561d78a6c470 2023-10-11T16:34:42.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4eaccb1b80 10 client.15969 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:42.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4eaccb1b80 10 client.15969 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:34:42.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4eaccb1b80 20 client.15969 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:42.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x561d78ada060 0x561d78adc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:42.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4eaccb1b80 4 client.15969 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:34:42.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/295828881 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561d7898b660 con 0x561d78ada060 2023-10-11T16:34:42.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.448+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x561d78ada060 0x561d78adc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:42.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4e76ff5700 10 client.15969 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:34:42.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4e92ffd700 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x561d78ada060 0x561d78adc470 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45acdeda9a0388ba server_cookie=932e59a6c348c1ca in_seq=0 out_seq=0 2023-10-11T16:34:42.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4e76ff5700 1 -- 172.21.15.111:0/295828881 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561d7898b660 con 0x561d78ada060 2023-10-11T16:34:42.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4e76ff5700 10 client.15969 handle_command_reply: tid=0 2023-10-11T16:34:42.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 client.15969 shutdown 2023-10-11T16:34:42.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/295828881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 msgr2=0x561d78a7aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a7aed0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4e88004f10 tx=0x7f4e88005700 comp rx=0 tx=0).stop 2023-10-11T16:34:42.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/295828881 shutdown_connections 2023-10-11T16:34:42.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x561d78ada060 0x561d78adc470 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561d78a708b0 0x561d78a7b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x561d78a6bb60 0x561d78a7a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4e76ff5700 0 client.15969 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:34:42.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 --2- 172.21.15.111:0/295828881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x561d78a6c470 0x561d78a7aed0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/295828881 >> 172.21.15.111:0/295828881 conn(0x561d78a61360 msgr2=0x561d78a6e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:42.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/295828881 shutdown_connections 2023-10-11T16:34:42.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.449+0000 7f4eaccb1b80 1 -- 172.21.15.111:0/295828881 wait complete. 2023-10-11T16:34:42.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4eaccb1b80 20 client.15969 trim_cache size 0 max 16384 2023-10-11T16:34:42.467 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:34:42.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 -- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4e7c0a12c0 msgr2=0x7f4e7c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4e7c0a12c0 0x7f4e7c0a3770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009990 tx=0x7f4e8401a040 comp rx=0 tx=0).stop 2023-10-11T16:34:42.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 -- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94101660 msgr2=0x7f4e94073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:42.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94101660 0x7f4e94073330 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800e910 tx=0x7f4e8800ec20 comp rx=0 tx=0).stop 2023-10-11T16:34:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 -- 172.21.15.111:0/2124659344 shutdown_connections 2023-10-11T16:34:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4e7c0a12c0 0x7f4e7c0a3770 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e94105620 0x7f4e941a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e94100cc0 0x7f4e94072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 --2- 172.21.15.111:0/2124659344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e94101660 0x7f4e94073330 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:42.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 -- 172.21.15.111:0/2124659344 >> 172.21.15.111:0/2124659344 conn(0x7f4e94077560 msgr2=0x7f4e940feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:42.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 -- 172.21.15.111:0/2124659344 shutdown_connections 2023-10-11T16:34:42.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:42.450+0000 7f4e993ff700 1 -- 172.21.15.111:0/2124659344 wait complete. 2023-10-11T16:34:42.499 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:34:42.809 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:42.810 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:42.874 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:42.874 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:42.883 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:42.883 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:42.884 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 92 -lt 100 ']' 2023-10-11T16:34:42.884 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 92 + 1 2023-10-11T16:34:42.928 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=93 2023-10-11T16:34:42.928 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:42.939 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:42.939 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:42.942 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:42.942 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:42.943 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:43.259 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:43.260 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:43.339 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:43.339 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:43.352 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:43.352 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:43.353 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 93 -lt 100 ']' 2023-10-11T16:34:43.353 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 93 + 1 2023-10-11T16:34:43.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:43 smithi111 ceph-mon[98527]: pgmap v65: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-10-11T16:34:43.386 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=94 2023-10-11T16:34:43.386 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:43.442 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:43.442 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:43.444 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:43.445 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:43.445 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:43.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:43 smithi130 ceph-mon[131859]: pgmap v65: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-10-11T16:34:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:43 smithi150 ceph-mon[132217]: pgmap v65: 129 pgs: 129 active+clean; 2.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-10-11T16:34:43.943 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:43.943 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:44.084 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:44.085 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:44.118 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:44.118 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:44.119 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 94 -lt 100 ']' 2023-10-11T16:34:44.119 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 94 + 1 2023-10-11T16:34:44.227 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=95 2023-10-11T16:34:44.228 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:44.265 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:44.265 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:44.267 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:44.268 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:44.268 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:45.235 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:45.235 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:45.508 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:45.508 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:45.527 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:45.528 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:45.528 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 95 -lt 100 ']' 2023-10-11T16:34:45.528 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 95 + 1 2023-10-11T16:34:45.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:45 smithi130 ceph-mon[131859]: pgmap v66: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 11 op/s 2023-10-11T16:34:45.596 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=96 2023-10-11T16:34:45.597 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:45 smithi111 ceph-mon[98527]: pgmap v66: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 11 op/s 2023-10-11T16:34:45.679 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:45.679 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:45.682 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:45.683 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:45.683 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:45 smithi150 ceph-mon[132217]: pgmap v66: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 11 op/s 2023-10-11T16:34:45.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/839474026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e81063f0 msgr2=0x7fb3e81067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/839474026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e81063f0 0x7fb3e81067c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009920 tx=0x7fb3d8009c30 comp rx=0 tx=0).stop 2023-10-11T16:34:45.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/839474026 shutdown_connections 2023-10-11T16:34:45.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/839474026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8101400 0x7fb3e810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/839474026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e81063f0 0x7fb3e81067c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/839474026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3e8100a50 0x7fb3e8100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/839474026 >> 172.21.15.111:0/839474026 conn(0x7fb3e8077590 msgr2=0x7fb3e8077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:45.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/839474026 shutdown_connections 2023-10-11T16:34:45.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.773+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/839474026 wait complete. 2023-10-11T16:34:45.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.774+0000 7fb3ee3ff700 1 Processor -- start 2023-10-11T16:34:45.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.774+0000 7fb3ee3ff700 1 -- start start 2023-10-11T16:34:45.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3ee3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e8100a50 0x7fb3e81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3ee3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8101400 0x7fb3e8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3ee3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3e81063f0 0x7fb3e819bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3ee3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e819c590 con 0x7fb3e8101400 2023-10-11T16:34:45.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3ee3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e819c700 con 0x7fb3e8100a50 2023-10-11T16:34:45.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3ee3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e819c8c0 con 0x7fb3e81063f0 2023-10-11T16:34:45.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3e77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8101400 0x7fb3e8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3e77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8101400 0x7fb3e8198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35906/0 (socket says 172.21.15.111:35906) 2023-10-11T16:34:45.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3e77fe700 1 -- 172.21.15.111:0/144359258 learned_addr learned my addr 172.21.15.111:0/144359258 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:45.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e8100a50 0x7fb3e81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.775+0000 7fb3ec99c700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3e81063f0 0x7fb3e819bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e7fff700 1 -- 172.21.15.111:0/144359258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3e81063f0 msgr2=0x7fb3e819bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3e81063f0 0x7fb3e819bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e7fff700 1 -- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8101400 msgr2=0x7fb3e8198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8101400 0x7fb3e8198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e7fff700 1 -- 172.21.15.111:0/144359258 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8009520 con 0x7fb3e8100a50 2023-10-11T16:34:45.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8101400 0x7fb3e8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:45.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e8100a50 0x7fb3e81986f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009230 tx=0x7fb3d800fa20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:45.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e57fa700 1 -- 172.21.15.111:0/144359258 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d801c070 con 0x7fb3e8100a50 2023-10-11T16:34:45.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e57fa700 1 -- 172.21.15.111:0/144359258 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3d800baa0 con 0x7fb3e8100a50 2023-10-11T16:34:45.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3e57fa700 1 -- 172.21.15.111:0/144359258 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80038e0 con 0x7fb3e8100a50 2023-10-11T16:34:45.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.776+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/144359258 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e81103d0 con 0x7fb3e8100a50 2023-10-11T16:34:45.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.777+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/144359258 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e8110240 con 0x7fb3e8100a50 2023-10-11T16:34:45.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.780+0000 7fb3e57fa700 1 -- 172.21.15.111:0/144359258 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fb3d8020030 con 0x7fb3e8100a50 2023-10-11T16:34:45.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.781+0000 7fb3e57fa700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb3d00a1340 0x7fb3d00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.781+0000 7fb3e57fa700 1 -- 172.21.15.111:0/144359258 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb3d80cdf30 con 0x7fb3e8100a50 2023-10-11T16:34:45.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.781+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb3d00a1340 0x7fb3d00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.782+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb3d00a1340 0x7fb3d00a37f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8199cb0 tx=0x7fb3dc009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:45.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.862+0000 7fb401d2bb80 1 Processor -- start 2023-10-11T16:34:45.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.862+0000 7fb401d2bb80 1 -- start start 2023-10-11T16:34:45.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b58f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 0x558884b5d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b5dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884a5e060 con 0x558884b5d8b0 2023-10-11T16:34:45.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884ab08a0 con 0x558884b59470 2023-10-11T16:34:45.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884aa9d30 con 0x558884b58b60 2023-10-11T16:34:45.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b58f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b58f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55640/0 (socket says 172.21.15.111:55640) 2023-10-11T16:34:45.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e7fff700 1 -- 172.21.15.111:0/2880650033 learned_addr learned my addr 172.21.15.111:0/2880650033 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:45.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3ec99c700 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b5dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 0x558884b5d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e7fff700 1 -- 172.21.15.111:0/2880650033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 msgr2=0x558884b5d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 0x558884b5d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e7fff700 1 -- 172.21.15.111:0/2880650033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 msgr2=0x558884b5dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:45.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b5dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3e7fff700 1 -- 172.21.15.111:0/2880650033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55888492b630 con 0x558884b58b60 2023-10-11T16:34:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.863+0000 7fb3ec99c700 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b5dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.864+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b58f30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80a2b10 tx=0x7fb3d809a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7410e01a2fb89af server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.864+0000 7fb3ce7fc700 1 -- 172.21.15.111:0/2880650033 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80a3040 con 0x558884b58b60 2023-10-11T16:34:45.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.864+0000 7fb3ce7fc700 1 -- 172.21.15.111:0/2880650033 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3d80cb4f0 con 0x558884b58b60 2023-10-11T16:34:45.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.864+0000 7fb3ce7fc700 1 -- 172.21.15.111:0/2880650033 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d809a170 con 0x558884b58b60 2023-10-11T16:34:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2880650033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 msgr2=0x558884b58f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b58f30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80a2b10 tx=0x7fb3d809a8f0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2880650033 shutdown_connections 2023-10-11T16:34:45.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b5dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b58f30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2880650033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 0x558884b5d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2880650033 >> 172.21.15.111:0/2880650033 conn(0x558884b4e360 msgr2=0x558884b53b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:45.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2880650033 shutdown_connections 2023-10-11T16:34:45.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2880650033 wait complete. 2023-10-11T16:34:45.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.865+0000 7fb401d2bb80 1 -- start start 2023-10-11T16:34:45.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.866+0000 7fb401d2bb80 1 Processor -- start 2023-10-11T16:34:45.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.866+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b64fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.866+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b59470 0x558884b621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.866+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b5d8b0 0x558884b62750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.866+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884a4b8e0 con 0x558884b59470 2023-10-11T16:34:45.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.866+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884a4b580 con 0x558884b58b60 2023-10-11T16:34:45.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.866+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55888492d920 con 0x558884b5d8b0 2023-10-11T16:34:45.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.866+0000 7fb401d2bb80 10 client.0 _ll_register_callbacks cb 0x558884a69780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:45.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b59470 0x558884b621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b59470 0x558884b621b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35920/0 (socket says 172.21.15.111:35920) 2023-10-11T16:34:45.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 -- 172.21.15.111:0/2681851101 learned_addr learned my addr 172.21.15.111:0/2681851101 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:45.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b64fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3ec99c700 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b5d8b0 0x558884b62750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 -- 172.21.15.111:0/2681851101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b5d8b0 msgr2=0x558884b62750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:45.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b5d8b0 0x558884b62750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 -- 172.21.15.111:0/2681851101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 msgr2=0x558884b64fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:45.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b64fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 -- 172.21.15.111:0/2681851101 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e8110240 con 0x558884b59470 2023-10-11T16:34:45.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b64fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:45.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b59470 0x558884b621b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00ed30 tx=0x7fb3dc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bfa06fe6801065 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:45.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.867+0000 7fb3c77fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:45.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.868+0000 7fb3c77fe700 1 -- 172.21.15.111:0/2681851101 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc01ed90 con 0x558884b59470 2023-10-11T16:34:45.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.868+0000 7fb3c77fe700 1 -- 172.21.15.111:0/2681851101 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb3dc0263f0 con 0x558884b59470 2023-10-11T16:34:45.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.868+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2681851101 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558884b4bb20 con 0x558884b59470 2023-10-11T16:34:45.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.868+0000 7fb3c77fe700 1 -- 172.21.15.111:0/2681851101 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3dc0265b0 con 0x558884b59470 2023-10-11T16:34:45.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.868+0000 7fb401d2bb80 10 client.15981 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:45.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.868+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2681851101 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558884a70190 con 0x558884b59470 2023-10-11T16:34:45.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.868+0000 7fb3c77fe700 1 -- 172.21.15.111:0/2681851101 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558884b4bb20 con 0x558884b59470 2023-10-11T16:34:45.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.868+0000 7fb3c77fe700 1 -- 172.21.15.111:0/2681851101 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb3dc01c7b0 con 0x558884b59470 2023-10-11T16:34:45.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.869+0000 7fb401d2bb80 10 client.15981 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:45.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.869+0000 7fb401d2bb80 10 client.15981 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:34:45.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.869+0000 7fb401d2bb80 20 client.15981 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:45.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.869+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558884b72650 0x558884b74a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.869+0000 7fb401d2bb80 4 client.15981 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:45.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.869+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2681851101 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558884a6c0a0 con 0x558884b72650 2023-10-11T16:34:45.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.869+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558884b72650 0x558884b74a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.870+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558884b72650 0x558884b74a60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e383d343393a15e0 server_cookie=3891fbd60e120b2f in_seq=0 out_seq=0 2023-10-11T16:34:45.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.870+0000 7fb3c77fe700 10 client.15981 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:34:45.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.870+0000 7fb3c77fe700 1 -- 172.21.15.111:0/2681851101 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558884a6c0a0 con 0x558884b72650 2023-10-11T16:34:45.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.870+0000 7fb3c77fe700 10 client.15981 handle_command_reply: tid=0 2023-10-11T16:34:45.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.870+0000 7fb401d2bb80 1 client.15981 shutdown 2023-10-11T16:34:45.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2681851101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b59470 msgr2=0x558884b621b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b59470 0x558884b621b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00ed30 tx=0x7fb3dc0047c0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2681851101 shutdown_connections 2023-10-11T16:34:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558884b72650 0x558884b74a60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b5d8b0 0x558884b62750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b64fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb3c77fe700 0 client.15981 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:34:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/2681851101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b59470 0x558884b621b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2681851101 >> 172.21.15.111:0/2681851101 conn(0x558884b4e360 msgr2=0x558884b5ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2681851101 shutdown_connections 2023-10-11T16:34:45.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 1 -- 172.21.15.111:0/2681851101 wait complete. 2023-10-11T16:34:45.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.871+0000 7fb401d2bb80 20 client.15981 trim_cache size 0 max 16384 2023-10-11T16:34:45.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.885+0000 7fb401d2bb80 1 Processor -- start 2023-10-11T16:34:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.885+0000 7fb401d2bb80 1 -- start start 2023-10-11T16:34:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b53ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 0x558884b697b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b610c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884b54710 con 0x558884b5d8b0 2023-10-11T16:34:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884b632f0 con 0x558884b59470 2023-10-11T16:34:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884b6ef70 con 0x558884b58b60 2023-10-11T16:34:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b53ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3ec99c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b53ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55658/0 (socket says 172.21.15.111:55658) 2023-10-11T16:34:45.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e7fff700 1 -- 172.21.15.111:0/3638173123 learned_addr learned my addr 172.21.15.111:0/3638173123 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:45.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/3638173123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 0x558884b697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e7fff700 1 -- 172.21.15.111:0/3638173123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 msgr2=0x558884b697b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:45.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/3638173123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 0x558884b697b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e7fff700 1 -- 172.21.15.111:0/3638173123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 msgr2=0x558884b610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:45.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/3638173123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.886+0000 7fb3e7fff700 1 -- 172.21.15.111:0/3638173123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558884a6fee0 con 0x558884b58b60 2023-10-11T16:34:45.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.887+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/3638173123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b53ba0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb3d809ac10 tx=0x7fb3d80a0040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d46cd2e39762caa5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:45.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.887+0000 7fb3ceffd700 1 -- 172.21.15.111:0/3638173123 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80c9070 con 0x558884b58b60 2023-10-11T16:34:45.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.887+0000 7fb3ceffd700 1 -- 172.21.15.111:0/3638173123 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3d80990e0 con 0x558884b58b60 2023-10-11T16:34:45.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.887+0000 7fb3ceffd700 1 -- 172.21.15.111:0/3638173123 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d809a390 con 0x558884b58b60 2023-10-11T16:34:45.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.887+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3638173123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 msgr2=0x558884b53ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.887+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3638173123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b53ba0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb3d809ac10 tx=0x7fb3d80a0040 comp rx=0 tx=0).stop 2023-10-11T16:34:45.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3638173123 shutdown_connections 2023-10-11T16:34:45.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3638173123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3638173123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b58b60 0x558884b53ba0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3638173123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b59470 0x558884b697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3638173123 >> 172.21.15.111:0/3638173123 conn(0x558884b4e360 msgr2=0x558884b609b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:45.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3638173123 shutdown_connections 2023-10-11T16:34:45.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3638173123 wait complete. 2023-10-11T16:34:45.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 -- start start 2023-10-11T16:34:45.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.888+0000 7fb401d2bb80 1 Processor -- start 2023-10-11T16:34:45.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b65730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b59470 0x558884b65c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb401d2bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b661b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884bc55e0 con 0x558884b5d8b0 2023-10-11T16:34:45.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884bc5720 con 0x558884b58b60 2023-10-11T16:34:45.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb401d2bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558884bc58e0 con 0x558884b59470 2023-10-11T16:34:45.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb401d2bb80 10 client.0 _ll_register_callbacks cb 0x7fb3dc00e8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:45.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3ec99c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3e7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b65730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3ec99c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b661b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35942/0 (socket says 172.21.15.111:35942) 2023-10-11T16:34:45.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3ec99c700 1 -- 172.21.15.111:0/3381658141 learned_addr learned my addr 172.21.15.111:0/3381658141 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:45.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b59470 0x558884b65c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3e7fff700 1 -- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b59470 msgr2=0x558884b65c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:45.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b59470 0x558884b65c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3e7fff700 1 -- 172.21.15.111:0/3381658141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 msgr2=0x558884b661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:45.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b661b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3e7fff700 1 -- 172.21.15.111:0/3381658141 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55888492b630 con 0x558884b58b60 2023-10-11T16:34:45.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3ec99c700 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:45.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.889+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b59470 0x558884b65c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:45.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.890+0000 7fb3e7fff700 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b65730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009920 tx=0x7fb3d809a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf06708f23791b2e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:45.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.890+0000 7fb3c77fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:34:45.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.890+0000 7fb3c77fe700 1 -- 172.21.15.111:0/3381658141 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80c9030 con 0x558884b58b60 2023-10-11T16:34:45.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.890+0000 7fb3c77fe700 1 -- 172.21.15.111:0/3381658141 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb3d807d020 con 0x558884b58b60 2023-10-11T16:34:45.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.890+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3381658141 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3d80cb4f0 con 0x558884b58b60 2023-10-11T16:34:45.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.890+0000 7fb3c77fe700 1 -- 172.21.15.111:0/3381658141 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3d80b3250 con 0x558884b58b60 2023-10-11T16:34:45.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.890+0000 7fb401d2bb80 10 client.26030 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:45.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.890+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3381658141 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5588848c0c60 con 0x558884b58b60 2023-10-11T16:34:45.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb3c77fe700 1 -- 172.21.15.111:0/3381658141 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3d80cb4f0 con 0x558884b58b60 2023-10-11T16:34:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb3c77fe700 1 -- 172.21.15.111:0/3381658141 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb3d80a2bb0 con 0x558884b58b60 2023-10-11T16:34:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb401d2bb80 10 client.26030 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb401d2bb80 10 client.26030 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:34:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb401d2bb80 20 client.26030 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:45.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558884bc6b20 0x558884bc8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:45.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb401d2bb80 4 client.26030 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:34:45.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3381658141 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558884a78660 con 0x558884bc6b20 2023-10-11T16:34:45.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.891+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558884bc6b20 0x558884bc8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:45.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558884bc6b20 0x558884bc8f00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d9273c2dd939589 server_cookie=5a62973813e2c17f in_seq=0 out_seq=0 2023-10-11T16:34:45.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb3c77fe700 10 client.26030 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:34:45.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb3c77fe700 1 -- 172.21.15.111:0/3381658141 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558884a78660 con 0x558884bc6b20 2023-10-11T16:34:45.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb3c77fe700 10 client.26030 handle_command_reply: tid=0 2023-10-11T16:34:45.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 client.26030 shutdown 2023-10-11T16:34:45.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3381658141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 msgr2=0x558884b65730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b65730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009920 tx=0x7fb3d809a8f0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3381658141 shutdown_connections 2023-10-11T16:34:45.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x558884bc6b20 0x558884bc8f00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558884b5d8b0 0x558884b661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558884b58b60 0x558884b65730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 --2- 172.21.15.111:0/3381658141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558884b59470 0x558884b65c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb3c77fe700 0 client.26030 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:34:45.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3381658141 >> 172.21.15.111:0/3381658141 conn(0x558884b4e360 msgr2=0x558884b5c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:45.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3381658141 shutdown_connections 2023-10-11T16:34:45.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.892+0000 7fb401d2bb80 1 -- 172.21.15.111:0/3381658141 wait complete. 2023-10-11T16:34:45.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb401d2bb80 20 client.26030 trim_cache size 0 max 16384 2023-10-11T16:34:45.911 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:34:45.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb3d00a1340 msgr2=0x7fb3d00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb3d00a1340 0x7fb3d00a37f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8199cb0 tx=0x7fb3dc009260 comp rx=0 tx=0).stop 2023-10-11T16:34:45.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/144359258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e8100a50 msgr2=0x7fb3e81986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:45.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e8100a50 0x7fb3e81986f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009230 tx=0x7fb3d800fa20 comp rx=0 tx=0).stop 2023-10-11T16:34:45.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/144359258 shutdown_connections 2023-10-11T16:34:45.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb3d00a1340 0x7fb3d00a37f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3e81063f0 0x7fb3e819bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3e8100a50 0x7fb3e81986f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 --2- 172.21.15.111:0/144359258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e8101400 0x7fb3e8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:45.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/144359258 >> 172.21.15.111:0/144359258 conn(0x7fb3e8077590 msgr2=0x7fb3e80ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:45.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/144359258 shutdown_connections 2023-10-11T16:34:45.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:45.893+0000 7fb3ee3ff700 1 -- 172.21.15.111:0/144359258 wait complete. 2023-10-11T16:34:45.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:34:46.392 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:46.393 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:46.457 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:46.457 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:46.477 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:46.477 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:46.478 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 96 -lt 100 ']' 2023-10-11T16:34:46.478 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 96 + 1 2023-10-11T16:34:46.502 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=97 2023-10-11T16:34:46.502 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:46.514 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:46.514 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:46.516 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:46.517 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:46.517 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:46.892 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:46.893 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:46.973 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:46.973 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:46.986 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:46.987 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:46.987 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 97 -lt 100 ']' 2023-10-11T16:34:46.987 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 97 + 1 2023-10-11T16:34:47.038 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=98 2023-10-11T16:34:47.038 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:47.067 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:47.067 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:47.069 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:47.070 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:47.070 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:47.491 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:47 smithi111 ceph-mon[98527]: pgmap v67: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-10-11T16:34:47.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:47 smithi130 ceph-mon[131859]: pgmap v67: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-10-11T16:34:47.633 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:47.634 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:47 smithi150 ceph-mon[132217]: pgmap v67: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-10-11T16:34:47.809 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:47.809 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:47.826 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:47.826 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:47.826 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 98 -lt 100 ']' 2023-10-11T16:34:47.827 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 98 + 1 2023-10-11T16:34:47.885 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=99 2023-10-11T16:34:47.885 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:47.964 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:47.965 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:47.967 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:47.968 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:47.969 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:48.675 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:48.676 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:48.875 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:48.876 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:48.893 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:48.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:48.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 99 -lt 100 ']' 2023-10-11T16:34:48.894 INFO:tasks.workunit.client.0.smithi111.stderr:++ expr 99 + 1 2023-10-11T16:34:48.952 INFO:tasks.workunit.client.0.smithi111.stderr:+ c=100 2023-10-11T16:34:48.952 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:49.007 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:49.008 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:49.010 INFO:tasks.workunit.client.0.smithi111.stderr:+ c1=2 2023-10-11T16:34:49.011 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo 3 2023-10-11T16:34:49.011 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-11T16:34:49.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 -- 172.21.15.111:0/4137851525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708101660 msgr2=0x7f0708104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 --2- 172.21.15.111:0/4137851525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708101660 0x7f0708104ef0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0099c0 tx=0x7f06fc009cd0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 -- 172.21.15.111:0/4137851525 shutdown_connections 2023-10-11T16:34:49.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 --2- 172.21.15.111:0/4137851525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0708105620 0x7f070810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 --2- 172.21.15.111:0/4137851525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708100cc0 0x7f0708101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 --2- 172.21.15.111:0/4137851525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708101660 0x7f0708104ef0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 -- 172.21.15.111:0/4137851525 >> 172.21.15.111:0/4137851525 conn(0x7f0708077560 msgr2=0x7f0708077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:49.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 -- 172.21.15.111:0/4137851525 shutdown_connections 2023-10-11T16:34:49.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.389+0000 7f070d3ff700 1 -- 172.21.15.111:0/4137851525 wait complete. 2023-10-11T16:34:49.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.390+0000 7f070d3ff700 1 Processor -- start 2023-10-11T16:34:49.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.390+0000 7f070d3ff700 1 -- start start 2023-10-11T16:34:49.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f070d3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708100cc0 0x7f07081986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f070d3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0708101660 0x7f0708198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f070d3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708105620 0x7f070819bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f070d3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070819c5a0 con 0x7f0708100cc0 2023-10-11T16:34:49.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f070d3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070819c710 con 0x7f0708101660 2023-10-11T16:34:49.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f070d3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070819c8d0 con 0x7f0708105620 2023-10-11T16:34:49.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708100cc0 0x7f07081986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708100cc0 0x7f07081986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35970/0 (socket says 172.21.15.111:35970) 2023-10-11T16:34:49.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f0706ffd700 1 -- 172.21.15.111:0/1402627725 learned_addr learned my addr 172.21.15.111:0/1402627725 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:49.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f07067fc700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0708101660 0x7f0708198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f07077fe700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708105620 0x7f070819bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f0706ffd700 1 -- 172.21.15.111:0/1402627725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708105620 msgr2=0x7f070819bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f0706ffd700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708105620 0x7f070819bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f0706ffd700 1 -- 172.21.15.111:0/1402627725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0708101660 msgr2=0x7f0708198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.391+0000 7f0706ffd700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0708101660 0x7f0708198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.392+0000 7f0706ffd700 1 -- 172.21.15.111:0/1402627725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8009520 con 0x7f0708100cc0 2023-10-11T16:34:49.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.392+0000 7f0706ffd700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708100cc0 0x7f07081986b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f06f800eb30 tx=0x7f06f800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:49.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.392+0000 7f06effff700 1 -- 172.21.15.111:0/1402627725 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8009880 con 0x7f0708100cc0 2023-10-11T16:34:49.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.392+0000 7f06effff700 1 -- 172.21.15.111:0/1402627725 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06f80044e0 con 0x7f0708100cc0 2023-10-11T16:34:49.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.392+0000 7f070d3ff700 1 -- 172.21.15.111:0/1402627725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06fc0095f0 con 0x7f0708100cc0 2023-10-11T16:34:49.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.392+0000 7f06effff700 1 -- 172.21.15.111:0/1402627725 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8003e90 con 0x7f0708100cc0 2023-10-11T16:34:49.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.392+0000 7f070d3ff700 1 -- 172.21.15.111:0/1402627725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0708110370 con 0x7f0708100cc0 2023-10-11T16:34:49.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.395+0000 7f06effff700 1 -- 172.21.15.111:0/1402627725 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f06f8004000 con 0x7f0708100cc0 2023-10-11T16:34:49.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.397+0000 7f06effff700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f06f00a1340 0x7f06f00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.397+0000 7f06effff700 1 -- 172.21.15.111:0/1402627725 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f06f8014070 con 0x7f0708100cc0 2023-10-11T16:34:49.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.397+0000 7f07067fc700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f06f00a1340 0x7f06f00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.398+0000 7f07067fc700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f06f00a1340 0x7f06f00a37f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00b580 tx=0x7f06fc000bc0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.494+0000 7f0720dc7b80 1 Processor -- start 2023-10-11T16:34:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.494+0000 7f0720dc7b80 1 -- start start 2023-10-11T16:34:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bc3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 0x565463bc8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc88b0 0x565463bc8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463ac9060 con 0x565463bc3b60 2023-10-11T16:34:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463b1b8a0 con 0x565463bc4470 2023-10-11T16:34:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463b14d30 con 0x565463bc88b0 2023-10-11T16:34:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bc3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bc3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35984/0 (socket says 172.21.15.111:35984) 2023-10-11T16:34:49.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0706ffd700 1 -- 172.21.15.111:0/2316050418 learned_addr learned my addr 172.21.15.111:0/2316050418 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:49.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f07077fe700 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc88b0 0x565463bc8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f07067fc700 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 0x565463bc8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0706ffd700 1 -- 172.21.15.111:0/2316050418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc88b0 msgr2=0x565463bc8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:49.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0706ffd700 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc88b0 0x565463bc8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0706ffd700 1 -- 172.21.15.111:0/2316050418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 msgr2=0x565463bc8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0706ffd700 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 0x565463bc8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.495+0000 7f0706ffd700 1 -- 172.21.15.111:0/2316050418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565463996630 con 0x565463bc3b60 2023-10-11T16:34:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.496+0000 7f0706ffd700 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bc3f30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f06f8096c90 tx=0x7f06f809a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa79af110bc509f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.496+0000 7f06ed7fa700 1 -- 172.21.15.111:0/2316050418 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f800cc90 con 0x565463bc3b60 2023-10-11T16:34:49.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.496+0000 7f06ed7fa700 1 -- 172.21.15.111:0/2316050418 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06f809a220 con 0x565463bc3b60 2023-10-11T16:34:49.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.496+0000 7f06ed7fa700 1 -- 172.21.15.111:0/2316050418 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f80cd110 con 0x565463bc3b60 2023-10-11T16:34:49.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2316050418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 msgr2=0x565463bc3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bc3f30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f06f8096c90 tx=0x7f06f809a100 comp rx=0 tx=0).stop 2023-10-11T16:34:49.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2316050418 shutdown_connections 2023-10-11T16:34:49.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc88b0 0x565463bc8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bc3f30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2316050418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 0x565463bc8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2316050418 >> 172.21.15.111:0/2316050418 conn(0x565463bb9360 msgr2=0x565463bbeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2316050418 shutdown_connections 2023-10-11T16:34:49.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.497+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2316050418 wait complete. 2023-10-11T16:34:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 1 -- start start 2023-10-11T16:34:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 1 Processor -- start 2023-10-11T16:34:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc3b60 0x565463bd0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 0x565463bcd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463bcd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463ab68e0 con 0x565463bc88b0 2023-10-11T16:34:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463ab6580 con 0x565463bc3b60 2023-10-11T16:34:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463998920 con 0x565463bc4470 2023-10-11T16:34:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.498+0000 7f0720dc7b80 10 client.0 _ll_register_callbacks cb 0x565463ad4780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463bcd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc3b60 0x565463bd0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463bcd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35996/0 (socket says 172.21.15.111:35996) 2023-10-11T16:34:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 -- 172.21.15.111:0/3724503038 learned_addr learned my addr 172.21.15.111:0/3724503038 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07067fc700 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 0x565463bcd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 -- 172.21.15.111:0/3724503038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 msgr2=0x565463bcd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 0x565463bcd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 -- 172.21.15.111:0/3724503038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc3b60 msgr2=0x565463bd0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc3b60 0x565463bd0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 -- 172.21.15.111:0/3724503038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0708110370 con 0x565463bc88b0 2023-10-11T16:34:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f0706ffd700 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc3b60 0x565463bd0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:49.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f07077fe700 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463bcd980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0700007ed0 tx=0x7f0700007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7244dd884546a06a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:49.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.499+0000 7f06eaff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:49.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f06eaff5700 1 -- 172.21.15.111:0/3724503038 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070000e070 con 0x565463bc88b0 2023-10-11T16:34:49.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 1 -- 172.21.15.111:0/3724503038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565463bb6b20 con 0x565463bc88b0 2023-10-11T16:34:49.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f06eaff5700 1 -- 172.21.15.111:0/3724503038 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f07000136e0 con 0x565463bc88b0 2023-10-11T16:34:49.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f06eaff5700 1 -- 172.21.15.111:0/3724503038 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07000138a0 con 0x565463bc88b0 2023-10-11T16:34:49.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 10 client.15999 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:49.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 1 -- 172.21.15.111:0/3724503038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565463adb190 con 0x565463bc88b0 2023-10-11T16:34:49.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f06eaff5700 1 -- 172.21.15.111:0/3724503038 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565463bb6b20 con 0x565463bc88b0 2023-10-11T16:34:49.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f06eaff5700 1 -- 172.21.15.111:0/3724503038 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0700016cb0 con 0x565463bc88b0 2023-10-11T16:34:49.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 10 client.15999 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:49.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:49.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 20 client.15999 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:49.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565463bdd910 0x565463bdfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 4 client.15999 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:49.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.500+0000 7f0720dc7b80 1 -- 172.21.15.111:0/3724503038 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565463ad70a0 con 0x565463bdd910 2023-10-11T16:34:49.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.501+0000 7f0706ffd700 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565463bdd910 0x565463bdfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.501+0000 7f0706ffd700 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565463bdd910 0x565463bdfd20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b67407aa1d7a042 server_cookie=82733076a71af71b in_seq=0 out_seq=0 2023-10-11T16:34:49.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.501+0000 7f06eaff5700 10 client.15999 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:49.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f06eaff5700 1 -- 172.21.15.111:0/3724503038 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x565463ad70a0 con 0x565463bdd910 2023-10-11T16:34:49.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f06eaff5700 10 client.15999 handle_command_reply: tid=0 2023-10-11T16:34:49.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 client.15999 shutdown 2023-10-11T16:34:49.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 -- 172.21.15.111:0/3724503038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 msgr2=0x565463bcd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463bcd980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0700007ed0 tx=0x7f0700007f00 comp rx=0 tx=0).stop 2023-10-11T16:34:49.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 -- 172.21.15.111:0/3724503038 shutdown_connections 2023-10-11T16:34:49.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565463bdd910 0x565463bdfd20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463bcd980 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc3b60 0x565463bd0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/3724503038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 0x565463bcd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f06eaff5700 0 client.15999 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:49.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 -- 172.21.15.111:0/3724503038 >> 172.21.15.111:0/3724503038 conn(0x565463bb9360 msgr2=0x565463bc9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:49.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.502+0000 7f0720dc7b80 1 -- 172.21.15.111:0/3724503038 shutdown_connections 2023-10-11T16:34:49.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.503+0000 7f0720dc7b80 1 -- 172.21.15.111:0/3724503038 wait complete. 2023-10-11T16:34:49.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.503+0000 7f0720dc7b80 20 client.15999 trim_cache size 0 max 16384 2023-10-11T16:34:49.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.517+0000 7f0720dc7b80 1 Processor -- start 2023-10-11T16:34:49.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.517+0000 7f0720dc7b80 1 -- start start 2023-10-11T16:34:49.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.517+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bcfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.517+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 0x565463bc69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.517+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc88b0 0x565463bccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.517+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463bb8ee0 con 0x565463bc3b60 2023-10-11T16:34:49.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.517+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463bcb840 con 0x565463bc88b0 2023-10-11T16:34:49.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.517+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463bc7480 con 0x565463bc4470 2023-10-11T16:34:49.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bcfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bcfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36008/0 (socket says 172.21.15.111:36008) 2023-10-11T16:34:49.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 -- 172.21.15.111:0/4197653921 learned_addr learned my addr 172.21.15.111:0/4197653921 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:49.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f07067fc700 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 0x565463bc69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f07077fe700 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc88b0 0x565463bccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 -- 172.21.15.111:0/4197653921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 msgr2=0x565463bc69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:49.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 0x565463bc69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 -- 172.21.15.111:0/4197653921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc88b0 msgr2=0x565463bccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:49.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc88b0 0x565463bccaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 -- 172.21.15.111:0/4197653921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565463adaee0 con 0x565463bc3b60 2023-10-11T16:34:49.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f07077fe700 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc88b0 0x565463bccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:49.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f0706ffd700 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bcfc00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f06f809a1b0 tx=0x7f06f8095f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397b13f898ccad97 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:49.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.518+0000 7f06edffb700 1 -- 172.21.15.111:0/4197653921 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f80a2670 con 0x565463bc3b60 2023-10-11T16:34:49.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f06edffb700 1 -- 172.21.15.111:0/4197653921 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06f809e250 con 0x565463bc3b60 2023-10-11T16:34:49.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f06edffb700 1 -- 172.21.15.111:0/4197653921 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f809a480 con 0x565463bc3b60 2023-10-11T16:34:49.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 -- 172.21.15.111:0/4197653921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 msgr2=0x565463bcfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bcfc00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f06f809a1b0 tx=0x7f06f8095f30 comp rx=0 tx=0).stop 2023-10-11T16:34:49.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 -- 172.21.15.111:0/4197653921 shutdown_connections 2023-10-11T16:34:49.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc88b0 0x565463bccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc3b60 0x565463bcfc00 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/4197653921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc4470 0x565463bc69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 -- 172.21.15.111:0/4197653921 >> 172.21.15.111:0/4197653921 conn(0x565463bb9360 msgr2=0x565463bbd550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 -- 172.21.15.111:0/4197653921 shutdown_connections 2023-10-11T16:34:49.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.519+0000 7f0720dc7b80 1 -- 172.21.15.111:0/4197653921 wait complete. 2023-10-11T16:34:49.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.520+0000 7f0720dc7b80 1 -- start start 2023-10-11T16:34:49.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.520+0000 7f0720dc7b80 1 Processor -- start 2023-10-11T16:34:49.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc3b60 0x565463bd3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 0x565463bd40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0720dc7b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463c307d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463bd4810 con 0x565463bc88b0 2023-10-11T16:34:49.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463bd4980 con 0x565463bc4470 2023-10-11T16:34:49.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0720dc7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x565463c30d10 con 0x565463bc3b60 2023-10-11T16:34:49.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0720dc7b80 10 client.0 _ll_register_callbacks cb 0x7f07000078a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:49.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f07077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463c307d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc3b60 0x565463bd3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0706ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc3b60 0x565463bd3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55690/0 (socket says 172.21.15.111:55690) 2023-10-11T16:34:49.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f07077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463c307d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36010/0 (socket says 172.21.15.111:36010) 2023-10-11T16:34:49.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f07067fc700 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 0x565463bd40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.521+0000 7f0706ffd700 1 -- 172.21.15.111:0/2520358936 learned_addr learned my addr 172.21.15.111:0/2520358936 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:49.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f07077fe700 1 -- 172.21.15.111:0/2520358936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc3b60 msgr2=0x565463bd3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:49.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f07077fe700 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc3b60 0x565463bd3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f07077fe700 1 -- 172.21.15.111:0/2520358936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 msgr2=0x565463bd40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f07077fe700 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 0x565463bd40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f07077fe700 1 -- 172.21.15.111:0/2520358936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565463adb190 con 0x565463bc88b0 2023-10-11T16:34:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f07077fe700 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463c307d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x565463bce3a0 tx=0x7f0700004d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd329b60b626ae5d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f06eaff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f06eaff5700 1 -- 172.21.15.111:0/2520358936 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070000e030 con 0x565463bc88b0 2023-10-11T16:34:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f06eaff5700 1 -- 172.21.15.111:0/2520358936 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f070001fa70 con 0x565463bc88b0 2023-10-11T16:34:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2520358936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06f809a220 con 0x565463bc88b0 2023-10-11T16:34:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f06eaff5700 1 -- 172.21.15.111:0/2520358936 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f070001f670 con 0x565463bc88b0 2023-10-11T16:34:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f0720dc7b80 10 client.16005 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2520358936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565463996630 con 0x565463bc88b0 2023-10-11T16:34:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.522+0000 7f06eaff5700 1 -- 172.21.15.111:0/2520358936 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06f809a220 con 0x565463bc88b0 2023-10-11T16:34:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f06eaff5700 1 -- 172.21.15.111:0/2520358936 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0700013800 con 0x565463bc88b0 2023-10-11T16:34:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f0720dc7b80 10 client.16005 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f0720dc7b80 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f0720dc7b80 20 client.16005 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565463c32390 0x565463c347a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f0720dc7b80 4 client.16005 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:34:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2520358936 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565463ae3660 con 0x565463c32390 2023-10-11T16:34:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f0706ffd700 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565463c32390 0x565463c347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:49.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f06eaff5700 10 client.16005 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:49.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.523+0000 7f0706ffd700 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565463c32390 0x565463c347a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8599124a729dbd9e server_cookie=81fda4995e8e301b in_seq=0 out_seq=0 2023-10-11T16:34:49.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f06eaff5700 1 -- 172.21.15.111:0/2520358936 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565463ae3660 con 0x565463c32390 2023-10-11T16:34:49.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f06eaff5700 10 client.16005 handle_command_reply: tid=0 2023-10-11T16:34:49.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 client.16005 shutdown 2023-10-11T16:34:49.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2520358936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 msgr2=0x565463c307d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463c307d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x565463bce3a0 tx=0x7f0700004d40 comp rx=0 tx=0).stop 2023-10-11T16:34:49.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2520358936 shutdown_connections 2023-10-11T16:34:49.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565463c32390 0x565463c347a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565463bc88b0 0x565463c307d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565463bc3b60 0x565463bd3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f06eaff5700 0 client.16005 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:49.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 --2- 172.21.15.111:0/2520358936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565463bc4470 0x565463bd40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2520358936 >> 172.21.15.111:0/2520358936 conn(0x565463bb9360 msgr2=0x565463bc69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:49.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2520358936 shutdown_connections 2023-10-11T16:34:49.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 1 -- 172.21.15.111:0/2520358936 wait complete. 2023-10-11T16:34:49.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.524+0000 7f0720dc7b80 20 client.16005 trim_cache size 0 max 16384 2023-10-11T16:34:49.543 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:34:49.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 -- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f06f00a1340 msgr2=0x7f06f00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f06f00a1340 0x7f06f00a37f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f06fc00b580 tx=0x7f06fc000bc0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 -- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708100cc0 msgr2=0x7f07081986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:49.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708100cc0 0x7f07081986b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f06f800eb30 tx=0x7f06f800ee40 comp rx=0 tx=0).stop 2023-10-11T16:34:49.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 -- 172.21.15.111:0/1402627725 shutdown_connections 2023-10-11T16:34:49.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f06f00a1340 0x7f06f00a37f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0708105620 0x7f070819bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0708100cc0 0x7f07081986b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 --2- 172.21.15.111:0/1402627725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0708101660 0x7f0708198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:49.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 -- 172.21.15.111:0/1402627725 >> 172.21.15.111:0/1402627725 conn(0x7f0708077560 msgr2=0x7f07081001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:49.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 -- 172.21.15.111:0/1402627725 shutdown_connections 2023-10-11T16:34:49.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:49.525+0000 7f070d3ff700 1 -- 172.21.15.111:0/1402627725 wait complete. 2023-10-11T16:34:49.570 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:49 smithi111 ceph-mon[98527]: pgmap v68: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-10-11T16:34:49.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:49 smithi130 ceph-mon[131859]: pgmap v68: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-10-11T16:34:49.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:49 smithi150 ceph-mon[132217]: pgmap v68: 129 pgs: 129 active+clean; 2.5 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-10-11T16:34:49.817 INFO:tasks.workunit.client.0.smithi111.stderr:++ getfacl d1 2023-10-11T16:34:49.817 INFO:tasks.workunit.client.0.smithi111.stderr:++ grep -c nobody:rw 2023-10-11T16:34:50.025 INFO:tasks.workunit.client.0.smithi111.stderr:+ c2=2 2023-10-11T16:34:50.025 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:50.035 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:50.036 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 2 -ne 2 ']' 2023-10-11T16:34:50.036 INFO:tasks.workunit.client.0.smithi111.stderr:+ '[' 100 -lt 100 ']' 2023-10-11T16:34:50.040 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir d1 2023-10-11T16:34:50.071 INFO:tasks.workunit.client.0.smithi111.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2023-10-11T16:34:50.078 INFO:tasks.workunit.client.0.smithi111.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2023-10-11T16:34:50.082 INFO:tasks.workunit.client.0.smithi111.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2023-10-11T16:34:50.082 INFO:tasks.workunit.client.0.smithi111.stderr:+ getfattr -n system.posix_acl_access d1 2023-10-11T16:34:50.103 INFO:tasks.workunit.client.0.smithi111.stderr:d1: system.posix_acl_access: No such attribute 2023-10-11T16:34:50.103 INFO:tasks.workunit.client.0.smithi111.stderr:+ return 0 2023-10-11T16:34:50.103 INFO:tasks.workunit.client.0.smithi111.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2023-10-11T16:34:50.104 INFO:tasks.workunit.client.0.smithi111.stderr:+ getfattr -n system.posix_acl_default . 2023-10-11T16:34:50.105 INFO:tasks.workunit.client.0.smithi111.stderr:.: system.posix_acl_default: No such attribute 2023-10-11T16:34:50.105 INFO:tasks.workunit.client.0.smithi111.stderr:+ return 0 2023-10-11T16:34:50.105 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir d1 2023-10-11T16:34:50.109 INFO:tasks.workunit.client.0.smithi111.stderr:+ cd .. 2023-10-11T16:34:50.109 INFO:tasks.workunit.client.0.smithi111.stderr:+ rmdir testdir 2023-10-11T16:34:50.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo OK 2023-10-11T16:34:50.123 INFO:tasks.workunit.client.0.smithi111.stdout:OK 2023-10-11T16:34:50.124 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-11T16:34:50.124 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-11T16:34:50.573 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:34:50.574 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:34:50.587 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2023-10-11T16:34:50.587 DEBUG:teuthology.orchestra.run.smithi111:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2023-10-11T16:34:50.768 INFO:tasks.workunit.client.0.smithi111.stderr:+ set -e 2023-10-11T16:34:50.768 INFO:tasks.workunit.client.0.smithi111.stderr:+ file=test_chmod.154917 2023-10-11T16:34:50.768 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo foo 2023-10-11T16:34:50.773 INFO:tasks.workunit.client.0.smithi111.stderr:+ test 0 '!=' 0 2023-10-11T16:34:50.773 INFO:tasks.workunit.client.0.smithi111.stderr:+ chmod 400 test_chmod.154917 2023-10-11T16:34:50.778 INFO:tasks.workunit.client.0.smithi111.stderr:+ test 0 '!=' 0 2023-10-11T16:34:50.778 INFO:tasks.workunit.client.0.smithi111.stderr:+ check_perms test_chmod.154917 -r-------- 2023-10-11T16:34:50.779 INFO:tasks.workunit.client.0.smithi111.stderr:+ file=test_chmod.154917 2023-10-11T16:34:50.779 INFO:tasks.workunit.client.0.smithi111.stderr:++ ls -la test_chmod.154917 2023-10-11T16:34:50.785 INFO:tasks.workunit.client.0.smithi111.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 Oct 11 16:34 test_chmod.154917' 2023-10-11T16:34:50.785 INFO:tasks.workunit.client.0.smithi111.stderr:+ test 0 '!=' 0 2023-10-11T16:34:50.786 INFO:tasks.workunit.client.0.smithi111.stderr:+ perms=-r-------- 2023-10-11T16:34:50.786 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Oct 11 16:34 test_chmod.154917 2023-10-11T16:34:50.787 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2023-10-11T16:34:50.894 INFO:tasks.workunit.client.0.smithi111.stderr:+ test -r-------- '!=' -r--------. 2023-10-11T16:34:50.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Oct 11 16:34 test_chmod.154917 2023-10-11T16:34:50.896 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2023-10-11T16:34:50.900 INFO:tasks.workunit.client.0.smithi111.stderr:+ test -r--------. '!=' -r--------. 2023-10-11T16:34:50.900 INFO:tasks.workunit.client.0.smithi111.stderr:+ set +e 2023-10-11T16:34:50.901 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo bar 2023-10-11T16:34:50.901 INFO:tasks.workunit.client.0.smithi111.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.154917: Permission denied 2023-10-11T16:34:50.901 INFO:tasks.workunit.client.0.smithi111.stderr:+ test 1 = 0 2023-10-11T16:34:50.901 INFO:tasks.workunit.client.0.smithi111.stderr:+ set -e 2023-10-11T16:34:50.901 INFO:tasks.workunit.client.0.smithi111.stderr:+ chmod 600 test_chmod.154917 2023-10-11T16:34:50.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo bar 2023-10-11T16:34:50.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ test 0 '!=' 0 2023-10-11T16:34:50.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ check_perms test_chmod.154917 -rw------- 2023-10-11T16:34:50.903 INFO:tasks.workunit.client.0.smithi111.stderr:+ file=test_chmod.154917 2023-10-11T16:34:50.904 INFO:tasks.workunit.client.0.smithi111.stderr:++ ls -la test_chmod.154917 2023-10-11T16:34:50.906 INFO:tasks.workunit.client.0.smithi111.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 Oct 11 16:34 test_chmod.154917' 2023-10-11T16:34:50.906 INFO:tasks.workunit.client.0.smithi111.stderr:+ test 0 '!=' 0 2023-10-11T16:34:50.907 INFO:tasks.workunit.client.0.smithi111.stderr:+ perms=-rw------- 2023-10-11T16:34:50.907 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Oct 11 16:34 test_chmod.154917 2023-10-11T16:34:50.908 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2023-10-11T16:34:50.911 INFO:tasks.workunit.client.0.smithi111.stderr:+ test -rw------- '!=' -rw-------. 2023-10-11T16:34:50.913 INFO:tasks.workunit.client.0.smithi111.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Oct 11 16:34 test_chmod.154917 2023-10-11T16:34:50.913 INFO:tasks.workunit.client.0.smithi111.stderr:++ awk '{print $1}' 2023-10-11T16:34:50.917 INFO:tasks.workunit.client.0.smithi111.stderr:+ test -rw-------. '!=' -rw-------. 2023-10-11T16:34:50.917 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo foo 2023-10-11T16:34:50.917 INFO:tasks.workunit.client.0.smithi111.stderr:+ test 0 '!=' 0 2023-10-11T16:34:50.918 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-11T16:34:50.919 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-11T16:34:51.010 INFO:tasks.workunit:Running workunit fs/misc/dac_override.sh... 2023-10-11T16:34:51.010 DEBUG:teuthology.orchestra.run.smithi111:workunit test fs/misc/dac_override.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2023-10-11T16:34:51.075 INFO:tasks.workunit.client.0.smithi111.stderr:+ set -e 2023-10-11T16:34:51.076 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir -p testdir 2023-10-11T16:34:51.079 INFO:tasks.workunit.client.0.smithi111.stderr:+ file=test_chmod.154977 2023-10-11T16:34:51.080 INFO:tasks.workunit.client.0.smithi111.stderr:+ echo foo 2023-10-11T16:34:51.080 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo chmod 600 testdir 2023-10-11T16:34:51.119 INFO:tasks.workunit.client.0.smithi111.stderr:+ expect_failure cat testdir/test_chmod.154977 2023-10-11T16:34:51.119 INFO:tasks.workunit.client.0.smithi111.stderr:+ cat testdir/test_chmod.154977 2023-10-11T16:34:51.121 INFO:tasks.workunit.client.0.smithi111.stderr:cat: testdir/test_chmod.154977: Permission denied 2023-10-11T16:34:51.121 INFO:tasks.workunit.client.0.smithi111.stderr:+ return 0 2023-10-11T16:34:51.122 INFO:tasks.workunit.client.0.smithi111.stderr:+ sudo cat testdir/test_chmod.154977 2023-10-11T16:34:51.148 INFO:tasks.workunit.client.0.smithi111.stdout:foo 2023-10-11T16:34:51.152 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-11T16:34:51.152 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-11T16:34:51.243 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2023-10-11T16:34:51.243 DEBUG:teuthology.orchestra.run.smithi111:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2023-10-11T16:34:51.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:51 smithi130 ceph-mon[131859]: pgmap v69: 129 pgs: 129 active+clean; 2.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 11 op/s 2023-10-11T16:34:51.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:51 smithi111 ceph-mon[98527]: pgmap v69: 129 pgs: 129 active+clean; 2.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 11 op/s 2023-10-11T16:34:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:51 smithi150 ceph-mon[132217]: pgmap v69: 129 pgs: 129 active+clean; 2.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 11 op/s 2023-10-11T16:34:51.722 INFO:tasks.workunit.client.0.smithi111.stdout:ok 2023-10-11T16:34:51.728 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-11T16:34:51.729 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-11T16:34:51.814 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2023-10-11T16:34:51.815 DEBUG:teuthology.orchestra.run.smithi111:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2023-10-11T16:34:51.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 -- 172.21.15.111:0/1594742689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ffdf0 msgr2=0x7f5f54103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:51.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 --2- 172.21.15.111:0/1594742689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ffdf0 0x7f5f54103db0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5f480099c0 tx=0x7f5f48009cd0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 -- 172.21.15.111:0/1594742689 shutdown_connections 2023-10-11T16:34:51.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 --2- 172.21.15.111:0/1594742689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f54104490 0x7f5f5410c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 --2- 172.21.15.111:0/1594742689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f540ff450 0x7f5f540ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 --2- 172.21.15.111:0/1594742689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ffdf0 0x7f5f54103db0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 -- 172.21.15.111:0/1594742689 >> 172.21.15.111:0/1594742689 conn(0x7f5f540770b0 msgr2=0x7f5f540774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:51.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 -- 172.21.15.111:0/1594742689 shutdown_connections 2023-10-11T16:34:51.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.856+0000 7f5f595ff700 1 -- 172.21.15.111:0/1594742689 wait complete. 2023-10-11T16:34:51.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.857+0000 7f5f595ff700 1 Processor -- start 2023-10-11T16:34:51.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.857+0000 7f5f595ff700 1 -- start start 2023-10-11T16:34:51.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f595ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ff450 0x7f5f54198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:51.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f595ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f540ffdf0 0x7f5f54198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:51.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f595ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f54104490 0x7f5f5419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:51.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f595ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5419d650 con 0x7f5f540ff450 2023-10-11T16:34:51.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f595ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5419d7c0 con 0x7f5f54104490 2023-10-11T16:34:51.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f595ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5419d980 con 0x7f5f540ffdf0 2023-10-11T16:34:51.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ff450 0x7f5f54198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:51.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ff450 0x7f5f54198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36102/0 (socket says 172.21.15.111:36102) 2023-10-11T16:34:51.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f527fc700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f540ffdf0 0x7f5f54198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:51.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f52ffd700 1 -- 172.21.15.111:0/3831036645 learned_addr learned my addr 172.21.15.111:0/3831036645 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:51.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.858+0000 7f5f537fe700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f54104490 0x7f5f5419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:51.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f52ffd700 1 -- 172.21.15.111:0/3831036645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f540ffdf0 msgr2=0x7f5f54198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:51.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f540ffdf0 0x7f5f54198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f52ffd700 1 -- 172.21.15.111:0/3831036645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f54104490 msgr2=0x7f5f5419d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:51.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f54104490 0x7f5f5419d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f52ffd700 1 -- 172.21.15.111:0/3831036645 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f480095f0 con 0x7f5f540ff450 2023-10-11T16:34:51.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f527fc700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f540ffdf0 0x7f5f54198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:51.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ff450 0x7f5f54198670 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e9a0 tx=0x7f5f4400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:51.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f3bfff700 1 -- 172.21.15.111:0/3831036645 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44009790 con 0x7f5f540ff450 2023-10-11T16:34:51.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f3bfff700 1 -- 172.21.15.111:0/3831036645 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f44010430 con 0x7f5f540ff450 2023-10-11T16:34:51.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f595ff700 1 -- 172.21.15.111:0/3831036645 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5410f260 con 0x7f5f540ff450 2023-10-11T16:34:51.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f3bfff700 1 -- 172.21.15.111:0/3831036645 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44018680 con 0x7f5f540ff450 2023-10-11T16:34:51.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.859+0000 7f5f595ff700 1 -- 172.21.15.111:0/3831036645 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5410f0d0 con 0x7f5f540ff450 2023-10-11T16:34:51.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.862+0000 7f5f3bfff700 1 -- 172.21.15.111:0/3831036645 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f5f440105a0 con 0x7f5f540ff450 2023-10-11T16:34:51.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.863+0000 7f5f3bfff700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5f3c0a12c0 0x7f5f3c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.863+0000 7f5f3bfff700 1 -- 172.21.15.111:0/3831036645 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f5f44014070 con 0x7f5f540ff450 2023-10-11T16:34:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.864+0000 7f5f527fc700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5f3c0a12c0 0x7f5f3c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.864+0000 7f5f527fc700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5f3c0a12c0 0x7f5f3c0a3770 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800b580 tx=0x7f5f4801a040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:51.935 INFO:tasks.workunit.client.0.smithi111.stdout:creating folder hierarchy 2023-10-11T16:34:51.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.987+0000 7f5f6cebab80 1 Processor -- start 2023-10-11T16:34:51.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.987+0000 7f5f6cebab80 1 -- start start 2023-10-11T16:34:51.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.987+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b5892bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:51.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.987+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892c4f0 0x562b589303f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:51.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.987+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 0x562b58930d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:51.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.987+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b588310e0 con 0x562b5892bbe0 2023-10-11T16:34:51.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.987+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b58883920 con 0x562b58930930 2023-10-11T16:34:51.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.987+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b5887cdb0 con 0x562b5892c4f0 2023-10-11T16:34:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f527fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892c4f0 0x562b589303f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f527fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892c4f0 0x562b589303f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55760/0 (socket says 172.21.15.111:55760) 2023-10-11T16:34:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f527fc700 1 -- 172.21.15.111:0/105457730 learned_addr learned my addr 172.21.15.111:0/105457730 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:51.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b5892bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f537fe700 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 0x562b58930d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f527fc700 1 -- 172.21.15.111:0/105457730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 msgr2=0x562b58930d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f527fc700 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 0x562b58930d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f527fc700 1 -- 172.21.15.111:0/105457730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 msgr2=0x562b5892bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:51.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f527fc700 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b5892bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.988+0000 7f5f527fc700 1 -- 172.21.15.111:0/105457730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b586fe6b0 con 0x562b5892c4f0 2023-10-11T16:34:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.989+0000 7f5f527fc700 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892c4f0 0x562b589303f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5f48015000 tx=0x7f5f48009cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=723f66e077fa5bd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.989+0000 7f5f397fa700 1 -- 172.21.15.111:0/105457730 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4802a070 con 0x562b5892c4f0 2023-10-11T16:34:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.989+0000 7f5f397fa700 1 -- 172.21.15.111:0/105457730 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f4801e040 con 0x562b5892c4f0 2023-10-11T16:34:51.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.989+0000 7f5f397fa700 1 -- 172.21.15.111:0/105457730 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4800f6c0 con 0x562b5892c4f0 2023-10-11T16:34:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 -- 172.21.15.111:0/105457730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892c4f0 msgr2=0x562b589303f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892c4f0 0x562b589303f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5f48015000 tx=0x7f5f48009cd0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 -- 172.21.15.111:0/105457730 shutdown_connections 2023-10-11T16:34:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 0x562b58930d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b5892bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/105457730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892c4f0 0x562b589303f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:51.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 -- 172.21.15.111:0/105457730 >> 172.21.15.111:0/105457730 conn(0x562b589213e0 msgr2=0x562b58926bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:51.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 -- 172.21.15.111:0/105457730 shutdown_connections 2023-10-11T16:34:51.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.990+0000 7f5f6cebab80 1 -- 172.21.15.111:0/105457730 wait complete. 2023-10-11T16:34:51.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.991+0000 7f5f6cebab80 1 -- start start 2023-10-11T16:34:51.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.991+0000 7f5f6cebab80 1 Processor -- start 2023-10-11T16:34:52.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.991+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892bbe0 0x562b58938140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.991+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 0x562b589353c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.991+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 0x562b58935ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.991+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b5881e960 con 0x562b5892c4f0 2023-10-11T16:34:52.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.991+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b5881e600 con 0x562b58930930 2023-10-11T16:34:52.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b587009a0 con 0x562b5892bbe0 2023-10-11T16:34:52.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f6cebab80 10 client.0 _ll_register_callbacks cb 0x562b5883c800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:52.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892bbe0 0x562b58938140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892bbe0 0x562b58938140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55766/0 (socket says 172.21.15.111:55766) 2023-10-11T16:34:52.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 -- 172.21.15.111:0/3791102285 learned_addr learned my addr 172.21.15.111:0/3791102285 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:52.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f527fc700 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 0x562b589353c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f537fe700 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 0x562b58935ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 -- 172.21.15.111:0/3791102285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 msgr2=0x562b58935ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 0x562b58935ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 -- 172.21.15.111:0/3791102285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 msgr2=0x562b589353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 0x562b589353c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 -- 172.21.15.111:0/3791102285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5410c8a0 con 0x562b5892bbe0 2023-10-11T16:34:52.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892bbe0 0x562b58938140 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5f44096a20 tx=0x7f5f440cd200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=931e120ae9ef3aa1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.992+0000 7f5f36ff5700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:34:52.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3791102285 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f440996a0 con 0x562b5892bbe0 2023-10-11T16:34:52.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3791102285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b5891eba0 con 0x562b5892bbe0 2023-10-11T16:34:52.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3791102285 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f5f440a2400 con 0x562b5892bbe0 2023-10-11T16:34:52.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3791102285 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f440b4ea0 con 0x562b5892bbe0 2023-10-11T16:34:52.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f6cebab80 10 client.26110 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:52.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3791102285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562b58843210 con 0x562b5892bbe0 2023-10-11T16:34:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3791102285 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f440c8080 con 0x562b5892bbe0 2023-10-11T16:34:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3791102285 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5f44094bc0 con 0x562b5892bbe0 2023-10-11T16:34:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f6cebab80 10 client.26110 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f6cebab80 10 client.26110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:52.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f6cebab80 20 client.26110 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:52.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.993+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562b58945900 0x562b58947ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.994+0000 7f5f6cebab80 4 client.26110 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:52.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.994+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3791102285 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b5883f120 con 0x562b58945900 2023-10-11T16:34:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.994+0000 7f5f527fc700 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562b58945900 0x562b58947ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.994+0000 7f5f36ff5700 10 client.26110 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.994+0000 7f5f527fc700 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562b58945900 0x562b58947ce0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f377387a6260cec8 server_cookie=18b1f248326b7d71 in_seq=0 out_seq=0 2023-10-11T16:34:52.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.995+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3791102285 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562b5883f120 con 0x562b58945900 2023-10-11T16:34:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.995+0000 7f5f36ff5700 10 client.26110 handle_command_reply: tid=0 2023-10-11T16:34:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.995+0000 7f5f6cebab80 1 client.26110 shutdown 2023-10-11T16:34:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3791102285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892bbe0 msgr2=0x562b58938140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892bbe0 0x562b58938140 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5f44096a20 tx=0x7f5f440cd200 comp rx=0 tx=0).stop 2023-10-11T16:34:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3791102285 shutdown_connections 2023-10-11T16:34:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562b58945900 0x562b58947ce0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b58930930 0x562b58935ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b5892bbe0 0x562b58938140 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3791102285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 0x562b589353c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3791102285 >> 172.21.15.111:0/3791102285 conn(0x562b589213e0 msgr2=0x562b58931de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f36ff5700 0 client.26110 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:52.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3791102285 shutdown_connections 2023-10-11T16:34:52.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.996+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3791102285 wait complete. 2023-10-11T16:34:52.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:51.997+0000 7f5f6cebab80 20 client.26110 trim_cache size 0 max 16384 2023-10-11T16:34:52.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.027+0000 7f5f6cebab80 1 Processor -- start 2023-10-11T16:34:52.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.027+0000 7f5f6cebab80 1 -- start start 2023-10-11T16:34:52.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.027+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b58926a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.027+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892c4f0 0x562b58937ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.027+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b58934cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.027+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b58928280 con 0x562b5892bbe0 2023-10-11T16:34:52.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.027+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b58933920 con 0x562b5892c4f0 2023-10-11T16:34:52.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.027+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b58936940 con 0x562b58930930 2023-10-11T16:34:52.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.028+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b58926a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.028+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b58926a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36128/0 (socket says 172.21.15.111:36128) 2023-10-11T16:34:52.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.028+0000 7f5f52ffd700 1 -- 172.21.15.111:0/2620130066 learned_addr learned my addr 172.21.15.111:0/2620130066 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:52.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.028+0000 7f5f527fc700 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892c4f0 0x562b58937ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.028+0000 7f5f537fe700 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b58934cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.029+0000 7f5f52ffd700 1 -- 172.21.15.111:0/2620130066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 msgr2=0x562b58934cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.029+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b58934cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.029+0000 7f5f52ffd700 1 -- 172.21.15.111:0/2620130066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892c4f0 msgr2=0x562b58937ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.029+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892c4f0 0x562b58937ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.029+0000 7f5f52ffd700 1 -- 172.21.15.111:0/2620130066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b58930fd0 con 0x562b5892bbe0 2023-10-11T16:34:52.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.029+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b58926a80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5f44000c00 tx=0x7f5f44010a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6be921d97f8250 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.029+0000 7f5f39ffb700 1 -- 172.21.15.111:0/2620130066 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f440b8170 con 0x562b5892bbe0 2023-10-11T16:34:52.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.030+0000 7f5f39ffb700 1 -- 172.21.15.111:0/2620130066 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562b5891eba0 con 0x562b5892bbe0 2023-10-11T16:34:52.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.030+0000 7f5f39ffb700 1 -- 172.21.15.111:0/2620130066 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44099290 con 0x562b5892bbe0 2023-10-11T16:34:52.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 -- 172.21.15.111:0/2620130066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 msgr2=0x562b58926a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b58926a80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5f44000c00 tx=0x7f5f44010a90 comp rx=0 tx=0).stop 2023-10-11T16:34:52.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 -- 172.21.15.111:0/2620130066 shutdown_connections 2023-10-11T16:34:52.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b58934cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892bbe0 0x562b58926a80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/2620130066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892c4f0 0x562b58937ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 -- 172.21.15.111:0/2620130066 >> 172.21.15.111:0/2620130066 conn(0x562b589213e0 msgr2=0x562b5892ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 -- 172.21.15.111:0/2620130066 shutdown_connections 2023-10-11T16:34:52.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.031+0000 7f5f6cebab80 1 -- 172.21.15.111:0/2620130066 wait complete. 2023-10-11T16:34:52.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.032+0000 7f5f6cebab80 1 -- start start 2023-10-11T16:34:52.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.032+0000 7f5f6cebab80 1 Processor -- start 2023-10-11T16:34:52.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.033+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892bbe0 0x562b5893c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.033+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 0x562b5893aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.033+0000 7f5f6cebab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b5893af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.033+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b5893b4d0 con 0x562b5892c4f0 2023-10-11T16:34:52.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.033+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b5893b640 con 0x562b5892bbe0 2023-10-11T16:34:52.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.033+0000 7f5f6cebab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b5893b780 con 0x562b58930930 2023-10-11T16:34:52.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.033+0000 7f5f6cebab80 10 client.0 _ll_register_callbacks cb 0x7f5f440cd0c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.034+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892bbe0 0x562b5893c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.034+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892bbe0 0x562b5893c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:53382/0 (socket says 172.21.15.111:53382) 2023-10-11T16:34:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.034+0000 7f5f537fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b5893af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.034+0000 7f5f537fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b5893af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55786/0 (socket says 172.21.15.111:55786) 2023-10-11T16:34:52.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.034+0000 7f5f537fe700 1 -- 172.21.15.111:0/3989702794 learned_addr learned my addr 172.21.15.111:0/3989702794 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:52.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.035+0000 7f5f537fe700 1 -- 172.21.15.111:0/3989702794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892bbe0 msgr2=0x562b5893c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.035+0000 7f5f537fe700 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892bbe0 0x562b5893c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.035+0000 7f5f537fe700 1 -- 172.21.15.111:0/3989702794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 msgr2=0x562b5893aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.035+0000 7f5f537fe700 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 0x562b5893aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.035+0000 7f5f537fe700 1 -- 172.21.15.111:0/3989702794 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b58843210 con 0x562b58930930 2023-10-11T16:34:52.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.035+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892bbe0 0x562b5893c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:52.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.036+0000 7f5f537fe700 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b5893af90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c004660 tx=0x7f5f4c004970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5987c475dac88e8b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.036+0000 7f5f36ff5700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:34:52.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.036+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3989702794 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c01c070 con 0x562b58930930 2023-10-11T16:34:52.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.036+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3989702794 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f4801e040 con 0x562b58930930 2023-10-11T16:34:52.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.037+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3989702794 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f5f4c017a10 con 0x562b58930930 2023-10-11T16:34:52.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.037+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3989702794 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f4c00e040 con 0x562b58930930 2023-10-11T16:34:52.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.037+0000 7f5f6cebab80 10 client.26116 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:52.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.037+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3989702794 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562b586aae60 con 0x562b58930930 2023-10-11T16:34:52.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.038+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3989702794 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f4c00f040 con 0x562b58930930 2023-10-11T16:34:52.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.038+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3989702794 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5f4c021bc0 con 0x562b58930930 2023-10-11T16:34:52.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.039+0000 7f5f6cebab80 10 client.26116 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:52.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.039+0000 7f5f6cebab80 10 client.26116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:52.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.039+0000 7f5f6cebab80 20 client.26116 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:52.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.039+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562b58994b90 0x562b58994f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.039+0000 7f5f6cebab80 4 client.26116 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:34:52.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.039+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3989702794 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x7f5f4801e040 con 0x562b58994b90 2023-10-11T16:34:52.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.039+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562b58994b90 0x562b58994f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.040+0000 7f5f52ffd700 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562b58994b90 0x562b58994f90 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da7b09881d7c7b6 server_cookie=e2775152051e3223 in_seq=0 out_seq=0 2023-10-11T16:34:52.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.040+0000 7f5f36ff5700 10 client.26116 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:52.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f36ff5700 1 -- 172.21.15.111:0/3989702794 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f5f4801e040 con 0x562b58994b90 2023-10-11T16:34:52.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f36ff5700 10 client.26116 handle_command_reply: tid=0 2023-10-11T16:34:52.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f6cebab80 1 client.26116 shutdown 2023-10-11T16:34:52.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3989702794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 msgr2=0x562b5893af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b5893af90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c004660 tx=0x7f5f4c004970 comp rx=0 tx=0).stop 2023-10-11T16:34:52.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3989702794 shutdown_connections 2023-10-11T16:34:52.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x562b58994b90 0x562b58994f90 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x562b58930930 0x562b5893af90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562b5892bbe0 0x562b5893c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.042+0000 7f5f6cebab80 1 --2- 172.21.15.111:0/3989702794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x562b5892c4f0 0x562b5893aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.043+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3989702794 >> 172.21.15.111:0/3989702794 conn(0x562b589213e0 msgr2=0x562b5892f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.043+0000 7f5f36ff5700 0 client.26116 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:52.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.043+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3989702794 shutdown_connections 2023-10-11T16:34:52.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.043+0000 7f5f6cebab80 1 -- 172.21.15.111:0/3989702794 wait complete. 2023-10-11T16:34:52.051 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:34:52.051 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:34:52.052 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "84170411-7f5f-4475-8c14-55f9e93adff6", 2023-10-11T16:34:52.052 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:34:52.052 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:34:52.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.044+0000 7f5f6cebab80 20 client.26116 trim_cache size 0 max 16384 2023-10-11T16:34:52.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 -- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5f3c0a12c0 msgr2=0x7f5f3c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5f3c0a12c0 0x7f5f3c0a3770 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800b580 tx=0x7f5f4801a040 comp rx=0 tx=0).stop 2023-10-11T16:34:52.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 -- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ff450 msgr2=0x7f5f54198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ff450 0x7f5f54198670 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e9a0 tx=0x7f5f4400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 -- 172.21.15.111:0/3831036645 shutdown_connections 2023-10-11T16:34:52.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5f3c0a12c0 0x7f5f3c0a3770 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f54104490 0x7f5f5419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f540ff450 0x7f5f54198670 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 --2- 172.21.15.111:0/3831036645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f540ffdf0 0x7f5f54198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 -- 172.21.15.111:0/3831036645 >> 172.21.15.111:0/3831036645 conn(0x7f5f540770b0 msgr2=0x7f5f5410ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 -- 172.21.15.111:0/3831036645 shutdown_connections 2023-10-11T16:34:52.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.045+0000 7f5f595ff700 1 -- 172.21.15.111:0/3831036645 wait complete. 2023-10-11T16:34:52.075 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:34:52.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:52 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:34:52.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:52 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:52.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.434+0000 7fb7e13ff700 1 -- 172.21.15.111:0/1245425702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 msgr2=0x7fb7dc112e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.434+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/1245425702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 0x7fb7dc112e70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00098f0 tx=0x7fb7d0009c00 comp rx=0 tx=0).stop 2023-10-11T16:34:52.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.434+0000 7fb7e13ff700 1 -- 172.21.15.111:0/1245425702 shutdown_connections 2023-10-11T16:34:52.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.434+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/1245425702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc1133b0 0x7fb7dc115790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.434+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/1245425702 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7dc107af0 0x7fb7dc107ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.434+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/1245425702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 0x7fb7dc112e70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.434+0000 7fb7e13ff700 1 -- 172.21.15.111:0/1245425702 >> 172.21.15.111:0/1245425702 conn(0x7fb7dc06ca60 msgr2=0x7fb7dc071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.435+0000 7fb7e13ff700 1 -- 172.21.15.111:0/1245425702 shutdown_connections 2023-10-11T16:34:52.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.435+0000 7fb7e13ff700 1 -- 172.21.15.111:0/1245425702 wait complete. 2023-10-11T16:34:52.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.437+0000 7fb7e13ff700 1 Processor -- start 2023-10-11T16:34:52.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.437+0000 7fb7e13ff700 1 -- start start 2023-10-11T16:34:52.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7e13ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc107af0 0x7fb7dc1a2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7e13ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 0x7fb7dc1a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7e13ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7dc1133b0 0x7fb7dc19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7e13ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc19cf70 con 0x7fb7dc1133b0 2023-10-11T16:34:52.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7e13ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc19d0e0 con 0x7fb7dc108400 2023-10-11T16:34:52.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7e13ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc19d2a0 con 0x7fb7dc107af0 2023-10-11T16:34:52.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc107af0 0x7fb7dc1a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc107af0 0x7fb7dc1a2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55798/0 (socket says 172.21.15.111:55798) 2023-10-11T16:34:52.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.438+0000 7fb7daffd700 1 -- 172.21.15.111:0/706477338 learned_addr learned my addr 172.21.15.111:0/706477338 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:52.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.439+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 0x7fb7dc1a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.439+0000 7fb7daffd700 1 -- 172.21.15.111:0/706477338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 msgr2=0x7fb7dc1a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.439+0000 7fb7db7fe700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7dc1133b0 0x7fb7dc19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.439+0000 7fb7daffd700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 0x7fb7dc1a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.439+0000 7fb7daffd700 1 -- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7dc1133b0 msgr2=0x7fb7dc19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.439+0000 7fb7daffd700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7dc1133b0 0x7fb7dc19c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.439+0000 7fb7daffd700 1 -- 172.21.15.111:0/706477338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d0009520 con 0x7fb7dc107af0 2023-10-11T16:34:52.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.439+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 0x7fb7dc1a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:34:52.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.440+0000 7fb7db7fe700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7dc1133b0 0x7fb7dc19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:34:52.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.440+0000 7fb7daffd700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc107af0 0x7fb7dc1a2890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00b550 tx=0x7fb7cc00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.440+0000 7fb7c3fff700 1 -- 172.21.15.111:0/706477338 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc011830 con 0x7fb7dc107af0 2023-10-11T16:34:52.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.440+0000 7fb7c3fff700 1 -- 172.21.15.111:0/706477338 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7cc011e50 con 0x7fb7dc107af0 2023-10-11T16:34:52.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.440+0000 7fb7c3fff700 1 -- 172.21.15.111:0/706477338 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00f2e0 con 0x7fb7dc107af0 2023-10-11T16:34:52.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.442+0000 7fb7e13ff700 1 -- 172.21.15.111:0/706477338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc116970 con 0x7fb7dc107af0 2023-10-11T16:34:52.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.442+0000 7fb7e13ff700 1 -- 172.21.15.111:0/706477338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc1167e0 con 0x7fb7dc107af0 2023-10-11T16:34:52.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.447+0000 7fb7c3fff700 1 -- 172.21.15.111:0/706477338 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fb7cc011990 con 0x7fb7dc107af0 2023-10-11T16:34:52.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.449+0000 7fb7f4e0eb80 1 Processor -- start 2023-10-11T16:34:52.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.450+0000 7fb7f4e0eb80 1 -- start start 2023-10-11T16:34:52.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.450+0000 7fb7c3fff700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb7c40a13c0 0x7fb7c40a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.450+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d3b60 0x55ebba1d3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.450+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 0x55ebba1d8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.450+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1d8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba0d9060 con 0x55ebba1d88b0 2023-10-11T16:34:52.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1d8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d3b60 0x55ebba1d3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7db7fe700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb7c40a13c0 0x7fb7c40a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d3b60 0x55ebba1d3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55814/0 (socket says 172.21.15.111:55814) 2023-10-11T16:34:52.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7daffd700 1 -- 172.21.15.111:0/4173053819 learned_addr learned my addr 172.21.15.111:0/4173053819 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:52.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/4173053819 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba12b8a0 con 0x55ebba1d4470 2023-10-11T16:34:52.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7c3fff700 1 -- 172.21.15.111:0/706477338 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb7cc0ccb00 con 0x7fb7dc107af0 2023-10-11T16:34:52.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.451+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/4173053819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba124d30 con 0x55ebba1d3b60 2023-10-11T16:34:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.452+0000 7fb7daffd700 1 -- 172.21.15.111:0/4173053819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 msgr2=0x55ebba1d8370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:34:52.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.452+0000 7fb7daffd700 1 --2- 172.21.15.111:0/4173053819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 0x55ebba1d8370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.452+0000 7fb7daffd700 1 -- 172.21.15.111:0/4173053819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 msgr2=0x55ebba1d8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.452+0000 7fb7daffd700 1 --2- 172.21.15.111:0/4173053819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1d8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.452+0000 7fb7daffd700 1 -- 172.21.15.111:0/4173053819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebb9fa6630 con 0x55ebba1d3b60 2023-10-11T16:34:52.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.453+0000 7fb7db7fe700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb7c40a13c0 0x7fb7c40a3870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009750 tx=0x7fb7d40096a0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.454+0000 7fb7daffd700 1 --2- 172.21.15.111:0/4173053819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d3b60 0x55ebba1d3f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0c9ef0 tx=0x7fb7cc099540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6557ed68c9e8c9e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.454+0000 7fb7c17fa700 1 -- 172.21.15.111:0/4173053819 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0998d0 con 0x55ebba1d3b60 2023-10-11T16:34:52.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.455+0000 7fb7c17fa700 1 -- 172.21.15.111:0/4173053819 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7cc09a6b0 con 0x55ebba1d3b60 2023-10-11T16:34:52.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.455+0000 7fb7c17fa700 1 -- 172.21.15.111:0/4173053819 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc09e760 con 0x55ebba1d3b60 2023-10-11T16:34:52.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.456+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/4173053819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d3b60 msgr2=0x55ebba1d3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.456+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/4173053819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d3b60 0x55ebba1d3f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0c9ef0 tx=0x7fb7cc099540 comp rx=0 tx=0).stop 2023-10-11T16:34:52.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.456+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/4173053819 shutdown_connections 2023-10-11T16:34:52.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.456+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/4173053819 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1d8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.456+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/4173053819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d3b60 0x55ebba1d3f30 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0c9ef0 tx=0x7fb7cc099540 comp rx=0 tx=0).stop 2023-10-11T16:34:52.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.456+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/4173053819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 0x55ebba1d8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.456+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/4173053819 >> 172.21.15.111:0/4173053819 conn(0x55ebba1c9360 msgr2=0x55ebba1ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.457+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/4173053819 shutdown_connections 2023-10-11T16:34:52.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.457+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/4173053819 wait complete. 2023-10-11T16:34:52.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.458+0000 7fb7f4e0eb80 1 -- start start 2023-10-11T16:34:52.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.458+0000 7fb7f4e0eb80 1 Processor -- start 2023-10-11T16:34:52.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 0x55ebba1dd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1dda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1ddf40 0x55ebba1faa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba0c68e0 con 0x55ebba1d88b0 2023-10-11T16:34:52.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba0c6580 con 0x55ebba1d4470 2023-10-11T16:34:52.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebb9fa8920 con 0x55ebba1ddf40 2023-10-11T16:34:52.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7f4e0eb80 10 client.0 _ll_register_callbacks cb 0x55ebba0e4780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:52.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7da7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1dda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7da7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1dda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36158/0 (socket says 172.21.15.111:36158) 2023-10-11T16:34:52.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7da7fc700 1 -- 172.21.15.111:0/2905596307 learned_addr learned my addr 172.21.15.111:0/2905596307 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:52.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.459+0000 7fb7daffd700 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 0x55ebba1dd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.460+0000 7fb7daffd700 1 -- 172.21.15.111:0/2905596307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1ddf40 msgr2=0x55ebba1faa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.460+0000 7fb7daffd700 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1ddf40 0x55ebba1faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.460+0000 7fb7daffd700 1 -- 172.21.15.111:0/2905596307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 msgr2=0x55ebba1dda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.460+0000 7fb7daffd700 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1dda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.460+0000 7fb7daffd700 1 -- 172.21.15.111:0/2905596307 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc1167e0 con 0x55ebba1d4470 2023-10-11T16:34:52.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.460+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1dda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:52.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.460+0000 7fb7daffd700 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 0x55ebba1dd4c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55ebba1cfae0 tx=0x7fb7cc0c9fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a79142d9c5355871 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.460+0000 7fb7beff5700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:34:52.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.461+0000 7fb7beff5700 1 -- 172.21.15.111:0/2905596307 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0a1d20 con 0x55ebba1d4470 2023-10-11T16:34:52.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.461+0000 7fb7beff5700 1 -- 172.21.15.111:0/2905596307 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb7cc09e760 con 0x55ebba1d4470 2023-10-11T16:34:52.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.461+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2905596307 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebba1c6b20 con 0x55ebba1d4470 2023-10-11T16:34:52.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.461+0000 7fb7beff5700 1 -- 172.21.15.111:0/2905596307 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7cc0998d0 con 0x55ebba1d4470 2023-10-11T16:34:52.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.461+0000 7fb7f4e0eb80 10 client.26090 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:52.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.461+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2905596307 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ebba0eb190 con 0x55ebba1d4470 2023-10-11T16:34:52.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.462+0000 7fb7beff5700 1 -- 172.21.15.111:0/2905596307 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ebba1c6b20 con 0x55ebba1d4470 2023-10-11T16:34:52.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.463+0000 7fb7beff5700 1 -- 172.21.15.111:0/2905596307 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb7cc0b3070 con 0x55ebba1d4470 2023-10-11T16:34:52.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.463+0000 7fb7f4e0eb80 10 client.26090 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:52.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.463+0000 7fb7f4e0eb80 10 client.26090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:52.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.463+0000 7fb7f4e0eb80 20 client.26090 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:52.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.463+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ebba1feb50 0x55ebba200f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.463+0000 7fb7f4e0eb80 4 client.26090 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:52.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.463+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2905596307 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebba0e70a0 con 0x55ebba1feb50 2023-10-11T16:34:52.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.463+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ebba1feb50 0x55ebba200f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.464+0000 7fb7beff5700 10 client.26090 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:52.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.464+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ebba1feb50 0x55ebba200f60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ae279d0a9cf93d5 server_cookie=4275b292f067e09d in_seq=0 out_seq=0 2023-10-11T16:34:52.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.464+0000 7fb7beff5700 1 -- 172.21.15.111:0/2905596307 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ebba0e70a0 con 0x55ebba1feb50 2023-10-11T16:34:52.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.465+0000 7fb7beff5700 10 client.26090 handle_command_reply: tid=0 2023-10-11T16:34:52.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.465+0000 7fb7f4e0eb80 1 client.26090 shutdown 2023-10-11T16:34:52.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.465+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 msgr2=0x55ebba1dd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.465+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 0x55ebba1dd4c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55ebba1cfae0 tx=0x7fb7cc0c9fa0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.466+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2905596307 shutdown_connections 2023-10-11T16:34:52.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.466+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ebba1feb50 0x55ebba200f60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.466+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1ddf40 0x55ebba1faa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.466+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d4470 0x55ebba1dd4c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.466+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2905596307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d88b0 0x55ebba1dda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.466+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2905596307 >> 172.21.15.111:0/2905596307 conn(0x55ebba1c9360 msgr2=0x55ebba1d2960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.466+0000 7fb7beff5700 0 client.26090 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:52.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.467+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2905596307 shutdown_connections 2023-10-11T16:34:52.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.467+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2905596307 wait complete. 2023-10-11T16:34:52.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.468+0000 7fb7f4e0eb80 20 client.26090 trim_cache size 0 max 16384 2023-10-11T16:34:52.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.496+0000 7fb7f4e0eb80 1 Processor -- start 2023-10-11T16:34:52.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.496+0000 7fb7f4e0eb80 1 -- start start 2023-10-11T16:34:52.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.497+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d4470 0x55ebba1d61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.497+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d88b0 0x55ebba1f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.497+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1ddf40 0x55ebba1dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.497+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba1deaa0 con 0x55ebba1ddf40 2023-10-11T16:34:52.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.497+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba1e02e0 con 0x55ebba1d88b0 2023-10-11T16:34:52.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.497+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba1d0430 con 0x55ebba1d4470 2023-10-11T16:34:52.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.497+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d4470 0x55ebba1d61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d4470 0x55ebba1d61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55836/0 (socket says 172.21.15.111:55836) 2023-10-11T16:34:52.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7daffd700 1 -- 172.21.15.111:0/2594639223 learned_addr learned my addr 172.21.15.111:0/2594639223 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:52.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1ddf40 0x55ebba1dc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/2594639223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d88b0 0x55ebba1f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7db7fe700 1 -- 172.21.15.111:0/2594639223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d4470 msgr2=0x55ebba1d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7db7fe700 1 --2- 172.21.15.111:0/2594639223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d4470 0x55ebba1d61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7db7fe700 1 -- 172.21.15.111:0/2594639223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d88b0 msgr2=0x55ebba1f59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7db7fe700 1 --2- 172.21.15.111:0/2594639223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d88b0 0x55ebba1f59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.498+0000 7fb7db7fe700 1 -- 172.21.15.111:0/2594639223 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebba0eaee0 con 0x55ebba1ddf40 2023-10-11T16:34:52.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.499+0000 7fb7db7fe700 1 --2- 172.21.15.111:0/2594639223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1ddf40 0x55ebba1dc7a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc06a150 tx=0x7fb7d4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f112bf0a47d167 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.499+0000 7fb7c1ffb700 1 -- 172.21.15.111:0/2594639223 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d400e050 con 0x55ebba1ddf40 2023-10-11T16:34:52.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.499+0000 7fb7c1ffb700 1 -- 172.21.15.111:0/2594639223 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7d400f040 con 0x55ebba1ddf40 2023-10-11T16:34:52.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.499+0000 7fb7c1ffb700 1 -- 172.21.15.111:0/2594639223 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4007ce0 con 0x55ebba1ddf40 2023-10-11T16:34:52.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.500+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2594639223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1ddf40 msgr2=0x55ebba1dc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.501+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2594639223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1ddf40 0x55ebba1dc7a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc06a150 tx=0x7fb7d4019000 comp rx=0 tx=0).stop 2023-10-11T16:34:52.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.501+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2594639223 shutdown_connections 2023-10-11T16:34:52.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.501+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2594639223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1ddf40 0x55ebba1dc7a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.501+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2594639223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d4470 0x55ebba1d61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.501+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/2594639223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1d88b0 0x55ebba1f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.502+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2594639223 >> 172.21.15.111:0/2594639223 conn(0x55ebba1c9360 msgr2=0x55ebba1d6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.502+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2594639223 shutdown_connections 2023-10-11T16:34:52.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.502+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/2594639223 wait complete. 2023-10-11T16:34:52.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.502+0000 7fb7f4e0eb80 1 -- start start 2023-10-11T16:34:52.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.503+0000 7fb7f4e0eb80 1 Processor -- start 2023-10-11T16:34:52.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.503+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d4470 0x55ebba1f3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d88b0 0x55ebba1f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7f4e0eb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1ddf40 0x55ebba251a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba251f50 con 0x55ebba1d4470 2023-10-11T16:34:52.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba252090 con 0x55ebba1ddf40 2023-10-11T16:34:52.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7f4e0eb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebba2521d0 con 0x55ebba1d88b0 2023-10-11T16:34:52.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7f4e0eb80 10 client.0 _ll_register_callbacks cb 0x55ebba0e4780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:52.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d4470 0x55ebba1f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7daffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d4470 0x55ebba1f3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36176/0 (socket says 172.21.15.111:36176) 2023-10-11T16:34:52.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7daffd700 1 -- 172.21.15.111:0/3738137320 learned_addr learned my addr 172.21.15.111:0/3738137320 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:52.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.504+0000 7fb7daffd700 1 -- 172.21.15.111:0/3738137320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d88b0 msgr2=0x55ebba1f3cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:34:52.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.505+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d88b0 0x55ebba1f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.505+0000 7fb7daffd700 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d88b0 0x55ebba1f3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.505+0000 7fb7daffd700 1 -- 172.21.15.111:0/3738137320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1ddf40 msgr2=0x55ebba251a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:52.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.505+0000 7fb7daffd700 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1ddf40 0x55ebba251a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.505+0000 7fb7daffd700 1 -- 172.21.15.111:0/3738137320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d4009070 con 0x55ebba1d4470 2023-10-11T16:34:52.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.505+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d88b0 0x55ebba1f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:34:52.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.505+0000 7fb7daffd700 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d4470 0x55ebba1f3780 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc095dd0 tx=0x7fb7cc099540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d9c753abf403c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:52.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.505+0000 7fb7beff5700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:52.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.506+0000 7fb7beff5700 1 -- 172.21.15.111:0/3738137320 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc010660 con 0x55ebba1d4470 2023-10-11T16:34:52.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.506+0000 7fb7beff5700 1 -- 172.21.15.111:0/3738137320 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb7cc09e760 con 0x55ebba1d4470 2023-10-11T16:34:52.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.506+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/3738137320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7cc09a6b0 con 0x55ebba1d4470 2023-10-11T16:34:52.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.506+0000 7fb7beff5700 1 -- 172.21.15.111:0/3738137320 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7cc0a1d20 con 0x55ebba1d4470 2023-10-11T16:34:52.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.507+0000 7fb7f4e0eb80 10 client.16050 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:52.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.507+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/3738137320 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ebb9f3bc60 con 0x55ebba1d4470 2023-10-11T16:34:52.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.507+0000 7fb7beff5700 1 -- 172.21.15.111:0/3738137320 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7cc09a6b0 con 0x55ebba1d4470 2023-10-11T16:34:52.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.507+0000 7fb7beff5700 1 -- 172.21.15.111:0/3738137320 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb7cc0b3040 con 0x55ebba1d4470 2023-10-11T16:34:52.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.507+0000 7fb7f4e0eb80 10 client.16050 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:52.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.507+0000 7fb7f4e0eb80 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:52.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.507+0000 7fb7f4e0eb80 20 client.16050 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:52.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.507+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ebba2533e0 0x55ebba2557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:52.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.508+0000 7fb7f4e0eb80 4 client.16050 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:34:52.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.508+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/3738137320 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ebba0f3660 con 0x55ebba2533e0 2023-10-11T16:34:52.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.508+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ebba2533e0 0x55ebba2557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:52.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.509+0000 7fb7beff5700 10 client.16050 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:52.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.509+0000 7fb7da7fc700 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ebba2533e0 0x55ebba2557c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9792f8ff85e91f60 server_cookie=741f49d113d308c5 in_seq=0 out_seq=0 2023-10-11T16:34:52.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.509+0000 7fb7beff5700 1 -- 172.21.15.111:0/3738137320 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ebba0f3660 con 0x55ebba2533e0 2023-10-11T16:34:52.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.510+0000 7fb7beff5700 10 client.16050 handle_command_reply: tid=0 2023-10-11T16:34:52.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.510+0000 7fb7f4e0eb80 1 client.16050 shutdown 2023-10-11T16:34:52.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.510+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/3738137320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d4470 msgr2=0x55ebba1f3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.510+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d4470 0x55ebba1f3780 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc095dd0 tx=0x7fb7cc099540 comp rx=0 tx=0).stop 2023-10-11T16:34:52.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.510+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/3738137320 shutdown_connections 2023-10-11T16:34:52.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.510+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ebba2533e0 0x55ebba2557c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.511+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ebba1ddf40 0x55ebba251a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.511+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ebba1d4470 0x55ebba1f3780 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.511+0000 7fb7f4e0eb80 1 --2- 172.21.15.111:0/3738137320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ebba1d88b0 0x55ebba1f3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.511+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/3738137320 >> 172.21.15.111:0/3738137320 conn(0x55ebba1c9360 msgr2=0x55ebba1dd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.511+0000 7fb7beff5700 0 client.16050 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.511+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/3738137320 shutdown_connections 2023-10-11T16:34:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.511+0000 7fb7f4e0eb80 1 -- 172.21.15.111:0/3738137320 wait complete. 2023-10-11T16:34:52.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.511+0000 7fb7f4e0eb80 20 client.16050 trim_cache size 0 max 16384 2023-10-11T16:34:52.520 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:34:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:34:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:34:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "84170411-7f5f-4475-8c14-55f9e93adff6": { 2023-10-11T16:34:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:34:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "84170411-7f5f-4475-8c14-55f9e93adff6", 2023-10-11T16:34:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:34:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:34:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:34:52.521 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:34:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 -- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb7c40a13c0 msgr2=0x7fb7c40a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb7c40a13c0 0x7fb7c40a3870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009750 tx=0x7fb7d40096a0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 -- 172.21.15.111:0/706477338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc107af0 msgr2=0x7fb7dc1a2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc107af0 0x7fb7dc1a2890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00b550 tx=0x7fb7cc00b910 comp rx=0 tx=0).stop 2023-10-11T16:34:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 -- 172.21.15.111:0/706477338 shutdown_connections 2023-10-11T16:34:52.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb7c40a13c0 0x7fb7c40a3870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7dc1133b0 0x7fb7dc19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc107af0 0x7fb7dc1a2890 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 --2- 172.21.15.111:0/706477338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7dc108400 0x7fb7dc1a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.512+0000 7fb7e13ff700 1 -- 172.21.15.111:0/706477338 >> 172.21.15.111:0/706477338 conn(0x7fb7dc06ca60 msgr2=0x7fb7dc10ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.513+0000 7fb7e13ff700 1 -- 172.21.15.111:0/706477338 shutdown_connections 2023-10-11T16:34:52.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:34:52.513+0000 7fb7e13ff700 1 -- 172.21.15.111:0/706477338 wait complete. 2023-10-11T16:34:52.543 INFO:tasks.cephfs.filesystem:scrub status for tag:84170411-7f5f-4475-8c14-55f9e93adff6 - {'path': '/', 'tag': '84170411-7f5f-4475-8c14-55f9e93adff6', 'options': 'recursive,force'} 2023-10-11T16:34:52.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:52 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:34:52.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:52 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:52.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:52 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:34:52.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:52 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:53.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:53 smithi130 ceph-mon[131859]: pgmap v70: 129 pgs: 129 active+clean; 2.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-10-11T16:34:53.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:53 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:34:53.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:53 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:53.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:53 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:34:53.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:53 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:53.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:53 smithi111 ceph-mon[98527]: pgmap v70: 129 pgs: 129 active+clean; 2.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-10-11T16:34:53.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:53 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:34:53.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:53 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:53.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:53 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:34:53.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:53 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:53 smithi150 ceph-mon[132217]: pgmap v70: 129 pgs: 129 active+clean; 2.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-10-11T16:34:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:53 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:34:53.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:53 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:53.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:53 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:34:53.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:53 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:34:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.792+0000 7f2ae27ff700 1 -- 172.21.15.111:0/3324032832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc10a810 msgr2=0x7f2adc10cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.792+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/3324032832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc10a810 0x7f2adc10cc30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400b0a0 tx=0x7f2ad400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.793+0000 7f2ae27ff700 1 -- 172.21.15.111:0/3324032832 shutdown_connections 2023-10-11T16:34:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.793+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/3324032832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc10a810 0x7f2adc10cc30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.793+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/3324032832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc101440 0x7f2adc101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.793+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/3324032832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2adc101d50 0x7f2adc10a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.793+0000 7f2ae27ff700 1 -- 172.21.15.111:0/3324032832 >> 172.21.15.111:0/3324032832 conn(0x7f2adc0763a0 msgr2=0x7f2adc0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.794+0000 7f2ae27ff700 1 -- 172.21.15.111:0/3324032832 shutdown_connections 2023-10-11T16:34:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.794+0000 7f2ae27ff700 1 -- 172.21.15.111:0/3324032832 wait complete. 2023-10-11T16:34:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.795+0000 7f2ae27ff700 1 Processor -- start 2023-10-11T16:34:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.795+0000 7f2ae27ff700 1 -- start start 2023-10-11T16:34:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.796+0000 7f2ae27ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc101440 0x7f2adc1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.796+0000 7f2ae27ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc101d50 0x7f2adc1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.796+0000 7f2ae27ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2adc10a810 0x7f2adc19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.796+0000 7f2ae27ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc19b490 con 0x7f2adc101d50 2023-10-11T16:34:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.796+0000 7f2ae27ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc19b600 con 0x7f2adc10a810 2023-10-11T16:34:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.796+0000 7f2ae27ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc19b7c0 con 0x7f2adc101440 2023-10-11T16:34:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2adbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc101440 0x7f2adc1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.798 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2adbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc101440 0x7f2adc1964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55854/0 (socket says 172.21.15.111:55854) 2023-10-11T16:34:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2adbfff700 1 -- 172.21.15.111:0/521019365 learned_addr learned my addr 172.21.15.111:0/521019365 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2adc10a810 0x7f2adc19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2adbfff700 1 -- 172.21.15.111:0/521019365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2adc10a810 msgr2=0x7f2adc19ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:53.799 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2adbfff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2adc10a810 0x7f2adc19ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2adbfff700 1 -- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc101d50 msgr2=0x7f2adc1969f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:34:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc101d50 0x7f2adc1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.797+0000 7f2adbfff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc101d50 0x7f2adc1969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.798+0000 7f2adbfff700 1 -- 172.21.15.111:0/521019365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad4009d00 con 0x7f2adc101440 2023-10-11T16:34:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.798+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc101d50 0x7f2adc1969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:34:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.798+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2adc10a810 0x7f2adc19ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:34:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.798+0000 7f2adbfff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc101440 0x7f2adc1964b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00b530 tx=0x7f2acc00b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.798+0000 7f2ad97fa700 1 -- 172.21.15.111:0/521019365 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc011830 con 0x7f2adc101440 2023-10-11T16:34:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.798+0000 7f2ad97fa700 1 -- 172.21.15.111:0/521019365 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2acc011e50 con 0x7f2adc101440 2023-10-11T16:34:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.798+0000 7f2ad97fa700 1 -- 172.21.15.111:0/521019365 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc00f2e0 con 0x7f2adc101440 2023-10-11T16:34:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.798+0000 7f2ae27ff700 1 -- 172.21.15.111:0/521019365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2adc1103d0 con 0x7f2adc101440 2023-10-11T16:34:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.799+0000 7f2ae27ff700 1 -- 172.21.15.111:0/521019365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2adc110240 con 0x7f2adc101440 2023-10-11T16:34:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.804+0000 7f2ad97fa700 1 -- 172.21.15.111:0/521019365 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2acc0105a0 con 0x7f2adc101440 2023-10-11T16:34:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.805+0000 7f2af60edb80 1 Processor -- start 2023-10-11T16:34:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.806+0000 7f2af60edb80 1 -- start start 2023-10-11T16:34:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.806+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d02d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.806+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d027560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.806+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d027b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.806+0000 7f2af60edb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4cf2d000 con 0x555f4d02d720 2023-10-11T16:34:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.806+0000 7f2af60edb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4cf7f840 con 0x555f4ce836b0 2023-10-11T16:34:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.806+0000 7f2af60edb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4cf78cd0 con 0x555f4d02db90 2023-10-11T16:34:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.807+0000 7f2adb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d027b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.807+0000 7f2adb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d027b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:55862/0 (socket says 172.21.15.111:55862) 2023-10-11T16:34:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.807+0000 7f2adb7fe700 1 -- 172.21.15.111:0/149363643 learned_addr learned my addr 172.21.15.111:0/149363643 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.807+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d02d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.807+0000 7f2adb7fe700 1 -- 172.21.15.111:0/149363643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 msgr2=0x555f4d02d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.807+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d02d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.807+0000 7f2adb7fe700 1 -- 172.21.15.111:0/149363643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 msgr2=0x555f4d027560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:34:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.808+0000 7f2adbfff700 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d027560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.808+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d027560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.808+0000 7f2adb7fe700 1 -- 172.21.15.111:0/149363643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f4cdfa5d0 con 0x555f4d02db90 2023-10-11T16:34:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.808+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d02d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.808+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d027b00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000aa10 tx=0x7f2ad000ad20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df5b040b2364836c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.809+0000 7f2adbfff700 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d027560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:34:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.809+0000 7f2ac27fc700 1 -- 172.21.15.111:0/149363643 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000ca90 con 0x555f4d02db90 2023-10-11T16:34:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.809+0000 7f2ad97fa700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ac40a1340 0x7f2ac40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.809+0000 7f2ac27fc700 1 -- 172.21.15.111:0/149363643 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ad000cbf0 con 0x555f4d02db90 2023-10-11T16:34:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.809+0000 7f2ac27fc700 1 -- 172.21.15.111:0/149363643 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad00185c0 con 0x555f4d02db90 2023-10-11T16:34:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.809+0000 7f2ad97fa700 1 -- 172.21.15.111:0/521019365 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2acc0ccb10 con 0x7f2adc101440 2023-10-11T16:34:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.810+0000 7f2af60edb80 1 -- 172.21.15.111:0/149363643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 msgr2=0x555f4d027b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.810+0000 7f2af60edb80 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d027b00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000aa10 tx=0x7f2ad000ad20 comp rx=0 tx=0).stop 2023-10-11T16:34:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.810+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ac40a1340 0x7f2ac40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.810+0000 7f2af60edb80 1 -- 172.21.15.111:0/149363643 shutdown_connections 2023-10-11T16:34:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.810+0000 7f2af60edb80 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d027b00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.810+0000 7f2af60edb80 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d02d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.810+0000 7f2af60edb80 1 --2- 172.21.15.111:0/149363643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d027560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.810+0000 7f2af60edb80 1 -- 172.21.15.111:0/149363643 >> 172.21.15.111:0/149363643 conn(0x555f4d01cf50 msgr2=0x555f4d022270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.811+0000 7f2af60edb80 1 -- 172.21.15.111:0/149363643 shutdown_connections 2023-10-11T16:34:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.812+0000 7f2af60edb80 1 -- 172.21.15.111:0/149363643 wait complete. 2023-10-11T16:34:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.812+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ac40a1340 0x7f2ac40a37f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400b070 tx=0x7f2ad400b7d0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.812+0000 7f2af60edb80 1 -- start start 2023-10-11T16:34:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.813+0000 7f2af60edb80 1 Processor -- start 2023-10-11T16:34:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.814+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d033a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.814+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02d720 0x555f4d030c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.814+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02db90 0x555f4d0312c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.814+0000 7f2af60edb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4cf1a880 con 0x555f4d02db90 2023-10-11T16:34:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.814+0000 7f2af60edb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4cf1a520 con 0x555f4ce836b0 2023-10-11T16:34:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.814+0000 7f2af60edb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4cdfc8c0 con 0x555f4d02d720 2023-10-11T16:34:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.814+0000 7f2af60edb80 10 client.0 _ll_register_callbacks cb 0x555f4cf38720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.815+0000 7f2ae0d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02db90 0x555f4d0312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.815+0000 7f2ae0d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02db90 0x555f4d0312c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36212/0 (socket says 172.21.15.111:36212) 2023-10-11T16:34:53.819 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.815+0000 7f2ae0d9c700 1 -- 172.21.15.111:0/31706335 learned_addr learned my addr 172.21.15.111:0/31706335 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.815+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02d720 0x555f4d030c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.816+0000 7f2ae0d9c700 1 -- 172.21.15.111:0/31706335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02d720 msgr2=0x555f4d030c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.816+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02d720 0x555f4d030c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.816+0000 7f2ae0d9c700 1 -- 172.21.15.111:0/31706335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 msgr2=0x555f4d033a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.816+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d033a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.816+0000 7f2ae0d9c700 1 -- 172.21.15.111:0/31706335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad000a640 con 0x555f4d02db90 2023-10-11T16:34:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.817+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02db90 0x555f4d0312c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2ad40092c0 tx=0x7f2ad400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99ff468e2522321 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.817+0000 7f2abb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:34:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.817+0000 7f2abb7fe700 1 -- 172.21.15.111:0/31706335 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad400e040 con 0x555f4d02db90 2023-10-11T16:34:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.817+0000 7f2abb7fe700 1 -- 172.21.15.111:0/31706335 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2ad4024040 con 0x555f4d02db90 2023-10-11T16:34:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.818+0000 7f2af60edb80 1 -- 172.21.15.111:0/31706335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f4d01a710 con 0x555f4d02db90 2023-10-11T16:34:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.818+0000 7f2abb7fe700 1 -- 172.21.15.111:0/31706335 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ad401cce0 con 0x555f4d02db90 2023-10-11T16:34:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.818+0000 7f2af60edb80 10 client.16053 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.818+0000 7f2af60edb80 1 -- 172.21.15.111:0/31706335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555f4cf3f130 con 0x555f4d02db90 2023-10-11T16:34:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.819+0000 7f2abb7fe700 1 -- 172.21.15.111:0/31706335 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555f4d01a710 con 0x555f4d02db90 2023-10-11T16:34:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.820+0000 7f2abb7fe700 1 -- 172.21.15.111:0/31706335 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ad40293f0 con 0x555f4d02db90 2023-10-11T16:34:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.820+0000 7f2af60edb80 10 client.16053 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.820+0000 7f2af60edb80 10 client.16053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.820+0000 7f2af60edb80 20 client.16053 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.820+0000 7f2af60edb80 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555f4d040d20 0x555f4d043100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.820+0000 7f2af60edb80 4 client.16053 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:34:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.820+0000 7f2af60edb80 1 -- 172.21.15.111:0/31706335 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f4cf3b040 con 0x555f4d040d20 2023-10-11T16:34:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.821+0000 7f2adbfff700 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555f4d040d20 0x555f4d043100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.822+0000 7f2adbfff700 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555f4d040d20 0x555f4d043100 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dff85f1ed67bd230 server_cookie=a20e472c268eca23 in_seq=0 out_seq=0 2023-10-11T16:34:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.822+0000 7f2abb7fe700 10 client.16053 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.823+0000 7f2abb7fe700 1 -- 172.21.15.111:0/31706335 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555f4cf3b040 con 0x555f4d040d20 2023-10-11T16:34:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.823+0000 7f2abb7fe700 10 client.16053 handle_command_reply: tid=0 2023-10-11T16:34:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.823+0000 7f2af60edb80 1 client.16053 shutdown 2023-10-11T16:34:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.823+0000 7f2af60edb80 1 -- 172.21.15.111:0/31706335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02db90 msgr2=0x555f4d0312c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.823+0000 7f2af60edb80 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02db90 0x555f4d0312c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2ad40092c0 tx=0x7f2ad400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.824+0000 7f2af60edb80 1 -- 172.21.15.111:0/31706335 shutdown_connections 2023-10-11T16:34:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.824+0000 7f2af60edb80 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555f4d040d20 0x555f4d043100 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.824+0000 7f2af60edb80 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02db90 0x555f4d0312c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.824+0000 7f2af60edb80 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d033a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.824+0000 7f2af60edb80 1 --2- 172.21.15.111:0/31706335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02d720 0x555f4d030c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.824+0000 7f2af60edb80 1 -- 172.21.15.111:0/31706335 >> 172.21.15.111:0/31706335 conn(0x555f4d01cf50 msgr2=0x555f4d0217e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:53.828 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.824+0000 7f2abb7fe700 0 client.16053 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.824+0000 7f2af60edb80 1 -- 172.21.15.111:0/31706335 shutdown_connections 2023-10-11T16:34:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.825+0000 7f2af60edb80 1 -- 172.21.15.111:0/31706335 wait complete. 2023-10-11T16:34:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.825+0000 7f2af60edb80 20 client.16053 trim_cache size 0 max 16384 2023-10-11T16:34:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.847+0000 7f2af60edb80 1 Processor -- start 2023-10-11T16:34:53.848 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.847+0000 7f2af60edb80 1 -- start start 2023-10-11T16:34:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.848+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d038090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.848+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.848+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d021f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.848+0000 7f2af60edb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4d023410 con 0x555f4d02d720 2023-10-11T16:34:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.848+0000 7f2af60edb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4d02f1f0 con 0x555f4ce836b0 2023-10-11T16:34:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.848+0000 7f2af60edb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4d02b940 con 0x555f4d02db90 2023-10-11T16:34:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.848+0000 7f2adbfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d038090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adbfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d038090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:53456/0 (socket says 172.21.15.111:53456) 2023-10-11T16:34:53.851 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adbfff700 1 -- 172.21.15.111:0/3675438954 learned_addr learned my addr 172.21.15.111:0/3675438954 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d021f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adbfff700 1 -- 172.21.15.111:0/3675438954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 msgr2=0x555f4d021f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adbfff700 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d021f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adbfff700 1 -- 172.21.15.111:0/3675438954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 msgr2=0x555f4d0335b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adbfff700 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0335b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adbfff700 1 -- 172.21.15.111:0/3675438954 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f4d030800 con 0x555f4ce836b0 2023-10-11T16:34:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d021f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:34:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.849+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0335b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:53.853 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.850+0000 7f2adbfff700 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d038090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2acc096350 tx=0x7f2acc0b9040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16f14f651d61fb61 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:53.854 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.850+0000 7f2ac2ffd700 1 -- 172.21.15.111:0/3675438954 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc09d910 con 0x555f4ce836b0 2023-10-11T16:34:53.854 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.850+0000 7f2ac2ffd700 1 -- 172.21.15.111:0/3675438954 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2acc0a1d30 con 0x555f4ce836b0 2023-10-11T16:34:53.854 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.850+0000 7f2ac2ffd700 1 -- 172.21.15.111:0/3675438954 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc099b40 con 0x555f4ce836b0 2023-10-11T16:34:53.854 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 -- 172.21.15.111:0/3675438954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 msgr2=0x555f4d038090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:53.855 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d038090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2acc096350 tx=0x7f2acc0b9040 comp rx=0 tx=0).stop 2023-10-11T16:34:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 -- 172.21.15.111:0/3675438954 shutdown_connections 2023-10-11T16:34:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4d02db90 0x555f4d021f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4ce836b0 0x555f4d038090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 --2- 172.21.15.111:0/3675438954 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0335b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 -- 172.21.15.111:0/3675438954 >> 172.21.15.111:0/3675438954 conn(0x555f4d01cf50 msgr2=0x555f4d030c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 -- 172.21.15.111:0/3675438954 shutdown_connections 2023-10-11T16:34:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.851+0000 7f2af60edb80 1 -- 172.21.15.111:0/3675438954 wait complete. 2023-10-11T16:34:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.852+0000 7f2af60edb80 1 -- start start 2023-10-11T16:34:53.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.852+0000 7f2af60edb80 1 Processor -- start 2023-10-11T16:34:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4ce836b0 0x555f4d028c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0291a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2af60edb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4d02db90 0x555f4d0296e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2af60edb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4d029e20 con 0x555f4d02d720 2023-10-11T16:34:53.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2af60edb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4d029f90 con 0x555f4d02db90 2023-10-11T16:34:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2af60edb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4d02a150 con 0x555f4ce836b0 2023-10-11T16:34:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2af60edb80 10 client.0 _ll_register_callbacks cb 0x7f2ad4004d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:34:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2ae0d9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4d02db90 0x555f4d0296e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.853+0000 7f2ae0d9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4d02db90 0x555f4d0296e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:53472/0 (socket says 172.21.15.111:53472) 2023-10-11T16:34:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2ae0d9c700 1 -- 172.21.15.111:0/1435773122 learned_addr learned my addr 172.21.15.111:0/1435773122 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:34:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2adbfff700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4ce836b0 0x555f4d028c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0291a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2ae0d9c700 1 -- 172.21.15.111:0/1435773122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4ce836b0 msgr2=0x555f4d028c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4ce836b0 0x555f4d028c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2ae0d9c700 1 -- 172.21.15.111:0/1435773122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 msgr2=0x555f4d0291a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:34:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0291a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2ae0d9c700 1 -- 172.21.15.111:0/1435773122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f4cdfa5d0 con 0x555f4d02db90 2023-10-11T16:34:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.854+0000 7f2adb7fe700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0291a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:34:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.855+0000 7f2adbfff700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4ce836b0 0x555f4d028c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:34:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.855+0000 7f2ae0d9c700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4d02db90 0x555f4d0296e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4009a20 tx=0x7f2ad401cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bedb8f9785f51ae7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:34:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.855+0000 7f2abb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:34:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.855+0000 7f2abb7fe700 1 -- 172.21.15.111:0/1435773122 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad400e040 con 0x555f4d02db90 2023-10-11T16:34:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.855+0000 7f2af60edb80 1 -- 172.21.15.111:0/1435773122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ad000cbf0 con 0x555f4d02db90 2023-10-11T16:34:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.855+0000 7f2abb7fe700 1 -- 172.21.15.111:0/1435773122 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2ad4029980 con 0x555f4d02db90 2023-10-11T16:34:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.856+0000 7f2abb7fe700 1 -- 172.21.15.111:0/1435773122 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ad40092f0 con 0x555f4d02db90 2023-10-11T16:34:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.857+0000 7f2af60edb80 10 client.26102 fetch_fsmap learned FSMap version 28 2023-10-11T16:34:53.864 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.857+0000 7f2af60edb80 1 -- 172.21.15.111:0/1435773122 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f2adc110240 con 0x555f4d02db90 2023-10-11T16:34:53.864 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.857+0000 7f2abb7fe700 1 -- 172.21.15.111:0/1435773122 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ad000cbf0 con 0x555f4d02db90 2023-10-11T16:34:53.864 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.858+0000 7f2abb7fe700 1 -- 172.21.15.111:0/1435773122 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ad40052e0 con 0x555f4d02db90 2023-10-11T16:34:53.864 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.858+0000 7f2af60edb80 10 client.26102 fetch_fsmap finished waiting for FSMap version 28 2023-10-11T16:34:53.865 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.858+0000 7f2af60edb80 10 client.26102 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:34:53.865 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.858+0000 7f2af60edb80 20 client.26102 populate_metadata read hostname 'smithi111' 2023-10-11T16:34:53.865 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.858+0000 7f2af60edb80 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555f4d095450 0x555f4d097860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:34:53.865 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.858+0000 7f2af60edb80 4 client.26102 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:34:53.865 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.858+0000 7f2af60edb80 1 -- 172.21.15.111:0/1435773122 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555f4cf47600 con 0x555f4d095450 2023-10-11T16:34:53.866 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.859+0000 7f2adbfff700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555f4d095450 0x555f4d097860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:34:53.866 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.860+0000 7f2abb7fe700 10 client.26102 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:53.866 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.860+0000 7f2adbfff700 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555f4d095450 0x555f4d097860 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cca3548abb8d5d2 server_cookie=93b1d84fb31bda2b in_seq=0 out_seq=0 2023-10-11T16:34:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.870+0000 7f2abb7fe700 1 -- 172.21.15.111:0/1435773122 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555f4cf47600 con 0x555f4d095450 2023-10-11T16:34:53.870 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.870+0000 7f2abb7fe700 10 client.26102 handle_command_reply: tid=0 2023-10-11T16:34:53.871 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.870+0000 7f2af60edb80 1 client.26102 shutdown 2023-10-11T16:34:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.870+0000 7f2af60edb80 1 -- 172.21.15.111:0/1435773122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4d02db90 msgr2=0x555f4d0296e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.870+0000 7f2af60edb80 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4d02db90 0x555f4d0296e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4009a20 tx=0x7f2ad401cd10 comp rx=0 tx=0).stop 2023-10-11T16:34:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.871+0000 7f2af60edb80 1 -- 172.21.15.111:0/1435773122 shutdown_connections 2023-10-11T16:34:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.871+0000 7f2af60edb80 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555f4d095450 0x555f4d097860 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.872 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.871+0000 7f2af60edb80 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555f4d02db90 0x555f4d0296e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.871+0000 7f2af60edb80 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555f4ce836b0 0x555f4d028c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.871+0000 7f2af60edb80 1 --2- 172.21.15.111:0/1435773122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555f4d02d720 0x555f4d0291a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.873 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.871+0000 7f2af60edb80 1 -- 172.21.15.111:0/1435773122 >> 172.21.15.111:0/1435773122 conn(0x555f4d01cf50 msgr2=0x555f4d02df90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.872+0000 7f2abb7fe700 0 client.26102 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:34:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.872+0000 7f2af60edb80 1 -- 172.21.15.111:0/1435773122 shutdown_connections 2023-10-11T16:34:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.872+0000 7f2af60edb80 1 -- 172.21.15.111:0/1435773122 wait complete. 2023-10-11T16:34:53.874 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.873+0000 7f2af60edb80 20 client.26102 trim_cache size 0 max 16384 2023-10-11T16:34:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:34:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:34:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:34:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:34:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 -- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ac40a1340 msgr2=0x7f2ac40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ac40a1340 0x7f2ac40a37f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400b070 tx=0x7f2ad400b7d0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.876 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 -- 172.21.15.111:0/521019365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc101440 msgr2=0x7f2adc1964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:34:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc101440 0x7f2adc1964b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00b530 tx=0x7f2acc00b8f0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 -- 172.21.15.111:0/521019365 shutdown_connections 2023-10-11T16:34:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ac40a1340 0x7f2ac40a37f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2adc10a810 0x7f2adc19ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2adc101440 0x7f2adc1964b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 --2- 172.21.15.111:0/521019365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2adc101d50 0x7f2adc1969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:34:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.874+0000 7f2ae27ff700 1 -- 172.21.15.111:0/521019365 >> 172.21.15.111:0/521019365 conn(0x7f2adc0763a0 msgr2=0x7f2adc10b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:34:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.875+0000 7f2ae27ff700 1 -- 172.21.15.111:0/521019365 shutdown_connections 2023-10-11T16:34:53.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:34:53.875+0000 7f2ae27ff700 1 -- 172.21.15.111:0/521019365 wait complete. 2023-10-11T16:34:55.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:55 smithi130 ceph-mon[131859]: pgmap v71: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 314 KiB/s wr, 23 op/s 2023-10-11T16:34:55.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:55 smithi111 ceph-mon[98527]: pgmap v71: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 314 KiB/s wr, 23 op/s 2023-10-11T16:34:55.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:55 smithi150 ceph-mon[132217]: pgmap v71: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 314 KiB/s wr, 23 op/s 2023-10-11T16:34:57.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:57 smithi130 ceph-mon[131859]: pgmap v72: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 305 KiB/s wr, 18 op/s 2023-10-11T16:34:57.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:57 smithi111 ceph-mon[98527]: pgmap v72: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 305 KiB/s wr, 18 op/s 2023-10-11T16:34:57.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:57 smithi150 ceph-mon[132217]: pgmap v72: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 305 KiB/s wr, 18 op/s 2023-10-11T16:34:58.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:58 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:34:58.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:58 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:34:58.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:58 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:34:59.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:34:59 smithi130 ceph-mon[131859]: pgmap v73: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 305 KiB/s wr, 18 op/s 2023-10-11T16:34:59.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:34:59 smithi111 ceph-mon[98527]: pgmap v73: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 305 KiB/s wr, 18 op/s 2023-10-11T16:34:59.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:34:59 smithi150 ceph-mon[132217]: pgmap v73: 129 pgs: 129 active+clean; 5.4 MiB data, 92 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 305 KiB/s wr, 18 op/s 2023-10-11T16:35:01.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:01 smithi130 ceph-mon[131859]: pgmap v74: 129 pgs: 129 active+clean; 16 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 78 op/s 2023-10-11T16:35:01.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:01 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:35:01.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:01 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:35:01.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:01 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:35:01.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:01 smithi111 ceph-mon[98527]: pgmap v74: 129 pgs: 129 active+clean; 16 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 78 op/s 2023-10-11T16:35:01.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:01 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:35:01.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:01 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:35:01.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:01 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:35:01.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:01 smithi150 ceph-mon[132217]: pgmap v74: 129 pgs: 129 active+clean; 16 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 78 op/s 2023-10-11T16:35:01.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:01 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:35:01.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:01 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:35:01.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:01 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:35:02.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:02 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:02.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:02 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:02.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:02 smithi150 ceph-mon[132217]: pgmap v75: 129 pgs: 129 active+clean; 16 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 75 op/s 2023-10-11T16:35:02.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:02 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:02.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:02 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:03.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:02 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:02 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:02 smithi130 ceph-mon[131859]: pgmap v75: 129 pgs: 129 active+clean; 16 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 75 op/s 2023-10-11T16:35:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:02 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:02 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:03.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:02 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:02 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:02 smithi111 ceph-mon[98527]: pgmap v75: 129 pgs: 129 active+clean; 16 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 75 op/s 2023-10-11T16:35:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:02 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:03.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:02 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:03 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:03 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:35:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:03 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:35:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:03 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:35:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:03 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:03 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:35:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:03 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:35:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:03 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:35:04.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:03 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:03 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:35:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:03 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:35:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:03 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:35:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:03 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:03 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:35:04.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:03 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:35:04.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:03 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:35:04.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:03 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:04.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:03 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:35:04.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:03 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:35:04.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:03 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:35:04.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:03 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:35:04.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:03 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:35:04.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:03 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:35:04.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:03 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:35:05.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:05 smithi130 ceph-mon[131859]: pgmap v76: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 105 op/s 2023-10-11T16:35:05.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:05 smithi111 ceph-mon[98527]: pgmap v76: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 105 op/s 2023-10-11T16:35:05.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:05 smithi150 ceph-mon[132217]: pgmap v76: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 105 op/s 2023-10-11T16:35:07.075 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.074+0000 7f265b9ff700 1 -- 172.21.15.111:0/3200346516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26540ff230 msgr2=0x7f26540ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.074+0000 7f265b9ff700 1 --2- 172.21.15.111:0/3200346516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26540ff230 0x7f26540ff600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f265001a9d0 tx=0x7f265001ace0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.075+0000 7f265b9ff700 1 -- 172.21.15.111:0/3200346516 shutdown_connections 2023-10-11T16:35:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.075+0000 7f265b9ff700 1 --2- 172.21.15.111:0/3200346516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2654104630 0x7f2654104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.075+0000 7f265b9ff700 1 --2- 172.21.15.111:0/3200346516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26540ff230 0x7f26540ff600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.076 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.075+0000 7f265b9ff700 1 --2- 172.21.15.111:0/3200346516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26540ffb40 0x7f2654104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.075+0000 7f265b9ff700 1 -- 172.21.15.111:0/3200346516 >> 172.21.15.111:0/3200346516 conn(0x7f26540763a0 msgr2=0x7f26540767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.075+0000 7f265b9ff700 1 -- 172.21.15.111:0/3200346516 shutdown_connections 2023-10-11T16:35:07.077 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.075+0000 7f265b9ff700 1 -- 172.21.15.111:0/3200346516 wait complete. 2023-10-11T16:35:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.076+0000 7f265b9ff700 1 Processor -- start 2023-10-11T16:35:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.077+0000 7f265b9ff700 1 -- start start 2023-10-11T16:35:07.078 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.077+0000 7f265b9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26540ff230 0x7f265410f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.077+0000 7f265b9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26540ffb40 0x7f265410f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.077+0000 7f265b9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2654104630 0x7f265419ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.077+0000 7f265b9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26541a0470 con 0x7f26540ffb40 2023-10-11T16:35:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.077+0000 7f265b9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26541a05e0 con 0x7f26540ff230 2023-10-11T16:35:07.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.077+0000 7f265b9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26541a07a0 con 0x7f2654104630 2023-10-11T16:35:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2658f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26540ffb40 0x7f265410f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2658f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26540ffb40 0x7f265410f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57858/0 (socket says 172.21.15.111:57858) 2023-10-11T16:35:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2658f9a700 1 -- 172.21.15.111:0/1818320402 learned_addr learned my addr 172.21.15.111:0/1818320402 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2659f9c700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2654104630 0x7f265419ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2658f9a700 1 -- 172.21.15.111:0/1818320402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2654104630 msgr2=0x7f265419ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2654104630 0x7f265419ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2658f9a700 1 -- 172.21.15.111:0/1818320402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26540ff230 msgr2=0x7f265410f140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:35:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26540ff230 0x7f265410f140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.078+0000 7f2658f9a700 1 -- 172.21.15.111:0/1818320402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265001a600 con 0x7f26540ffb40 2023-10-11T16:35:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.079+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26540ffb40 0x7f265410f680 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f264400b530 tx=0x7f264400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:07.083 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.079+0000 7f264a7fc700 1 -- 172.21.15.111:0/1818320402 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644011830 con 0x7f26540ffb40 2023-10-11T16:35:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.079+0000 7f264a7fc700 1 -- 172.21.15.111:0/1818320402 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2644011e50 con 0x7f26540ffb40 2023-10-11T16:35:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.079+0000 7f264a7fc700 1 -- 172.21.15.111:0/1818320402 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264400f2e0 con 0x7f26540ffb40 2023-10-11T16:35:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.079+0000 7f265b9ff700 1 -- 172.21.15.111:0/1818320402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2654107db0 con 0x7f26540ffb40 2023-10-11T16:35:07.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.079+0000 7f265b9ff700 1 -- 172.21.15.111:0/1818320402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2654107c20 con 0x7f26540ffb40 2023-10-11T16:35:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.084+0000 7f264a7fc700 1 -- 172.21.15.111:0/1818320402 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2644010380 con 0x7f26540ffb40 2023-10-11T16:35:07.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.085+0000 7f266f22bb80 1 Processor -- start 2023-10-11T16:35:07.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.085+0000 7f266f22bb80 1 -- start start 2023-10-11T16:35:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f264a7fc700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f26400a12c0 0x7f26400a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f991121e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112720 0x563f9910c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f264a7fc700 1 -- 172.21.15.111:0/1818320402 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f26440ccb10 con 0x7f26540ffb40 2023-10-11T16:35:07.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112b90 0x563f9910cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f266f22bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f99012000 con 0x563f99112720 2023-10-11T16:35:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f266f22bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f99064840 con 0x563f98f686b0 2023-10-11T16:35:07.088 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f266f22bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9905dcd0 con 0x563f99112b90 2023-10-11T16:35:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.086+0000 7f265979b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f991121e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f2658f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112720 0x563f9910c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f265979b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f991121e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54974/0 (socket says 172.21.15.111:54974) 2023-10-11T16:35:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f265979b700 1 -- 172.21.15.111:0/1614332164 learned_addr learned my addr 172.21.15.111:0/1614332164 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f2659f9c700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f26400a12c0 0x7f26400a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f2658f9a700 1 -- 172.21.15.111:0/1614332164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112b90 msgr2=0x563f9910cb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:35:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1614332164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112b90 0x563f9910cb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f2658f9a700 1 -- 172.21.15.111:0/1614332164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 msgr2=0x563f991121e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1614332164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f991121e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.087+0000 7f2658f9a700 1 -- 172.21.15.111:0/1614332164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f98edf5d0 con 0x563f99112720 2023-10-11T16:35:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.088+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1614332164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112720 0x563f9910c560 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f26440c9fc0 tx=0x7f2644004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b8355f5e90088f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.088+0000 7f263f7fe700 1 -- 172.21.15.111:0/1614332164 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440a1d90 con 0x563f99112720 2023-10-11T16:35:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.088+0000 7f263f7fe700 1 -- 172.21.15.111:0/1614332164 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2644084070 con 0x563f99112720 2023-10-11T16:35:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.088+0000 7f263f7fe700 1 -- 172.21.15.111:0/1614332164 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644099090 con 0x563f99112720 2023-10-11T16:35:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.089+0000 7f266f22bb80 1 -- 172.21.15.111:0/1614332164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112720 msgr2=0x563f9910c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.089+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1614332164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112720 0x563f9910c560 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f26440c9fc0 tx=0x7f2644004860 comp rx=0 tx=0).stop 2023-10-11T16:35:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.089+0000 7f2659f9c700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f26400a12c0 0x7f26400a3770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f264c00f410 tx=0x7f264c007950 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.089+0000 7f266f22bb80 1 -- 172.21.15.111:0/1614332164 shutdown_connections 2023-10-11T16:35:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.089+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1614332164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112b90 0x563f9910cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.089+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1614332164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f991121e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.089+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1614332164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112720 0x563f9910c560 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.089+0000 7f266f22bb80 1 -- 172.21.15.111:0/1614332164 >> 172.21.15.111:0/1614332164 conn(0x563f99101f50 msgr2=0x563f99107270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:07.097 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.090+0000 7f266f22bb80 1 -- 172.21.15.111:0/1614332164 shutdown_connections 2023-10-11T16:35:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.090+0000 7f266f22bb80 1 -- 172.21.15.111:0/1614332164 wait complete. 2023-10-11T16:35:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.091+0000 7f266f22bb80 1 -- start start 2023-10-11T16:35:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.091+0000 7f266f22bb80 1 Processor -- start 2023-10-11T16:35:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f99115e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.098 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99116360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f991168a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f266f22bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f98fff880 con 0x563f99112b90 2023-10-11T16:35:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f266f22bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f98fff520 con 0x563f98f686b0 2023-10-11T16:35:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f266f22bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f98ee18c0 con 0x563f99112720 2023-10-11T16:35:07.099 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f266f22bb80 10 client.0 _ll_register_callbacks cb 0x563f9901d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f265979b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f99115e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f265979b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f99115e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54986/0 (socket says 172.21.15.111:54986) 2023-10-11T16:35:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f265979b700 1 -- 172.21.15.111:0/1282835180 learned_addr learned my addr 172.21.15.111:0/1282835180 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99116360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.100 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.092+0000 7f2659f9c700 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f991168a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.093+0000 7f265979b700 1 -- 172.21.15.111:0/1282835180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 msgr2=0x563f99116360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.093+0000 7f265979b700 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99116360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.093+0000 7f265979b700 1 -- 172.21.15.111:0/1282835180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 msgr2=0x563f991168a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.093+0000 7f265979b700 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f991168a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.101 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.093+0000 7f265979b700 1 -- 172.21.15.111:0/1282835180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2654107c20 con 0x563f98f686b0 2023-10-11T16:35:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.093+0000 7f2659f9c700 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f991168a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.093+0000 7f265979b700 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f99115e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f265001cc30 tx=0x7f265001cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e94c19a6e123a0b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.093+0000 7f263cff9700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:07.102 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.094+0000 7f263cff9700 1 -- 172.21.15.111:0/1282835180 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650005430 con 0x563f98f686b0 2023-10-11T16:35:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.094+0000 7f266f22bb80 1 -- 172.21.15.111:0/1282835180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f990ff710 con 0x563f98f686b0 2023-10-11T16:35:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.095+0000 7f263cff9700 1 -- 172.21.15.111:0/1282835180 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2650005590 con 0x563f98f686b0 2023-10-11T16:35:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.095+0000 7f263cff9700 1 -- 172.21.15.111:0/1282835180 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26500038a0 con 0x563f98f686b0 2023-10-11T16:35:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.095+0000 7f266f22bb80 10 client.26108 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:07.103 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.095+0000 7f266f22bb80 1 -- 172.21.15.111:0/1282835180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563f99024130 con 0x563f98f686b0 2023-10-11T16:35:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.095+0000 7f263cff9700 1 -- 172.21.15.111:0/1282835180 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563f990ff710 con 0x563f98f686b0 2023-10-11T16:35:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.096+0000 7f263cff9700 1 -- 172.21.15.111:0/1282835180 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f26500349b0 con 0x563f98f686b0 2023-10-11T16:35:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.097+0000 7f266f22bb80 10 client.26108 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.097+0000 7f266f22bb80 10 client.26108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.097+0000 7f266f22bb80 20 client.26108 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.097+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563f991260c0 0x563f991284a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.097+0000 7f266f22bb80 4 client.26108 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.097+0000 7f266f22bb80 1 -- 172.21.15.111:0/1282835180 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f99020040 con 0x563f991260c0 2023-10-11T16:35:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.097+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563f991260c0 0x563f991284a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.098+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563f991260c0 0x563f991284a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffd427e883c39ce8 server_cookie=c69ee3a0b86c4a1e in_seq=0 out_seq=0 2023-10-11T16:35:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.098+0000 7f263cff9700 10 client.26108 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.098+0000 7f263cff9700 1 -- 172.21.15.111:0/1282835180 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563f99020040 con 0x563f991260c0 2023-10-11T16:35:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.098+0000 7f263cff9700 10 client.26108 handle_command_reply: tid=0 2023-10-11T16:35:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.098+0000 7f266f22bb80 1 client.26108 shutdown 2023-10-11T16:35:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.099+0000 7f266f22bb80 1 -- 172.21.15.111:0/1282835180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 msgr2=0x563f99115e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.099+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f99115e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f265001cc30 tx=0x7f265001cc60 comp rx=0 tx=0).stop 2023-10-11T16:35:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.100+0000 7f266f22bb80 1 -- 172.21.15.111:0/1282835180 shutdown_connections 2023-10-11T16:35:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.100+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563f991260c0 0x563f991284a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.100+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f991168a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.100+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f98f686b0 0x563f99115e20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.109 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.100+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1282835180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99116360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.109 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.100+0000 7f266f22bb80 1 -- 172.21.15.111:0/1282835180 >> 172.21.15.111:0/1282835180 conn(0x563f99101f50 msgr2=0x563f991067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:07.109 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.101+0000 7f263cff9700 0 client.26108 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:07.109 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.101+0000 7f266f22bb80 1 -- 172.21.15.111:0/1282835180 shutdown_connections 2023-10-11T16:35:07.109 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.101+0000 7f266f22bb80 1 -- 172.21.15.111:0/1282835180 wait complete. 2023-10-11T16:35:07.109 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.102+0000 7f266f22bb80 20 client.26108 trim_cache size 0 max 16384 2023-10-11T16:35:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.123+0000 7f266f22bb80 1 Processor -- start 2023-10-11T16:35:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.123+0000 7f266f22bb80 1 -- start start 2023-10-11T16:35:07.124 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.123+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f98f686b0 0x563f9911d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.123+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99106f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112b90 0x563f9910c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f266f22bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f99101b70 con 0x563f98f686b0 2023-10-11T16:35:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f266f22bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f991038e0 con 0x563f99112b90 2023-10-11T16:35:07.125 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f266f22bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f991143a0 con 0x563f99112720 2023-10-11T16:35:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f265979b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f98f686b0 0x563f9911d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f2658f9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99106f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f265979b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f98f686b0 0x563f9911d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57880/0 (socket says 172.21.15.111:57880) 2023-10-11T16:35:07.127 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f265979b700 1 -- 172.21.15.111:0/1469210088 learned_addr learned my addr 172.21.15.111:0/1469210088 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f2659f9c700 1 --2- 172.21.15.111:0/1469210088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112b90 0x563f9910c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f2658f9a700 1 -- 172.21.15.111:0/1469210088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112b90 msgr2=0x563f9910c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1469210088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112b90 0x563f9910c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f2658f9a700 1 -- 172.21.15.111:0/1469210088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f98f686b0 msgr2=0x563f9911d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:07.128 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1469210088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f98f686b0 0x563f9911d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.124+0000 7f2658f9a700 1 -- 172.21.15.111:0/1469210088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f99023e80 con 0x563f99112720 2023-10-11T16:35:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.125+0000 7f2658f9a700 1 --2- 172.21.15.111:0/1469210088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99106f60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2644010900 tx=0x7f2644099710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=455b6745524576af server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.126+0000 7f263ffff700 1 -- 172.21.15.111:0/1469210088 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440b7ad0 con 0x563f99112720 2023-10-11T16:35:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.126+0000 7f263ffff700 1 -- 172.21.15.111:0/1469210088 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26440b7c30 con 0x563f99112720 2023-10-11T16:35:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.126+0000 7f263ffff700 1 -- 172.21.15.111:0/1469210088 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644099e00 con 0x563f99112720 2023-10-11T16:35:07.131 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.127+0000 7f266f22bb80 1 -- 172.21.15.111:0/1469210088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 msgr2=0x563f99106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:07.131 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.128+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1469210088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99106f60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2644010900 tx=0x7f2644099710 comp rx=0 tx=0).stop 2023-10-11T16:35:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.128+0000 7f266f22bb80 1 -- 172.21.15.111:0/1469210088 shutdown_connections 2023-10-11T16:35:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.128+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1469210088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112b90 0x563f9910c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.128+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1469210088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f98f686b0 0x563f9911d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.128+0000 7f266f22bb80 1 --2- 172.21.15.111:0/1469210088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f99112720 0x563f99106f60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.128+0000 7f266f22bb80 1 -- 172.21.15.111:0/1469210088 >> 172.21.15.111:0/1469210088 conn(0x563f99101f50 msgr2=0x563f9910ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.128+0000 7f266f22bb80 1 -- 172.21.15.111:0/1469210088 shutdown_connections 2023-10-11T16:35:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.128+0000 7f266f22bb80 1 -- 172.21.15.111:0/1469210088 wait complete. 2023-10-11T16:35:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.129+0000 7f266f22bb80 1 -- start start 2023-10-11T16:35:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.129+0000 7f266f22bb80 1 Processor -- start 2023-10-11T16:35:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f98f686b0 0x563f9911d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112720 0x563f9911af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f266f22bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f9911b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f266f22bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9911bb50 con 0x563f99112b90 2023-10-11T16:35:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f266f22bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f9911bcc0 con 0x563f99112720 2023-10-11T16:35:07.135 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f266f22bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f99178f20 con 0x563f98f686b0 2023-10-11T16:35:07.135 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f266f22bb80 10 client.0 _ll_register_callbacks cb 0x7f265001a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:07.135 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f265979b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f98f686b0 0x563f9911d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.135 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f265979b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f98f686b0 0x563f9911d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:37570/0 (socket says 172.21.15.111:37570) 2023-10-11T16:35:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f265979b700 1 -- 172.21.15.111:0/2919246118 learned_addr learned my addr 172.21.15.111:0/2919246118 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.130+0000 7f2659f9c700 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f9911b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f2658f9a700 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112720 0x563f9911af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f265979b700 1 -- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112720 msgr2=0x563f9911af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f265979b700 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112720 0x563f9911af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f265979b700 1 -- 172.21.15.111:0/2919246118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 msgr2=0x563f9911b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f265979b700 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f9911b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f265979b700 1 -- 172.21.15.111:0/2919246118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f98edf5d0 con 0x563f98f686b0 2023-10-11T16:35:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f2658f9a700 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112720 0x563f9911af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f265979b700 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f98f686b0 0x563f9911d180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2650005e20 tx=0x7f26500283f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cd1404cc8661799 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.131+0000 7f263cff9700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:35:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.132+0000 7f263cff9700 1 -- 172.21.15.111:0/2919246118 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650003ab0 con 0x563f98f686b0 2023-10-11T16:35:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.132+0000 7f263cff9700 1 -- 172.21.15.111:0/2919246118 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2650000e40 con 0x563f98f686b0 2023-10-11T16:35:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.132+0000 7f266f22bb80 1 -- 172.21.15.111:0/2919246118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2644084070 con 0x563f98f686b0 2023-10-11T16:35:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.133+0000 7f263cff9700 1 -- 172.21.15.111:0/2919246118 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2650003c10 con 0x563f98f686b0 2023-10-11T16:35:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.133+0000 7f266f22bb80 10 client.26158 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.133+0000 7f266f22bb80 1 -- 172.21.15.111:0/2919246118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563f98e74c00 con 0x563f98f686b0 2023-10-11T16:35:07.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.134+0000 7f263cff9700 1 -- 172.21.15.111:0/2919246118 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2644084070 con 0x563f98f686b0 2023-10-11T16:35:07.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.134+0000 7f263cff9700 1 -- 172.21.15.111:0/2919246118 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f2650028660 con 0x563f98f686b0 2023-10-11T16:35:07.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.134+0000 7f266f22bb80 10 client.26158 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:07.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.134+0000 7f266f22bb80 10 client.26158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:07.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.134+0000 7f266f22bb80 20 client.26158 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:07.142 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.134+0000 7f266f22bb80 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563f9917a080 0x563f9917c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:07.142 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.134+0000 7f266f22bb80 4 client.26158 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:35:07.142 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.134+0000 7f266f22bb80 1 -- 172.21.15.111:0/2919246118 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563f9902c600 con 0x563f9917a080 2023-10-11T16:35:07.142 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.135+0000 7f2658f9a700 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563f9917a080 0x563f9917c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:07.142 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.135+0000 7f263cff9700 10 client.26158 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:07.143 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.135+0000 7f2658f9a700 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563f9917a080 0x563f9917c460 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c0752f7d99448af server_cookie=63e9f70c9b0e7dd8 in_seq=0 out_seq=0 2023-10-11T16:35:07.146 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f263cff9700 1 -- 172.21.15.111:0/2919246118 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563f9902c600 con 0x563f9917a080 2023-10-11T16:35:07.146 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f263cff9700 10 client.26158 handle_command_reply: tid=0 2023-10-11T16:35:07.146 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 client.26158 shutdown 2023-10-11T16:35:07.147 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 -- 172.21.15.111:0/2919246118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f98f686b0 msgr2=0x563f9911d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f98f686b0 0x563f9911d180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2650005e20 tx=0x7f26500283f0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 -- 172.21.15.111:0/2919246118 shutdown_connections 2023-10-11T16:35:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563f9917a080 0x563f9917c460 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563f99112b90 0x563f9911b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.148 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563f98f686b0 0x563f9911d180 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2650005e20 tx=0x7f26500283f0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 --2- 172.21.15.111:0/2919246118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f99112720 0x563f9911af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f266f22bb80 1 -- 172.21.15.111:0/2919246118 >> 172.21.15.111:0/2919246118 conn(0x563f99101f50 msgr2=0x563f99110ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.146+0000 7f263cff9700 0 client.26158 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.147+0000 7f266f22bb80 1 -- 172.21.15.111:0/2919246118 shutdown_connections 2023-10-11T16:35:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:35:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:35:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:35:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:35:07.151 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.147+0000 7f266f22bb80 1 -- 172.21.15.111:0/2919246118 wait complete. 2023-10-11T16:35:07.151 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.147+0000 7f266f22bb80 20 client.26158 trim_cache size 0 max 16384 2023-10-11T16:35:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 -- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f26400a12c0 msgr2=0x7f26400a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f26400a12c0 0x7f26400a3770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f264c00f410 tx=0x7f264c007950 comp rx=0 tx=0).stop 2023-10-11T16:35:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 -- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26540ffb40 msgr2=0x7f265410f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26540ffb40 0x7f265410f680 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f264400b530 tx=0x7f264400b840 comp rx=0 tx=0).stop 2023-10-11T16:35:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 -- 172.21.15.111:0/1818320402 shutdown_connections 2023-10-11T16:35:07.153 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f26400a12c0 0x7f26400a3770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.153 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2654104630 0x7f265419ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.153 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26540ff230 0x7f265410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.153 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 --2- 172.21.15.111:0/1818320402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26540ffb40 0x7f265410f680 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:07.153 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 -- 172.21.15.111:0/1818320402 >> 172.21.15.111:0/1818320402 conn(0x7f26540763a0 msgr2=0x7f2654101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:07.154 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.148+0000 7f265b9ff700 1 -- 172.21.15.111:0/1818320402 shutdown_connections 2023-10-11T16:35:07.154 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:07.149+0000 7f265b9ff700 1 -- 172.21.15.111:0/1818320402 wait complete. 2023-10-11T16:35:07.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:07 smithi130 ceph-mon[131859]: pgmap v77: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 90 op/s 2023-10-11T16:35:07.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:07 smithi111 ceph-mon[98527]: pgmap v77: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 90 op/s 2023-10-11T16:35:07.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:07 smithi150 ceph-mon[132217]: pgmap v77: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 90 op/s 2023-10-11T16:35:09.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:09 smithi130 ceph-mon[131859]: pgmap v78: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 90 op/s 2023-10-11T16:35:09.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:09 smithi111 ceph-mon[98527]: pgmap v78: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 90 op/s 2023-10-11T16:35:09.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:09 smithi150 ceph-mon[132217]: pgmap v78: 129 pgs: 129 active+clean; 33 MiB data, 124 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 90 op/s 2023-10-11T16:35:11.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:11 smithi130 ceph-mon[131859]: pgmap v79: 129 pgs: 129 active+clean; 46 MiB data, 135 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 101 op/s 2023-10-11T16:35:11.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:11 smithi111 ceph-mon[98527]: pgmap v79: 129 pgs: 129 active+clean; 46 MiB data, 135 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 101 op/s 2023-10-11T16:35:11.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:11 smithi150 ceph-mon[132217]: pgmap v79: 129 pgs: 129 active+clean; 46 MiB data, 135 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 101 op/s 2023-10-11T16:35:13.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:13 smithi130 ceph-mon[131859]: pgmap v80: 129 pgs: 129 active+clean; 46 MiB data, 135 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 41 op/s 2023-10-11T16:35:13.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:13 smithi111 ceph-mon[98527]: pgmap v80: 129 pgs: 129 active+clean; 46 MiB data, 135 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 41 op/s 2023-10-11T16:35:13.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:13 smithi150 ceph-mon[132217]: pgmap v80: 129 pgs: 129 active+clean; 46 MiB data, 135 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 41 op/s 2023-10-11T16:35:15.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:15 smithi130 ceph-mon[131859]: pgmap v81: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 62 op/s 2023-10-11T16:35:15.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:15 smithi111 ceph-mon[98527]: pgmap v81: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 62 op/s 2023-10-11T16:35:15.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:15 smithi150 ceph-mon[132217]: pgmap v81: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 62 op/s 2023-10-11T16:35:17.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:17 smithi130 ceph-mon[131859]: pgmap v82: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 32 op/s 2023-10-11T16:35:17.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:17 smithi111 ceph-mon[98527]: pgmap v82: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 32 op/s 2023-10-11T16:35:17.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:17 smithi150 ceph-mon[132217]: pgmap v82: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 32 op/s 2023-10-11T16:35:19.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:19 smithi111 ceph-mon[98527]: pgmap v83: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 32 op/s 2023-10-11T16:35:19.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:19 smithi150 ceph-mon[132217]: pgmap v83: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 32 op/s 2023-10-11T16:35:19.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:19 smithi130 ceph-mon[131859]: pgmap v83: 129 pgs: 129 active+clean; 62 MiB data, 152 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 32 op/s 2023-10-11T16:35:20.355 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/1783664630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c2810a810 msgr2=0x7f8c2810cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:20.355 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/1783664630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c2810a810 0x7f8c2810cc30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c20009f90 tx=0x7f8c2000b330 comp rx=0 tx=0).stop 2023-10-11T16:35:20.355 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/1783664630 shutdown_connections 2023-10-11T16:35:20.356 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/1783664630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c2810a810 0x7f8c2810cc30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.356 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/1783664630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c28101440 0x7f8c28101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.356 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/1783664630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c28101d50 0x7f8c2810a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.356 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/1783664630 >> 172.21.15.111:0/1783664630 conn(0x7f8c280763a0 msgr2=0x7f8c280767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:20.356 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/1783664630 shutdown_connections 2023-10-11T16:35:20.357 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.353+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/1783664630 wait complete. 2023-10-11T16:35:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.354+0000 7f8c2d9ff700 1 Processor -- start 2023-10-11T16:35:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c2d9ff700 1 -- start start 2023-10-11T16:35:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c2d9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c28101440 0x7f8c281985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c2d9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c28101d50 0x7f8c28198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c2d9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c2810a810 0x7f8c2819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c2d9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2819d5d0 con 0x7f8c2810a810 2023-10-11T16:35:20.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c2d9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2819d740 con 0x7f8c28101d50 2023-10-11T16:35:20.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c2d9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2819d900 con 0x7f8c28101440 2023-10-11T16:35:20.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c28101440 0x7f8c281985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c28101440 0x7f8c281985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36024/0 (socket says 172.21.15.111:36024) 2023-10-11T16:35:20.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c26ffd700 1 -- 172.21.15.111:0/4139970704 learned_addr learned my addr 172.21.15.111:0/4139970704 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:20.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c267fc700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c28101d50 0x7f8c28198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.355+0000 7f8c277fe700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c2810a810 0x7f8c2819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c26ffd700 1 -- 172.21.15.111:0/4139970704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c28101d50 msgr2=0x7f8c28198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:20.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c28101d50 0x7f8c28198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c26ffd700 1 -- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c2810a810 msgr2=0x7f8c2819cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:20.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c2810a810 0x7f8c2819cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c26ffd700 1 -- 172.21.15.111:0/4139970704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c20009c40 con 0x7f8c28101440 2023-10-11T16:35:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c277fe700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c2810a810 0x7f8c2819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c28101440 0x7f8c281985f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800c9b0 tx=0x7f8c1800cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c2c9fd700 1 -- 172.21.15.111:0/4139970704 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18007b00 con 0x7f8c28101440 2023-10-11T16:35:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/4139970704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c281102b0 con 0x7f8c28101440 2023-10-11T16:35:20.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c2c9fd700 1 -- 172.21.15.111:0/4139970704 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c18007c60 con 0x7f8c28101440 2023-10-11T16:35:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c2c9fd700 1 -- 172.21.15.111:0/4139970704 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18018660 con 0x7f8c28101440 2023-10-11T16:35:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.356+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/4139970704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c28110120 con 0x7f8c28101440 2023-10-11T16:35:20.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.362+0000 7f8c2c9fd700 1 -- 172.21.15.111:0/4139970704 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f8c1800f450 con 0x7f8c28101440 2023-10-11T16:35:20.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.364+0000 7f8c413a4b80 1 Processor -- start 2023-10-11T16:35:20.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.364+0000 7f8c413a4b80 1 -- start start 2023-10-11T16:35:20.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.364+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9c9bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.364+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9ca05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.364+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c910 0x55a3d9ca0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9b99000 con 0x55a3d9aef6b0 2023-10-11T16:35:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9beb840 con 0x55a3d9c9c4a0 2023-10-11T16:35:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9be4cd0 con 0x55a3d9c9c910 2023-10-11T16:35:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9c9bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c2c9fd700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8c100a13c0 0x7f8c100a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9c9bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56298/0 (socket says 172.21.15.111:56298) 2023-10-11T16:35:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c277fe700 1 -- 172.21.15.111:0/3895195992 learned_addr learned my addr 172.21.15.111:0/3895195992 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c2c9fd700 1 -- 172.21.15.111:0/4139970704 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f8c180cd0a0 con 0x7f8c28101440 2023-10-11T16:35:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.365+0000 7f8c277fe700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8c100a13c0 0x7f8c100a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.366+0000 7f8c277fe700 1 -- 172.21.15.111:0/3895195992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c910 msgr2=0x55a3d9ca0b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:35:20.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.366+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3895195992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c910 0x55a3d9ca0b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.366+0000 7f8c277fe700 1 -- 172.21.15.111:0/3895195992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 msgr2=0x55a3d9ca05f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:35:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.366+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3895195992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9ca05f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.366+0000 7f8c277fe700 1 -- 172.21.15.111:0/3895195992 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3d9a665d0 con 0x55a3d9aef6b0 2023-10-11T16:35:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.366+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3895195992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9c9bf60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8c20009790 tx=0x7f8c200074a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943f5aa6c5b60ce2 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:20.372 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.366+0000 7f8c277fe700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8c100a13c0 0x7f8c100a3870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c200075c0 tx=0x7f8c20025000 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:20.372 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.366+0000 7f8c0dffb700 1 -- 172.21.15.111:0/3895195992 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000e040 con 0x55a3d9aef6b0 2023-10-11T16:35:20.372 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.367+0000 7f8c0dffb700 1 -- 172.21.15.111:0/3895195992 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c200092e0 con 0x55a3d9aef6b0 2023-10-11T16:35:20.372 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.367+0000 7f8c0dffb700 1 -- 172.21.15.111:0/3895195992 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20005940 con 0x55a3d9aef6b0 2023-10-11T16:35:20.373 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.368+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3895195992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 msgr2=0x55a3d9c9bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:20.373 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.368+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3895195992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9c9bf60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8c20009790 tx=0x7f8c200074a0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.373 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.369+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3895195992 shutdown_connections 2023-10-11T16:35:20.373 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.369+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3895195992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c910 0x55a3d9ca0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.373 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.369+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3895195992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9c9bf60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.374 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.369+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3895195992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9ca05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.374 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.369+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3895195992 >> 172.21.15.111:0/3895195992 conn(0x55a3d9c88f50 msgr2=0x55a3d9c8e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:20.374 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.369+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3895195992 shutdown_connections 2023-10-11T16:35:20.374 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.370+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3895195992 wait complete. 2023-10-11T16:35:20.374 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.370+0000 7f8c413a4b80 1 -- start start 2023-10-11T16:35:20.375 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.371+0000 7f8c413a4b80 1 Processor -- start 2023-10-11T16:35:20.376 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.371+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cab2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.376 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.371+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.376 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.371+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c910 0x55a3d9ca5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.371+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9b86880 con 0x55a3d9c9c910 2023-10-11T16:35:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.371+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9b86520 con 0x55a3d9c9c4a0 2023-10-11T16:35:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.371+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9a688c0 con 0x55a3d9aef6b0 2023-10-11T16:35:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.371+0000 7f8c413a4b80 10 client.0 _ll_register_callbacks cb 0x55a3d9ba4720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:20.377 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c267fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cab2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c267fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cab7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33284/0 (socket says 172.21.15.111:33284) 2023-10-11T16:35:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c267fc700 1 -- 172.21.15.111:0/3131625832 learned_addr learned my addr 172.21.15.111:0/3131625832 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c277fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c910 0x55a3d9ca5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.378 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c277fe700 1 -- 172.21.15.111:0/3131625832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 msgr2=0x55a3d9cab2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cab2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c277fe700 1 -- 172.21.15.111:0/3131625832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 msgr2=0x55a3d9cab7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cab7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.372+0000 7f8c277fe700 1 -- 172.21.15.111:0/3131625832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c2810d8f0 con 0x55a3d9c9c910 2023-10-11T16:35:20.379 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.373+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c910 0x55a3d9ca5540 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8c20008fd0 tx=0x7f8c20009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b28d263dd76ac7a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.373+0000 7f8c06ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:35:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.373+0000 7f8c06ffd700 1 -- 172.21.15.111:0/3131625832 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000e040 con 0x55a3d9c9c910 2023-10-11T16:35:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.373+0000 7f8c06ffd700 1 -- 172.21.15.111:0/3131625832 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f8c200057d0 con 0x55a3d9c9c910 2023-10-11T16:35:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.373+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3131625832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a3d9c86710 con 0x55a3d9c9c910 2023-10-11T16:35:20.380 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.374+0000 7f8c06ffd700 1 -- 172.21.15.111:0/3131625832 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c2000b680 con 0x55a3d9c9c910 2023-10-11T16:35:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.378+0000 7f8c413a4b80 10 client.16083 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.378+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3131625832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a3d9bab130 con 0x55a3d9c9c910 2023-10-11T16:35:20.381 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.378+0000 7f8c06ffd700 1 -- 172.21.15.111:0/3131625832 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a3d9c86710 con 0x55a3d9c9c910 2023-10-11T16:35:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.378+0000 7f8c06ffd700 1 -- 172.21.15.111:0/3131625832 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f8c2002f410 con 0x55a3d9c9c910 2023-10-11T16:35:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.378+0000 7f8c413a4b80 10 client.16083 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.379+0000 7f8c413a4b80 10 client.16083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.379+0000 7f8c413a4b80 20 client.16083 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:20.382 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.379+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a3d9cb5500 0x55a3d9cb7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.383 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.379+0000 7f8c413a4b80 4 client.16083 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:20.383 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.379+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3131625832 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a3d9ba7040 con 0x55a3d9cb5500 2023-10-11T16:35:20.383 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.380+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a3d9cb5500 0x55a3d9cb7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.384 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.380+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a3d9cb5500 0x55a3d9cb7910 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e2a575ce2f445aa server_cookie=6a88586a90a9c3ce in_seq=0 out_seq=0 2023-10-11T16:35:20.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.380+0000 7f8c06ffd700 10 client.16083 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:20.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.381+0000 7f8c06ffd700 1 -- 172.21.15.111:0/3131625832 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a3d9ba7040 con 0x55a3d9cb5500 2023-10-11T16:35:20.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.381+0000 7f8c06ffd700 10 client.16083 handle_command_reply: tid=0 2023-10-11T16:35:20.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.381+0000 7f8c413a4b80 1 client.16083 shutdown 2023-10-11T16:35:20.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3131625832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c910 msgr2=0x55a3d9ca5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:20.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c910 0x55a3d9ca5540 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8c20008fd0 tx=0x7f8c20009030 comp rx=0 tx=0).stop 2023-10-11T16:35:20.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3131625832 shutdown_connections 2023-10-11T16:35:20.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a3d9cb5500 0x55a3d9cb7910 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c910 0x55a3d9ca5540 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cab2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3131625832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cab7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3131625832 >> 172.21.15.111:0/3131625832 conn(0x55a3d9c88f50 msgr2=0x55a3d9ca1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:20.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c06ffd700 0 client.16083 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:20.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3131625832 shutdown_connections 2023-10-11T16:35:20.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3131625832 wait complete. 2023-10-11T16:35:20.388 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.382+0000 7f8c413a4b80 20 client.16083 trim_cache size 0 max 16384 2023-10-11T16:35:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.405+0000 7f8c413a4b80 1 Processor -- start 2023-10-11T16:35:20.407 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.405+0000 7f8c413a4b80 1 -- start start 2023-10-11T16:35:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9ca54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.408 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9c9e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9c9e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9ca38f0 con 0x55a3d9c9c4a0 2023-10-11T16:35:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9c9ce70 con 0x55a3d9c9c910 2023-10-11T16:35:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9ca5a80 con 0x55a3d9aef6b0 2023-10-11T16:35:20.409 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9ca54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9ca54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36046/0 (socket says 172.21.15.111:36046) 2023-10-11T16:35:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c26ffd700 1 -- 172.21.15.111:0/3207077302 learned_addr learned my addr 172.21.15.111:0/3207077302 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c277fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9c9e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.406+0000 7f8c267fc700 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9c9e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.410 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.407+0000 7f8c277fe700 1 -- 172.21.15.111:0/3207077302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 msgr2=0x55a3d9ca54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.407+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9ca54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.407+0000 7f8c277fe700 1 -- 172.21.15.111:0/3207077302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c4a0 msgr2=0x55a3d9c9e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.407+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9c9e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.407+0000 7f8c277fe700 1 -- 172.21.15.111:0/3207077302 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3d9baae80 con 0x55a3d9c9c910 2023-10-11T16:35:20.411 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.407+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9ca54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:20.412 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.407+0000 7f8c267fc700 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9c9e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:20.413 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.407+0000 7f8c277fe700 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9c9e960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8c2002f870 tx=0x7f8c2000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3edf9e9b1ace2f3e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.408+0000 7f8c0e7fc700 1 -- 172.21.15.111:0/3207077302 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000e040 con 0x55a3d9c9c910 2023-10-11T16:35:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.408+0000 7f8c0e7fc700 1 -- 172.21.15.111:0/3207077302 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c20007db0 con 0x55a3d9c9c910 2023-10-11T16:35:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c0e7fc700 1 -- 172.21.15.111:0/3207077302 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c200059d0 con 0x55a3d9c9c910 2023-10-11T16:35:20.414 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3207077302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 msgr2=0x55a3d9c9e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9c9e960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8c2002f870 tx=0x7f8c2000b330 comp rx=0 tx=0).stop 2023-10-11T16:35:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3207077302 shutdown_connections 2023-10-11T16:35:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9c9e960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9aef6b0 0x55a3d9ca54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.415 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/3207077302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9c9e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3207077302 >> 172.21.15.111:0/3207077302 conn(0x55a3d9c88f50 msgr2=0x55a3d9ca4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:20.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3207077302 shutdown_connections 2023-10-11T16:35:20.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.409+0000 7f8c413a4b80 1 -- 172.21.15.111:0/3207077302 wait complete. 2023-10-11T16:35:20.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.410+0000 7f8c413a4b80 1 -- start start 2023-10-11T16:35:20.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.410+0000 7f8c413a4b80 1 Processor -- start 2023-10-11T16:35:20.416 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.411+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cfd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.411+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cfd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.411+0000 7f8c413a4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9d08390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.411+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9cfdfa0 con 0x55a3d9aef6b0 2023-10-11T16:35:20.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.411+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9cfe110 con 0x55a3d9c9c910 2023-10-11T16:35:20.417 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.411+0000 7f8c413a4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3d9c93560 con 0x55a3d9c9c4a0 2023-10-11T16:35:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.411+0000 7f8c413a4b80 10 client.0 _ll_register_callbacks cb 0x7f8c1c0091e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.412+0000 7f8c267fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cfd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.412+0000 7f8c277fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9d08390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.412+0000 7f8c277fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9d08390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33304/0 (socket says 172.21.15.111:33304) 2023-10-11T16:35:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.412+0000 7f8c277fe700 1 -- 172.21.15.111:0/1129650840 learned_addr learned my addr 172.21.15.111:0/1129650840 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:20.419 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.412+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cfd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.420 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c277fe700 1 -- 172.21.15.111:0/1129650840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c4a0 msgr2=0x55a3d9cfd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:20.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c277fe700 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cfd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c277fe700 1 -- 172.21.15.111:0/1129650840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 msgr2=0x55a3d9cfd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:20.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c277fe700 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cfd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:20.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c277fe700 1 -- 172.21.15.111:0/1129650840 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c28110120 con 0x55a3d9c9c910 2023-10-11T16:35:20.421 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c267fc700 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cfd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cfd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c277fe700 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9d08390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c2002f770 tx=0x7f8c2000b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f3477e57e86d327 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.413+0000 7f8c06ffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.414+0000 7f8c06ffd700 1 -- 172.21.15.111:0/1129650840 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c200059e0 con 0x55a3d9c9c910 2023-10-11T16:35:20.422 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.414+0000 7f8c06ffd700 1 -- 172.21.15.111:0/1129650840 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f8c2003e450 con 0x55a3d9c9c910 2023-10-11T16:35:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.414+0000 7f8c413a4b80 1 -- 172.21.15.111:0/1129650840 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c200092e0 con 0x55a3d9c9c910 2023-10-11T16:35:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.414+0000 7f8c06ffd700 1 -- 172.21.15.111:0/1129650840 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c2003e610 con 0x55a3d9c9c910 2023-10-11T16:35:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.414+0000 7f8c413a4b80 10 client.26129 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.414+0000 7f8c413a4b80 1 -- 172.21.15.111:0/1129650840 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f8c1c0099e0 con 0x55a3d9c9c910 2023-10-11T16:35:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c06ffd700 1 -- 172.21.15.111:0/1129650840 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c200092e0 con 0x55a3d9c9c910 2023-10-11T16:35:20.423 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c06ffd700 1 -- 172.21.15.111:0/1129650840 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f8c20050070 con 0x55a3d9c9c910 2023-10-11T16:35:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c413a4b80 10 client.26129 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c413a4b80 10 client.26129 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c413a4b80 20 client.26129 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a3d9c94770 0x55a3d9c96b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:20.424 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c413a4b80 4 client.26129 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:35:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c413a4b80 1 -- 172.21.15.111:0/1129650840 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a3d9bb3600 con 0x55a3d9c94770 2023-10-11T16:35:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.415+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a3d9c94770 0x55a3d9c96b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.416+0000 7f8c26ffd700 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a3d9c94770 0x55a3d9c96b80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f22a073801db88b9 server_cookie=2a11a2402f2054f3 in_seq=0 out_seq=0 2023-10-11T16:35:20.425 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:20.416+0000 7f8c06ffd700 10 client.26129 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:21.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:21 smithi111 ceph-mon[98527]: pgmap v84: 129 pgs: 129 active+clean; 78 MiB data, 174 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 57 op/s 2023-10-11T16:35:21.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:21 smithi150 ceph-mon[132217]: pgmap v84: 129 pgs: 129 active+clean; 78 MiB data, 174 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 57 op/s 2023-10-11T16:35:21.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:21 smithi130 ceph-mon[131859]: pgmap v84: 129 pgs: 129 active+clean; 78 MiB data, 174 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 57 op/s 2023-10-11T16:35:22.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:35:22.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.974+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/3989589632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095ff0 msgr2=0x7fdcb0099fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.974+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/3989589632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095ff0 0x7fdcb0099fb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009990 tx=0x7fdca4009ca0 comp rx=0 tx=0).stop 2023-10-11T16:35:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.978+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/3989589632 shutdown_connections 2023-10-11T16:35:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.978+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/3989589632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb009a690 0x7fdcb00a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.978+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/3989589632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcb0095650 0x7fdcb0095a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:22.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.978+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/3989589632 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095ff0 0x7fdcb0099fb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:22.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.978+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/3989589632 >> 172.21.15.111:0/3989589632 conn(0x7fdcb000c1b0 msgr2=0x7fdcb000c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:22.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.979+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/3989589632 shutdown_connections 2023-10-11T16:35:22.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.979+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/3989589632 wait complete. 2023-10-11T16:35:22.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.981+0000 7fdcbcf9d700 1 Processor -- start 2023-10-11T16:35:22.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.982+0000 7fdcbcf9d700 1 -- start start 2023-10-11T16:35:22.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcbcf9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095650 0x7fdcb013c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:22.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcbcf9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb0095ff0 0x7fdcb013cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:22.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcbcf9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcb009a690 0x7fdcb0141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:22.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcbcf9d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb0141660 con 0x7fdcb009a690 2023-10-11T16:35:22.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcbcf9d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb01417d0 con 0x7fdcb0095650 2023-10-11T16:35:22.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcbcf9d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb0141990 con 0x7fdcb0095ff0 2023-10-11T16:35:22.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcb77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095650 0x7fdcb013c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:22.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcb77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095650 0x7fdcb013c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33326/0 (socket says 172.21.15.111:33326) 2023-10-11T16:35:22.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcb77fe700 1 -- 172.21.15.111:0/2151060289 learned_addr learned my addr 172.21.15.111:0/2151060289 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:22.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb0095ff0 0x7fdcb013cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:22.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcb009a690 0x7fdcb0141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:22.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcb77fe700 1 -- 172.21.15.111:0/2151060289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb0095ff0 msgr2=0x7fdcb013cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:22.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb0095ff0 0x7fdcb013cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.983+0000 7fdcb77fe700 1 -- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcb009a690 msgr2=0x7fdcb0141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.984+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcb009a690 0x7fdcb0141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.984+0000 7fdcb77fe700 1 -- 172.21.15.111:0/2151060289 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40095f0 con 0x7fdcb0095650 2023-10-11T16:35:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.984+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcb009a690 0x7fdcb0141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.985+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095650 0x7fdcb013c680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00e9a0 tx=0x7fdcac00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:22.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.986+0000 7fdcb4ff9700 1 -- 172.21.15.111:0/2151060289 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac009790 con 0x7fdcb0095650 2023-10-11T16:35:22.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.986+0000 7fdcb4ff9700 1 -- 172.21.15.111:0/2151060289 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcac00cbd0 con 0x7fdcb0095650 2023-10-11T16:35:22.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.986+0000 7fdcb4ff9700 1 -- 172.21.15.111:0/2151060289 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac018680 con 0x7fdcb0095650 2023-10-11T16:35:22.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.987+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/2151060289 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb00a5500 con 0x7fdcb0095650 2023-10-11T16:35:22.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:22.987+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/2151060289 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb00a5370 con 0x7fdcb0095650 2023-10-11T16:35:23.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.002+0000 7fdcd1b09b80 1 Processor -- start 2023-10-11T16:35:23.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.004+0000 7fdcd1b09b80 1 -- start start 2023-10-11T16:35:23.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.004+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:23.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.004+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dd2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:23.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.004+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dd36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:23.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.004+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469ccb060 con 0x565469dceed0 2023-10-11T16:35:23.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.004+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469d1d8a0 con 0x565469dce5c0 2023-10-11T16:35:23.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.005+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469d16d30 con 0x565469dd3280 2023-10-11T16:35:23.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.005+0000 7fdcb6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dd36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:23.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.005+0000 7fdcb6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dd36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36072/0 (socket says 172.21.15.111:36072) 2023-10-11T16:35:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.005+0000 7fdcb6ffd700 1 -- 172.21.15.111:0/1928570597 learned_addr learned my addr 172.21.15.111:0/1928570597 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.006+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.006+0000 7fdcb4ff9700 1 -- 172.21.15.111:0/2151060289 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fdcac00cd40 con 0x7fdcb0095650 2023-10-11T16:35:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.006+0000 7fdcb7fff700 1 -- 172.21.15.111:0/1928570597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 msgr2=0x565469dd36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.006+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dd36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:23.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.006+0000 7fdcb7fff700 1 -- 172.21.15.111:0/1928570597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 msgr2=0x565469dd2d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:35:23.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.008+0000 7fdcb4ff9700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a1340 0x7fdca80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:23.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.009+0000 7fdcb4ff9700 1 -- 172.21.15.111:0/2151060289 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fdcac014070 con 0x7fdcb0095650 2023-10-11T16:35:23.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.009+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a1340 0x7fdca80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:23.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.010+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dd2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:23.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.010+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dd2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.010+0000 7fdcb7fff700 1 -- 172.21.15.111:0/1928570597 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565469b98630 con 0x565469dce5c0 2023-10-11T16:35:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.010+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dce990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8066770 tx=0x7fdcb80718c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7486513c6a9e119 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.011+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a1340 0x7fdca80a37f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdca400b2e0 tx=0x7fdca4005ec0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:23.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.011+0000 7fdca1ffb700 1 -- 172.21.15.111:0/1928570597 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb806d8e0 con 0x565469dce5c0 2023-10-11T16:35:23.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.011+0000 7fdca1ffb700 1 -- 172.21.15.111:0/1928570597 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcb806f3f0 con 0x565469dce5c0 2023-10-11T16:35:23.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.011+0000 7fdca1ffb700 1 -- 172.21.15.111:0/1928570597 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8079600 con 0x565469dce5c0 2023-10-11T16:35:23.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.012+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1928570597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 msgr2=0x565469dce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:23.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.012+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dce990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8066770 tx=0x7fdcb80718c0 comp rx=0 tx=0).stop 2023-10-11T16:35:23.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.012+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1928570597 shutdown_connections 2023-10-11T16:35:23.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.012+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dd36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:23.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.012+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dce990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:23.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.012+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1928570597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dd2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:23.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.012+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1928570597 >> 172.21.15.111:0/1928570597 conn(0x565469dbb5b0 msgr2=0x565469dc0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:23.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.013+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1928570597 shutdown_connections 2023-10-11T16:35:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.013+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1928570597 wait complete. 2023-10-11T16:35:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.014+0000 7fdcd1b09b80 1 -- start start 2023-10-11T16:35:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.014+0000 7fdcd1b09b80 1 Processor -- start 2023-10-11T16:35:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.015+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469ddaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:23.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.015+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:23.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.015+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dd83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.015+0000 7fdcb77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469ddaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469ddaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36086/0 (socket says 172.21.15.111:36086) 2023-10-11T16:35:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469cb88e0 con 0x565469dd3280 2023-10-11T16:35:23.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469cb8580 con 0x565469dceed0 2023-10-11T16:35:23.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469b9a920 con 0x565469dce5c0 2023-10-11T16:35:23.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb77fe700 1 -- 172.21.15.111:0/1120137538 learned_addr learned my addr 172.21.15.111:0/1120137538 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:23.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:23.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dd83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:23.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcd1b09b80 10 client.0 _ll_register_callbacks cb 0x565469cd6780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:23.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb77fe700 1 -- 172.21.15.111:0/1120137538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 msgr2=0x565469dd7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb77fe700 1 -- 172.21.15.111:0/1120137538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 msgr2=0x565469dd83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dd83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.016+0000 7fdcb77fe700 1 -- 172.21.15.111:0/1120137538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb804f7c0 con 0x565469dce5c0 2023-10-11T16:35:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.017+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469ddaa80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00f010 tx=0x7fdcac09a240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9758a8c61e03d78 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.017+0000 7fdc96ffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:35:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.017+0000 7fdc96ffd700 1 -- 172.21.15.111:0/1120137538 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac09a620 con 0x565469dce5c0 2023-10-11T16:35:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.017+0000 7fdc96ffd700 1 -- 172.21.15.111:0/1120137538 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fdcac0996b0 con 0x565469dce5c0 2023-10-11T16:35:23.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.018+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1120137538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565469db8d70 con 0x565469dce5c0 2023-10-11T16:35:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.019+0000 7fdc96ffd700 1 -- 172.21.15.111:0/1120137538 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcac0cd370 con 0x565469dce5c0 2023-10-11T16:35:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.020+0000 7fdcd1b09b80 10 client.26182 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:23.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.020+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1120137538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x565469cdd190 con 0x565469dce5c0 2023-10-11T16:35:23.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.023+0000 7fdc96ffd700 1 -- 172.21.15.111:0/1120137538 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565469db8d70 con 0x565469dce5c0 2023-10-11T16:35:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.023+0000 7fdc96ffd700 1 -- 172.21.15.111:0/1120137538 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdcac0b2ba0 con 0x565469dce5c0 2023-10-11T16:35:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.024+0000 7fdcd1b09b80 10 client.26182 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.024+0000 7fdcd1b09b80 10 client.26182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:23.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.024+0000 7fdcd1b09b80 20 client.26182 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:23.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.024+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565469df0ac0 0x565469df2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:23.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.024+0000 7fdcd1b09b80 4 client.26182 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:23.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.024+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1120137538 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565469cd90a0 con 0x565469df0ac0 2023-10-11T16:35:23.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.027+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565469df0ac0 0x565469df2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:23.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.030+0000 7fdc96ffd700 10 client.26182 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:23.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:23.030+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565469df0ac0 0x565469df2ed0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b67c89d27aa4c8 server_cookie=7e36cccb86a51f08 in_seq=0 out_seq=0 2023-10-11T16:35:23.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:23 smithi111 ceph-mon[98527]: pgmap v85: 129 pgs: 129 active+clean; 78 MiB data, 174 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 45 op/s 2023-10-11T16:35:23.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:23 smithi150 ceph-mon[132217]: pgmap v85: 129 pgs: 129 active+clean; 78 MiB data, 174 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 45 op/s 2023-10-11T16:35:23.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:23 smithi130 ceph-mon[131859]: pgmap v85: 129 pgs: 129 active+clean; 78 MiB data, 174 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 45 op/s 2023-10-11T16:35:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.282+0000 7f8c06ffd700 1 -- 172.21.15.111:0/1129650840 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a3d9bb3600 con 0x55a3d9c94770 2023-10-11T16:35:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.282+0000 7f8c06ffd700 10 client.26129 handle_command_reply: tid=0 2023-10-11T16:35:24.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdc96ffd700 1 -- 172.21.15.111:0/1120137538 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x565469cd90a0 con 0x565469df0ac0 2023-10-11T16:35:24.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdc96ffd700 10 client.26182 handle_command_reply: tid=0 2023-10-11T16:35:24.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 client.26182 shutdown 2023-10-11T16:35:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 client.26129 shutdown 2023-10-11T16:35:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 -- 172.21.15.111:0/1129650840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 msgr2=0x55a3d9d08390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9d08390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c2002f770 tx=0x7f8c2000b780 comp rx=0 tx=0).stop 2023-10-11T16:35:24.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1120137538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 msgr2=0x565469ddaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469ddaa80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00f010 tx=0x7fdcac09a240 comp rx=0 tx=0).stop 2023-10-11T16:35:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 -- 172.21.15.111:0/1129650840 shutdown_connections 2023-10-11T16:35:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a3d9c94770 0x55a3d9c96b80 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a3d9c9c910 0x55a3d9d08390 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a3d9aef6b0 0x55a3d9cfd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 --2- 172.21.15.111:0/1129650840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a3d9c9c4a0 0x55a3d9cfd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c413a4b80 1 -- 172.21.15.111:0/1129650840 >> 172.21.15.111:0/1129650840 conn(0x55a3d9c88f50 msgr2=0x55a3d9c8cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.283+0000 7f8c06ffd700 0 client.26129 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:24.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1120137538 shutdown_connections 2023-10-11T16:35:24.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565469df0ac0 0x565469df2ed0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dd83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469ddaa80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1120137538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1120137538 >> 172.21.15.111:0/1120137538 conn(0x565469dbb5b0 msgr2=0x565469dbf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:24.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.283+0000 7fdc96ffd700 0 client.26182 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.284+0000 7f8c413a4b80 1 -- 172.21.15.111:0/1129650840 shutdown_connections 2023-10-11T16:35:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.284+0000 7f8c413a4b80 1 -- 172.21.15.111:0/1129650840 wait complete. 2023-10-11T16:35:24.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.284+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1120137538 shutdown_connections 2023-10-11T16:35:24.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.284+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1120137538 wait complete. 2023-10-11T16:35:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.284+0000 7f8c413a4b80 20 client.26129 trim_cache size 0 max 16384 2023-10-11T16:35:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:35:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:35:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:35:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:35:24.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.284+0000 7fdcd1b09b80 20 client.26182 trim_cache size 0 max 16384 2023-10-11T16:35:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.286+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8c100a13c0 msgr2=0x7f8c100a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.286+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8c100a13c0 0x7f8c100a3870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c200075c0 tx=0x7f8c20025000 comp rx=0 tx=0).stop 2023-10-11T16:35:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.286+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/4139970704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c28101440 msgr2=0x7f8c281985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.286+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c28101440 0x7f8c281985f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800c9b0 tx=0x7f8c1800cd70 comp rx=0 tx=0).stop 2023-10-11T16:35:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.287+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/4139970704 shutdown_connections 2023-10-11T16:35:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.287+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8c100a13c0 0x7f8c100a3870 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.287+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c2810a810 0x7f8c2819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.287+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c28101440 0x7f8c281985f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.287+0000 7f8c2d9ff700 1 --2- 172.21.15.111:0/4139970704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c28101d50 0x7f8c28198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.287+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/4139970704 >> 172.21.15.111:0/4139970704 conn(0x7f8c280763a0 msgr2=0x7f8c280ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.287+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/4139970704 shutdown_connections 2023-10-11T16:35:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:24.287+0000 7f8c2d9ff700 1 -- 172.21.15.111:0/4139970704 wait complete. 2023-10-11T16:35:24.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.311+0000 7fdcd1b09b80 1 Processor -- start 2023-10-11T16:35:24.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.311+0000 7fdcd1b09b80 1 -- start start 2023-10-11T16:35:24.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.311+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469dd7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.311+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.311+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dc07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.311+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469dd0ea0 con 0x565469dd3280 2023-10-11T16:35:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.311+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469df08a0 con 0x565469dceed0 2023-10-11T16:35:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.311+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469dbb0e0 con 0x565469dce5c0 2023-10-11T16:35:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb6ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb6ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33352/0 (socket says 172.21.15.111:33352) 2023-10-11T16:35:24.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb6ffd700 1 -- 172.21.15.111:0/1556424779 learned_addr learned my addr 172.21.15.111:0/1556424779 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:24.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469dd7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dc07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb77fe700 1 -- 172.21.15.111:0/1556424779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 msgr2=0x565469dd0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:24.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb77fe700 1 -- 172.21.15.111:0/1556424779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 msgr2=0x565469dc07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dc07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb77fe700 1 -- 172.21.15.111:0/1556424779 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565469dd3920 con 0x565469dce5c0 2023-10-11T16:35:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.312+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dc07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:24.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.313+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469dd7c40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdcac0ca8f0 tx=0x7fdcac099e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=693088213db1f0ae server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:24.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.313+0000 7fdca27fc700 1 -- 172.21.15.111:0/1556424779 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac0998b0 con 0x565469dce5c0 2023-10-11T16:35:24.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.313+0000 7fdca27fc700 1 -- 172.21.15.111:0/1556424779 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcac096500 con 0x565469dce5c0 2023-10-11T16:35:24.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.313+0000 7fdca27fc700 1 -- 172.21.15.111:0/1556424779 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac09a6a0 con 0x565469dce5c0 2023-10-11T16:35:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1556424779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 msgr2=0x565469dd7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469dd7c40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdcac0ca8f0 tx=0x7fdcac099e30 comp rx=0 tx=0).stop 2023-10-11T16:35:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1556424779 shutdown_connections 2023-10-11T16:35:24.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dd3280 0x565469dc07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dce5c0 0x565469dd7c40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/1556424779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dceed0 0x565469dd0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1556424779 >> 172.21.15.111:0/1556424779 conn(0x565469dbb5b0 msgr2=0x565469dd7530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1556424779 shutdown_connections 2023-10-11T16:35:24.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/1556424779 wait complete. 2023-10-11T16:35:24.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.314+0000 7fdcd1b09b80 1 -- start start 2023-10-11T16:35:24.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.315+0000 7fdcd1b09b80 1 Processor -- start 2023-10-11T16:35:24.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.315+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dc9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.315+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dc9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.315+0000 7fdcd1b09b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dc7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.315+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469dc8560 con 0x565469dceed0 2023-10-11T16:35:24.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.315+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469dc86d0 con 0x565469dce5c0 2023-10-11T16:35:24.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.315+0000 7fdcd1b09b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x565469dc8890 con 0x565469dd3280 2023-10-11T16:35:24.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.315+0000 7fdcd1b09b80 10 client.0 _ll_register_callbacks cb 0x7fdcac099bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:24.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dc9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dc9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33358/0 (socket says 172.21.15.111:33358) 2023-10-11T16:35:24.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 -- 172.21.15.111:0/180044664 learned_addr learned my addr 172.21.15.111:0/180044664 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:24.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb7fff700 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dc7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dc9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 -- 172.21.15.111:0/180044664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 msgr2=0x565469dc7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:24.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dc7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 -- 172.21.15.111:0/180044664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 msgr2=0x565469dc9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:24.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dc9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 -- 172.21.15.111:0/180044664 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565469cdd190 con 0x565469dce5c0 2023-10-11T16:35:24.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dc9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:24.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdcb77fe700 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dc9940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c5b0 tx=0x7fdcac09a2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb5d6922f5e72ee server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:24.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.316+0000 7fdc96ffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:24.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdc96ffd700 1 -- 172.21.15.111:0/180044664 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac0998c0 con 0x565469dce5c0 2023-10-11T16:35:24.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/180044664 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcb806f3f0 con 0x565469dce5c0 2023-10-11T16:35:24.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdc96ffd700 1 -- 172.21.15.111:0/180044664 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fdcac095fb0 con 0x565469dce5c0 2023-10-11T16:35:24.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdc96ffd700 1 -- 172.21.15.111:0/180044664 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcac096170 con 0x565469dce5c0 2023-10-11T16:35:24.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 10 client.26147 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:24.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/180044664 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x565469b44de0 con 0x565469dce5c0 2023-10-11T16:35:24.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdc96ffd700 1 -- 172.21.15.111:0/180044664 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcb806f3f0 con 0x565469dce5c0 2023-10-11T16:35:24.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdc96ffd700 1 -- 172.21.15.111:0/180044664 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdcac0c49f0 con 0x565469dce5c0 2023-10-11T16:35:24.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 10 client.26147 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:24.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 10 client.26147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:24.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 20 client.26147 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:24.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565469e455c0 0x565469e479a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 4 client.26147 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:35:24.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.317+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/180044664 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x565469ce5660 con 0x565469e455c0 2023-10-11T16:35:24.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.318+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565469e455c0 0x565469e479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.318+0000 7fdcb6ffd700 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565469e455c0 0x565469e479a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a953170ff6d49580 server_cookie=94ae38fd2188542f in_seq=0 out_seq=0 2023-10-11T16:35:24.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.318+0000 7fdc96ffd700 10 client.26147 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:24.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.319+0000 7fdc96ffd700 1 -- 172.21.15.111:0/180044664 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x565469ce5660 con 0x565469e455c0 2023-10-11T16:35:24.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.319+0000 7fdc96ffd700 10 client.26147 handle_command_reply: tid=0 2023-10-11T16:35:24.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 client.26147 shutdown 2023-10-11T16:35:24.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/180044664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 msgr2=0x565469dc9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dc9940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c5b0 tx=0x7fdcac09a2c0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/180044664 shutdown_connections 2023-10-11T16:35:24.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x565469e455c0 0x565469e479a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x565469dd3280 0x565469dc7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565469dce5c0 0x565469dc9940 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 --2- 172.21.15.111:0/180044664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x565469dceed0 0x565469dc9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.320+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/180044664 >> 172.21.15.111:0/180044664 conn(0x565469dbb5b0 msgr2=0x565469dd2250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:24.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.321+0000 7fdc96ffd700 0 client.26147 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:24.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.321+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/180044664 shutdown_connections 2023-10-11T16:35:24.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.321+0000 7fdcd1b09b80 1 -- 172.21.15.111:0/180044664 wait complete. 2023-10-11T16:35:24.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.321+0000 7fdcd1b09b80 20 client.26147 trim_cache size 0 max 16384 2023-10-11T16:35:24.337 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:35:24.337 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:35:24.337 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:35:24.338 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:35:24.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a1340 msgr2=0x7fdca80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a1340 0x7fdca80a37f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdca400b2e0 tx=0x7fdca4005ec0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/2151060289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095650 msgr2=0x7fdcb013c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095650 0x7fdcb013c680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00e9a0 tx=0x7fdcac00ed60 comp rx=0 tx=0).stop 2023-10-11T16:35:24.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/2151060289 shutdown_connections 2023-10-11T16:35:24.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a1340 0x7fdca80a37f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcb009a690 0x7fdcb0141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcb0095650 0x7fdcb013c680 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 --2- 172.21.15.111:0/2151060289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb0095ff0 0x7fdcb013cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/2151060289 >> 172.21.15.111:0/2151060289 conn(0x7fdcb000c1b0 msgr2=0x7fdcb0098f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:24.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.322+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/2151060289 shutdown_connections 2023-10-11T16:35:24.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.323+0000 7fdcbcf9d700 1 -- 172.21.15.111:0/2151060289 wait complete. 2023-10-11T16:35:24.349 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:35:24.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:35:24.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.611+0000 7f65f119d700 1 -- 172.21.15.111:0/2758458249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec072e90 msgr2=0x7f65ec073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.611+0000 7f65f119d700 1 --2- 172.21.15.111:0/2758458249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec072e90 0x7f65ec073260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0098f0 tx=0x7f65dc009c00 comp rx=0 tx=0).stop 2023-10-11T16:35:24.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.611+0000 7f65f119d700 1 -- 172.21.15.111:0/2758458249 shutdown_connections 2023-10-11T16:35:24.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.611+0000 7f65f119d700 1 --2- 172.21.15.111:0/2758458249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65ec1110d0 0x7f65ec1134b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.611+0000 7f65f119d700 1 --2- 172.21.15.111:0/2758458249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec072e90 0x7f65ec073260 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0098f0 tx=0x7f65dc009c00 comp rx=0 tx=0).stop 2023-10-11T16:35:24.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.611+0000 7f65f119d700 1 --2- 172.21.15.111:0/2758458249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec0737a0 0x7f65ec110b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.611+0000 7f65f119d700 1 -- 172.21.15.111:0/2758458249 >> 172.21.15.111:0/2758458249 conn(0x7f65ec0fbbc0 msgr2=0x7f65ec0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:24.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.612+0000 7f65f119d700 1 -- 172.21.15.111:0/2758458249 shutdown_connections 2023-10-11T16:35:24.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.613+0000 7f65f119d700 1 -- 172.21.15.111:0/2758458249 wait complete. 2023-10-11T16:35:24.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.614+0000 7f65f119d700 1 Processor -- start 2023-10-11T16:35:24.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.614+0000 7f65f119d700 1 -- start start 2023-10-11T16:35:24.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65f119d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec0737a0 0x7f65ec1a2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65f119d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65ec1110d0 0x7f65ec1a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65f119d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec19c870 0x7f65ec19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65f119d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec19d2e0 con 0x7f65ec0737a0 2023-10-11T16:35:24.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65f119d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec19d420 con 0x7f65ec1110d0 2023-10-11T16:35:24.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65f119d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec19d5e0 con 0x7f65ec19c870 2023-10-11T16:35:24.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65eb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65ec1110d0 0x7f65ec1a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65ebfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec0737a0 0x7f65ec1a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65f099c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec19c870 0x7f65ec19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65eb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65ec1110d0 0x7f65ec1a2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33382/0 (socket says 172.21.15.111:33382) 2023-10-11T16:35:24.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.615+0000 7f65eb7fe700 1 -- 172.21.15.111:0/3094822383 learned_addr learned my addr 172.21.15.111:0/3094822383 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:24.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.616+0000 7f65f099c700 1 -- 172.21.15.111:0/3094822383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65ec1110d0 msgr2=0x7f65ec1a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.616+0000 7f65f099c700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65ec1110d0 0x7f65ec1a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.616+0000 7f65f099c700 1 -- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec0737a0 msgr2=0x7f65ec1a2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.616+0000 7f65f099c700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec0737a0 0x7f65ec1a2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.616+0000 7f65f099c700 1 -- 172.21.15.111:0/3094822383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc009520 con 0x7f65ec19c870 2023-10-11T16:35:24.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.616+0000 7f65f099c700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec19c870 0x7f65ec19cd10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f65e400ef90 tx=0x7f65e400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:24.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.617+0000 7f65e97fa700 1 -- 172.21.15.111:0/3094822383 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4018070 con 0x7f65ec19c870 2023-10-11T16:35:24.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.617+0000 7f65e97fa700 1 -- 172.21.15.111:0/3094822383 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65e400f040 con 0x7f65ec19c870 2023-10-11T16:35:24.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.617+0000 7f65e97fa700 1 -- 172.21.15.111:0/3094822383 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e400cce0 con 0x7f65ec19c870 2023-10-11T16:35:24.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.617+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ec116770 con 0x7f65ec19c870 2023-10-11T16:35:24.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.617+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ec1165e0 con 0x7f65ec19c870 2023-10-11T16:35:24.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.619+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ec116900 con 0x7f65ec19c870 2023-10-11T16:35:24.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.621+0000 7f65e97fa700 1 -- 172.21.15.111:0/3094822383 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f65e4003a00 con 0x7f65ec19c870 2023-10-11T16:35:24.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.623+0000 7f65e97fa700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65d40a1340 0x7f65d40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:24.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.623+0000 7f65e97fa700 1 -- 172.21.15.111:0/3094822383 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f65e40ccbb0 con 0x7f65ec19c870 2023-10-11T16:35:24.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.623+0000 7f65ebfff700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65d40a1340 0x7f65d40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:24.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.624+0000 7f65ebfff700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65d40a1340 0x7f65d40a37f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009230 tx=0x7f65dc019040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:24.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.626+0000 7f65e97fa700 1 -- 172.21.15.111:0/3094822383 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f65e4095ca0 con 0x7f65ec19c870 2023-10-11T16:35:24.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.967+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65ec067980 con 0x7f65ec19c870 2023-10-11T16:35:24.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.968+0000 7f65e97fa700 1 -- 172.21.15.111:0/3094822383 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f65e40992c0 con 0x7f65ec19c870 2023-10-11T16:35:24.970 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:35:24.970 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:35:02.750894+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:35:24.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65d40a1340 msgr2=0x7f65d40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65d40a1340 0x7f65d40a37f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009230 tx=0x7f65dc019040 comp rx=0 tx=0).stop 2023-10-11T16:35:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec19c870 msgr2=0x7f65ec19cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec19c870 0x7f65ec19cd10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f65e400ef90 tx=0x7f65e400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 shutdown_connections 2023-10-11T16:35:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f65d40a1340 0x7f65d40a37f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ec19c870 0x7f65ec19cd10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ec0737a0 0x7f65ec1a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 --2- 172.21.15.111:0/3094822383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f65ec1110d0 0x7f65ec1a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 >> 172.21.15.111:0/3094822383 conn(0x7f65ec0fbbc0 msgr2=0x7f65ec0fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 shutdown_connections 2023-10-11T16:35:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:24.971+0000 7f65f119d700 1 -- 172.21.15.111:0/3094822383 wait complete. 2023-10-11T16:35:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 30 2023-10-11T16:35:24.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:35:25.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.230+0000 7f57c37ff700 1 -- 172.21.15.111:0/4087715286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 msgr2=0x7f57bc102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.230+0000 7f57c37ff700 1 --2- 172.21.15.111:0/4087715286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 0x7f57bc102990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009990 tx=0x7f57ac009ca0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.230+0000 7f57c37ff700 1 -- 172.21.15.111:0/4087715286 shutdown_connections 2023-10-11T16:35:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.230+0000 7f57c37ff700 1 --2- 172.21.15.111:0/4087715286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57bc102ed0 0x7f57bc10f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.230+0000 7f57c37ff700 1 --2- 172.21.15.111:0/4087715286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57bc108550 0x7f57bc108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.230+0000 7f57c37ff700 1 --2- 172.21.15.111:0/4087715286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 0x7f57bc102990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.230+0000 7f57c37ff700 1 -- 172.21.15.111:0/4087715286 >> 172.21.15.111:0/4087715286 conn(0x7f57bc0fdfc0 msgr2=0x7f57bc1003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.231+0000 7f57c37ff700 1 -- 172.21.15.111:0/4087715286 shutdown_connections 2023-10-11T16:35:25.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.231+0000 7f57c37ff700 1 -- 172.21.15.111:0/4087715286 wait complete. 2023-10-11T16:35:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.232+0000 7f57c37ff700 1 Processor -- start 2023-10-11T16:35:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.232+0000 7f57c37ff700 1 -- start start 2023-10-11T16:35:25.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.232+0000 7f57c37ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 0x7f57bc075000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.232+0000 7f57c37ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57bc102ed0 0x7f57bc075540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.232+0000 7f57c37ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57bc108550 0x7f57bc075a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.232+0000 7f57c37ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc1a8c10 con 0x7f57bc108550 2023-10-11T16:35:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.232+0000 7f57c37ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc1a8d80 con 0x7f57bc102520 2023-10-11T16:35:25.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c37ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc1a8f40 con 0x7f57bc102ed0 2023-10-11T16:35:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c159b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 0x7f57bc075000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c0d9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57bc102ed0 0x7f57bc075540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c159b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 0x7f57bc075000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48078/0 (socket says 172.21.15.111:48078) 2023-10-11T16:35:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c159b700 1 -- 172.21.15.111:0/1497163553 learned_addr learned my addr 172.21.15.111:0/1497163553 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c0d9a700 1 -- 172.21.15.111:0/1497163553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 msgr2=0x7f57bc075000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 0x7f57bc075000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c0d9a700 1 -- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57bc108550 msgr2=0x7f57bc075a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:35:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57bc108550 0x7f57bc075a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c0d9a700 1 -- 172.21.15.111:0/1497163553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac0095f0 con 0x7f57bc102ed0 2023-10-11T16:35:25.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.233+0000 7f57c159b700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 0x7f57bc075000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.234+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57bc102ed0 0x7f57bc075540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f57ac000c00 tx=0x7f57ac005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.234+0000 7f57b27fc700 1 -- 172.21.15.111:0/1497163553 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac01d070 con 0x7f57bc102ed0 2023-10-11T16:35:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.234+0000 7f57b27fc700 1 -- 172.21.15.111:0/1497163553 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57ac021410 con 0x7f57bc102ed0 2023-10-11T16:35:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.234+0000 7f57c37ff700 1 -- 172.21.15.111:0/1497163553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57bc112570 con 0x7f57bc102ed0 2023-10-11T16:35:25.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.234+0000 7f57b27fc700 1 -- 172.21.15.111:0/1497163553 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac017650 con 0x7f57bc102ed0 2023-10-11T16:35:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.234+0000 7f57c37ff700 1 -- 172.21.15.111:0/1497163553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc1123e0 con 0x7f57bc102ed0 2023-10-11T16:35:25.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.237+0000 7f57b27fc700 1 -- 172.21.15.111:0/1497163553 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f57ac021580 con 0x7f57bc102ed0 2023-10-11T16:35:25.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.239+0000 7f57b27fc700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f57a80a12c0 0x7f57a80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.239+0000 7f57b27fc700 1 -- 172.21.15.111:0/1497163553 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f57ac0ce4a0 con 0x7f57bc102ed0 2023-10-11T16:35:25.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.239+0000 7f57c159b700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f57a80a12c0 0x7f57a80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.240+0000 7f57c159b700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f57a80a12c0 0x7f57a80a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f57b80104e0 tx=0x7f57b8007480 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.240+0000 7f57d71feb80 1 Processor -- start 2023-10-11T16:35:25.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.240+0000 7f57d71feb80 1 -- start start 2023-10-11T16:35:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a804f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 0x55943a809370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a8098b0 0x55943a809d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57d71feb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a70a060 con 0x55943a805470 2023-10-11T16:35:25.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57d71feb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a75c8a0 con 0x55943a8098b0 2023-10-11T16:35:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57d71feb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a755d30 con 0x55943a804b60 2023-10-11T16:35:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a804f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c1d9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a8098b0 0x55943a809d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a804f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52566/0 (socket says 172.21.15.111:52566) 2023-10-11T16:35:25.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c159b700 1 -- 172.21.15.111:0/877242856 learned_addr learned my addr 172.21.15.111:0/877242856 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/877242856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 0x55943a809370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c159b700 1 -- 172.21.15.111:0/877242856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a8098b0 msgr2=0x55943a809d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c159b700 1 --2- 172.21.15.111:0/877242856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a8098b0 0x55943a809d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c159b700 1 -- 172.21.15.111:0/877242856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 msgr2=0x55943a809370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c159b700 1 --2- 172.21.15.111:0/877242856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 0x55943a809370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.241+0000 7f57c159b700 1 -- 172.21.15.111:0/877242856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55943a5d7630 con 0x55943a804b60 2023-10-11T16:35:25.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.242+0000 7f57c159b700 1 --2- 172.21.15.111:0/877242856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a804f30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f57b80116e0 tx=0x7f57b8013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c7b6437dff3ff0e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.242+0000 7f57a77fe700 1 -- 172.21.15.111:0/877242856 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b801e620 con 0x55943a804b60 2023-10-11T16:35:25.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.242+0000 7f57a77fe700 1 -- 172.21.15.111:0/877242856 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57b8015070 con 0x55943a804b60 2023-10-11T16:35:25.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.242+0000 7f57a77fe700 1 -- 172.21.15.111:0/877242856 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b801cb60 con 0x55943a804b60 2023-10-11T16:35:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 -- 172.21.15.111:0/877242856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 msgr2=0x55943a804f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 --2- 172.21.15.111:0/877242856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a804f30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f57b80116e0 tx=0x7f57b8013000 comp rx=0 tx=0).stop 2023-10-11T16:35:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 -- 172.21.15.111:0/877242856 shutdown_connections 2023-10-11T16:35:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 --2- 172.21.15.111:0/877242856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a8098b0 0x55943a809d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 --2- 172.21.15.111:0/877242856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a804f30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 --2- 172.21.15.111:0/877242856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 0x55943a809370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 -- 172.21.15.111:0/877242856 >> 172.21.15.111:0/877242856 conn(0x55943a7fa360 msgr2=0x55943a7ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 -- 172.21.15.111:0/877242856 shutdown_connections 2023-10-11T16:35:25.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.243+0000 7f57d71feb80 1 -- 172.21.15.111:0/877242856 wait complete. 2023-10-11T16:35:25.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.244+0000 7f57d71feb80 1 -- start start 2023-10-11T16:35:25.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.244+0000 7f57d71feb80 1 Processor -- start 2023-10-11T16:35:25.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a811090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a80e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a8098b0 0x55943a80ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57d71feb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a6f78e0 con 0x55943a8098b0 2023-10-11T16:35:25.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57d71feb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a6f7580 con 0x55943a805470 2023-10-11T16:35:25.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57d71feb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a5d9920 con 0x55943a804b60 2023-10-11T16:35:25.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57d71feb80 10 client.0 _ll_register_callbacks cb 0x55943a715780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:25.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57c159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a811090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57c159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a811090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52578/0 (socket says 172.21.15.111:52578) 2023-10-11T16:35:25.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57c159b700 1 -- 172.21.15.111:0/585611560 learned_addr learned my addr 172.21.15.111:0/585611560 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.245+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a80e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57c159b700 1 -- 172.21.15.111:0/585611560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 msgr2=0x55943a80e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57c159b700 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a80e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57c159b700 1 -- 172.21.15.111:0/585611560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a8098b0 msgr2=0x55943a80ea50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:35:25.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57c1d9c700 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a8098b0 0x55943a80ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57c159b700 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a8098b0 0x55943a80ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57c159b700 1 -- 172.21.15.111:0/585611560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b8011440 con 0x55943a804b60 2023-10-11T16:35:25.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57c159b700 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a811090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57b80116b0 tx=0x7f57b801ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e14ded80811321e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:35:25.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57a4ff9700 1 -- 172.21.15.111:0/585611560 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b802c070 con 0x55943a804b60 2023-10-11T16:35:25.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57a4ff9700 1 -- 172.21.15.111:0/585611560 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f57b8017070 con 0x55943a804b60 2023-10-11T16:35:25.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.246+0000 7f57d71feb80 1 -- 172.21.15.111:0/585611560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55943a7f7b20 con 0x55943a804b60 2023-10-11T16:35:25.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.247+0000 7f57a4ff9700 1 -- 172.21.15.111:0/585611560 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57b801ce80 con 0x55943a804b60 2023-10-11T16:35:25.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.247+0000 7f57d71feb80 10 client.26206 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:25.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.247+0000 7f57d71feb80 1 -- 172.21.15.111:0/585611560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55943a71c190 con 0x55943a804b60 2023-10-11T16:35:25.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.247+0000 7f57a4ff9700 1 -- 172.21.15.111:0/585611560 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55943a7f7b20 con 0x55943a804b60 2023-10-11T16:35:25.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.247+0000 7f57a4ff9700 1 -- 172.21.15.111:0/585611560 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f57b8004cf0 con 0x55943a804b60 2023-10-11T16:35:25.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.248+0000 7f57d71feb80 10 client.26206 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:25.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.248+0000 7f57d71feb80 10 client.26206 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:35:25.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.248+0000 7f57d71feb80 20 client.26206 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:25.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.248+0000 7f57d71feb80 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55943a81e770 0x55943a820b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.248+0000 7f57d71feb80 4 client.26206 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:25.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.248+0000 7f57d71feb80 1 -- 172.21.15.111:0/585611560 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55943a7180a0 con 0x55943a81e770 2023-10-11T16:35:25.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.249+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55943a81e770 0x55943a820b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.249+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55943a81e770 0x55943a820b80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce455115988ad4c9 server_cookie=a1aeb47322d9b547 in_seq=0 out_seq=0 2023-10-11T16:35:25.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.249+0000 7f57a4ff9700 10 client.26206 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:35:25.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.250+0000 7f57a4ff9700 1 -- 172.21.15.111:0/585611560 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55943a7180a0 con 0x55943a81e770 2023-10-11T16:35:25.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.250+0000 7f57a4ff9700 10 client.26206 handle_command_reply: tid=0 2023-10-11T16:35:25.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.250+0000 7f57d71feb80 1 client.26206 shutdown 2023-10-11T16:35:25.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57d71feb80 1 -- 172.21.15.111:0/585611560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 msgr2=0x55943a811090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57d71feb80 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a811090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57b80116b0 tx=0x7f57b801ca20 comp rx=0 tx=0).stop 2023-10-11T16:35:25.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57d71feb80 1 -- 172.21.15.111:0/585611560 shutdown_connections 2023-10-11T16:35:25.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57d71feb80 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55943a81e770 0x55943a820b80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57d71feb80 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a8098b0 0x55943a80ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57d71feb80 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a804b60 0x55943a811090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57d71feb80 1 --2- 172.21.15.111:0/585611560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a80e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57d71feb80 1 -- 172.21.15.111:0/585611560 >> 172.21.15.111:0/585611560 conn(0x55943a7fa360 msgr2=0x55943a80ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.251+0000 7f57a4ff9700 0 client.26206 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:35:25.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.252+0000 7f57d71feb80 1 -- 172.21.15.111:0/585611560 shutdown_connections 2023-10-11T16:35:25.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.252+0000 7f57d71feb80 1 -- 172.21.15.111:0/585611560 wait complete. 2023-10-11T16:35:25.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.252+0000 7f57d71feb80 20 client.26206 trim_cache size 0 max 16384 2023-10-11T16:35:25.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.268+0000 7f57d71feb80 1 Processor -- start 2023-10-11T16:35:25.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.268+0000 7f57d71feb80 1 -- start start 2023-10-11T16:35:25.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a804b60 0x55943a8071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a820780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a810c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57d71feb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a805bc0 con 0x55943a804b60 2023-10-11T16:35:25.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57d71feb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a80f8f0 con 0x55943a805470 2023-10-11T16:35:25.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57d71feb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a80c890 con 0x55943a8098b0 2023-10-11T16:35:25.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57c159b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a804b60 0x55943a8071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57c159b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a804b60 0x55943a8071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42370/0 (socket says 172.21.15.111:42370) 2023-10-11T16:35:25.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57c159b700 1 -- 172.21.15.111:0/1925136461 learned_addr learned my addr 172.21.15.111:0/1925136461 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a820780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.269+0000 7f57c1d9c700 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a810c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.270+0000 7f57c0d9a700 1 -- 172.21.15.111:0/1925136461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 msgr2=0x55943a810c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.270+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a810c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.270+0000 7f57c0d9a700 1 -- 172.21.15.111:0/1925136461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a804b60 msgr2=0x55943a8071a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.270+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a804b60 0x55943a8071a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.270+0000 7f57c0d9a700 1 -- 172.21.15.111:0/1925136461 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55943a809f50 con 0x55943a805470 2023-10-11T16:35:25.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.270+0000 7f57c159b700 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a804b60 0x55943a8071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.271+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a820780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0cb7e0 tx=0x7f57ac0a1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=270123415b5f1f18 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.271+0000 7f57a7fff700 1 -- 172.21.15.111:0/1925136461 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac09b1f0 con 0x55943a805470 2023-10-11T16:35:25.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.271+0000 7f57a7fff700 1 -- 172.21.15.111:0/1925136461 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57ac09f220 con 0x55943a805470 2023-10-11T16:35:25.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.271+0000 7f57a7fff700 1 -- 172.21.15.111:0/1925136461 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac021860 con 0x55943a805470 2023-10-11T16:35:25.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 -- 172.21.15.111:0/1925136461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 msgr2=0x55943a820780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a820780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0cb7e0 tx=0x7f57ac0a1040 comp rx=0 tx=0).stop 2023-10-11T16:35:25.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 -- 172.21.15.111:0/1925136461 shutdown_connections 2023-10-11T16:35:25.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a810c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a804b60 0x55943a8071a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1925136461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a805470 0x55943a820780 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 -- 172.21.15.111:0/1925136461 >> 172.21.15.111:0/1925136461 conn(0x55943a7fa360 msgr2=0x55943a81ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 -- 172.21.15.111:0/1925136461 shutdown_connections 2023-10-11T16:35:25.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 -- 172.21.15.111:0/1925136461 wait complete. 2023-10-11T16:35:25.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.272+0000 7f57d71feb80 1 -- start start 2023-10-11T16:35:25.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57d71feb80 1 Processor -- start 2023-10-11T16:35:25.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a804b60 0x55943a815bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 0x55943a8139c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57d71feb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a813f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57d71feb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a814670 con 0x55943a805470 2023-10-11T16:35:25.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57d71feb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a8147e0 con 0x55943a804b60 2023-10-11T16:35:25.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57d71feb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943a871630 con 0x55943a8098b0 2023-10-11T16:35:25.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57d71feb80 10 client.0 _ll_register_callbacks cb 0x7f57ac009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:25.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57c159b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a804b60 0x55943a815bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c1d9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a813f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.273+0000 7f57c159b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a804b60 0x55943a815bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48110/0 (socket says 172.21.15.111:48110) 2023-10-11T16:35:25.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c159b700 1 -- 172.21.15.111:0/1323913053 learned_addr learned my addr 172.21.15.111:0/1323913053 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 0x55943a8139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c159b700 1 -- 172.21.15.111:0/1323913053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 msgr2=0x55943a813f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c159b700 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a813f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c159b700 1 -- 172.21.15.111:0/1323913053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 msgr2=0x55943a8139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c159b700 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 0x55943a8139c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c159b700 1 -- 172.21.15.111:0/1323913053 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55943a71c190 con 0x55943a804b60 2023-10-11T16:35:25.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c1d9c700 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a813f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57c159b700 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a804b60 0x55943a815bd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f57b801c510 tx=0x7f57b8005980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=993f2cc5a02eabf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:25.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57a4ff9700 1 -- 172.21.15.111:0/1323913053 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b802c060 con 0x55943a804b60 2023-10-11T16:35:25.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57a4ff9700 1 -- 172.21.15.111:0/1323913053 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f57b8017040 con 0x55943a804b60 2023-10-11T16:35:25.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.274+0000 7f57d71feb80 1 -- 172.21.15.111:0/1323913053 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57b8015070 con 0x55943a804b60 2023-10-11T16:35:25.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57a4ff9700 1 -- 172.21.15.111:0/1323913053 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57b801ce80 con 0x55943a804b60 2023-10-11T16:35:25.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57d71feb80 10 client.26168 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:25.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57d71feb80 1 -- 172.21.15.111:0/1323913053 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f57bc1123e0 con 0x55943a804b60 2023-10-11T16:35:25.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57a4ff9700 1 -- 172.21.15.111:0/1323913053 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57b8015070 con 0x55943a804b60 2023-10-11T16:35:25.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57a4ff9700 1 -- 172.21.15.111:0/1323913053 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f57b8004780 con 0x55943a804b60 2023-10-11T16:35:25.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57d71feb80 10 client.26168 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:25.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57d71feb80 10 client.26168 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:35:25.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57d71feb80 20 client.26168 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:25.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55943a872850 0x55943a874c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57d71feb80 4 client.26168 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:35:25.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.275+0000 7f57d71feb80 1 -- 172.21.15.111:0/1323913053 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55943a724660 con 0x55943a872850 2023-10-11T16:35:25.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.276+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55943a872850 0x55943a874c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.276+0000 7f57c0d9a700 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55943a872850 0x55943a874c30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de2dc409a48fc0a server_cookie=c992f64d78883a28 in_seq=0 out_seq=0 2023-10-11T16:35:25.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.276+0000 7f57a4ff9700 10 client.26168 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:35:25.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.277+0000 7f57a4ff9700 1 -- 172.21.15.111:0/1323913053 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55943a724660 con 0x55943a872850 2023-10-11T16:35:25.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.277+0000 7f57a4ff9700 10 client.26168 handle_command_reply: tid=0 2023-10-11T16:35:25.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.277+0000 7f57d71feb80 1 client.26168 shutdown 2023-10-11T16:35:25.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 -- 172.21.15.111:0/1323913053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a804b60 msgr2=0x55943a815bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a804b60 0x55943a815bd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f57b801c510 tx=0x7f57b8005980 comp rx=0 tx=0).stop 2023-10-11T16:35:25.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 -- 172.21.15.111:0/1323913053 shutdown_connections 2023-10-11T16:35:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55943a872850 0x55943a874c30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55943a8098b0 0x55943a813f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55943a804b60 0x55943a815bd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57a4ff9700 0 client.26168 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:35:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 --2- 172.21.15.111:0/1323913053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55943a805470 0x55943a8139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 -- 172.21.15.111:0/1323913053 >> 172.21.15.111:0/1323913053 conn(0x55943a7fa360 msgr2=0x55943a820780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 -- 172.21.15.111:0/1323913053 shutdown_connections 2023-10-11T16:35:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 1 -- 172.21.15.111:0/1323913053 wait complete. 2023-10-11T16:35:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.278+0000 7f57d71feb80 20 client.26168 trim_cache size 0 max 16384 2023-10-11T16:35:25.292 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:35:25.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 -- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f57a80a12c0 msgr2=0x7f57a80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f57a80a12c0 0x7f57a80a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f57b80104e0 tx=0x7f57b8007480 comp rx=0 tx=0).stop 2023-10-11T16:35:25.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 -- 172.21.15.111:0/1497163553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57bc102ed0 msgr2=0x7f57bc075540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57bc102ed0 0x7f57bc075540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f57ac000c00 tx=0x7f57ac005740 comp rx=0 tx=0).stop 2023-10-11T16:35:25.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 -- 172.21.15.111:0/1497163553 shutdown_connections 2023-10-11T16:35:25.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f57a80a12c0 0x7f57a80a3770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57bc108550 0x7f57bc075a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57bc102520 0x7f57bc075000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 --2- 172.21.15.111:0/1497163553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57bc102ed0 0x7f57bc075540 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 -- 172.21.15.111:0/1497163553 >> 172.21.15.111:0/1497163553 conn(0x7f57bc0fdfc0 msgr2=0x7f57bc106120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 -- 172.21.15.111:0/1497163553 shutdown_connections 2023-10-11T16:35:25.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.279+0000 7f57c37ff700 1 -- 172.21.15.111:0/1497163553 wait complete. 2023-10-11T16:35:25.303 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:35:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 -- 172.21.15.111:0/2620985968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80fc107ec0 msgr2=0x7f80fc108330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 --2- 172.21.15.111:0/2620985968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80fc107ec0 0x7f80fc108330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f80e40099c0 tx=0x7f80e4009cd0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 -- 172.21.15.111:0/2620985968 shutdown_connections 2023-10-11T16:35:25.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 --2- 172.21.15.111:0/2620985968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc119a40 0x7f80fc11be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 --2- 172.21.15.111:0/2620985968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80fc107af0 0x7f80fc10eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 --2- 172.21.15.111:0/2620985968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80fc107ec0 0x7f80fc108330 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 -- 172.21.15.111:0/2620985968 >> 172.21.15.111:0/2620985968 conn(0x7f80fc06c7b0 msgr2=0x7f80fc071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 -- 172.21.15.111:0/2620985968 shutdown_connections 2023-10-11T16:35:25.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.581+0000 7f8101bff700 1 -- 172.21.15.111:0/2620985968 wait complete. 2023-10-11T16:35:25.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.583+0000 7f8101bff700 1 Processor -- start 2023-10-11T16:35:25.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.583+0000 7f8101bff700 1 -- start start 2023-10-11T16:35:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.583+0000 7f8101bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc107af0 0x7f80fc1a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.583+0000 7f8101bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80fc107ec0 0x7f80fc1a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.583+0000 7f8101bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80fc119a40 0x7f80fc1a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.583+0000 7f8101bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc1aa260 con 0x7f80fc107af0 2023-10-11T16:35:25.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.583+0000 7f8101bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc1aa3d0 con 0x7f80fc119a40 2023-10-11T16:35:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.583+0000 7f8101bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc1aa590 con 0x7f80fc107ec0 2023-10-11T16:35:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc107af0 0x7f80fc1a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc107af0 0x7f80fc1a5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42378/0 (socket says 172.21.15.111:42378) 2023-10-11T16:35:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fb7fe700 1 -- 172.21.15.111:0/3285786980 learned_addr learned my addr 172.21.15.111:0/3285786980 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80fc107ec0 0x7f80fc1a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fbfff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80fc119a40 0x7f80fc1a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fb7fe700 1 -- 172.21.15.111:0/3285786980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80fc107ec0 msgr2=0x7f80fc1a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80fc107ec0 0x7f80fc1a57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fb7fe700 1 -- 172.21.15.111:0/3285786980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80fc119a40 msgr2=0x7f80fc1a9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80fc119a40 0x7f80fc1a9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80fb7fe700 1 -- 172.21.15.111:0/3285786980 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40095f0 con 0x7f80fc107af0 2023-10-11T16:35:25.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.584+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80fc107ec0 0x7f80fc1a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.585+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc107af0 0x7f80fc1a5280 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00ef10 tx=0x7f80ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.585+0000 7f80f97fa700 1 -- 172.21.15.111:0/3285786980 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec00ca70 con 0x7f80fc107af0 2023-10-11T16:35:25.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.585+0000 7f80f97fa700 1 -- 172.21.15.111:0/3285786980 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80ec00f040 con 0x7f80fc107af0 2023-10-11T16:35:25.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.585+0000 7f8101bff700 1 -- 172.21.15.111:0/3285786980 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc11f1d0 con 0x7f80fc107af0 2023-10-11T16:35:25.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.585+0000 7f80f97fa700 1 -- 172.21.15.111:0/3285786980 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec0146e0 con 0x7f80fc107af0 2023-10-11T16:35:25.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.586+0000 7f8101bff700 1 -- 172.21.15.111:0/3285786980 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc11f040 con 0x7f80fc107af0 2023-10-11T16:35:25.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.590+0000 7f80f97fa700 1 -- 172.21.15.111:0/3285786980 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f80ec00cbd0 con 0x7f80fc107af0 2023-10-11T16:35:25.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.591+0000 7f8115423b80 1 Processor -- start 2023-10-11T16:35:25.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.591+0000 7f8115423b80 1 -- start start 2023-10-11T16:35:25.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.591+0000 7f80f97fa700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f80e80a13c0 0x7f80e80a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.591+0000 7f80f97fa700 1 -- 172.21.15.111:0/3285786980 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f80ec0cd290 con 0x7f80fc107af0 2023-10-11T16:35:25.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f8115423b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0b9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f8115423b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0be370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f8115423b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0be8b0 0x55b65c0bed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f8115423b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65bfbf060 con 0x55b65c0b9b60 2023-10-11T16:35:25.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f8115423b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65c0118a0 con 0x55b65c0ba470 2023-10-11T16:35:25.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f8115423b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65c00ad30 con 0x55b65c0be8b0 2023-10-11T16:35:25.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f80fbfff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f80e80a13c0 0x7f80e80a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f80fb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0b9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f80f2ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f80f2ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0be370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48128/0 (socket says 172.21.15.111:48128) 2023-10-11T16:35:25.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.592+0000 7f80f2ffd700 1 -- 172.21.15.111:0/3231727222 learned_addr learned my addr 172.21.15.111:0/3231727222 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.593+0000 7f80f2ffd700 1 -- 172.21.15.111:0/3231727222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0be8b0 msgr2=0x55b65c0bed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.593+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/3231727222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0be8b0 0x55b65c0bed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.593+0000 7f80f2ffd700 1 -- 172.21.15.111:0/3231727222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 msgr2=0x55b65c0b9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.593+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/3231727222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0b9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.593+0000 7f80f2ffd700 1 -- 172.21.15.111:0/3231727222 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b65be8c630 con 0x55b65c0ba470 2023-10-11T16:35:25.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.593+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/3231727222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0b9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.593+0000 7f80fbfff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f80e80a13c0 0x7f80e80a3870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f80f4006fd0 tx=0x7f80f4009cb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.593+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/3231727222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0be370 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f80e40099c0 tx=0x7f80e4003bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=400e0cd1b4a2804 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.594+0000 7f80f1ffb700 1 -- 172.21.15.111:0/3231727222 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e401d030 con 0x55b65c0ba470 2023-10-11T16:35:25.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.594+0000 7f80f1ffb700 1 -- 172.21.15.111:0/3231727222 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80e401e040 con 0x55b65c0ba470 2023-10-11T16:35:25.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.594+0000 7f80f1ffb700 1 -- 172.21.15.111:0/3231727222 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e400bee0 con 0x55b65c0ba470 2023-10-11T16:35:25.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.595+0000 7f8115423b80 1 -- 172.21.15.111:0/3231727222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 msgr2=0x55b65c0be370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.595+0000 7f8115423b80 1 --2- 172.21.15.111:0/3231727222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0be370 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f80e40099c0 tx=0x7f80e4003bd0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.595+0000 7f8115423b80 1 -- 172.21.15.111:0/3231727222 shutdown_connections 2023-10-11T16:35:25.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.595+0000 7f8115423b80 1 --2- 172.21.15.111:0/3231727222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0be8b0 0x55b65c0bed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.595+0000 7f8115423b80 1 --2- 172.21.15.111:0/3231727222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0b9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.595+0000 7f8115423b80 1 --2- 172.21.15.111:0/3231727222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0be370 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.595+0000 7f8115423b80 1 -- 172.21.15.111:0/3231727222 >> 172.21.15.111:0/3231727222 conn(0x55b65c0af360 msgr2=0x55b65c0b4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.596+0000 7f8115423b80 1 -- 172.21.15.111:0/3231727222 shutdown_connections 2023-10-11T16:35:25.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.596+0000 7f8115423b80 1 -- 172.21.15.111:0/3231727222 wait complete. 2023-10-11T16:35:25.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 1 -- start start 2023-10-11T16:35:25.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 1 Processor -- start 2023-10-11T16:35:25.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0c3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0be8b0 0x55b65c0c3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65bfac8e0 con 0x55b65c0be8b0 2023-10-11T16:35:25.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65bfac580 con 0x55b65c0ba470 2023-10-11T16:35:25.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65be8e920 con 0x55b65c0b9b60 2023-10-11T16:35:25.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.597+0000 7f8115423b80 10 client.0 _ll_register_callbacks cb 0x55b65bfca780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:25.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80fb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80fb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52636/0 (socket says 172.21.15.111:52636) 2023-10-11T16:35:25.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80fb7fe700 1 -- 172.21.15.111:0/627361026 learned_addr learned my addr 172.21.15.111:0/627361026 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80fbfff700 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0be8b0 0x55b65c0c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80f2ffd700 1 -- 172.21.15.111:0/627361026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 msgr2=0x55b65c0c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80f2ffd700 1 -- 172.21.15.111:0/627361026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0be8b0 msgr2=0x55b65c0c3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0be8b0 0x55b65c0c3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80f2ffd700 1 -- 172.21.15.111:0/627361026 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc11c770 con 0x55b65c0ba470 2023-10-11T16:35:25.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.598+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.599+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0c3240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009fd0 tx=0x7f80e4005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85aac60c86b7f9c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.599+0000 7f80daffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:25.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.599+0000 7f80daffd700 1 -- 172.21.15.111:0/627361026 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e401d040 con 0x55b65c0ba470 2023-10-11T16:35:25.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.599+0000 7f80daffd700 1 -- 172.21.15.111:0/627361026 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f80e4005960 con 0x55b65c0ba470 2023-10-11T16:35:25.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.600+0000 7f8115423b80 1 -- 172.21.15.111:0/627361026 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b65c0acb20 con 0x55b65c0ba470 2023-10-11T16:35:25.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.600+0000 7f80daffd700 1 -- 172.21.15.111:0/627361026 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80e4005700 con 0x55b65c0ba470 2023-10-11T16:35:25.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.600+0000 7f8115423b80 10 client.26180 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:25.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.600+0000 7f8115423b80 1 -- 172.21.15.111:0/627361026 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b65bfd1190 con 0x55b65c0ba470 2023-10-11T16:35:25.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.600+0000 7f80daffd700 1 -- 172.21.15.111:0/627361026 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b65c0acb20 con 0x55b65c0ba470 2023-10-11T16:35:25.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.600+0000 7f80daffd700 1 -- 172.21.15.111:0/627361026 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f80e40234d0 con 0x55b65c0ba470 2023-10-11T16:35:25.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.601+0000 7f8115423b80 10 client.26180 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:25.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.601+0000 7f8115423b80 10 client.26180 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:35:25.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.601+0000 7f8115423b80 20 client.26180 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:25.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.601+0000 7f8115423b80 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55b65c0d33b0 0x55b65c0d5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.601+0000 7f8115423b80 4 client.26180 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.601+0000 7f8115423b80 1 -- 172.21.15.111:0/627361026 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b65bfcd0a0 con 0x55b65c0d33b0 2023-10-11T16:35:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.601+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55b65c0d33b0 0x55b65c0d5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.602+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55b65c0d33b0 0x55b65c0d5790 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b94e242a8d314e8 server_cookie=39eecc9671b8e5ba in_seq=0 out_seq=0 2023-10-11T16:35:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.602+0000 7f80daffd700 10 client.26180 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:35:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f80daffd700 1 -- 172.21.15.111:0/627361026 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b65bfcd0a0 con 0x55b65c0d33b0 2023-10-11T16:35:25.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f80daffd700 10 client.26180 handle_command_reply: tid=0 2023-10-11T16:35:25.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 client.26180 shutdown 2023-10-11T16:35:25.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 -- 172.21.15.111:0/627361026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 msgr2=0x55b65c0c3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0c3240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009fd0 tx=0x7f80e4005280 comp rx=0 tx=0).stop 2023-10-11T16:35:25.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 -- 172.21.15.111:0/627361026 shutdown_connections 2023-10-11T16:35:25.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55b65c0d33b0 0x55b65c0d5790 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0be8b0 0x55b65c0c3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 --2- 172.21.15.111:0/627361026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0ba470 0x55b65c0c3240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f80daffd700 0 client.26180 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:35:25.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 -- 172.21.15.111:0/627361026 >> 172.21.15.111:0/627361026 conn(0x55b65c0af360 msgr2=0x55b65c0bfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 -- 172.21.15.111:0/627361026 shutdown_connections 2023-10-11T16:35:25.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.606+0000 7f8115423b80 1 -- 172.21.15.111:0/627361026 wait complete. 2023-10-11T16:35:25.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.607+0000 7f8115423b80 20 client.26180 trim_cache size 0 max 16384 2023-10-11T16:35:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.624+0000 7f8115423b80 1 Processor -- start 2023-10-11T16:35:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.624+0000 7f8115423b80 1 -- start start 2023-10-11T16:35:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f8115423b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f8115423b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0ba470 0x55b65c0bc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f8115423b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c0bcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f8115423b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65c0aeee0 con 0x55b65c0ba470 2023-10-11T16:35:25.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f8115423b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65bfcd0d0 con 0x55b65c0be8b0 2023-10-11T16:35:25.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f8115423b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65c0b61c0 con 0x55b65c0b9b60 2023-10-11T16:35:25.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f80fb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f80fb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52642/0 (socket says 172.21.15.111:52642) 2023-10-11T16:35:25.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f80fb7fe700 1 -- 172.21.15.111:0/1395230196 learned_addr learned my addr 172.21.15.111:0/1395230196 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f80fbfff700 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c0bcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.625+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0ba470 0x55b65c0bc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80fb7fe700 1 -- 172.21.15.111:0/1395230196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 msgr2=0x55b65c0bcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c0bcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80fb7fe700 1 -- 172.21.15.111:0/1395230196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0ba470 msgr2=0x55b65c0bc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0ba470 0x55b65c0bc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80fb7fe700 1 -- 172.21.15.111:0/1395230196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b65bfd0ee0 con 0x55b65c0b9b60 2023-10-11T16:35:25.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0ba470 0x55b65c0bc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:25.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0ca9d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0090a0 tx=0x7f80ec0cc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed91d4f84c267f28 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80f27fc700 1 -- 172.21.15.111:0/1395230196 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec0cd0c0 con 0x55b65c0b9b60 2023-10-11T16:35:25.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80f27fc700 1 -- 172.21.15.111:0/1395230196 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80ec099f50 con 0x55b65c0b9b60 2023-10-11T16:35:25.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.626+0000 7f80f27fc700 1 -- 172.21.15.111:0/1395230196 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec099440 con 0x55b65c0b9b60 2023-10-11T16:35:25.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.627+0000 7f8115423b80 1 -- 172.21.15.111:0/1395230196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 msgr2=0x55b65c0ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.627+0000 7f8115423b80 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0ca9d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0090a0 tx=0x7f80ec0cc040 comp rx=0 tx=0).stop 2023-10-11T16:35:25.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.628+0000 7f8115423b80 1 -- 172.21.15.111:0/1395230196 shutdown_connections 2023-10-11T16:35:25.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.628+0000 7f8115423b80 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c0bcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.628+0000 7f8115423b80 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0b9b60 0x55b65c0ca9d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.628+0000 7f8115423b80 1 --2- 172.21.15.111:0/1395230196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0ba470 0x55b65c0bc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.628+0000 7f8115423b80 1 -- 172.21.15.111:0/1395230196 >> 172.21.15.111:0/1395230196 conn(0x55b65c0af360 msgr2=0x55b65c0cfc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.628+0000 7f8115423b80 1 -- 172.21.15.111:0/1395230196 shutdown_connections 2023-10-11T16:35:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.628+0000 7f8115423b80 1 -- 172.21.15.111:0/1395230196 wait complete. 2023-10-11T16:35:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.629+0000 7f8115423b80 1 -- start start 2023-10-11T16:35:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.629+0000 7f8115423b80 1 Processor -- start 2023-10-11T16:35:25.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.629+0000 7f8115423b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f8115423b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0ba470 0x55b65c0ca140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f8115423b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c126200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f8115423b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65c0caa50 con 0x55b65c0b9b60 2023-10-11T16:35:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f8115423b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65c126740 con 0x55b65c0be8b0 2023-10-11T16:35:25.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f8115423b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b65c126880 con 0x55b65c0ba470 2023-10-11T16:35:25.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f8115423b80 10 client.0 _ll_register_callbacks cb 0x7f80e4005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:25.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f80fb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f80f2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0ba470 0x55b65c0ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f80f2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0ba470 0x55b65c0ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52646/0 (socket says 172.21.15.111:52646) 2023-10-11T16:35:25.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f80f2ffd700 1 -- 172.21.15.111:0/1016105121 learned_addr learned my addr 172.21.15.111:0/1016105121 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f80fbfff700 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c126200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f80f2ffd700 1 -- 172.21.15.111:0/1016105121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 msgr2=0x55b65c126200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.630+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c126200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80f2ffd700 1 -- 172.21.15.111:0/1016105121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 msgr2=0x55b65c0c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80f2ffd700 1 -- 172.21.15.111:0/1016105121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec023070 con 0x55b65c0ba470 2023-10-11T16:35:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80fbfff700 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c126200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80f2ffd700 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0ba470 0x55b65c0ca140 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f80e40238e0 tx=0x7f80e4023a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb8e0479e381817 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80daffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:35:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80daffd700 1 -- 172.21.15.111:0/1016105121 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e401d030 con 0x55b65c0ba470 2023-10-11T16:35:25.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f80daffd700 1 -- 172.21.15.111:0/1016105121 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f80e4000e40 con 0x55b65c0ba470 2023-10-11T16:35:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.631+0000 7f8115423b80 1 -- 172.21.15.111:0/1016105121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80e401e040 con 0x55b65c0ba470 2023-10-11T16:35:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.632+0000 7f80daffd700 1 -- 172.21.15.111:0/1016105121 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80e4005ce0 con 0x55b65c0ba470 2023-10-11T16:35:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.632+0000 7f8115423b80 10 client.26227 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.632+0000 7f8115423b80 1 -- 172.21.15.111:0/1016105121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b65be21c60 con 0x55b65c0ba470 2023-10-11T16:35:25.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.632+0000 7f80daffd700 1 -- 172.21.15.111:0/1016105121 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80e401e040 con 0x55b65c0ba470 2023-10-11T16:35:25.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f80daffd700 1 -- 172.21.15.111:0/1016105121 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f80e4046070 con 0x55b65c0ba470 2023-10-11T16:35:25.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f8115423b80 10 client.26227 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:25.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f8115423b80 10 client.26227 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:35:25.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f8115423b80 20 client.26227 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:25.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f8115423b80 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55b65c127b70 0x55b65c129f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f8115423b80 4 client.26227 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:35:25.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f8115423b80 1 -- 172.21.15.111:0/1016105121 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b65bfd9660 con 0x55b65c127b70 2023-10-11T16:35:25.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55b65c127b70 0x55b65c129f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.633+0000 7f80fb7fe700 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55b65c127b70 0x55b65c129f80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ef7073d5c1df849 server_cookie=c8f3f17a6b18db87 in_seq=0 out_seq=0 2023-10-11T16:35:25.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.634+0000 7f80daffd700 10 client.26227 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:35:25.648 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:25 smithi111 ceph-mon[98527]: pgmap v86: 129 pgs: 129 active+clean; 81 MiB data, 285 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 1.55k op/s 2023-10-11T16:35:25.648 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:25 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3094822383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:35:25.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f80daffd700 1 -- 172.21.15.111:0/1016105121 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b65bfd9660 con 0x55b65c127b70 2023-10-11T16:35:25.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f80daffd700 10 client.26227 handle_command_reply: tid=0 2023-10-11T16:35:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 client.26227 shutdown 2023-10-11T16:35:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 -- 172.21.15.111:0/1016105121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0ba470 msgr2=0x55b65c0ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0ba470 0x55b65c0ca140 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f80e40238e0 tx=0x7f80e4023a80 comp rx=0 tx=0).stop 2023-10-11T16:35:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 -- 172.21.15.111:0/1016105121 shutdown_connections 2023-10-11T16:35:25.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55b65c127b70 0x55b65c129f80 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b65c0be8b0 0x55b65c126200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b65c0b9b60 0x55b65c0c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 --2- 172.21.15.111:0/1016105121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b65c0ba470 0x55b65c0ca140 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f80daffd700 0 client.26227 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:35:25.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 -- 172.21.15.111:0/1016105121 >> 172.21.15.111:0/1016105121 conn(0x55b65c0af360 msgr2=0x55b65c0b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 -- 172.21.15.111:0/1016105121 shutdown_connections 2023-10-11T16:35:25.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 1 -- 172.21.15.111:0/1016105121 wait complete. 2023-10-11T16:35:25.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.637+0000 7f8115423b80 20 client.26227 trim_cache size 0 max 16384 2023-10-11T16:35:25.652 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:35:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.638+0000 7f8101bff700 1 -- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f80e80a13c0 msgr2=0x7f80e80a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.638+0000 7f8101bff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f80e80a13c0 0x7f80e80a3870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f80f4006fd0 tx=0x7f80f4009cb0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.638+0000 7f8101bff700 1 -- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc107af0 msgr2=0x7f80fc1a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.638+0000 7f8101bff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc107af0 0x7f80fc1a5280 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00ef10 tx=0x7f80ec00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.639+0000 7f8101bff700 1 -- 172.21.15.111:0/3285786980 shutdown_connections 2023-10-11T16:35:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.639+0000 7f8101bff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f80e80a13c0 0x7f80e80a3870 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.639+0000 7f8101bff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80fc119a40 0x7f80fc1a9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.639+0000 7f8101bff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80fc107af0 0x7f80fc1a5280 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.639+0000 7f8101bff700 1 --2- 172.21.15.111:0/3285786980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80fc107ec0 0x7f80fc1a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.639+0000 7f8101bff700 1 -- 172.21.15.111:0/3285786980 >> 172.21.15.111:0/3285786980 conn(0x7f80fc06c7b0 msgr2=0x7f80fc11a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.639+0000 7f8101bff700 1 -- 172.21.15.111:0/3285786980 shutdown_connections 2023-10-11T16:35:25.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.640+0000 7f8101bff700 1 -- 172.21.15.111:0/3285786980 wait complete. 2023-10-11T16:35:25.665 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:35:25.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:25 smithi150 ceph-mon[132217]: pgmap v86: 129 pgs: 129 active+clean; 81 MiB data, 285 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 1.55k op/s 2023-10-11T16:35:25.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:25 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3094822383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:35:25.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:25 smithi130 ceph-mon[131859]: pgmap v86: 129 pgs: 129 active+clean; 81 MiB data, 285 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 1.55k op/s 2023-10-11T16:35:25.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:25 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3094822383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:35:25.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 -- 172.21.15.111:0/3671262160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6474101600 msgr2=0x7f64741019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 --2- 172.21.15.111:0/3671262160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6474101600 0x7f64741019d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6464009990 tx=0x7f6464009ca0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 -- 172.21.15.111:0/3671262160 shutdown_connections 2023-10-11T16:35:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 --2- 172.21.15.111:0/3671262160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f647410a940 0x7f647410cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 --2- 172.21.15.111:0/3671262160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6474101600 0x7f64741019d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 --2- 172.21.15.111:0/3671262160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6474101f10 0x7f647410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 -- 172.21.15.111:0/3671262160 >> 172.21.15.111:0/3671262160 conn(0x7f6474076100 msgr2=0x7f6474076500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 -- 172.21.15.111:0/3671262160 shutdown_connections 2023-10-11T16:35:25.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.924+0000 7f647a1ff700 1 -- 172.21.15.111:0/3671262160 wait complete. 2023-10-11T16:35:25.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.925+0000 7f647a1ff700 1 Processor -- start 2023-10-11T16:35:25.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f647a1ff700 1 -- start start 2023-10-11T16:35:25.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f647a1ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6474101600 0x7f6474194220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f647a1ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6474101f10 0x7f6474194760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f647a1ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647410a940 0x7f6474198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f647a1ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6474199200 con 0x7f6474101600 2023-10-11T16:35:25.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f647a1ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6474199370 con 0x7f6474101f10 2023-10-11T16:35:25.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f647a1ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6474199530 con 0x7f647410a940 2023-10-11T16:35:25.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f64737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6474101600 0x7f6474194220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f64737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6474101600 0x7f6474194220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42426/0 (socket says 172.21.15.111:42426) 2023-10-11T16:35:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.926+0000 7f64737fe700 1 -- 172.21.15.111:0/2008227487 learned_addr learned my addr 172.21.15.111:0/2008227487 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.927+0000 7f64737fe700 1 -- 172.21.15.111:0/2008227487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647410a940 msgr2=0x7f6474198bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:35:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.927+0000 7f64737fe700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647410a940 0x7f6474198bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.927+0000 7f64737fe700 1 -- 172.21.15.111:0/2008227487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6474101f10 msgr2=0x7f6474194760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.927+0000 7f64737fe700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6474101f10 0x7f6474194760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.927+0000 7f64737fe700 1 -- 172.21.15.111:0/2008227487 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64640095f0 con 0x7f6474101600 2023-10-11T16:35:25.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.927+0000 7f64737fe700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6474101600 0x7f6474194220 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6464005910 tx=0x7f64640057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.928+0000 7f6470ff9700 1 -- 172.21.15.111:0/2008227487 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646401d070 con 0x7f6474101600 2023-10-11T16:35:25.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.928+0000 7f6470ff9700 1 -- 172.21.15.111:0/2008227487 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6464022470 con 0x7f6474101600 2023-10-11T16:35:25.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.928+0000 7f6470ff9700 1 -- 172.21.15.111:0/2008227487 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464017650 con 0x7f6474101600 2023-10-11T16:35:25.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.928+0000 7f647a1ff700 1 -- 172.21.15.111:0/2008227487 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647410e280 con 0x7f6474101600 2023-10-11T16:35:25.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.928+0000 7f647a1ff700 1 -- 172.21.15.111:0/2008227487 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647410e0f0 con 0x7f6474101600 2023-10-11T16:35:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.932+0000 7f6470ff9700 1 -- 172.21.15.111:0/2008227487 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f64640177b0 con 0x7f6474101600 2023-10-11T16:35:25.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.934+0000 7f6470ff9700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f645c0a5750 0x7f645c0a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.934+0000 7f6470ff9700 1 -- 172.21.15.111:0/2008227487 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f64640ce330 con 0x7f6474101600 2023-10-11T16:35:25.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.934+0000 7f6472ffd700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f645c0a5750 0x7f645c0a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.935+0000 7f648daa4b80 1 Processor -- start 2023-10-11T16:35:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.935+0000 7f648daa4b80 1 -- start start 2023-10-11T16:35:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.935+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.935+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.935+0000 7f6472ffd700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f645c0a5750 0x7f645c0a7c00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6468005950 tx=0x7f64680058e0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad54d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f648daa4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bac55060 con 0x5650bad4fb60 2023-10-11T16:35:25.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f648daa4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650baca78a0 con 0x5650bad548b0 2023-10-11T16:35:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f648daa4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650baca0d30 con 0x5650bad50470 2023-10-11T16:35:25.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6473fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad54d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6473fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad54d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48196/0 (socket says 172.21.15.111:48196) 2023-10-11T16:35:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6473fff700 1 -- 172.21.15.111:0/2440505342 learned_addr learned my addr 172.21.15.111:0/2440505342 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6472ffd700 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f64737fe700 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6473fff700 1 -- 172.21.15.111:0/2440505342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 msgr2=0x5650bad54370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6473fff700 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6473fff700 1 -- 172.21.15.111:0/2440505342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 msgr2=0x5650bad4ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6473fff700 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f6473fff700 1 -- 172.21.15.111:0/2440505342 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650bab22630 con 0x5650bad548b0 2023-10-11T16:35:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.936+0000 7f64737fe700 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:35:25.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.937+0000 7f6473fff700 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad54d00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e380 tx=0x7f646c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c84c0bf1d01b6640 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.937+0000 7f6459ffb700 1 -- 172.21.15.111:0/2440505342 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c018070 con 0x5650bad548b0 2023-10-11T16:35:25.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.937+0000 7f6459ffb700 1 -- 172.21.15.111:0/2440505342 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f646c00ee30 con 0x5650bad548b0 2023-10-11T16:35:25.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.937+0000 7f6459ffb700 1 -- 172.21.15.111:0/2440505342 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c010040 con 0x5650bad548b0 2023-10-11T16:35:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 -- 172.21.15.111:0/2440505342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 msgr2=0x5650bad54d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad54d00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e380 tx=0x7f646c00b040 comp rx=0 tx=0).stop 2023-10-11T16:35:25.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 -- 172.21.15.111:0/2440505342 shutdown_connections 2023-10-11T16:35:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad54d00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 --2- 172.21.15.111:0/2440505342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 -- 172.21.15.111:0/2440505342 >> 172.21.15.111:0/2440505342 conn(0x5650bad45360 msgr2=0x5650bad4ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 -- 172.21.15.111:0/2440505342 shutdown_connections 2023-10-11T16:35:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.939+0000 7f648daa4b80 1 -- 172.21.15.111:0/2440505342 wait complete. 2023-10-11T16:35:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.940+0000 7f648daa4b80 1 -- start start 2023-10-11T16:35:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.940+0000 7f648daa4b80 1 Processor -- start 2023-10-11T16:35:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad4fb60 0x5650bad5bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad548b0 0x5650bad59740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f648daa4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bac428e0 con 0x5650bad548b0 2023-10-11T16:35:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f648daa4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bac42580 con 0x5650bad4fb60 2023-10-11T16:35:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f648daa4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bab24920 con 0x5650bad50470 2023-10-11T16:35:25.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f648daa4b80 10 client.0 _ll_register_callbacks cb 0x5650bac60780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f64737fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad4fb60 0x5650bad5bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f64737fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad4fb60 0x5650bad5bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48200/0 (socket says 172.21.15.111:48200) 2023-10-11T16:35:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f6473fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad548b0 0x5650bad59740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.941+0000 7f64737fe700 1 -- 172.21.15.111:0/1513023640 learned_addr learned my addr 172.21.15.111:0/1513023640 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.942+0000 7f64737fe700 1 -- 172.21.15.111:0/1513023640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 msgr2=0x5650bad591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.942+0000 7f64737fe700 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.942+0000 7f64737fe700 1 -- 172.21.15.111:0/1513023640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad548b0 msgr2=0x5650bad59740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.942+0000 7f64737fe700 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad548b0 0x5650bad59740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.942+0000 7f64737fe700 1 -- 172.21.15.111:0/1513023640 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c00e0c0 con 0x5650bad4fb60 2023-10-11T16:35:25.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.942+0000 7f6473fff700 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad548b0 0x5650bad59740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.942+0000 7f64737fe700 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad4fb60 0x5650bad5bfb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f64640cbcd0 tx=0x7f646409ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8790df18d20c5d2e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.943+0000 7f6452ffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:25.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.943+0000 7f6452ffd700 1 -- 172.21.15.111:0/1513023640 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64640ce170 con 0x5650bad4fb60 2023-10-11T16:35:25.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.943+0000 7f6452ffd700 1 -- 172.21.15.111:0/1513023640 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f646409f110 con 0x5650bad4fb60 2023-10-11T16:35:25.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.943+0000 7f648daa4b80 1 -- 172.21.15.111:0/1513023640 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5650bad42b20 con 0x5650bad4fb60 2023-10-11T16:35:25.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.943+0000 7f6452ffd700 1 -- 172.21.15.111:0/1513023640 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64640a34a0 con 0x5650bad4fb60 2023-10-11T16:35:25.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.943+0000 7f648daa4b80 10 client.26195 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:25.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.943+0000 7f648daa4b80 1 -- 172.21.15.111:0/1513023640 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5650bac67190 con 0x5650bad4fb60 2023-10-11T16:35:25.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.943+0000 7f6452ffd700 1 -- 172.21.15.111:0/1513023640 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5650bad42b20 con 0x5650bad4fb60 2023-10-11T16:35:25.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.944+0000 7f6452ffd700 1 -- 172.21.15.111:0/1513023640 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f6464096ed0 con 0x5650bad4fb60 2023-10-11T16:35:25.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.944+0000 7f648daa4b80 10 client.26195 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:25.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.944+0000 7f648daa4b80 10 client.26195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:25.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.944+0000 7f648daa4b80 20 client.26195 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:25.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.944+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5650bad69370 0x5650bad6b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.944+0000 7f648daa4b80 4 client.26195 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:25.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.944+0000 7f648daa4b80 1 -- 172.21.15.111:0/1513023640 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5650bac630a0 con 0x5650bad69370 2023-10-11T16:35:25.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.945+0000 7f6472ffd700 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5650bad69370 0x5650bad6b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.945+0000 7f6472ffd700 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5650bad69370 0x5650bad6b780 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fe523d9199622bf server_cookie=f5ae2c0224e18cb8 in_seq=0 out_seq=0 2023-10-11T16:35:25.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.945+0000 7f6452ffd700 10 client.26195 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:25.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.946+0000 7f6452ffd700 1 -- 172.21.15.111:0/1513023640 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5650bac630a0 con 0x5650bad69370 2023-10-11T16:35:25.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.946+0000 7f6452ffd700 10 client.26195 handle_command_reply: tid=0 2023-10-11T16:35:25.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.946+0000 7f648daa4b80 1 client.26195 shutdown 2023-10-11T16:35:25.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.946+0000 7f648daa4b80 1 -- 172.21.15.111:0/1513023640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad4fb60 msgr2=0x5650bad5bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.946+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad4fb60 0x5650bad5bfb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f64640cbcd0 tx=0x7f646409ad10 comp rx=0 tx=0).stop 2023-10-11T16:35:25.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.946+0000 7f648daa4b80 1 -- 172.21.15.111:0/1513023640 shutdown_connections 2023-10-11T16:35:25.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5650bad69370 0x5650bad6b780 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad548b0 0x5650bad59740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad4fb60 0x5650bad5bfb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1513023640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f648daa4b80 1 -- 172.21.15.111:0/1513023640 >> 172.21.15.111:0/1513023640 conn(0x5650bad45360 msgr2=0x5650bad55d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f6452ffd700 0 client.26195 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:25.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f648daa4b80 1 -- 172.21.15.111:0/1513023640 shutdown_connections 2023-10-11T16:35:25.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f648daa4b80 1 -- 172.21.15.111:0/1513023640 wait complete. 2023-10-11T16:35:25.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.947+0000 7f648daa4b80 20 client.26195 trim_cache size 0 max 16384 2023-10-11T16:35:25.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f648daa4b80 1 Processor -- start 2023-10-11T16:35:25.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f648daa4b80 1 -- start start 2023-10-11T16:35:25.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad56570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f648daa4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bad5a1a0 con 0x5650bad4fb60 2023-10-11T16:35:25.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f648daa4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bad54710 con 0x5650bad548b0 2023-10-11T16:35:25.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f648daa4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bad42a20 con 0x5650bad50470 2023-10-11T16:35:25.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.966+0000 7f64737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f6472ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f64737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42440/0 (socket says 172.21.15.111:42440) 2023-10-11T16:35:25.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f64737fe700 1 -- 172.21.15.111:0/1554962850 learned_addr learned my addr 172.21.15.111:0/1554962850 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f6473fff700 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad56570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f64737fe700 1 -- 172.21.15.111:0/1554962850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 msgr2=0x5650bad609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f64737fe700 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f64737fe700 1 -- 172.21.15.111:0/1554962850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 msgr2=0x5650bad56570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f64737fe700 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad56570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f64737fe700 1 -- 172.21.15.111:0/1554962850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650bad703b0 con 0x5650bad4fb60 2023-10-11T16:35:25.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f6473fff700 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad56570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.967+0000 7f64737fe700 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ad10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6464097b50 tx=0x7f646409ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f83192bd1c88559 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.968+0000 7f645a7fc700 1 -- 172.21.15.111:0/1554962850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64640ce170 con 0x5650bad4fb60 2023-10-11T16:35:25.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.968+0000 7f645a7fc700 1 -- 172.21.15.111:0/1554962850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f646409f110 con 0x5650bad4fb60 2023-10-11T16:35:25.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.968+0000 7f645a7fc700 1 -- 172.21.15.111:0/1554962850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64640a3010 con 0x5650bad4fb60 2023-10-11T16:35:25.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 -- 172.21.15.111:0/1554962850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 msgr2=0x5650bad4ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ad10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6464097b50 tx=0x7f646409ab70 comp rx=0 tx=0).stop 2023-10-11T16:35:25.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 -- 172.21.15.111:0/1554962850 shutdown_connections 2023-10-11T16:35:25.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad56570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad4ad10 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 --2- 172.21.15.111:0/1554962850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 -- 172.21.15.111:0/1554962850 >> 172.21.15.111:0/1554962850 conn(0x5650bad45360 msgr2=0x5650bad4a600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 -- 172.21.15.111:0/1554962850 shutdown_connections 2023-10-11T16:35:25.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.969+0000 7f648daa4b80 1 -- 172.21.15.111:0/1554962850 wait complete. 2023-10-11T16:35:25.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.970+0000 7f648daa4b80 1 -- start start 2023-10-11T16:35:25.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.970+0000 7f648daa4b80 1 Processor -- start 2023-10-11T16:35:25.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad5bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f648daa4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad55260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f648daa4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bad5c170 con 0x5650bad4fb60 2023-10-11T16:35:25.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f648daa4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bad5c2e0 con 0x5650bad548b0 2023-10-11T16:35:25.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f648daa4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650bad557a0 con 0x5650bad50470 2023-10-11T16:35:25.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f648daa4b80 10 client.0 _ll_register_callbacks cb 0x7f64640974d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:25.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f64737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad5bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f6473fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad55260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f6472ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.971+0000 7f64737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad5bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42450/0 (socket says 172.21.15.111:42450) 2023-10-11T16:35:25.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f64737fe700 1 -- 172.21.15.111:0/972139943 learned_addr learned my addr 172.21.15.111:0/972139943 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:25.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f6472ffd700 1 -- 172.21.15.111:0/972139943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 msgr2=0x5650bad55260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f6472ffd700 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad55260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f6472ffd700 1 -- 172.21.15.111:0/972139943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 msgr2=0x5650bad5bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:25.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f6472ffd700 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad5bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f6472ffd700 1 -- 172.21.15.111:0/972139943 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650bab22630 con 0x5650bad50470 2023-10-11T16:35:25.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f6473fff700 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad55260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:25.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f6472ffd700 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54d20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5650bad6c090 tx=0x7f6468009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d320c514cc60621 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:25.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.972+0000 7f6452ffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:35:25.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.973+0000 7f6452ffd700 1 -- 172.21.15.111:0/972139943 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468024860 con 0x5650bad50470 2023-10-11T16:35:25.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.973+0000 7f648daa4b80 1 -- 172.21.15.111:0/972139943 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f646c00ee30 con 0x5650bad50470 2023-10-11T16:35:25.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.973+0000 7f6452ffd700 1 -- 172.21.15.111:0/972139943 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f6468022050 con 0x5650bad50470 2023-10-11T16:35:25.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.973+0000 7f6452ffd700 1 -- 172.21.15.111:0/972139943 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64680249c0 con 0x5650bad50470 2023-10-11T16:35:25.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.973+0000 7f648daa4b80 10 client.26239 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:25.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.973+0000 7f648daa4b80 1 -- 172.21.15.111:0/972139943 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5650bac67190 con 0x5650bad50470 2023-10-11T16:35:25.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.973+0000 7f6452ffd700 1 -- 172.21.15.111:0/972139943 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f646c00ee30 con 0x5650bad50470 2023-10-11T16:35:25.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.974+0000 7f6452ffd700 1 -- 172.21.15.111:0/972139943 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f6468005020 con 0x5650bad50470 2023-10-11T16:35:25.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.974+0000 7f648daa4b80 10 client.26239 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:25.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.974+0000 7f648daa4b80 10 client.26239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:25.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.974+0000 7f648daa4b80 20 client.26239 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:25.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.974+0000 7f648daa4b80 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5650badbd380 0x5650badbf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:25.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.974+0000 7f648daa4b80 4 client.26239 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:35:25.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.974+0000 7f648daa4b80 1 -- 172.21.15.111:0/972139943 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5650bac6f660 con 0x5650badbd380 2023-10-11T16:35:25.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.974+0000 7f64737fe700 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5650badbd380 0x5650badbf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:25.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.975+0000 7f64737fe700 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5650badbd380 0x5650badbf790 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f141bcc6e09e3021 server_cookie=89380823d65edd0e in_seq=0 out_seq=0 2023-10-11T16:35:25.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.975+0000 7f6452ffd700 10 client.26239 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:25.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.975+0000 7f6452ffd700 1 -- 172.21.15.111:0/972139943 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5650bac6f660 con 0x5650badbd380 2023-10-11T16:35:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.975+0000 7f6452ffd700 10 client.26239 handle_command_reply: tid=0 2023-10-11T16:35:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 client.26239 shutdown 2023-10-11T16:35:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 -- 172.21.15.111:0/972139943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 msgr2=0x5650bad54d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54d20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5650bad6c090 tx=0x7f6468009260 comp rx=0 tx=0).stop 2023-10-11T16:35:25.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 -- 172.21.15.111:0/972139943 shutdown_connections 2023-10-11T16:35:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5650badbd380 0x5650badbf790 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5650bad548b0 0x5650bad55260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5650bad4fb60 0x5650bad5bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 --2- 172.21.15.111:0/972139943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5650bad50470 0x5650bad54d20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 -- 172.21.15.111:0/972139943 >> 172.21.15.111:0/972139943 conn(0x5650bad45360 msgr2=0x5650bad56570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f6452ffd700 0 client.26239 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:25.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 -- 172.21.15.111:0/972139943 shutdown_connections 2023-10-11T16:35:25.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 1 -- 172.21.15.111:0/972139943 wait complete. 2023-10-11T16:35:25.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.976+0000 7f648daa4b80 20 client.26239 trim_cache size 0 max 16384 2023-10-11T16:35:25.993 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:35:25.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 -- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f645c0a5750 msgr2=0x7f645c0a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f645c0a5750 0x7f645c0a7c00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6468005950 tx=0x7f64680058e0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 -- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6474101600 msgr2=0x7f6474194220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:25.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6474101600 0x7f6474194220 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6464005910 tx=0x7f64640057d0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 -- 172.21.15.111:0/2008227487 shutdown_connections 2023-10-11T16:35:25.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f645c0a5750 0x7f645c0a7c00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647410a940 0x7f6474198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6474101600 0x7f6474194220 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 --2- 172.21.15.111:0/2008227487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6474101f10 0x7f6474194760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:25.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 -- 172.21.15.111:0/2008227487 >> 172.21.15.111:0/2008227487 conn(0x7f6474076100 msgr2=0x7f64741023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:25.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 -- 172.21.15.111:0/2008227487 shutdown_connections 2023-10-11T16:35:25.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:25.977+0000 7f647a1ff700 1 -- 172.21.15.111:0/2008227487 wait complete. 2023-10-11T16:35:27.006 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:35:27.006 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:35:27.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 -- 172.21.15.111:0/1857038155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7ec0ffa90 msgr2=0x7fd7ec104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/1857038155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7ec0ffa90 0x7fd7ec104140 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd7e00098f0 tx=0x7fd7e0009c00 comp rx=0 tx=0).stop 2023-10-11T16:35:27.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 -- 172.21.15.111:0/1857038155 shutdown_connections 2023-10-11T16:35:27.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/1857038155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7ec104680 0x7fd7ec104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/1857038155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7ec0ff180 0x7fd7ec0ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/1857038155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7ec0ffa90 0x7fd7ec104140 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 -- 172.21.15.111:0/1857038155 >> 172.21.15.111:0/1857038155 conn(0x7fd7ec0770b0 msgr2=0x7fd7ec0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 -- 172.21.15.111:0/1857038155 shutdown_connections 2023-10-11T16:35:27.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.249+0000 7fd7f1fff700 1 -- 172.21.15.111:0/1857038155 wait complete. 2023-10-11T16:35:27.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.250+0000 7fd7f1fff700 1 Processor -- start 2023-10-11T16:35:27.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7f1fff700 1 -- start start 2023-10-11T16:35:27.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7f1fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7ec0ff180 0x7fd7ec1a28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7f1fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7ec0ffa90 0x7fd7ec1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7f1fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7ec104680 0x7fd7ec19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7f1fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec19d090 con 0x7fd7ec0ffa90 2023-10-11T16:35:27.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7f1fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec19d200 con 0x7fd7ec104680 2023-10-11T16:35:27.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7f1fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec19d3c0 con 0x7fd7ec0ff180 2023-10-11T16:35:27.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7ec0ff180 0x7fd7ec1a28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7ebfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7ec104680 0x7fd7ec19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7ebfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7ec104680 0x7fd7ec19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48236/0 (socket says 172.21.15.111:48236) 2023-10-11T16:35:27.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7ec0ff180 0x7fd7ec1a28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52696/0 (socket says 172.21.15.111:52696) 2023-10-11T16:35:27.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.251+0000 7fd7ebfff700 1 -- 172.21.15.111:0/2513477333 learned_addr learned my addr 172.21.15.111:0/2513477333 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.252+0000 7fd7ebfff700 1 -- 172.21.15.111:0/2513477333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7ec0ff180 msgr2=0x7fd7ec1a28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.252+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7ec0ff180 0x7fd7ec1a28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.252+0000 7fd7ebfff700 1 -- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7ec0ffa90 msgr2=0x7fd7ec1a2e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:35:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.252+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7ec0ffa90 0x7fd7ec1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.252+0000 7fd7ebfff700 1 -- 172.21.15.111:0/2513477333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc0095f0 con 0x7fd7ec104680 2023-10-11T16:35:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.253+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7ec104680 0x7fd7ec19c8c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400ef30 tx=0x7fd7e400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.253+0000 7fd7e8ff9700 1 -- 172.21.15.111:0/2513477333 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4018070 con 0x7fd7ec104680 2023-10-11T16:35:27.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.253+0000 7fd7e8ff9700 1 -- 172.21.15.111:0/2513477333 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7e400f040 con 0x7fd7ec104680 2023-10-11T16:35:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.253+0000 7fd7f1fff700 1 -- 172.21.15.111:0/2513477333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e0009520 con 0x7fd7ec104680 2023-10-11T16:35:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.253+0000 7fd7e8ff9700 1 -- 172.21.15.111:0/2513477333 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e400cdb0 con 0x7fd7ec104680 2023-10-11T16:35:27.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.253+0000 7fd7f1fff700 1 -- 172.21.15.111:0/2513477333 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec116950 con 0x7fd7ec104680 2023-10-11T16:35:27.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.259+0000 7fd7e8ff9700 1 -- 172.21.15.111:0/2513477333 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fd7e4003a00 con 0x7fd7ec104680 2023-10-11T16:35:27.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.259+0000 7fd805845b80 1 Processor -- start 2023-10-11T16:35:27.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.260+0000 7fd805845b80 1 -- start start 2023-10-11T16:35:27.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.260+0000 7fd805845b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fb8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.260+0000 7fd805845b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fb94f0 0x563135fbd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.260+0000 7fd805845b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fbd930 0x563135fbdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.260+0000 7fd805845b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135ebe0e0 con 0x563135fbd930 2023-10-11T16:35:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.260+0000 7fd805845b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135f10920 con 0x563135fb94f0 2023-10-11T16:35:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.260+0000 7fd805845b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135f09db0 con 0x563135fb8be0 2023-10-11T16:35:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fb94f0 0x563135fbd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fb94f0 0x563135fbd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48244/0 (socket says 172.21.15.111:48244) 2023-10-11T16:35:27.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eb7fe700 1 -- 172.21.15.111:0/1244822783 learned_addr learned my addr 172.21.15.111:0/1244822783 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/1244822783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fb8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eb7fe700 1 -- 172.21.15.111:0/1244822783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 msgr2=0x563135fb8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/1244822783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fb8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eb7fe700 1 -- 172.21.15.111:0/1244822783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fbd930 msgr2=0x563135fbdd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:35:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/1244822783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fbd930 0x563135fbdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7eb7fe700 1 -- 172.21.15.111:0/1244822783 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563135d8b6b0 con 0x563135fb94f0 2023-10-11T16:35:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.261+0000 7fd7e8ff9700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7d40a1340 0x7fd7d40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.262+0000 7fd7e8ff9700 1 -- 172.21.15.111:0/2513477333 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd7e40ccbf0 con 0x7fd7ec104680 2023-10-11T16:35:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.262+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/1244822783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fb94f0 0x563135fbd3f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009fc0 tx=0x7fd7dc011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb04866a001a7960 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.262+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7d40a1340 0x7fd7d40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.262+0000 7fd7d1ffb700 1 -- 172.21.15.111:0/1244822783 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00e670 con 0x563135fb94f0 2023-10-11T16:35:27.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.262+0000 7fd7d1ffb700 1 -- 172.21.15.111:0/1244822783 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7dc00ec90 con 0x563135fb94f0 2023-10-11T16:35:27.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.262+0000 7fd7d1ffb700 1 -- 172.21.15.111:0/1244822783 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc019d30 con 0x563135fb94f0 2023-10-11T16:35:27.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.263+0000 7fd805845b80 1 -- 172.21.15.111:0/1244822783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fb94f0 msgr2=0x563135fbd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.263+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7d40a1340 0x7fd7d40a37f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0005950 tx=0x7fd7e000f720 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.263+0000 7fd805845b80 1 --2- 172.21.15.111:0/1244822783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fb94f0 0x563135fbd3f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009fc0 tx=0x7fd7dc011720 comp rx=0 tx=0).stop 2023-10-11T16:35:27.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.264+0000 7fd805845b80 1 -- 172.21.15.111:0/1244822783 shutdown_connections 2023-10-11T16:35:27.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.264+0000 7fd805845b80 1 --2- 172.21.15.111:0/1244822783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fbd930 0x563135fbdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.264+0000 7fd805845b80 1 --2- 172.21.15.111:0/1244822783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fb8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.264+0000 7fd805845b80 1 --2- 172.21.15.111:0/1244822783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fb94f0 0x563135fbd3f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.264+0000 7fd805845b80 1 -- 172.21.15.111:0/1244822783 >> 172.21.15.111:0/1244822783 conn(0x563135fae3e0 msgr2=0x563135fb3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.264+0000 7fd805845b80 1 -- 172.21.15.111:0/1244822783 shutdown_connections 2023-10-11T16:35:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.264+0000 7fd805845b80 1 -- 172.21.15.111:0/1244822783 wait complete. 2023-10-11T16:35:27.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.264+0000 7fd805845b80 1 -- start start 2023-10-11T16:35:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.265+0000 7fd805845b80 1 Processor -- start 2023-10-11T16:35:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.265+0000 7fd805845b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fc5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.265+0000 7fd805845b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.265+0000 7fd805845b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.265+0000 7fd805845b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135eab960 con 0x563135fb94f0 2023-10-11T16:35:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.265+0000 7fd805845b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135eab600 con 0x563135fbd930 2023-10-11T16:35:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.265+0000 7fd805845b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135d8d9a0 con 0x563135fb8be0 2023-10-11T16:35:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd805845b80 10 client.0 _ll_register_callbacks cb 0x563135ec9800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fc5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fc5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52716/0 (socket says 172.21.15.111:52716) 2023-10-11T16:35:27.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eb7fe700 1 -- 172.21.15.111:0/3678154997 learned_addr learned my addr 172.21.15.111:0/3678154997 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eb7fe700 1 -- 172.21.15.111:0/3678154997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 msgr2=0x563135fc2ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:35:27.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc2ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eb7fe700 1 -- 172.21.15.111:0/3678154997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 msgr2=0x563135fc2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eb7fe700 1 -- 172.21.15.111:0/3678154997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc009c20 con 0x563135fb8be0 2023-10-11T16:35:27.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.266+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:35:27.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.267+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fc5160 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc000c00 tx=0x7fd7dc01ac00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c5499641af43a45 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.267+0000 7fd7caffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:35:27.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.267+0000 7fd7caffd700 1 -- 172.21.15.111:0/3678154997 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00e680 con 0x563135fb8be0 2023-10-11T16:35:27.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.267+0000 7fd7caffd700 1 -- 172.21.15.111:0/3678154997 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd7dc00e7e0 con 0x563135fb8be0 2023-10-11T16:35:27.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.268+0000 7fd805845b80 1 -- 172.21.15.111:0/3678154997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563135fabba0 con 0x563135fb8be0 2023-10-11T16:35:27.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.268+0000 7fd7caffd700 1 -- 172.21.15.111:0/3678154997 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7dc00e9a0 con 0x563135fb8be0 2023-10-11T16:35:27.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.268+0000 7fd805845b80 10 client.26248 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:27.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.268+0000 7fd805845b80 1 -- 172.21.15.111:0/3678154997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563135ed0210 con 0x563135fb8be0 2023-10-11T16:35:27.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.268+0000 7fd7caffd700 1 -- 172.21.15.111:0/3678154997 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563135fabba0 con 0x563135fb8be0 2023-10-11T16:35:27.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.269+0000 7fd7caffd700 1 -- 172.21.15.111:0/3678154997 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fd7dc0049b0 con 0x563135fb8be0 2023-10-11T16:35:27.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.269+0000 7fd805845b80 10 client.26248 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:27.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.269+0000 7fd805845b80 10 client.26248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:27.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.269+0000 7fd805845b80 20 client.26248 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:27.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.269+0000 7fd805845b80 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563135fd2880 0x563135fd4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.269+0000 7fd805845b80 4 client.26248 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:27.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.269+0000 7fd805845b80 1 -- 172.21.15.111:0/3678154997 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563135ecc120 con 0x563135fd2880 2023-10-11T16:35:27.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.269+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563135fd2880 0x563135fd4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.270+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563135fd2880 0x563135fd4c60 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90fc2bdf735a6d5e server_cookie=d1890a8e32071e77 in_seq=0 out_seq=0 2023-10-11T16:35:27.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.270+0000 7fd7caffd700 10 client.26248 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:27.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.270+0000 7fd7caffd700 1 -- 172.21.15.111:0/3678154997 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563135ecc120 con 0x563135fd2880 2023-10-11T16:35:27.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.270+0000 7fd7caffd700 10 client.26248 handle_command_reply: tid=0 2023-10-11T16:35:27.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.270+0000 7fd805845b80 1 client.26248 shutdown 2023-10-11T16:35:27.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 -- 172.21.15.111:0/3678154997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 msgr2=0x563135fc5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fc5160 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc000c00 tx=0x7fd7dc01ac00 comp rx=0 tx=0).stop 2023-10-11T16:35:27.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 -- 172.21.15.111:0/3678154997 shutdown_connections 2023-10-11T16:35:27.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563135fd2880 0x563135fd4c60 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fc5160 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 --2- 172.21.15.111:0/3678154997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 -- 172.21.15.111:0/3678154997 >> 172.21.15.111:0/3678154997 conn(0x563135fae3e0 msgr2=0x563135fbede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd7caffd700 0 client.26248 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:27.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 -- 172.21.15.111:0/3678154997 shutdown_connections 2023-10-11T16:35:27.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.271+0000 7fd805845b80 1 -- 172.21.15.111:0/3678154997 wait complete. 2023-10-11T16:35:27.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.272+0000 7fd805845b80 20 client.26248 trim_cache size 0 max 16384 2023-10-11T16:35:27.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.295+0000 7fd805845b80 1 Processor -- start 2023-10-11T16:35:27.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd805845b80 1 -- start start 2023-10-11T16:35:27.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd805845b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fb3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd805845b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd805845b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd805845b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135fb5240 con 0x563135fb94f0 2023-10-11T16:35:27.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd805845b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135fc30b0 con 0x563135fbd930 2023-10-11T16:35:27.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd805845b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135fbc500 con 0x563135fb8be0 2023-10-11T16:35:27.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd7ebfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd7ebfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48270/0 (socket says 172.21.15.111:48270) 2023-10-11T16:35:27.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.296+0000 7fd7ebfff700 1 -- 172.21.15.111:0/2748345140 learned_addr learned my addr 172.21.15.111:0/2748345140 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.297+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fb3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.297+0000 7fd7ebfff700 1 -- 172.21.15.111:0/2748345140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 msgr2=0x563135fb3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.297+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fb3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.297+0000 7fd7ebfff700 1 -- 172.21.15.111:0/2748345140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 msgr2=0x563135fc4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.297+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.297+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.297+0000 7fd7ebfff700 1 -- 172.21.15.111:0/2748345140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563135fafb80 con 0x563135fbd930 2023-10-11T16:35:27.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.297+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:35:27.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.298+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc1c60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40963b0 tx=0x7fd7e4095c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b00631f78d5ed9ca server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.298+0000 7fd7d27fc700 1 -- 172.21.15.111:0/2748345140 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4004ad0 con 0x563135fbd930 2023-10-11T16:35:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.298+0000 7fd7d27fc700 1 -- 172.21.15.111:0/2748345140 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7e4084040 con 0x563135fbd930 2023-10-11T16:35:27.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.298+0000 7fd7d27fc700 1 -- 172.21.15.111:0/2748345140 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4099ba0 con 0x563135fbd930 2023-10-11T16:35:27.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.299+0000 7fd805845b80 1 -- 172.21.15.111:0/2748345140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 msgr2=0x563135fc1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.299+0000 7fd805845b80 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc1c60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40963b0 tx=0x7fd7e4095c80 comp rx=0 tx=0).stop 2023-10-11T16:35:27.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.299+0000 7fd805845b80 1 -- 172.21.15.111:0/2748345140 shutdown_connections 2023-10-11T16:35:27.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.299+0000 7fd805845b80 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc1c60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.299+0000 7fd805845b80 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb8be0 0x563135fb3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.299+0000 7fd805845b80 1 --2- 172.21.15.111:0/2748345140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb94f0 0x563135fc4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.299+0000 7fd805845b80 1 -- 172.21.15.111:0/2748345140 >> 172.21.15.111:0/2748345140 conn(0x563135fae3e0 msgr2=0x563135fbfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.300+0000 7fd805845b80 1 -- 172.21.15.111:0/2748345140 shutdown_connections 2023-10-11T16:35:27.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.301+0000 7fd805845b80 1 -- 172.21.15.111:0/2748345140 wait complete. 2023-10-11T16:35:27.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.301+0000 7fd805845b80 1 -- start start 2023-10-11T16:35:27.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.302+0000 7fd805845b80 1 Processor -- start 2023-10-11T16:35:27.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.302+0000 7fd805845b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb8be0 0x563135fc99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.302+0000 7fd805845b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb94f0 0x563135fc7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.302+0000 7fd805845b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.302+0000 7fd805845b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135fc8aa0 con 0x563135fb8be0 2023-10-11T16:35:27.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.302+0000 7fd805845b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563135fc8c10 con 0x563135fbd930 2023-10-11T16:35:27.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.302+0000 7fd805845b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563136020710 con 0x563135fb94f0 2023-10-11T16:35:27.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.302+0000 7fd805845b80 10 client.0 _ll_register_callbacks cb 0x7fd7dc009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:27.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48286/0 (socket says 172.21.15.111:48286) 2023-10-11T16:35:27.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 -- 172.21.15.111:0/1982489369 learned_addr learned my addr 172.21.15.111:0/1982489369 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb94f0 0x563135fc7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 -- 172.21.15.111:0/1982489369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb94f0 msgr2=0x563135fc7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb94f0 0x563135fc7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 -- 172.21.15.111:0/1982489369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb8be0 msgr2=0x563135fc99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb8be0 0x563135fc99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 -- 172.21.15.111:0/1982489369 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec116950 con 0x563135fbd930 2023-10-11T16:35:27.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7eaffd700 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb94f0 0x563135fc7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:27.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7ebfff700 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc83a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40956c0 tx=0x7fd7e40092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8860116d94b26995 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.303+0000 7fd7caffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:27.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.304+0000 7fd7caffd700 1 -- 172.21.15.111:0/1982489369 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4099940 con 0x563135fbd930 2023-10-11T16:35:27.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.304+0000 7fd7caffd700 1 -- 172.21.15.111:0/1982489369 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd7e409d970 con 0x563135fbd930 2023-10-11T16:35:27.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.304+0000 7fd805845b80 1 -- 172.21.15.111:0/1982489369 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7dc00ec90 con 0x563135fbd930 2023-10-11T16:35:27.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.305+0000 7fd7caffd700 1 -- 172.21.15.111:0/1982489369 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7e4097ce0 con 0x563135fbd930 2023-10-11T16:35:27.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.305+0000 7fd805845b80 10 client.26216 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:27.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.305+0000 7fd805845b80 1 -- 172.21.15.111:0/1982489369 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563135d37e60 con 0x563135fbd930 2023-10-11T16:35:27.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.306+0000 7fd7caffd700 1 -- 172.21.15.111:0/1982489369 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7dc00ec90 con 0x563135fbd930 2023-10-11T16:35:27.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.306+0000 7fd7caffd700 1 -- 172.21.15.111:0/1982489369 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fd7e40c9030 con 0x563135fbd930 2023-10-11T16:35:27.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.306+0000 7fd805845b80 10 client.26216 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:27.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.306+0000 7fd805845b80 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:27.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.306+0000 7fd805845b80 20 client.26216 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:27.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.306+0000 7fd805845b80 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563136021810 0x563136021be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.306+0000 7fd805845b80 4 client.26216 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:35:27.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.306+0000 7fd805845b80 1 -- 172.21.15.111:0/1982489369 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563135ed86e0 con 0x563136021810 2023-10-11T16:35:27.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.307+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563136021810 0x563136021be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.307+0000 7fd7caffd700 10 client.26216 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:27.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.307+0000 7fd7eb7fe700 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563136021810 0x563136021be0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e74c29e6d6dfd4 server_cookie=fdad2f637353739b in_seq=0 out_seq=0 2023-10-11T16:35:27.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.308+0000 7fd7caffd700 1 -- 172.21.15.111:0/1982489369 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563135ed86e0 con 0x563136021810 2023-10-11T16:35:27.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.308+0000 7fd7caffd700 10 client.26216 handle_command_reply: tid=0 2023-10-11T16:35:27.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.308+0000 7fd805845b80 1 client.26216 shutdown 2023-10-11T16:35:27.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.311+0000 7fd805845b80 1 -- 172.21.15.111:0/1982489369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 msgr2=0x563135fc83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.311+0000 7fd805845b80 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc83a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40956c0 tx=0x7fd7e40092e0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.311+0000 7fd805845b80 1 -- 172.21.15.111:0/1982489369 shutdown_connections 2023-10-11T16:35:27.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.311+0000 7fd805845b80 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563136021810 0x563136021be0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.312+0000 7fd805845b80 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563135fbd930 0x563135fc83a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.312+0000 7fd805845b80 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563135fb8be0 0x563135fc99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.312+0000 7fd805845b80 1 --2- 172.21.15.111:0/1982489369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563135fb94f0 0x563135fc7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.312+0000 7fd805845b80 1 -- 172.21.15.111:0/1982489369 >> 172.21.15.111:0/1982489369 conn(0x563135fae3e0 msgr2=0x563135fb79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.312+0000 7fd7caffd700 0 client.26216 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:27.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.312+0000 7fd805845b80 1 -- 172.21.15.111:0/1982489369 shutdown_connections 2023-10-11T16:35:27.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.312+0000 7fd805845b80 1 -- 172.21.15.111:0/1982489369 wait complete. 2023-10-11T16:35:27.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.312+0000 7fd805845b80 20 client.26216 trim_cache size 0 max 16384 2023-10-11T16:35:27.320 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:35:27.320 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:35:27.321 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "e4b353b6-bcd8-4549-a24b-3247f2c5740c", 2023-10-11T16:35:27.321 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:35:27.321 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:35:27.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.313+0000 7fd7f1fff700 1 -- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7d40a1340 msgr2=0x7fd7d40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.313+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7d40a1340 0x7fd7d40a37f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0005950 tx=0x7fd7e000f720 comp rx=0 tx=0).stop 2023-10-11T16:35:27.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.313+0000 7fd7f1fff700 1 -- 172.21.15.111:0/2513477333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7ec104680 msgr2=0x7fd7ec19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.313+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7ec104680 0x7fd7ec19c8c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400ef30 tx=0x7fd7e400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.314+0000 7fd7f1fff700 1 -- 172.21.15.111:0/2513477333 shutdown_connections 2023-10-11T16:35:27.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.314+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7d40a1340 0x7fd7d40a37f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.314+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7ec104680 0x7fd7ec19c8c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.314+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7ec0ff180 0x7fd7ec1a28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.314+0000 7fd7f1fff700 1 --2- 172.21.15.111:0/2513477333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7ec0ffa90 0x7fd7ec1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.314+0000 7fd7f1fff700 1 -- 172.21.15.111:0/2513477333 >> 172.21.15.111:0/2513477333 conn(0x7fd7ec0770b0 msgr2=0x7fd7ec103190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.315+0000 7fd7f1fff700 1 -- 172.21.15.111:0/2513477333 shutdown_connections 2023-10-11T16:35:27.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.315+0000 7fd7f1fff700 1 -- 172.21.15.111:0/2513477333 wait complete. 2023-10-11T16:35:27.340 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:35:27.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.603+0000 7fb6b4bff700 1 -- 172.21.15.111:0/2879206023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6b0104680 msgr2=0x7fb6b0104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.603+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/2879206023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6b0104680 0x7fb6b0104af0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800b0a0 tx=0x7fb6a800b3b0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.604+0000 7fb6b4bff700 1 -- 172.21.15.111:0/2879206023 shutdown_connections 2023-10-11T16:35:27.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.604+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/2879206023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6b0104680 0x7fb6b0104af0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.604+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/2879206023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6b00ff180 0x7fb6b00ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.604+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/2879206023 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b00ffa90 0x7fb6b0104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.604+0000 7fb6b4bff700 1 -- 172.21.15.111:0/2879206023 >> 172.21.15.111:0/2879206023 conn(0x7fb6b00770b0 msgr2=0x7fb6b00774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.604+0000 7fb6b4bff700 1 -- 172.21.15.111:0/2879206023 shutdown_connections 2023-10-11T16:35:27.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.604+0000 7fb6b4bff700 1 -- 172.21.15.111:0/2879206023 wait complete. 2023-10-11T16:35:27.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.606+0000 7fb6b4bff700 1 Processor -- start 2023-10-11T16:35:27.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.606+0000 7fb6b4bff700 1 -- start start 2023-10-11T16:35:27.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.606+0000 7fb6b4bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b00ff180 0x7fb6b01a2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6b4bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6b00ffa90 0x7fb6b01a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6b4bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6b0104680 0x7fb6b019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6b4bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b019d000 con 0x7fb6b00ffa90 2023-10-11T16:35:27.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6b4bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b019d170 con 0x7fb6b00ff180 2023-10-11T16:35:27.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6b4bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b019d330 con 0x7fb6b0104680 2023-10-11T16:35:27.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b00ff180 0x7fb6b01a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b00ff180 0x7fb6b01a2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48294/0 (socket says 172.21.15.111:48294) 2023-10-11T16:35:27.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6aed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6b0104680 0x7fb6b019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6ae59c700 1 -- 172.21.15.111:0/1766691833 learned_addr learned my addr 172.21.15.111:0/1766691833 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6add9b700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6b00ffa90 0x7fb6b01a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6ae59c700 1 -- 172.21.15.111:0/1766691833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6b0104680 msgr2=0x7fb6b019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6b0104680 0x7fb6b019c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6ae59c700 1 -- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6b00ffa90 msgr2=0x7fb6b01a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6b00ffa90 0x7fb6b01a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.607+0000 7fb6ae59c700 1 -- 172.21.15.111:0/1766691833 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8009d00 con 0x7fb6b00ff180 2023-10-11T16:35:27.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.608+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b00ff180 0x7fb6b01a2920 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000d980 tx=0x7fb6a000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.608+0000 7fb69f7fe700 1 -- 172.21.15.111:0/1766691833 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0004220 con 0x7fb6b00ff180 2023-10-11T16:35:27.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.608+0000 7fb69f7fe700 1 -- 172.21.15.111:0/1766691833 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6a0004cf0 con 0x7fb6b00ff180 2023-10-11T16:35:27.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.608+0000 7fb69f7fe700 1 -- 172.21.15.111:0/1766691833 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0005700 con 0x7fb6b00ff180 2023-10-11T16:35:27.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.608+0000 7fb6b4bff700 1 -- 172.21.15.111:0/1766691833 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b0116950 con 0x7fb6b00ff180 2023-10-11T16:35:27.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.609+0000 7fb6b4bff700 1 -- 172.21.15.111:0/1766691833 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b01167c0 con 0x7fb6b00ff180 2023-10-11T16:35:27.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.613+0000 7fb69f7fe700 1 -- 172.21.15.111:0/1766691833 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fb6a0004750 con 0x7fb6b00ff180 2023-10-11T16:35:27.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.615+0000 7fb69f7fe700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6980a1450 0x7fb6980a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.615+0000 7fb69f7fe700 1 -- 172.21.15.111:0/1766691833 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb6a00cce20 con 0x7fb6b00ff180 2023-10-11T16:35:27.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.615+0000 7fb6c852fb80 1 Processor -- start 2023-10-11T16:35:27.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.616+0000 7fb6add9b700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6980a1450 0x7fb6980a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.616+0000 7fb6c852fb80 1 -- start start 2023-10-11T16:35:27.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.616+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456cb60 0x557cc456cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.616+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc4571370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.616+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc45718b0 0x557cc4571d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.616+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc4472060 con 0x557cc456cb60 2023-10-11T16:35:27.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.616+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc44c48a0 con 0x557cc45718b0 2023-10-11T16:35:27.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.616+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc44bdd30 con 0x557cc456d470 2023-10-11T16:35:27.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6add9b700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6980a1450 0x7fb6980a3900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4005fd0 tx=0x7fb6a4005ee0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc45718b0 0x557cc4571d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc45718b0 0x557cc4571d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:48304/0 (socket says 172.21.15.111:48304) 2023-10-11T16:35:27.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 -- 172.21.15.111:0/2075263956 learned_addr learned my addr 172.21.15.111:0/2075263956 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6add9b700 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc4571370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456cb60 0x557cc456cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 -- 172.21.15.111:0/2075263956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 msgr2=0x557cc4571370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc4571370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 -- 172.21.15.111:0/2075263956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456cb60 msgr2=0x557cc456cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456cb60 0x557cc456cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 -- 172.21.15.111:0/2075263956 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cc433f630 con 0x557cc45718b0 2023-10-11T16:35:27.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.617+0000 7fb6aed9d700 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc45718b0 0x557cc4571d00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800ced0 tx=0x7fb6a800cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a27bc22e4ec745 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.618+0000 7fb69cff9700 1 -- 172.21.15.111:0/2075263956 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a800e040 con 0x557cc45718b0 2023-10-11T16:35:27.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.618+0000 7fb69cff9700 1 -- 172.21.15.111:0/2075263956 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6a80093d0 con 0x557cc45718b0 2023-10-11T16:35:27.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.618+0000 7fb69cff9700 1 -- 172.21.15.111:0/2075263956 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a801c680 con 0x557cc45718b0 2023-10-11T16:35:27.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.619+0000 7fb6c852fb80 1 -- 172.21.15.111:0/2075263956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc45718b0 msgr2=0x557cc4571d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.619+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc45718b0 0x557cc4571d00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800ced0 tx=0x7fb6a800cf00 comp rx=0 tx=0).stop 2023-10-11T16:35:27.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.619+0000 7fb6c852fb80 1 -- 172.21.15.111:0/2075263956 shutdown_connections 2023-10-11T16:35:27.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.619+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc45718b0 0x557cc4571d00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.619+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456cb60 0x557cc456cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.619+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/2075263956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc4571370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.619+0000 7fb6c852fb80 1 -- 172.21.15.111:0/2075263956 >> 172.21.15.111:0/2075263956 conn(0x557cc4562360 msgr2=0x557cc4567b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.620+0000 7fb6c852fb80 1 -- 172.21.15.111:0/2075263956 shutdown_connections 2023-10-11T16:35:27.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.620+0000 7fb6c852fb80 1 -- 172.21.15.111:0/2075263956 wait complete. 2023-10-11T16:35:27.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.620+0000 7fb6c852fb80 1 -- start start 2023-10-11T16:35:27.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.620+0000 7fb6c852fb80 1 Processor -- start 2023-10-11T16:35:27.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.621+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 0x557cc4579100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.621+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456d470 0x557cc45763f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.621+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc45718b0 0x557cc4576b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.621+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc445f8e0 con 0x557cc456d470 2023-10-11T16:35:27.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.621+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc445f580 con 0x557cc456cb60 2023-10-11T16:35:27.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.621+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc4341920 con 0x557cc45718b0 2023-10-11T16:35:27.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.621+0000 7fb6c852fb80 10 client.0 _ll_register_callbacks cb 0x557cc447d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:27.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.621+0000 7fb6aed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc45718b0 0x557cc4576b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6aed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc45718b0 0x557cc4576b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52776/0 (socket says 172.21.15.111:52776) 2023-10-11T16:35:27.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6aed9d700 1 -- 172.21.15.111:0/3440072568 learned_addr learned my addr 172.21.15.111:0/3440072568 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 0x557cc4579100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6aed9d700 1 -- 172.21.15.111:0/3440072568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 msgr2=0x557cc4579100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6aed9d700 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 0x557cc4579100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6aed9d700 1 -- 172.21.15.111:0/3440072568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456d470 msgr2=0x557cc45763f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6aed9d700 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456d470 0x557cc45763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6aed9d700 1 -- 172.21.15.111:0/3440072568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b0114090 con 0x557cc45718b0 2023-10-11T16:35:27.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.622+0000 7fb6aed9d700 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc45718b0 0x557cc4576b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80076d0 tx=0x7fb6a8007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d81fef4c50b2bfcb server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.623+0000 7fb695ffb700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:35:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.623+0000 7fb695ffb700 1 -- 172.21.15.111:0/3440072568 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a800e040 con 0x557cc45718b0 2023-10-11T16:35:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.623+0000 7fb695ffb700 1 -- 172.21.15.111:0/3440072568 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb6a80052d0 con 0x557cc45718b0 2023-10-11T16:35:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.623+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3440072568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557cc455fb20 con 0x557cc45718b0 2023-10-11T16:35:27.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.624+0000 7fb695ffb700 1 -- 172.21.15.111:0/3440072568 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6a80079d0 con 0x557cc45718b0 2023-10-11T16:35:27.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.624+0000 7fb6c852fb80 10 client.26263 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:27.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.624+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3440072568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557cc4484190 con 0x557cc45718b0 2023-10-11T16:35:27.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.624+0000 7fb695ffb700 1 -- 172.21.15.111:0/3440072568 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557cc455fb20 con 0x557cc45718b0 2023-10-11T16:35:27.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.624+0000 7fb695ffb700 1 -- 172.21.15.111:0/3440072568 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fb6a8019070 con 0x557cc45718b0 2023-10-11T16:35:27.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb6c852fb80 10 client.26263 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:27.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb6c852fb80 10 client.26263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:27.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb6c852fb80 20 client.26263 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:27.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557cc4586470 0x557cc4588880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb6c852fb80 4 client.26263 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3440072568 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557cc44800a0 con 0x557cc4586470 2023-10-11T16:35:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557cc4586470 0x557cc4588880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557cc4586470 0x557cc4588880 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=353002eb7160146e server_cookie=a764a3b462df6892 in_seq=0 out_seq=0 2023-10-11T16:35:27.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.625+0000 7fb695ffb700 10 client.26263 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.627+0000 7fb695ffb700 1 -- 172.21.15.111:0/3440072568 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557cc44800a0 con 0x557cc4586470 2023-10-11T16:35:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.627+0000 7fb695ffb700 10 client.26263 handle_command_reply: tid=0 2023-10-11T16:35:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.627+0000 7fb6c852fb80 1 client.26263 shutdown 2023-10-11T16:35:27.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3440072568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc45718b0 msgr2=0x557cc4576b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc45718b0 0x557cc4576b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80076d0 tx=0x7fb6a8007700 comp rx=0 tx=0).stop 2023-10-11T16:35:27.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3440072568 shutdown_connections 2023-10-11T16:35:27.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557cc4586470 0x557cc4588880 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc45718b0 0x557cc4576b30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 0x557cc4579100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3440072568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc456d470 0x557cc45763f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3440072568 >> 172.21.15.111:0/3440072568 conn(0x557cc4562360 msgr2=0x557cc4572d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb695ffb700 0 client.26263 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:27.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3440072568 shutdown_connections 2023-10-11T16:35:27.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.629+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3440072568 wait complete. 2023-10-11T16:35:27.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.630+0000 7fb6c852fb80 20 client.26263 trim_cache size 0 max 16384 2023-10-11T16:35:27.640 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:27 smithi111 ceph-mon[98527]: pgmap v87: 129 pgs: 129 active+clean; 86 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.17k op/s 2023-10-11T16:35:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.654+0000 7fb6c852fb80 1 Processor -- start 2023-10-11T16:35:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.654+0000 7fb6c852fb80 1 -- start start 2023-10-11T16:35:27.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 0x557cc4573c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc456f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 0x557cc456ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc45692c0 con 0x557cc45718b0 2023-10-11T16:35:27.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc4583300 con 0x557cc456cb60 2023-10-11T16:35:27.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc4570540 con 0x557cc456d470 2023-10-11T16:35:27.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6add9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc456f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6add9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc456f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52780/0 (socket says 172.21.15.111:52780) 2023-10-11T16:35:27.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6add9b700 1 -- 172.21.15.111:0/3830800565 learned_addr learned my addr 172.21.15.111:0/3830800565 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.655+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 0x557cc4573c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.656+0000 7fb6add9b700 1 -- 172.21.15.111:0/3830800565 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 msgr2=0x557cc4573c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.656+0000 7fb6add9b700 1 --2- 172.21.15.111:0/3830800565 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 0x557cc4573c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.656+0000 7fb6add9b700 1 -- 172.21.15.111:0/3830800565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 msgr2=0x557cc456ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.656+0000 7fb6add9b700 1 --2- 172.21.15.111:0/3830800565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 0x557cc456ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.656+0000 7fb6add9b700 1 -- 172.21.15.111:0/3830800565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cc4563b00 con 0x557cc456d470 2023-10-11T16:35:27.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.656+0000 7fb6add9b700 1 --2- 172.21.15.111:0/3830800565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc456f9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00fd4d0 tx=0x7fb6a40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76418f3dbb41a1e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.657+0000 7fb69d7fa700 1 -- 172.21.15.111:0/3830800565 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4011850 con 0x557cc456d470 2023-10-11T16:35:27.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.657+0000 7fb69d7fa700 1 -- 172.21.15.111:0/3830800565 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6a4011e70 con 0x557cc456d470 2023-10-11T16:35:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.657+0000 7fb69d7fa700 1 -- 172.21.15.111:0/3830800565 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a40103d0 con 0x557cc456d470 2023-10-11T16:35:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.658+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3830800565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 msgr2=0x557cc456f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.658+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3830800565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc456f9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00fd4d0 tx=0x7fb6a40047c0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.658+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3830800565 shutdown_connections 2023-10-11T16:35:27.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.658+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3830800565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 0x557cc456ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.658+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3830800565 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456cb60 0x557cc4573c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.658+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/3830800565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456d470 0x557cc456f9d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.658+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3830800565 >> 172.21.15.111:0/3830800565 conn(0x557cc4562360 msgr2=0x557cc4588480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.658+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3830800565 shutdown_connections 2023-10-11T16:35:27.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.659+0000 7fb6c852fb80 1 -- 172.21.15.111:0/3830800565 wait complete. 2023-10-11T16:35:27.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.659+0000 7fb6c852fb80 1 -- start start 2023-10-11T16:35:27.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.659+0000 7fb6c852fb80 1 Processor -- start 2023-10-11T16:35:27.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456cb60 0x557cc457b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456d470 0x557cc457be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6c852fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 0x557cc457c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc45d9440 con 0x557cc45718b0 2023-10-11T16:35:27.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc45d95b0 con 0x557cc456d470 2023-10-11T16:35:27.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6c852fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cc45d9770 con 0x557cc456cb60 2023-10-11T16:35:27.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6c852fb80 10 client.0 _ll_register_callbacks cb 0x7fb6a800ce20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:27.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456cb60 0x557cc457b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456cb60 0x557cc457b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52786/0 (socket says 172.21.15.111:52786) 2023-10-11T16:35:27.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6ae59c700 1 -- 172.21.15.111:0/245836332 learned_addr learned my addr 172.21.15.111:0/245836332 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:27.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6aed9d700 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 0x557cc457c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.660+0000 7fb6add9b700 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456d470 0x557cc457be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.661+0000 7fb6add9b700 1 -- 172.21.15.111:0/245836332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456cb60 msgr2=0x557cc457b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.661+0000 7fb6add9b700 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456cb60 0x557cc457b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.661+0000 7fb6add9b700 1 -- 172.21.15.111:0/245836332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 msgr2=0x557cc457c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:27.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.661+0000 7fb6add9b700 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 0x557cc457c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.661+0000 7fb6add9b700 1 -- 172.21.15.111:0/245836332 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a400e7b0 con 0x557cc456d470 2023-10-11T16:35:27.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.661+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456cb60 0x557cc457b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:27.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.661+0000 7fb6add9b700 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456d470 0x557cc457be70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x557cc44797c0 tx=0x7fb6a4009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec1c424eb61d1a84 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:27.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.661+0000 7fb695ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:27.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.662+0000 7fb695ffb700 1 -- 172.21.15.111:0/245836332 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4011a50 con 0x557cc456d470 2023-10-11T16:35:27.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.662+0000 7fb695ffb700 1 -- 172.21.15.111:0/245836332 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fb6a4011bb0 con 0x557cc456d470 2023-10-11T16:35:27.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.662+0000 7fb6c852fb80 1 -- 172.21.15.111:0/245836332 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb6a80093d0 con 0x557cc456d470 2023-10-11T16:35:27.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.662+0000 7fb695ffb700 1 -- 172.21.15.111:0/245836332 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6a4028450 con 0x557cc456d470 2023-10-11T16:35:27.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.662+0000 7fb6c852fb80 10 client.26234 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:27.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.662+0000 7fb6c852fb80 1 -- 172.21.15.111:0/245836332 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557cc42ebde0 con 0x557cc456d470 2023-10-11T16:35:27.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.663+0000 7fb695ffb700 1 -- 172.21.15.111:0/245836332 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6a80093d0 con 0x557cc456d470 2023-10-11T16:35:27.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.663+0000 7fb695ffb700 1 -- 172.21.15.111:0/245836332 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fb6a4032480 con 0x557cc456d470 2023-10-11T16:35:27.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.663+0000 7fb6c852fb80 10 client.26234 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:27.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.663+0000 7fb6c852fb80 10 client.26234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:27.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.663+0000 7fb6c852fb80 20 client.26234 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:27.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.663+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557cc45dacb0 0x557cc45dd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:27.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.663+0000 7fb6c852fb80 4 client.26234 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:35:27.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.663+0000 7fb6c852fb80 1 -- 172.21.15.111:0/245836332 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557cc448c660 con 0x557cc45dacb0 2023-10-11T16:35:27.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.665+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557cc45dacb0 0x557cc45dd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.665+0000 7fb6ae59c700 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557cc45dacb0 0x557cc45dd0c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df9329526eb29f2b server_cookie=b9adec172c1910f2 in_seq=0 out_seq=0 2023-10-11T16:35:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.665+0000 7fb695ffb700 10 client.26234 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.668+0000 7fb695ffb700 1 -- 172.21.15.111:0/245836332 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x557cc448c660 con 0x557cc45dacb0 2023-10-11T16:35:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.668+0000 7fb695ffb700 10 client.26234 handle_command_reply: tid=0 2023-10-11T16:35:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.668+0000 7fb6c852fb80 1 client.26234 shutdown 2023-10-11T16:35:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 -- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456d470 msgr2=0x557cc457be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456d470 0x557cc457be70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x557cc44797c0 tx=0x7fb6a4009310 comp rx=0 tx=0).stop 2023-10-11T16:35:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 -- 172.21.15.111:0/245836332 shutdown_connections 2023-10-11T16:35:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557cc45dacb0 0x557cc45dd0c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557cc45718b0 0x557cc457c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557cc456cb60 0x557cc457b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 --2- 172.21.15.111:0/245836332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557cc456d470 0x557cc457be70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 -- 172.21.15.111:0/245836332 >> 172.21.15.111:0/245836332 conn(0x557cc4562360 msgr2=0x557cc4573c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb695ffb700 0 client.26234 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 -- 172.21.15.111:0/245836332 shutdown_connections 2023-10-11T16:35:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 1 -- 172.21.15.111:0/245836332 wait complete. 2023-10-11T16:35:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.669+0000 7fb6c852fb80 20 client.26234 trim_cache size 0 max 16384 2023-10-11T16:35:27.681 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:35:27.682 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "scrub active (0 inodes in the stack)", 2023-10-11T16:35:27.682 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:35:27.682 INFO:teuthology.orchestra.run.smithi111.stdout: "e4b353b6-bcd8-4549-a24b-3247f2c5740c": { 2023-10-11T16:35:27.682 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:35:27.682 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "e4b353b6-bcd8-4549-a24b-3247f2c5740c", 2023-10-11T16:35:27.683 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:35:27.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:35:27.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:35:27.683 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:35:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.670+0000 7fb6b4bff700 1 -- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6980a1450 msgr2=0x7fb6980a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.670+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6980a1450 0x7fb6980a3900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4005fd0 tx=0x7fb6a4005ee0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.670+0000 7fb6b4bff700 1 -- 172.21.15.111:0/1766691833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b00ff180 msgr2=0x7fb6b01a2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.670+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b00ff180 0x7fb6b01a2920 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000d980 tx=0x7fb6a000dc90 comp rx=0 tx=0).stop 2023-10-11T16:35:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.671+0000 7fb6b4bff700 1 -- 172.21.15.111:0/1766691833 shutdown_connections 2023-10-11T16:35:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.671+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6980a1450 0x7fb6980a3900 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.671+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6b0104680 0x7fb6b019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.671+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b00ff180 0x7fb6b01a2920 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.671+0000 7fb6b4bff700 1 --2- 172.21.15.111:0/1766691833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6b00ffa90 0x7fb6b01a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.671+0000 7fb6b4bff700 1 -- 172.21.15.111:0/1766691833 >> 172.21.15.111:0/1766691833 conn(0x7fb6b00770b0 msgr2=0x7fb6b01021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.672+0000 7fb6b4bff700 1 -- 172.21.15.111:0/1766691833 shutdown_connections 2023-10-11T16:35:27.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:27.673+0000 7fb6b4bff700 1 -- 172.21.15.111:0/1766691833 wait complete. 2023-10-11T16:35:27.699 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b353b6-bcd8-4549-a24b-3247f2c5740c - {'path': '/', 'tag': 'e4b353b6-bcd8-4549-a24b-3247f2c5740c', 'options': 'recursive,force'} 2023-10-11T16:35:27.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:27 smithi150 ceph-mon[132217]: pgmap v87: 129 pgs: 129 active+clean; 86 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.17k op/s 2023-10-11T16:35:27.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:27 smithi130 ceph-mon[131859]: pgmap v87: 129 pgs: 129 active+clean; 86 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.17k op/s 2023-10-11T16:35:28.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:28 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:35:28.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:28 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:28.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:28 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:28.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:28 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:28.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:28 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:35:28.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:28 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:28.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:28 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:28.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:28 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:28.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:28 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:35:28.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:28 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:28.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:28 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:28.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:28 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:29.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:29 smithi111 ceph-mon[98527]: pgmap v88: 129 pgs: 129 active+clean; 86 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.17k op/s 2023-10-11T16:35:29.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:29 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:29.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:29 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:29.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:29 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:29.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:29 smithi150 ceph-mon[132217]: pgmap v88: 129 pgs: 129 active+clean; 86 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.17k op/s 2023-10-11T16:35:29.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:29 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:29.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:29 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:29.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:29 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:29.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:29 smithi130 ceph-mon[131859]: pgmap v88: 129 pgs: 129 active+clean; 86 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.17k op/s 2023-10-11T16:35:29.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:29 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:29.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:29 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:29.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:29 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:30.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:30 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:30.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:30 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:30.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:30 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:30.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:30 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:30.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:30 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:30.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:30 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:31.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:31 smithi150 ceph-mon[132217]: pgmap v89: 129 pgs: 129 active+clean; 89 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 2.33k op/s 2023-10-11T16:35:31.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:31 smithi130 ceph-mon[131859]: pgmap v89: 129 pgs: 129 active+clean; 89 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 2.33k op/s 2023-10-11T16:35:31.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:31 smithi111 ceph-mon[98527]: pgmap v89: 129 pgs: 129 active+clean; 89 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 2.33k op/s 2023-10-11T16:35:33.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: pgmap v90: 129 pgs: 129 active+clean; 89 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.31k op/s 2023-10-11T16:35:33.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.722 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.722 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.722 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.725 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.725 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.725 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.726 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.726 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.726 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.727 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.727 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.727 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.728 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.728 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.728 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.728 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.728 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.729 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.729 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.729 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.729 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.729 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.730 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.730 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.730 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.730 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.730 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.731 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.731 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.731 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.731 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.732 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.733 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.733 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.734 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.734 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.734 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.735 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.735 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.735 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.736 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.736 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.737 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.737 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.737 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:33.738 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:33 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: pgmap v90: 129 pgs: 129 active+clean; 89 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.31k op/s 2023-10-11T16:35:33.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.843 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.843 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.855 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.858 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.859 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.860 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.861 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:33.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:33 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: pgmap v90: 129 pgs: 129 active+clean; 89 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 2.31k op/s 2023-10-11T16:35:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.898 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.898 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.898 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.898 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.898 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.906 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.907 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.907 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.907 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.907 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.908 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.908 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.908 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.908 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.908 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.909 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.909 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.909 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.909 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.909 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.909 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.910 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.910 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.910 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.910 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.910 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.911 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.911 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.911 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:33.911 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:33.912 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:33 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:34 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:34.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:34 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:34 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:34 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:34 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:34 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:34 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:34.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:34 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:34 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:34.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:34 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:34 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:34.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:34 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:34 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:34.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:34 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:34 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:34.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:34 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:34 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:34.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:34 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:34 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:34.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:34 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:34 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:34.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:34 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:34.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:34 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:34.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:34 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: pgmap v91: 129 pgs: 129 active+clean; 104 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 2.36k op/s 2023-10-11T16:35:35.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.709 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.710 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.711 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.712 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.713 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.714 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.715 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.716 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.717 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.718 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.719 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.720 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.721 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.722 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.722 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.722 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.722 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.722 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.723 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.724 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:35:35.725 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:35 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: pgmap v91: 129 pgs: 129 active+clean; 104 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 2.36k op/s 2023-10-11T16:35:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.840 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.843 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.843 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.843 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.843 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.843 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.844 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.845 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.846 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.847 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.848 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.850 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:35:35.853 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:35 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: pgmap v91: 129 pgs: 129 active+clean; 104 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 2.36k op/s 2023-10-11T16:35:35.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.892 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.893 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.894 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.895 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.896 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.897 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.898 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.898 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.898 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.899 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.900 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.901 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.902 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.903 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.904 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.905 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.906 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.906 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.906 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:35.906 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.907 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:35:35.907 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:35 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 -- 172.21.15.111:0/3103830800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 msgr2=0x7fdd8c100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3103830800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 0x7fdd8c100930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c009990 tx=0x7fdd7c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 -- 172.21.15.111:0/3103830800 shutdown_connections 2023-10-11T16:35:37.470 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3103830800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c1055a0 0x7fdd8c10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3103830800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 0x7fdd8c100930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3103830800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd8c100f00 0x7fdd8c104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 -- 172.21.15.111:0/3103830800 >> 172.21.15.111:0/3103830800 conn(0x7fdd8c0781b0 msgr2=0x7fdd8c0785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 -- 172.21.15.111:0/3103830800 shutdown_connections 2023-10-11T16:35:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.468+0000 7fdd91dff700 1 -- 172.21.15.111:0/3103830800 wait complete. 2023-10-11T16:35:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.469+0000 7fdd91dff700 1 Processor -- start 2023-10-11T16:35:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.469+0000 7fdd91dff700 1 -- start start 2023-10-11T16:35:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd91dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 0x7fdd8c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd91dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd8c100f00 0x7fdd8c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd91dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c1055a0 0x7fdd8c1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd91dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c073b60 con 0x7fdd8c1055a0 2023-10-11T16:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd91dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c1a4b80 con 0x7fdd8c100560 2023-10-11T16:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd91dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c1a4cc0 con 0x7fdd8c100f00 2023-10-11T16:35:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 0x7fdd8c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd8c100f00 0x7fdd8c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c1055a0 0x7fdd8c1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 0x7fdd8c072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52238/0 (socket says 172.21.15.111:52238) 2023-10-11T16:35:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd8c100f00 0x7fdd8c073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:53080/0 (socket says 172.21.15.111:53080) 2023-10-11T16:35:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/3019473199 learned_addr learned my addr 172.21.15.111:0/3019473199 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/3019473199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd8c100f00 msgr2=0x7fdd8c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd8c100f00 0x7fdd8c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c1055a0 msgr2=0x7fdd8c1a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c1055a0 0x7fdd8c1a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.470+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/3019473199 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd80009520 con 0x7fdd8c100560 2023-10-11T16:35:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.471+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 0x7fdd8c072df0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c005910 tx=0x7fdd7c004710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.471+0000 7fdd88ff9700 1 -- 172.21.15.111:0/3019473199 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c01d070 con 0x7fdd8c100560 2023-10-11T16:35:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.471+0000 7fdd91dff700 1 -- 172.21.15.111:0/3019473199 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd84009c40 con 0x7fdd8c100560 2023-10-11T16:35:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.471+0000 7fdd88ff9700 1 -- 172.21.15.111:0/3019473199 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd7c022470 con 0x7fdd8c100560 2023-10-11T16:35:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.471+0000 7fdd88ff9700 1 -- 172.21.15.111:0/3019473199 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c017650 con 0x7fdd8c100560 2023-10-11T16:35:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.471+0000 7fdd91dff700 1 -- 172.21.15.111:0/3019473199 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c0095f0 con 0x7fdd8c100560 2023-10-11T16:35:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.474+0000 7fdd88ff9700 1 -- 172.21.15.111:0/3019473199 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fdd7c0225e0 con 0x7fdd8c100560 2023-10-11T16:35:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.475+0000 7fdda566bb80 1 Processor -- start 2023-10-11T16:35:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdd88ff9700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdd740a12c0 0x7fdd740a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdd88ff9700 1 -- 172.21.15.111:0/3019473199 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fdd7c0ce2f0 con 0x7fdd8c100560 2023-10-11T16:35:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdda566bb80 1 -- start start 2023-10-11T16:35:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdd8bfff700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdd740a12c0 0x7fdd740a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c220426b0 0x556c221ec1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ec720 0x556c221e6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdda566bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c220ec000 con 0x556c221ec720 2023-10-11T16:35:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdda566bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c2213e840 con 0x556c220426b0 2023-10-11T16:35:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.476+0000 7fdda566bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c22137cd0 con 0x556c221ecb90 2023-10-11T16:35:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c220426b0 0x556c221ec1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c220426b0 0x556c221ec1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52244/0 (socket says 172.21.15.111:52244) 2023-10-11T16:35:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/1483649371 learned_addr learned my addr 172.21.15.111:0/1483649371 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8bfff700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdd740a12c0 0x7fdd740a3770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c074f00 tx=0x7fdd8400b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8affd700 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ec720 0x556c221e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/1483649371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 msgr2=0x556c221e6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/1483649371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ec720 msgr2=0x556c221e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ec720 0x556c221e6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/1483649371 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c21fb95d0 con 0x556c220426b0 2023-10-11T16:35:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8bfff700 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c220426b0 0x556c221ec1e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c024000 tx=0x7fdd7c09ae10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f118eb5d71ab3b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.477+0000 7fdd71ffb700 1 -- 172.21.15.111:0/1483649371 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c09a9a0 con 0x556c220426b0 2023-10-11T16:35:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdd71ffb700 1 -- 172.21.15.111:0/1483649371 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd7c09b040 con 0x556c220426b0 2023-10-11T16:35:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdd71ffb700 1 -- 172.21.15.111:0/1483649371 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c09a530 con 0x556c220426b0 2023-10-11T16:35:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdda566bb80 1 -- 172.21.15.111:0/1483649371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c220426b0 msgr2=0x556c221ec1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdda566bb80 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c220426b0 0x556c221ec1e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c024000 tx=0x7fdd7c09ae10 comp rx=0 tx=0).stop 2023-10-11T16:35:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdda566bb80 1 -- 172.21.15.111:0/1483649371 shutdown_connections 2023-10-11T16:35:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdda566bb80 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdda566bb80 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c220426b0 0x556c221ec1e0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c024000 tx=0x7fdd7c09ae10 comp rx=0 tx=0).stop 2023-10-11T16:35:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdda566bb80 1 --2- 172.21.15.111:0/1483649371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ec720 0x556c221e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.478+0000 7fdda566bb80 1 -- 172.21.15.111:0/1483649371 >> 172.21.15.111:0/1483649371 conn(0x556c221dbf50 msgr2=0x556c221e1270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.479+0000 7fdda566bb80 1 -- 172.21.15.111:0/1483649371 shutdown_connections 2023-10-11T16:35:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.479+0000 7fdda566bb80 1 -- 172.21.15.111:0/1483649371 wait complete. 2023-10-11T16:35:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.479+0000 7fdda566bb80 1 -- start start 2023-10-11T16:35:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.479+0000 7fdda566bb80 1 Processor -- start 2023-10-11T16:35:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221efb40 0x556c221effb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdda566bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c220d9880 con 0x556c221ecb90 2023-10-11T16:35:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdda566bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c220d9520 con 0x556c221ec720 2023-10-11T16:35:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdda566bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c21fbb8c0 con 0x556c221efb40 2023-10-11T16:35:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdda566bb80 10 client.0 _ll_register_callbacks cb 0x556c220f7720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221f58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52256/0 (socket says 172.21.15.111:52256) 2023-10-11T16:35:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/3098518620 learned_addr learned my addr 172.21.15.111:0/3098518620 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdd8affd700 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.480+0000 7fdd8bfff700 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221efb40 0x556c221effb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/3098518620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221efb40 msgr2=0x556c221effb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221efb40 0x556c221effb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/3098518620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 msgr2=0x556c221f5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/3098518620 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c0095f0 con 0x556c221ec720 2023-10-11T16:35:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221f58f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c09ad40 tx=0x7fdd7c005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad28d5f5f3508dc server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd6affd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd6affd700 1 -- 172.21.15.111:0/3098518620 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c09a9a0 con 0x556c221ec720 2023-10-11T16:35:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd6affd700 1 -- 172.21.15.111:0/3098518620 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fdd7c0a3400 con 0x556c221ec720 2023-10-11T16:35:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdda566bb80 1 -- 172.21.15.111:0/3098518620 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556c221d9710 con 0x556c221ec720 2023-10-11T16:35:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdd6affd700 1 -- 172.21.15.111:0/3098518620 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd7c09b4b0 con 0x556c221ec720 2023-10-11T16:35:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdda566bb80 10 client.26246 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.481+0000 7fdda566bb80 1 -- 172.21.15.111:0/3098518620 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556c220fe130 con 0x556c221ec720 2023-10-11T16:35:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdd6affd700 1 -- 172.21.15.111:0/3098518620 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556c221d9710 con 0x556c221ec720 2023-10-11T16:35:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdd6affd700 1 -- 172.21.15.111:0/3098518620 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdd7c099fa0 con 0x556c221ec720 2023-10-11T16:35:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdda566bb80 10 client.26246 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdda566bb80 10 client.26246 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdda566bb80 20 client.26246 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdda566bb80 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x556c221ffc40 0x556c22202050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdda566bb80 4 client.26246 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdda566bb80 1 -- 172.21.15.111:0/3098518620 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556c220fa040 con 0x556c221ffc40 2023-10-11T16:35:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.482+0000 7fdd8affd700 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x556c221ffc40 0x556c22202050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.484+0000 7fdd8affd700 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x556c221ffc40 0x556c22202050 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2c6655b3339a1df server_cookie=a304ffc42e55ef10 in_seq=0 out_seq=0 2023-10-11T16:35:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.484+0000 7fdd6affd700 10 client.26246 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:37.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.487+0000 7fdd6affd700 1 -- 172.21.15.111:0/3098518620 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556c220fa040 con 0x556c221ffc40 2023-10-11T16:35:37.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.487+0000 7fdd6affd700 10 client.26246 handle_command_reply: tid=0 2023-10-11T16:35:37.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.487+0000 7fdda566bb80 1 client.26246 shutdown 2023-10-11T16:35:37.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 -- 172.21.15.111:0/3098518620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 msgr2=0x556c221f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:37.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221f58f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c09ad40 tx=0x7fdd7c005c70 comp rx=0 tx=0).stop 2023-10-11T16:35:37.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 -- 172.21.15.111:0/3098518620 shutdown_connections 2023-10-11T16:35:37.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x556c221ffc40 0x556c22202050 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221efb40 0x556c221effb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221f58f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 --2- 172.21.15.111:0/3098518620 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 -- 172.21.15.111:0/3098518620 >> 172.21.15.111:0/3098518620 conn(0x556c221dbf50 msgr2=0x556c221e07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdd6affd700 0 client.26246 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 -- 172.21.15.111:0/3098518620 shutdown_connections 2023-10-11T16:35:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.488+0000 7fdda566bb80 1 -- 172.21.15.111:0/3098518620 wait complete. 2023-10-11T16:35:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.489+0000 7fdda566bb80 20 client.26246 trim_cache size 0 max 16384 2023-10-11T16:35:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.506+0000 7fdda566bb80 1 Processor -- start 2023-10-11T16:35:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.506+0000 7fdda566bb80 1 -- start start 2023-10-11T16:35:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.506+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221e6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.506+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.506+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221efb40 0x556c221e8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.506+0000 7fdda566bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c221ddfd0 con 0x556c221efb40 2023-10-11T16:35:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.506+0000 7fdda566bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c221dbb70 con 0x556c221ec720 2023-10-11T16:35:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.506+0000 7fdda566bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c221f10d0 con 0x556c221ecb90 2023-10-11T16:35:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221efb40 0x556c221e8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221e6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52260/0 (socket says 172.21.15.111:52260) 2023-10-11T16:35:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/1891398767 learned_addr learned my addr 172.21.15.111:0/1891398767 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/1891398767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 msgr2=0x556c221e0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/1891398767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/1891398767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221efb40 msgr2=0x556c221e8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/1891398767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221efb40 0x556c221e8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 -- 172.21.15.111:0/1891398767 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c220fde80 con 0x556c221ec720 2023-10-11T16:35:37.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.507+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/1891398767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221e6560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c09b6a0 tx=0x7fdd7c09ae10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5380d64890e666b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:37.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdd727fc700 1 -- 172.21.15.111:0/1891398767 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c09a9a0 con 0x556c221ec720 2023-10-11T16:35:37.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdd727fc700 1 -- 172.21.15.111:0/1891398767 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd7c0a3400 con 0x556c221ec720 2023-10-11T16:35:37.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdd727fc700 1 -- 172.21.15.111:0/1891398767 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c0a31d0 con 0x556c221ec720 2023-10-11T16:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdda566bb80 1 -- 172.21.15.111:0/1891398767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 msgr2=0x556c221e6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdda566bb80 1 --2- 172.21.15.111:0/1891398767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221e6560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c09b6a0 tx=0x7fdd7c09ae10 comp rx=0 tx=0).stop 2023-10-11T16:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdda566bb80 1 -- 172.21.15.111:0/1891398767 shutdown_connections 2023-10-11T16:35:37.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdda566bb80 1 --2- 172.21.15.111:0/1891398767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221efb40 0x556c221e8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdda566bb80 1 --2- 172.21.15.111:0/1891398767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221ec720 0x556c221e6560 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdda566bb80 1 --2- 172.21.15.111:0/1891398767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ecb90 0x556c221e0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.508+0000 7fdda566bb80 1 -- 172.21.15.111:0/1891398767 >> 172.21.15.111:0/1891398767 conn(0x556c221dbf50 msgr2=0x556c221ef3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.509+0000 7fdda566bb80 1 -- 172.21.15.111:0/1891398767 shutdown_connections 2023-10-11T16:35:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.509+0000 7fdda566bb80 1 -- 172.21.15.111:0/1891398767 wait complete. 2023-10-11T16:35:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.509+0000 7fdda566bb80 1 -- start start 2023-10-11T16:35:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.509+0000 7fdda566bb80 1 Processor -- start 2023-10-11T16:35:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ec720 0x556c221f3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdda566bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221efb40 0x556c22252b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdda566bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c221f4490 con 0x556c221ecb90 2023-10-11T16:35:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdda566bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c22253040 con 0x556c221efb40 2023-10-11T16:35:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdda566bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c22253180 con 0x556c221ec720 2023-10-11T16:35:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdda566bb80 10 client.0 _ll_register_callbacks cb 0x556c220f7720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdd8b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ec720 0x556c221f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdd8bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221efb40 0x556c22252b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdd8bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221efb40 0x556c22252b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52270/0 (socket says 172.21.15.111:52270) 2023-10-11T16:35:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdd8bfff700 1 -- 172.21.15.111:0/964405547 learned_addr learned my addr 172.21.15.111:0/964405547 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.510+0000 7fdd8affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.512+0000 7fdd8bfff700 1 -- 172.21.15.111:0/964405547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ec720 msgr2=0x556c221f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.512+0000 7fdd8bfff700 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ec720 0x556c221f3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.512+0000 7fdd8bfff700 1 -- 172.21.15.111:0/964405547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 msgr2=0x556c221f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.512+0000 7fdd8bfff700 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.512+0000 7fdd8bfff700 1 -- 172.21.15.111:0/964405547 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c220fe130 con 0x556c221efb40 2023-10-11T16:35:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.512+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ec720 0x556c221f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.512+0000 7fdd8affd700 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.513+0000 7fdd8bfff700 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221efb40 0x556c22252b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd840234c0 tx=0x7fdd84004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caf5e76d00850d2f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.513+0000 7fdd6affd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.513+0000 7fdd6affd700 1 -- 172.21.15.111:0/964405547 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd84028070 con 0x556c221efb40 2023-10-11T16:35:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.513+0000 7fdd6affd700 1 -- 172.21.15.111:0/964405547 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fdd84009240 con 0x556c221efb40 2023-10-11T16:35:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.513+0000 7fdda566bb80 1 -- 172.21.15.111:0/964405547 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd7c09b040 con 0x556c221efb40 2023-10-11T16:35:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdd6affd700 1 -- 172.21.15.111:0/964405547 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd8400e040 con 0x556c221efb40 2023-10-11T16:35:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdda566bb80 10 client.26252 fetch_fsmap learned FSMap version 30 2023-10-11T16:35:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdda566bb80 1 -- 172.21.15.111:0/964405547 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556c21f4ec00 con 0x556c221efb40 2023-10-11T16:35:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdd6affd700 1 -- 172.21.15.111:0/964405547 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd7c09b040 con 0x556c221efb40 2023-10-11T16:35:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdd6affd700 1 -- 172.21.15.111:0/964405547 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 30) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdd84008b90 con 0x556c221efb40 2023-10-11T16:35:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdda566bb80 10 client.26252 fetch_fsmap finished waiting for FSMap version 30 2023-10-11T16:35:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdda566bb80 10 client.26252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdda566bb80 20 client.26252 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.514+0000 7fdda566bb80 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x556c22254940 0x556c22256d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.515+0000 7fdda566bb80 4 client.26252 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:35:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.515+0000 7fdda566bb80 1 -- 172.21.15.111:0/964405547 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556c22106600 con 0x556c22254940 2023-10-11T16:35:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.515+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x556c22254940 0x556c22256d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.516+0000 7fdd6affd700 10 client.26252 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:37.516+0000 7fdd8b7fe700 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x556c22254940 0x556c22256d50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b689a6ce3428ab9f server_cookie=1e3cdda6e304e7f1 in_seq=0 out_seq=0 2023-10-11T16:35:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:37 smithi150 ceph-mon[132217]: pgmap v92: 129 pgs: 129 active+clean; 109 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 870 op/s 2023-10-11T16:35:37.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:37 smithi130 ceph-mon[131859]: pgmap v92: 129 pgs: 129 active+clean; 109 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 870 op/s 2023-10-11T16:35:37.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:37 smithi111 ceph-mon[98527]: pgmap v92: 129 pgs: 129 active+clean; 109 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 870 op/s 2023-10-11T16:35:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:39 smithi150 ceph-mon[132217]: pgmap v93: 129 pgs: 129 active+clean; 109 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 224 op/s 2023-10-11T16:35:39.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:39 smithi130 ceph-mon[131859]: pgmap v93: 129 pgs: 129 active+clean; 109 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 224 op/s 2023-10-11T16:35:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:39 smithi111 ceph-mon[98527]: pgmap v93: 129 pgs: 129 active+clean; 109 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 224 op/s 2023-10-11T16:35:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdd6affd700 1 -- 172.21.15.111:0/964405547 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556c22106600 con 0x556c22254940 2023-10-11T16:35:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdd6affd700 10 client.26252 handle_command_reply: tid=0 2023-10-11T16:35:40.139 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 client.26252 shutdown 2023-10-11T16:35:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 -- 172.21.15.111:0/964405547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221efb40 msgr2=0x556c22252b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221efb40 0x556c22252b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd840234c0 tx=0x7fdd84004ad0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 -- 172.21.15.111:0/964405547 shutdown_connections 2023-10-11T16:35:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x556c22254940 0x556c22256d50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c221efb40 0x556c22252b00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556c221ec720 0x556c221f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 --2- 172.21.15.111:0/964405547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556c221ecb90 0x556c221f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdd6affd700 0 client.26252 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.137+0000 7fdda566bb80 1 -- 172.21.15.111:0/964405547 >> 172.21.15.111:0/964405547 conn(0x556c221dbf50 msgr2=0x556c221ee310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:40.141 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.138+0000 7fdda566bb80 1 -- 172.21.15.111:0/964405547 shutdown_connections 2023-10-11T16:35:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.138+0000 7fdda566bb80 1 -- 172.21.15.111:0/964405547 wait complete. 2023-10-11T16:35:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.138+0000 7fdda566bb80 20 client.26252 trim_cache size 0 max 16384 2023-10-11T16:35:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:35:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:35:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:35:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:35:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.140+0000 7fdd91dff700 1 -- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdd740a12c0 msgr2=0x7fdd740a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.140+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdd740a12c0 0x7fdd740a3770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c074f00 tx=0x7fdd8400b040 comp rx=0 tx=0).stop 2023-10-11T16:35:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.140+0000 7fdd91dff700 1 -- 172.21.15.111:0/3019473199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 msgr2=0x7fdd8c072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.140+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 0x7fdd8c072df0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c005910 tx=0x7fdd7c004710 comp rx=0 tx=0).stop 2023-10-11T16:35:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.141+0000 7fdd91dff700 1 -- 172.21.15.111:0/3019473199 shutdown_connections 2023-10-11T16:35:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.141+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdd740a12c0 0x7fdd740a3770 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.141+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd8c1055a0 0x7fdd8c1a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.141+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd8c100560 0x7fdd8c072df0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.141+0000 7fdd91dff700 1 --2- 172.21.15.111:0/3019473199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd8c100f00 0x7fdd8c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.141+0000 7fdd91dff700 1 -- 172.21.15.111:0/3019473199 >> 172.21.15.111:0/3019473199 conn(0x7fdd8c0781b0 msgr2=0x7fdd8c103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.141+0000 7fdd91dff700 1 -- 172.21.15.111:0/3019473199 shutdown_connections 2023-10-11T16:35:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:40.141+0000 7fdd91dff700 1 -- 172.21.15.111:0/3019473199 wait complete. 2023-10-11T16:35:41.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:41 smithi150 ceph-mon[132217]: pgmap v94: 129 pgs: 129 active+clean; 123 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.5 MiB/s wr, 1.59k op/s 2023-10-11T16:35:41.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:41 smithi130 ceph-mon[131859]: pgmap v94: 129 pgs: 129 active+clean; 123 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.5 MiB/s wr, 1.59k op/s 2023-10-11T16:35:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:41 smithi111 ceph-mon[98527]: pgmap v94: 129 pgs: 129 active+clean; 123 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.5 MiB/s wr, 1.59k op/s 2023-10-11T16:35:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:43 smithi150 ceph-mon[132217]: pgmap v95: 129 pgs: 129 active+clean; 123 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 1.43k op/s 2023-10-11T16:35:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:43 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:43.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:43 smithi130 ceph-mon[131859]: pgmap v95: 129 pgs: 129 active+clean; 123 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 1.43k op/s 2023-10-11T16:35:43.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:43 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:43 smithi111 ceph-mon[98527]: pgmap v95: 129 pgs: 129 active+clean; 123 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 1.43k op/s 2023-10-11T16:35:43.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:43 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:35:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:45 smithi150 ceph-mon[132217]: pgmap v96: 129 pgs: 129 active+clean; 121 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 1.65k op/s 2023-10-11T16:35:45.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:45 smithi130 ceph-mon[131859]: pgmap v96: 129 pgs: 129 active+clean; 121 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 1.65k op/s 2023-10-11T16:35:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:45 smithi111 ceph-mon[98527]: pgmap v96: 129 pgs: 129 active+clean; 121 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 1.65k op/s 2023-10-11T16:35:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:47 smithi150 ceph-mon[132217]: pgmap v97: 129 pgs: 129 active+clean; 117 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 2.89k op/s 2023-10-11T16:35:47.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:47 smithi130 ceph-mon[131859]: pgmap v97: 129 pgs: 129 active+clean; 117 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 2.89k op/s 2023-10-11T16:35:47.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:47 smithi111 ceph-mon[98527]: pgmap v97: 129 pgs: 129 active+clean; 117 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 2.89k op/s 2023-10-11T16:35:49.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:49 smithi150 ceph-mon[132217]: pgmap v98: 129 pgs: 129 active+clean; 117 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 2.87k op/s 2023-10-11T16:35:49.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:49 smithi130 ceph-mon[131859]: pgmap v98: 129 pgs: 129 active+clean; 117 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 2.87k op/s 2023-10-11T16:35:49.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:49 smithi111 ceph-mon[98527]: pgmap v98: 129 pgs: 129 active+clean; 117 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 2.87k op/s 2023-10-11T16:35:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:51 smithi150 ceph-mon[132217]: pgmap v99: 129 pgs: 129 active+clean; 145 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.2 MiB/s wr, 2.92k op/s 2023-10-11T16:35:51.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:51 smithi130 ceph-mon[131859]: pgmap v99: 129 pgs: 129 active+clean; 145 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.2 MiB/s wr, 2.92k op/s 2023-10-11T16:35:51.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:51 smithi111 ceph-mon[98527]: pgmap v99: 129 pgs: 129 active+clean; 145 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.2 MiB/s wr, 2.92k op/s 2023-10-11T16:35:52.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:52 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:52.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:52 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:52.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:52 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:53.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.312+0000 7f9d871ff700 1 -- 172.21.15.111:0/1625846465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d800ff230 msgr2=0x7f9d800ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.312+0000 7f9d871ff700 1 --2- 172.21.15.111:0/1625846465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d800ff230 0x7f9d800ff600 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009990 tx=0x7f9d74009ca0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.312+0000 7f9d871ff700 1 -- 172.21.15.111:0/1625846465 shutdown_connections 2023-10-11T16:35:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.312+0000 7f9d871ff700 1 --2- 172.21.15.111:0/1625846465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d80104630 0x7f9d80104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.312+0000 7f9d871ff700 1 --2- 172.21.15.111:0/1625846465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d800ff230 0x7f9d800ff600 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.312+0000 7f9d871ff700 1 --2- 172.21.15.111:0/1625846465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d800ffb40 0x7f9d80104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.312+0000 7f9d871ff700 1 -- 172.21.15.111:0/1625846465 >> 172.21.15.111:0/1625846465 conn(0x7f9d800763a0 msgr2=0x7f9d800767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:53.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.313+0000 7f9d871ff700 1 -- 172.21.15.111:0/1625846465 shutdown_connections 2023-10-11T16:35:53.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.313+0000 7f9d871ff700 1 -- 172.21.15.111:0/1625846465 wait complete. 2023-10-11T16:35:53.316 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d871ff700 1 Processor -- start 2023-10-11T16:35:53.317 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d871ff700 1 -- start start 2023-10-11T16:35:53.317 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d871ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d800ff230 0x7f9d8019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.317 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d871ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d800ffb40 0x7f9d8019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.317 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d871ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d80104630 0x7f9d801a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.317 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d871ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d801a1940 con 0x7f9d800ff230 2023-10-11T16:35:53.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d871ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d801a1ab0 con 0x7f9d80104630 2023-10-11T16:35:53.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d871ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d801a1c70 con 0x7f9d800ffb40 2023-10-11T16:35:53.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.314+0000 7f9d7ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d800ffb40 0x7f9d8019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d84f9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d800ff230 0x7f9d8019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d800ffb40 0x7f9d8019cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:33548/0 (socket says 172.21.15.111:33548) 2023-10-11T16:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7ffff700 1 -- 172.21.15.111:0/961012477 learned_addr learned my addr 172.21.15.111:0/961012477 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d8579c700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d80104630 0x7f9d801a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7ffff700 1 -- 172.21.15.111:0/961012477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d80104630 msgr2=0x7f9d801a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:53.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d80104630 0x7f9d801a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7ffff700 1 -- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d800ff230 msgr2=0x7f9d8019c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:53.321 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d800ff230 0x7f9d8019c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.321 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7ffff700 1 -- 172.21.15.111:0/961012477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d740095f0 con 0x7f9d800ffb40 2023-10-11T16:35:53.321 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d800ffb40 0x7f9d8019cee0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00d950 tx=0x7f9d6c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:53.321 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.315+0000 7f9d7dffb700 1 -- 172.21.15.111:0/961012477 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00f890 con 0x7f9d800ffb40 2023-10-11T16:35:53.321 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.316+0000 7f9d871ff700 1 -- 172.21.15.111:0/961012477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d801168c0 con 0x7f9d800ffb40 2023-10-11T16:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.316+0000 7f9d7dffb700 1 -- 172.21.15.111:0/961012477 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d6c010460 con 0x7f9d800ffb40 2023-10-11T16:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.316+0000 7f9d7dffb700 1 -- 172.21.15.111:0/961012477 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00e610 con 0x7f9d800ffb40 2023-10-11T16:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.316+0000 7f9d871ff700 1 -- 172.21.15.111:0/961012477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d80116730 con 0x7f9d800ffb40 2023-10-11T16:35:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.320+0000 7f9d7dffb700 1 -- 172.21.15.111:0/961012477 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f9d6c0108d0 con 0x7f9d800ffb40 2023-10-11T16:35:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.321+0000 7f9d9aa88b80 1 Processor -- start 2023-10-11T16:35:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.321+0000 7f9d9aa88b80 1 -- start start 2023-10-11T16:35:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.321+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.321+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4720 0x55e11f3ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.321+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.321+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f2f4000 con 0x55e11f24a6b0 2023-10-11T16:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.321+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f346840 con 0x55e11f3f4720 2023-10-11T16:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.321+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f33fcd0 con 0x55e11f3f4b90 2023-10-11T16:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d8579c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4720 0x55e11f3ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4720 0x55e11f3ee560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52704/0 (socket says 172.21.15.111:52704) 2023-10-11T16:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 -- 172.21.15.111:0/2381932039 learned_addr learned my addr 172.21.15.111:0/2381932039 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d8579c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3f41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37440/0 (socket says 172.21.15.111:37440) 2023-10-11T16:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 -- 172.21.15.111:0/2381932039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 msgr2=0x55e11f3eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 -- 172.21.15.111:0/2381932039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 msgr2=0x55e11f3f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 -- 172.21.15.111:0/2381932039 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e11f1c15d0 con 0x55e11f3f4720 2023-10-11T16:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d7dffb700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9d700a12c0 0x7f9d700a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d8579c700 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d7dffb700 1 -- 172.21.15.111:0/961012477 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f9d6c0ccea0 con 0x7f9d800ffb40 2023-10-11T16:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.322+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4720 0x55e11f3ee560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d74005f50 tx=0x7f9d7400b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf5ab229b965cf9d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.323+0000 7f9d8579c700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9d700a12c0 0x7f9d700a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.323+0000 7f9d6affd700 1 -- 172.21.15.111:0/2381932039 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7401d070 con 0x55e11f3f4720 2023-10-11T16:35:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.323+0000 7f9d6affd700 1 -- 172.21.15.111:0/2381932039 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d7401e040 con 0x55e11f3f4720 2023-10-11T16:35:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.323+0000 7f9d6affd700 1 -- 172.21.15.111:0/2381932039 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740049b0 con 0x55e11f3f4720 2023-10-11T16:35:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/2381932039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4720 msgr2=0x55e11f3ee560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4720 0x55e11f3ee560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d74005f50 tx=0x7f9d7400b610 comp rx=0 tx=0).stop 2023-10-11T16:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d8579c700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9d700a12c0 0x7f9d700a3770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9d78007500 tx=0x7f9d7800b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/2381932039 shutdown_connections 2023-10-11T16:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/2381932039 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4720 0x55e11f3ee560 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/2381932039 >> 172.21.15.111:0/2381932039 conn(0x55e11f3e3f50 msgr2=0x55e11f3e9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/2381932039 shutdown_connections 2023-10-11T16:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/2381932039 wait complete. 2023-10-11T16:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.324+0000 7f9d9aa88b80 1 -- start start 2023-10-11T16:35:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d9aa88b80 1 Processor -- start 2023-10-11T16:35:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3fda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3fdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f2e1880 con 0x55e11f3f4720 2023-10-11T16:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f2e1520 con 0x55e11f24a6b0 2023-10-11T16:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f1c38c0 con 0x55e11f3f4b90 2023-10-11T16:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d9aa88b80 10 client.0 _ll_register_callbacks cb 0x55e11f2ff720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:53.343 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d84f9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3fda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d84f9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3fda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52716/0 (socket says 172.21.15.111:52716) 2023-10-11T16:35:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.325+0000 7f9d84f9b700 1 -- 172.21.15.111:0/3191225550 learned_addr learned my addr 172.21.15.111:0/3191225550 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d8579c700 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3fdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d84f9b700 1 -- 172.21.15.111:0/3191225550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 msgr2=0x55e11f3f7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3f7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d84f9b700 1 -- 172.21.15.111:0/3191225550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 msgr2=0x55e11f3fdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3fdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d84f9b700 1 -- 172.21.15.111:0/3191225550 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d80113f60 con 0x55e11f24a6b0 2023-10-11T16:35:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3fda00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400b2e0 tx=0x7f9d7400fc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a54dbe8990fc26ff server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d5ffff700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.326+0000 7f9d5ffff700 1 -- 172.21.15.111:0/3191225550 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7401d040 con 0x55e11f24a6b0 2023-10-11T16:35:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d5ffff700 1 -- 172.21.15.111:0/3191225550 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f9d74005230 con 0x55e11f24a6b0 2023-10-11T16:35:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3191225550 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e11f3e1710 con 0x55e11f24a6b0 2023-10-11T16:35:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d5ffff700 1 -- 172.21.15.111:0/3191225550 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d740053f0 con 0x55e11f24a6b0 2023-10-11T16:35:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 10 client.26264 fetch_fsmap learned FSMap version 31 2023-10-11T16:35:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3191225550 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e11f306130 con 0x55e11f24a6b0 2023-10-11T16:35:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d5ffff700 1 -- 172.21.15.111:0/3191225550 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e11f3e1710 con 0x55e11f24a6b0 2023-10-11T16:35:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d5ffff700 1 -- 172.21.15.111:0/3191225550 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 31) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f9d74022400 con 0x55e11f24a6b0 2023-10-11T16:35:53.349 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 10 client.26264 fetch_fsmap finished waiting for FSMap version 31 2023-10-11T16:35:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 10 client.26264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 20 client.26264 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e11f407b60 0x55e11f409f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 4 client.26264 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.327+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3191225550 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e11f302040 con 0x55e11f407b60 2023-10-11T16:35:53.351 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.328+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e11f407b60 0x55e11f409f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.351 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.328+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e11f407b60 0x55e11f409f70 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adb3729709852403 server_cookie=2019f71210f0456e in_seq=0 out_seq=0 2023-10-11T16:35:53.351 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.328+0000 7f9d5ffff700 10 client.26264 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:53.351 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d5ffff700 1 -- 172.21.15.111:0/3191225550 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e11f302040 con 0x55e11f407b60 2023-10-11T16:35:53.351 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d5ffff700 10 client.26264 handle_command_reply: tid=0 2023-10-11T16:35:53.352 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 client.26264 shutdown 2023-10-11T16:35:53.352 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3191225550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 msgr2=0x55e11f3fda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:53.352 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3fda00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400b2e0 tx=0x7f9d7400fc20 comp rx=0 tx=0).stop 2023-10-11T16:35:53.352 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3191225550 shutdown_connections 2023-10-11T16:35:53.352 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e11f407b60 0x55e11f409f70 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3f7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3fda00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d5ffff700 0 client.26264 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3191225550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3fdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3191225550 >> 172.21.15.111:0/3191225550 conn(0x55e11f3e3f50 msgr2=0x55e11f3e87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3191225550 shutdown_connections 2023-10-11T16:35:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.329+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3191225550 wait complete. 2023-10-11T16:35:53.354 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.330+0000 7f9d9aa88b80 20 client.26264 trim_cache size 0 max 16384 2023-10-11T16:35:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d9aa88b80 1 Processor -- start 2023-10-11T16:35:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d9aa88b80 1 -- start start 2023-10-11T16:35:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3e8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f3ef6a0 con 0x55e11f3f4720 2023-10-11T16:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f3ef2d0 con 0x55e11f24a6b0 2023-10-11T16:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f3e9fe0 con 0x55e11f3f4b90 2023-10-11T16:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d84f9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d84f9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3f7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52730/0 (socket says 172.21.15.111:52730) 2023-10-11T16:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d84f9b700 1 -- 172.21.15.111:0/3469334709 learned_addr learned my addr 172.21.15.111:0/3469334709 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.346+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3e8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d8579c700 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d84f9b700 1 -- 172.21.15.111:0/3469334709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 msgr2=0x55e11f3ee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3ee560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d84f9b700 1 -- 172.21.15.111:0/3469334709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 msgr2=0x55e11f3e8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3e8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d84f9b700 1 -- 172.21.15.111:0/3469334709 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e11f305e80 con 0x55e11f24a6b0 2023-10-11T16:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3e8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3f7dc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009ce0 tx=0x7f9d74000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eca18fe4135908c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d6b7fe700 1 -- 172.21.15.111:0/3469334709 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74022a50 con 0x55e11f24a6b0 2023-10-11T16:35:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.347+0000 7f9d6b7fe700 1 -- 172.21.15.111:0/3469334709 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d74005530 con 0x55e11f24a6b0 2023-10-11T16:35:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.348+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3469334709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 msgr2=0x55e11f3f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.348+0000 7f9d6b7fe700 1 -- 172.21.15.111:0/3469334709 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74004b50 con 0x55e11f24a6b0 2023-10-11T16:35:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.348+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3f7dc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009ce0 tx=0x7f9d74000bc0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.349+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3469334709 shutdown_connections 2023-10-11T16:35:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.349+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4b90 0x55e11f3ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.349+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f24a6b0 0x55e11f3f7dc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.349+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/3469334709 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f3f4720 0x55e11f3e8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.349+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3469334709 >> 172.21.15.111:0/3469334709 conn(0x55e11f3e3f50 msgr2=0x55e11f3f85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.349+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3469334709 shutdown_connections 2023-10-11T16:35:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.349+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/3469334709 wait complete. 2023-10-11T16:35:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.349+0000 7f9d9aa88b80 1 -- start start 2023-10-11T16:35:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d9aa88b80 1 Processor -- start 2023-10-11T16:35:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3fdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4720 0x55e11f3fb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d9aa88b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4b90 0x55e11f3fbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f3fc140 con 0x55e11f24a6b0 2023-10-11T16:35:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f3fc2b0 con 0x55e11f3f4b90 2023-10-11T16:35:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d9aa88b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e11f3fc470 con 0x55e11f3f4720 2023-10-11T16:35:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d9aa88b80 10 client.0 _ll_register_callbacks cb 0x7f9d74005ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d7ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4720 0x55e11f3fb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d7ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4720 0x55e11f3fb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:33608/0 (socket says 172.21.15.111:33608) 2023-10-11T16:35:53.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d7ffff700 1 -- 172.21.15.111:0/611384942 learned_addr learned my addr 172.21.15.111:0/611384942 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:53.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d8579c700 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4b90 0x55e11f3fbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.350+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3fdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d7ffff700 1 -- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4b90 msgr2=0x55e11f3fbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4b90 0x55e11f3fbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d7ffff700 1 -- 172.21.15.111:0/611384942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 msgr2=0x55e11f3fdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3fdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d7ffff700 1 -- 172.21.15.111:0/611384942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e11f306130 con 0x55e11f3f4720 2023-10-11T16:35:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d8579c700 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4b90 0x55e11f3fbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d7ffff700 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4720 0x55e11f3fb5b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0998c0 tx=0x7f9d6c005910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33a58a6423abe52e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d5ffff700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:35:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d5ffff700 1 -- 172.21.15.111:0/611384942 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c014070 con 0x55e11f3f4720 2023-10-11T16:35:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d5ffff700 1 -- 172.21.15.111:0/611384942 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f9d6c099bc0 con 0x55e11f3f4720 2023-10-11T16:35:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/611384942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d7401e040 con 0x55e11f3f4720 2023-10-11T16:35:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d5ffff700 1 -- 172.21.15.111:0/611384942 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d6c09dbf0 con 0x55e11f3f4720 2023-10-11T16:35:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d9aa88b80 10 client.26305 fetch_fsmap learned FSMap version 31 2023-10-11T16:35:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.351+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/611384942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e11f156c00 con 0x55e11f3f4720 2023-10-11T16:35:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d5ffff700 1 -- 172.21.15.111:0/611384942 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d6c0c8020 con 0x55e11f3f4720 2023-10-11T16:35:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d5ffff700 1 -- 172.21.15.111:0/611384942 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 31) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f9d6c099f40 con 0x55e11f3f4720 2023-10-11T16:35:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d9aa88b80 10 client.26305 fetch_fsmap finished waiting for FSMap version 31 2023-10-11T16:35:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d9aa88b80 10 client.26305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d9aa88b80 20 client.26305 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e11f45c0a0 0x55e11f45e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d9aa88b80 4 client.26305 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:35:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/611384942 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e11f30e600 con 0x55e11f45c0a0 2023-10-11T16:35:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.352+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e11f45c0a0 0x55e11f45e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.353+0000 7f9d84f9b700 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e11f45c0a0 0x55e11f45e4b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=845b195fe0538769 server_cookie=7ed9bcde16983e2e in_seq=0 out_seq=0 2023-10-11T16:35:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:53.353+0000 7f9d5ffff700 10 client.26305 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:53.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:53 smithi130 ceph-mon[131859]: pgmap v100: 129 pgs: 129 active+clean; 145 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 1.56k op/s 2023-10-11T16:35:53.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:53 smithi111 ceph-mon[98527]: pgmap v100: 129 pgs: 129 active+clean; 145 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 1.56k op/s 2023-10-11T16:35:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:53 smithi150 ceph-mon[132217]: pgmap v100: 129 pgs: 129 active+clean; 145 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 1.56k op/s 2023-10-11T16:35:55.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:55 smithi130 ceph-mon[131859]: pgmap v101: 129 pgs: 129 active+clean; 154 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.8 MiB/s wr, 1.57k op/s 2023-10-11T16:35:55.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:55 smithi111 ceph-mon[98527]: pgmap v101: 129 pgs: 129 active+clean; 154 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.8 MiB/s wr, 1.57k op/s 2023-10-11T16:35:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:55 smithi150 ceph-mon[132217]: pgmap v101: 129 pgs: 129 active+clean; 154 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.8 MiB/s wr, 1.57k op/s 2023-10-11T16:35:57.558 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d5ffff700 1 -- 172.21.15.111:0/611384942 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e11f30e600 con 0x55e11f45c0a0 2023-10-11T16:35:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d5ffff700 10 client.26305 handle_command_reply: tid=0 2023-10-11T16:35:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 client.26305 shutdown 2023-10-11T16:35:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/611384942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4720 msgr2=0x55e11f3fb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4720 0x55e11f3fb5b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0998c0 tx=0x7f9d6c005910 comp rx=0 tx=0).stop 2023-10-11T16:35:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/611384942 shutdown_connections 2023-10-11T16:35:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e11f45c0a0 0x55e11f45e4b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e11f3f4b90 0x55e11f3fbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.560 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e11f24a6b0 0x55e11f3fdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 --2- 172.21.15.111:0/611384942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e11f3f4720 0x55e11f3fb5b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/611384942 >> 172.21.15.111:0/611384942 conn(0x55e11f3e3f50 msgr2=0x55e11f3f4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d5ffff700 0 client.26305 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:57.562 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.557+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/611384942 shutdown_connections 2023-10-11T16:35:57.562 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.558+0000 7f9d9aa88b80 1 -- 172.21.15.111:0/611384942 wait complete. 2023-10-11T16:35:57.562 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.558+0000 7f9d9aa88b80 20 client.26305 trim_cache size 0 max 16384 2023-10-11T16:35:57.562 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:35:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:35:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:35:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:35:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 -- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9d700a12c0 msgr2=0x7f9d700a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9d700a12c0 0x7f9d700a3770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9d78007500 tx=0x7f9d7800b040 comp rx=0 tx=0).stop 2023-10-11T16:35:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 -- 172.21.15.111:0/961012477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d800ffb40 msgr2=0x7f9d8019cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d800ffb40 0x7f9d8019cee0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00d950 tx=0x7f9d6c00dc60 comp rx=0 tx=0).stop 2023-10-11T16:35:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 -- 172.21.15.111:0/961012477 shutdown_connections 2023-10-11T16:35:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9d700a12c0 0x7f9d700a3770 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d80104630 0x7f9d801a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d800ff230 0x7f9d8019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 --2- 172.21.15.111:0/961012477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d800ffb40 0x7f9d8019cee0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 -- 172.21.15.111:0/961012477 >> 172.21.15.111:0/961012477 conn(0x7f9d800763a0 msgr2=0x7f9d80102050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 -- 172.21.15.111:0/961012477 shutdown_connections 2023-10-11T16:35:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:35:57.559+0000 7f9d871ff700 1 -- 172.21.15.111:0/961012477 wait complete. 2023-10-11T16:35:57.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:35:57.764 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:57 smithi111 ceph-mon[98527]: pgmap v102: 129 pgs: 129 active+clean; 174 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 1.36k op/s 2023-10-11T16:35:57.764 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:57 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:57.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:57 smithi130 ceph-mon[131859]: pgmap v102: 129 pgs: 129 active+clean; 174 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 1.36k op/s 2023-10-11T16:35:57.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:57 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:57.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 -- 172.21.15.111:0/3702912731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 msgr2=0x7ff830101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 --2- 172.21.15.111:0/3702912731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 0x7ff830101090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff820009990 tx=0x7ff820009ca0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 -- 172.21.15.111:0/3702912731 shutdown_connections 2023-10-11T16:35:57.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 --2- 172.21.15.111:0/3702912731 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff830105620 0x7ff83010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 --2- 172.21.15.111:0/3702912731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 0x7ff830101090 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 --2- 172.21.15.111:0/3702912731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff830101660 0x7ff830104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 -- 172.21.15.111:0/3702912731 >> 172.21.15.111:0/3702912731 conn(0x7ff830077560 msgr2=0x7ff830077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:57.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 -- 172.21.15.111:0/3702912731 shutdown_connections 2023-10-11T16:35:57.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.918+0000 7ff834fff700 1 -- 172.21.15.111:0/3702912731 wait complete. 2023-10-11T16:35:57.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.920+0000 7ff834fff700 1 Processor -- start 2023-10-11T16:35:57.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.920+0000 7ff834fff700 1 -- start start 2023-10-11T16:35:57.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.920+0000 7ff834fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 0x7ff8301964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.920+0000 7ff834fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff830101660 0x7ff830196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.920+0000 7ff834fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff830105620 0x7ff83019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.920+0000 7ff834fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83019b4a0 con 0x7ff830101660 2023-10-11T16:35:57.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.920+0000 7ff834fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83019b610 con 0x7ff830105620 2023-10-11T16:35:57.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.920+0000 7ff834fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83019b7d0 con 0x7ff830100cc0 2023-10-11T16:35:57.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 0x7ff8301964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 0x7ff8301964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:47474/0 (socket says 172.21.15.111:47474) 2023-10-11T16:35:57.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 -- 172.21.15.111:0/4081355891 learned_addr learned my addr 172.21.15.111:0/4081355891 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:57.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff827fff700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff830101660 0x7ff830196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82ed9d700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff830105620 0x7ff83019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 -- 172.21.15.111:0/4081355891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff830105620 msgr2=0x7ff83019ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff830105620 0x7ff83019ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 -- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff830101660 msgr2=0x7ff830196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff830101660 0x7ff830196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 -- 172.21.15.111:0/4081355891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8200095f0 con 0x7ff830100cc0 2023-10-11T16:35:57.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82ed9d700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff830105620 0x7ff83019ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:35:57.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.921+0000 7ff82e59c700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 0x7ff8301964c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff820005910 tx=0x7ff820004710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:57.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.922+0000 7ff8277fe700 1 -- 172.21.15.111:0/4081355891 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82001d070 con 0x7ff830100cc0 2023-10-11T16:35:57.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.922+0000 7ff834fff700 1 -- 172.21.15.111:0/4081355891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8301103d0 con 0x7ff830100cc0 2023-10-11T16:35:57.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.922+0000 7ff8277fe700 1 -- 172.21.15.111:0/4081355891 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff820022470 con 0x7ff830100cc0 2023-10-11T16:35:57.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.922+0000 7ff8277fe700 1 -- 172.21.15.111:0/4081355891 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff820017650 con 0x7ff830100cc0 2023-10-11T16:35:57.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.922+0000 7ff834fff700 1 -- 172.21.15.111:0/4081355891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff830110240 con 0x7ff830100cc0 2023-10-11T16:35:57.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.925+0000 7ff8277fe700 1 -- 172.21.15.111:0/4081355891 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7ff8200225e0 con 0x7ff830100cc0 2023-10-11T16:35:57.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.927+0000 7ff84887bb80 1 Processor -- start 2023-10-11T16:35:57.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.927+0000 7ff84887bb80 1 -- start start 2023-10-11T16:35:57.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.927+0000 7ff8277fe700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff81c0a12c0 0x7ff81c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.927+0000 7ff8277fe700 1 -- 172.21.15.111:0/4081355891 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff8200ce2e0 con 0x7ff830100cc0 2023-10-11T16:35:57.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f809785340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f809789780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82ed9d700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff81c0a12c0 0x7ff81c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff84887bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f809686060 con 0x55f809789cc0 2023-10-11T16:35:57.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff84887bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8096d88a0 con 0x55f809785880 2023-10-11T16:35:57.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff84887bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8096d1d30 con 0x55f809784f70 2023-10-11T16:35:57.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f809785340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f809785340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:47480/0 (socket says 172.21.15.111:47480) 2023-10-11T16:35:57.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82e59c700 1 -- 172.21.15.111:0/2619001534 learned_addr learned my addr 172.21.15.111:0/2619001534 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:57.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff827fff700 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f809789780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82ed9d700 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82e59c700 1 -- 172.21.15.111:0/2619001534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 msgr2=0x55f809789780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:57.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82e59c700 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f809789780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82e59c700 1 -- 172.21.15.111:0/2619001534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 msgr2=0x55f80978a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:57.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82e59c700 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.928+0000 7ff82e59c700 1 -- 172.21.15.111:0/2619001534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f809553630 con 0x55f809784f70 2023-10-11T16:35:57.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.929+0000 7ff82e59c700 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f809785340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff820024000 tx=0x7ff82009ae00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1216c53ee1b613a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:57.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.929+0000 7ff82ed9d700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff81c0a12c0 0x7ff81c0a3770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff828009fd0 tx=0x7ff828003cf0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:57.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.929+0000 7ff824ff9700 1 -- 172.21.15.111:0/2619001534 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82009b030 con 0x55f809784f70 2023-10-11T16:35:57.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.929+0000 7ff824ff9700 1 -- 172.21.15.111:0/2619001534 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff82009f060 con 0x55f809784f70 2023-10-11T16:35:57.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.929+0000 7ff824ff9700 1 -- 172.21.15.111:0/2619001534 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82009b4f0 con 0x55f809784f70 2023-10-11T16:35:57.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 -- 172.21.15.111:0/2619001534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 msgr2=0x55f809785340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f809785340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff820024000 tx=0x7ff82009ae00 comp rx=0 tx=0).stop 2023-10-11T16:35:57.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 -- 172.21.15.111:0/2619001534 shutdown_connections 2023-10-11T16:35:57.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f809785340 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 --2- 172.21.15.111:0/2619001534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f809789780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 -- 172.21.15.111:0/2619001534 >> 172.21.15.111:0/2619001534 conn(0x55f809776360 msgr2=0x55f80977bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:57.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 -- 172.21.15.111:0/2619001534 shutdown_connections 2023-10-11T16:35:57.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 -- 172.21.15.111:0/2619001534 wait complete. 2023-10-11T16:35:57.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.930+0000 7ff84887bb80 1 -- start start 2023-10-11T16:35:57.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.931+0000 7ff84887bb80 1 Processor -- start 2023-10-11T16:35:57.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.931+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f8097913c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.931+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f80978e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.931+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.931+0000 7ff84887bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8096738e0 con 0x55f809789cc0 2023-10-11T16:35:57.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.931+0000 7ff84887bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f809673580 con 0x55f809785880 2023-10-11T16:35:57.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.931+0000 7ff84887bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f809555920 con 0x55f809784f70 2023-10-11T16:35:57.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff84887bb80 10 client.0 _ll_register_callbacks cb 0x55f809691780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:57.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f80978e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f80978e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:35014/0 (socket says 172.21.15.111:35014) 2023-10-11T16:35:57.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff82ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 -- 172.21.15.111:0/1370350161 learned_addr learned my addr 172.21.15.111:0/1370350161 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:57.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff82e59c700 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f8097913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 -- 172.21.15.111:0/1370350161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 msgr2=0x55f8097913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:57.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f8097913c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 -- 172.21.15.111:0/1370350161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 msgr2=0x55f80978eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:57.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 -- 172.21.15.111:0/1370350161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff830110240 con 0x55f809785880 2023-10-11T16:35:57.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff827fff700 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f80978e5d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff818011760 tx=0x7ff818011a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd176a5ca4362ba1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:57.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.932+0000 7ff815ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:57.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.933+0000 7ff815ffb700 1 -- 172.21.15.111:0/1370350161 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818016070 con 0x55f809785880 2023-10-11T16:35:57.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.933+0000 7ff84887bb80 1 -- 172.21.15.111:0/1370350161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f809773b20 con 0x55f809785880 2023-10-11T16:35:57.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.933+0000 7ff815ffb700 1 -- 172.21.15.111:0/1370350161 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff818004830 con 0x55f809785880 2023-10-11T16:35:57.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.933+0000 7ff815ffb700 1 -- 172.21.15.111:0/1370350161 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8180049f0 con 0x55f809785880 2023-10-11T16:35:57.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.933+0000 7ff84887bb80 10 client.26282 fetch_fsmap learned FSMap version 32 2023-10-11T16:35:57.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.933+0000 7ff84887bb80 1 -- 172.21.15.111:0/1370350161 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f809698190 con 0x55f809785880 2023-10-11T16:35:57.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.933+0000 7ff815ffb700 1 -- 172.21.15.111:0/1370350161 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f809773b20 con 0x55f809785880 2023-10-11T16:35:57.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.933+0000 7ff815ffb700 1 -- 172.21.15.111:0/1370350161 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 32) v1 ==== 7486+0+0 (secure 0 0 0) 0x7ff81801a910 con 0x55f809785880 2023-10-11T16:35:57.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff84887bb80 10 client.26282 fetch_fsmap finished waiting for FSMap version 32 2023-10-11T16:35:57.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff84887bb80 10 client.26282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:57.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff84887bb80 20 client.26282 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:57.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff84887bb80 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f80979ea60 0x55f8097a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff84887bb80 4 client.26282 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:35:57.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff84887bb80 1 -- 172.21.15.111:0/1370350161 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8096940a0 con 0x55f80979ea60 2023-10-11T16:35:57.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff82e59c700 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f80979ea60 0x55f8097a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff82e59c700 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f80979ea60 0x55f8097a0e70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=babca97e05474528 server_cookie=16289af92f0ad101 in_seq=0 out_seq=0 2023-10-11T16:35:57.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.934+0000 7ff815ffb700 10 client.26282 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:57 smithi150 ceph-mon[132217]: pgmap v102: 129 pgs: 129 active+clean; 174 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 1.36k op/s 2023-10-11T16:35:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:57 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:35:57.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.952+0000 7ff815ffb700 1 -- 172.21.15.111:0/1370350161 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f8096940a0 con 0x55f80979ea60 2023-10-11T16:35:57.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.952+0000 7ff815ffb700 10 client.26282 handle_command_reply: tid=0 2023-10-11T16:35:57.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 client.26282 shutdown 2023-10-11T16:35:57.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 -- 172.21.15.111:0/1370350161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 msgr2=0x55f80978e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f80978e5d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff818011760 tx=0x7ff818011a70 comp rx=0 tx=0).stop 2023-10-11T16:35:57.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 -- 172.21.15.111:0/1370350161 shutdown_connections 2023-10-11T16:35:57.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f80979ea60 0x55f8097a0e70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809784f70 0x55f8097913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 --2- 172.21.15.111:0/1370350161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809785880 0x55f80978e5d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 -- 172.21.15.111:0/1370350161 >> 172.21.15.111:0/1370350161 conn(0x55f809776360 msgr2=0x55f80978b170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:57.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff815ffb700 0 client.26282 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:57.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 -- 172.21.15.111:0/1370350161 shutdown_connections 2023-10-11T16:35:57.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 1 -- 172.21.15.111:0/1370350161 wait complete. 2023-10-11T16:35:57.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.953+0000 7ff84887bb80 20 client.26282 trim_cache size 0 max 16384 2023-10-11T16:35:57.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff84887bb80 1 Processor -- start 2023-10-11T16:35:57.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff84887bb80 1 -- start start 2023-10-11T16:35:57.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f80977bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809785880 0x55f809795bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff84887bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f809775ee0 con 0x55f809789cc0 2023-10-11T16:35:57.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff84887bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f809785c50 con 0x55f809784f70 2023-10-11T16:35:57.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff84887bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f80979c010 con 0x55f809785880 2023-10-11T16:35:57.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff82e59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f80977bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff827fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809785880 0x55f809795bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff82e59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f80977bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:35016/0 (socket says 172.21.15.111:35016) 2023-10-11T16:35:57.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.975+0000 7ff82e59c700 1 -- 172.21.15.111:0/738618725 learned_addr learned my addr 172.21.15.111:0/738618725 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:57.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff82ed9d700 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff82e59c700 1 -- 172.21.15.111:0/738618725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809785880 msgr2=0x55f809795bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:57.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff82e59c700 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809785880 0x55f809795bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff82e59c700 1 -- 172.21.15.111:0/738618725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 msgr2=0x55f80978ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:57.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff82e59c700 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff82e59c700 1 -- 172.21.15.111:0/738618725 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f80978a5b0 con 0x55f809784f70 2023-10-11T16:35:57.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff82ed9d700 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:35:57.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff82e59c700 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f80977bca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff820096db0 tx=0x7ff820009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7534a6f7d2aaec28 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:57.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff8257fa700 1 -- 172.21.15.111:0/738618725 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82009b0b0 con 0x55f809784f70 2023-10-11T16:35:57.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff8257fa700 1 -- 172.21.15.111:0/738618725 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff820022900 con 0x55f809784f70 2023-10-11T16:35:57.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.976+0000 7ff8257fa700 1 -- 172.21.15.111:0/738618725 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82009b4f0 con 0x55f809784f70 2023-10-11T16:35:57.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 -- 172.21.15.111:0/738618725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 msgr2=0x55f80977bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:57.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f80977bca0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff820096db0 tx=0x7ff820009ca0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 -- 172.21.15.111:0/738618725 shutdown_connections 2023-10-11T16:35:57.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809789cc0 0x55f80978ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f80977bca0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 --2- 172.21.15.111:0/738618725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809785880 0x55f809795bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 -- 172.21.15.111:0/738618725 >> 172.21.15.111:0/738618725 conn(0x55f809776360 msgr2=0x55f80979acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 -- 172.21.15.111:0/738618725 shutdown_connections 2023-10-11T16:35:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.977+0000 7ff84887bb80 1 -- 172.21.15.111:0/738618725 wait complete. 2023-10-11T16:35:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 1 -- start start 2023-10-11T16:35:57.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 1 Processor -- start 2023-10-11T16:35:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f809791b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809785880 0x55f809792080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809789cc0 0x55f8097925c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8097f1a70 con 0x55f809785880 2023-10-11T16:35:57.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8097f1be0 con 0x55f809784f70 2023-10-11T16:35:57.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8097f1da0 con 0x55f809789cc0 2023-10-11T16:35:57.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff84887bb80 10 client.0 _ll_register_callbacks cb 0x7ff818009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:35:57.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.978+0000 7ff82e59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f809791b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff82e59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f809791b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:35018/0 (socket says 172.21.15.111:35018) 2023-10-11T16:35:57.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff82e59c700 1 -- 172.21.15.111:0/368720795 learned_addr learned my addr 172.21.15.111:0/368720795 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:35:57.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff82e59c700 1 -- 172.21.15.111:0/368720795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809789cc0 msgr2=0x55f8097925c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:35:57.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff82e59c700 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809789cc0 0x55f8097925c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff82e59c700 1 -- 172.21.15.111:0/368720795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809785880 msgr2=0x55f809792080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:35:57.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff827fff700 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809785880 0x55f809792080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff82e59c700 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809785880 0x55f809792080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:57.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff82e59c700 1 -- 172.21.15.111:0/368720795 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f809553630 con 0x55f809784f70 2023-10-11T16:35:57.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.979+0000 7ff827fff700 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809785880 0x55f809792080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:35:57.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff82e59c700 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f809791b40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff820006010 tx=0x7ff82000b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0dc9000bd758a05 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:35:57.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff815ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:35:57.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff815ffb700 1 -- 172.21.15.111:0/368720795 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8200b93b0 con 0x55f809784f70 2023-10-11T16:35:57.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff815ffb700 1 -- 172.21.15.111:0/368720795 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff82009b290 con 0x55f809784f70 2023-10-11T16:35:57.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff84887bb80 1 -- 172.21.15.111:0/368720795 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff82009f060 con 0x55f809784f70 2023-10-11T16:35:57.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff815ffb700 1 -- 172.21.15.111:0/368720795 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff82009b450 con 0x55f809784f70 2023-10-11T16:35:57.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff84887bb80 10 client.26288 fetch_fsmap learned FSMap version 32 2023-10-11T16:35:57.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff84887bb80 1 -- 172.21.15.111:0/368720795 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f8094ffde0 con 0x55f809784f70 2023-10-11T16:35:57.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff815ffb700 1 -- 172.21.15.111:0/368720795 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff82009f060 con 0x55f809784f70 2023-10-11T16:35:57.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff815ffb700 1 -- 172.21.15.111:0/368720795 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 32) v1 ==== 7486+0+0 (secure 0 0 0) 0x7ff8200a2f60 con 0x55f809784f70 2023-10-11T16:35:57.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff84887bb80 10 client.26288 fetch_fsmap finished waiting for FSMap version 32 2023-10-11T16:35:57.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff84887bb80 10 client.26288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:35:57.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.980+0000 7ff84887bb80 20 client.26288 populate_metadata read hostname 'smithi111' 2023-10-11T16:35:57.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.981+0000 7ff84887bb80 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8097f2fa0 0x55f8097f5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:35:57.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.981+0000 7ff84887bb80 4 client.26288 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:35:57.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.981+0000 7ff84887bb80 1 -- 172.21.15.111:0/368720795 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f8096a0660 con 0x55f8097f2fa0 2023-10-11T16:35:57.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.981+0000 7ff827fff700 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8097f2fa0 0x55f8097f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:35:57.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.981+0000 7ff815ffb700 10 client.26288 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:57.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:57.981+0000 7ff827fff700 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8097f2fa0 0x55f8097f5380 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=669caae754bf4d45 server_cookie=250f9697b091b79b in_seq=0 out_seq=0 2023-10-11T16:35:58.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff815ffb700 1 -- 172.21.15.111:0/368720795 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f8096a0660 con 0x55f8097f2fa0 2023-10-11T16:35:58.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff815ffb700 10 client.26288 handle_command_reply: tid=0 2023-10-11T16:35:58.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 client.26288 shutdown 2023-10-11T16:35:58.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 -- 172.21.15.111:0/368720795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 msgr2=0x55f809791b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:58.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f809791b40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff820006010 tx=0x7ff82000b610 comp rx=0 tx=0).stop 2023-10-11T16:35:58.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 -- 172.21.15.111:0/368720795 shutdown_connections 2023-10-11T16:35:58.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8097f2fa0 0x55f8097f5380 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f809789cc0 0x55f8097925c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f809784f70 0x55f809791b40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff815ffb700 0 client.26288 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:35:58.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 --2- 172.21.15.111:0/368720795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f809785880 0x55f809792080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 -- 172.21.15.111:0/368720795 >> 172.21.15.111:0/368720795 conn(0x55f809776360 msgr2=0x55f809788b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:58.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 -- 172.21.15.111:0/368720795 shutdown_connections 2023-10-11T16:35:58.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.002+0000 7ff84887bb80 1 -- 172.21.15.111:0/368720795 wait complete. 2023-10-11T16:35:58.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.003+0000 7ff84887bb80 20 client.26288 trim_cache size 0 max 16384 2023-10-11T16:35:58.008 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:35:58.008 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:35:58.008 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:35:58.009 INFO:teuthology.orchestra.run.smithi111.stdout: "e4b353b6-bcd8-4549-a24b-3247f2c5740c": { 2023-10-11T16:35:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:35:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "e4b353b6-bcd8-4549-a24b-3247f2c5740c", 2023-10-11T16:35:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:35:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:35:58.011 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:35:58.011 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:35:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.003+0000 7ff834fff700 1 -- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff81c0a12c0 msgr2=0x7ff81c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.003+0000 7ff834fff700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff81c0a12c0 0x7ff81c0a3770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff828009fd0 tx=0x7ff828003cf0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.003+0000 7ff834fff700 1 -- 172.21.15.111:0/4081355891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 msgr2=0x7ff8301964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:35:58.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.003+0000 7ff834fff700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 0x7ff8301964c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff820005910 tx=0x7ff820004710 comp rx=0 tx=0).stop 2023-10-11T16:35:58.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.004+0000 7ff834fff700 1 -- 172.21.15.111:0/4081355891 shutdown_connections 2023-10-11T16:35:58.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.004+0000 7ff834fff700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff81c0a12c0 0x7ff81c0a3770 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.004+0000 7ff834fff700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff830105620 0x7ff83019ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.004+0000 7ff834fff700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff830100cc0 0x7ff8301964c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.004+0000 7ff834fff700 1 --2- 172.21.15.111:0/4081355891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff830101660 0x7ff830196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:35:58.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.004+0000 7ff834fff700 1 -- 172.21.15.111:0/4081355891 >> 172.21.15.111:0/4081355891 conn(0x7ff830077560 msgr2=0x7ff8300feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:35:58.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.004+0000 7ff834fff700 1 -- 172.21.15.111:0/4081355891 shutdown_connections 2023-10-11T16:35:58.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:35:58.004+0000 7ff834fff700 1 -- 172.21.15.111:0/4081355891 wait complete. 2023-10-11T16:35:58.025 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b353b6-bcd8-4549-a24b-3247f2c5740c - {'path': '/', 'tag': 'e4b353b6-bcd8-4549-a24b-3247f2c5740c', 'options': 'recursive,force'} 2023-10-11T16:35:59.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:35:59 smithi130 ceph-mon[131859]: pgmap v103: 129 pgs: 129 active+clean; 174 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 66 op/s 2023-10-11T16:35:59.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:35:59 smithi111 ceph-mon[98527]: pgmap v103: 129 pgs: 129 active+clean; 174 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 66 op/s 2023-10-11T16:35:59.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:35:59 smithi150 ceph-mon[132217]: pgmap v103: 129 pgs: 129 active+clean; 174 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 66 op/s 2023-10-11T16:36:01.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:01 smithi130 ceph-mon[131859]: pgmap v104: 129 pgs: 129 active+clean; 189 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 8.7 MiB/s wr, 1.74k op/s 2023-10-11T16:36:01.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:01 smithi111 ceph-mon[98527]: pgmap v104: 129 pgs: 129 active+clean; 189 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 8.7 MiB/s wr, 1.74k op/s 2023-10-11T16:36:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:01 smithi150 ceph-mon[132217]: pgmap v104: 129 pgs: 129 active+clean; 189 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 8.7 MiB/s wr, 1.74k op/s 2023-10-11T16:36:03.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:03 smithi130 ceph-mon[131859]: pgmap v105: 129 pgs: 129 active+clean; 189 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.2 MiB/s wr, 1.69k op/s 2023-10-11T16:36:03.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:03 smithi111 ceph-mon[98527]: pgmap v105: 129 pgs: 129 active+clean; 189 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.2 MiB/s wr, 1.69k op/s 2023-10-11T16:36:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:03 smithi150 ceph-mon[132217]: pgmap v105: 129 pgs: 129 active+clean; 189 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.2 MiB/s wr, 1.69k op/s 2023-10-11T16:36:04.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:04 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:36:04.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:04 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:36:04.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:04 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:36:05.498 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:05 smithi130 ceph-mon[131859]: pgmap v106: 129 pgs: 129 active+clean; 209 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.9 MiB/s wr, 1.69k op/s 2023-10-11T16:36:05.499 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:05 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:36:05.499 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:05 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:36:05.499 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:05 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:36:05.499 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:05 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:36:05.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:05 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:36:05.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:05 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:36:05.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:05 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:36:05.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:05 smithi111 ceph-mon[98527]: pgmap v106: 129 pgs: 129 active+clean; 209 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.9 MiB/s wr, 1.69k op/s 2023-10-11T16:36:05.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:05 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:36:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:05 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:36:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:05 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:36:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:05 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:36:05.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:05 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:36:05.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:05 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:36:05.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:05 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:36:05.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:05 smithi150 ceph-mon[132217]: pgmap v106: 129 pgs: 129 active+clean; 209 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.9 MiB/s wr, 1.69k op/s 2023-10-11T16:36:05.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:05 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:36:05.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:05 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:36:05.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:05 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:36:05.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:05 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:36:05.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:05 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:36:05.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:05 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:36:05.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:05 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:36:07.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:07 smithi130 ceph-mon[131859]: pgmap v107: 129 pgs: 129 active+clean; 249 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 1.71k op/s 2023-10-11T16:36:07.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:07 smithi111 ceph-mon[98527]: pgmap v107: 129 pgs: 129 active+clean; 249 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 1.71k op/s 2023-10-11T16:36:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:07 smithi150 ceph-mon[132217]: pgmap v107: 129 pgs: 129 active+clean; 249 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 1.71k op/s 2023-10-11T16:36:09.784 INFO:tasks.workunit.client.0.smithi111.stdout:created hierarchy, now cleaning up 2023-10-11T16:36:09.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:09 smithi130 ceph-mon[131859]: pgmap v108: 129 pgs: 129 active+clean; 249 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.8 MiB/s wr, 1.70k op/s 2023-10-11T16:36:09.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:09 smithi111 ceph-mon[98527]: pgmap v108: 129 pgs: 129 active+clean; 249 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.8 MiB/s wr, 1.70k op/s 2023-10-11T16:36:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:09 smithi150 ceph-mon[132217]: pgmap v108: 129 pgs: 129 active+clean; 249 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.8 MiB/s wr, 1.70k op/s 2023-10-11T16:36:10.718 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/4237589061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8f4105560 msgr2=0x7fd8f410d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/4237589061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8f4105560 0x7fd8f410d170 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009fc0 tx=0x7fd8ec00b330 comp rx=0 tx=0).stop 2023-10-11T16:36:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/4237589061 shutdown_connections 2023-10-11T16:36:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/4237589061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8f4105560 0x7fd8f410d170 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/4237589061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8f4100c50 0x7fd8f4101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/4237589061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8f41015f0 0x7fd8f4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/4237589061 >> 172.21.15.111:0/4237589061 conn(0x7fd8f4077550 msgr2=0x7fd8f4077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/4237589061 shutdown_connections 2023-10-11T16:36:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.716+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/4237589061 wait complete. 2023-10-11T16:36:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.718+0000 7fd8fc1ff700 1 Processor -- start 2023-10-11T16:36:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.718+0000 7fd8fc1ff700 1 -- start start 2023-10-11T16:36:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.718+0000 7fd8fc1ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8f4100c50 0x7fd8f40741c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.718+0000 7fd8fc1ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8f41015f0 0x7fd8f4074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.718+0000 7fd8fc1ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8f4105560 0x7fd8f4070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.718+0000 7fd8fc1ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4070a10 con 0x7fd8f4105560 2023-10-11T16:36:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8fc1ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4070b80 con 0x7fd8f41015f0 2023-10-11T16:36:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8fc1ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4070d40 con 0x7fd8f4100c50 2023-10-11T16:36:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8f4100c50 0x7fd8f40741c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f979a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8f41015f0 0x7fd8f4074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8f4100c50 0x7fd8f40741c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:53352/0 (socket says 172.21.15.111:53352) 2023-10-11T16:36:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 -- 172.21.15.111:0/1599085107 learned_addr learned my addr 172.21.15.111:0/1599085107 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8f4105560 0x7fd8f4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 -- 172.21.15.111:0/1599085107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8f41015f0 msgr2=0x7fd8f4074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8f41015f0 0x7fd8f4074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 -- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8f4105560 msgr2=0x7fd8f4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8f4105560 0x7fd8f4070270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 -- 172.21.15.111:0/1599085107 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f0009520 con 0x7fd8f4100c50 2023-10-11T16:36:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f979a700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8f41015f0 0x7fd8f4074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.719+0000 7fd8f9f9b700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8f4100c50 0x7fd8f40741c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000eb30 tx=0x7fd8f000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.720+0000 7fd8eaffd700 1 -- 172.21.15.111:0/1599085107 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0009880 con 0x7fd8f4100c50 2023-10-11T16:36:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.720+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/1599085107 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e40095f0 con 0x7fd8f4100c50 2023-10-11T16:36:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.720+0000 7fd8eaffd700 1 -- 172.21.15.111:0/1599085107 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8f00044e0 con 0x7fd8f4100c50 2023-10-11T16:36:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.720+0000 7fd8eaffd700 1 -- 172.21.15.111:0/1599085107 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0003d80 con 0x7fd8f4100c50 2023-10-11T16:36:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.720+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/1599085107 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec009c40 con 0x7fd8f4100c50 2023-10-11T16:36:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.724+0000 7fd8eaffd700 1 -- 172.21.15.111:0/1599085107 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fd8f0003f20 con 0x7fd8f4100c50 2023-10-11T16:36:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.724+0000 7fd90faa9b80 1 Processor -- start 2023-10-11T16:36:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd90faa9b80 1 -- start start 2023-10-11T16:36:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 0x56105a8a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a89c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a89cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a7a2000 con 0x56105a6f86b0 2023-10-11T16:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a7f4840 con 0x56105a8a2b90 2023-10-11T16:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a7edcd0 con 0x56105a8a2720 2023-10-11T16:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd8f9f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a89c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd8f979a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a89cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd8f979a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a89cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:36734/0 (socket says 172.21.15.111:36734) 2023-10-11T16:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd8f9f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a89c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:53358/0 (socket says 172.21.15.111:53358) 2023-10-11T16:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd8f979a700 1 -- 172.21.15.111:0/4184143882 learned_addr learned my addr 172.21.15.111:0/4184143882 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.725+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/4184143882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 0x56105a8a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8f979a700 1 -- 172.21.15.111:0/4184143882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 msgr2=0x56105a89c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8f979a700 1 --2- 172.21.15.111:0/4184143882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a89c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8f979a700 1 -- 172.21.15.111:0/4184143882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 msgr2=0x56105a8a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8f979a700 1 --2- 172.21.15.111:0/4184143882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 0x56105a8a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8f979a700 1 -- 172.21.15.111:0/4184143882 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56105a66f5d0 con 0x56105a8a2b90 2023-10-11T16:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8eaffd700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8e00a12c0 0x7fd8e00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8f979a700 1 --2- 172.21.15.111:0/4184143882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a89cb00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009d10 tx=0x7fd8e4009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d45c7db4b71c9b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8eaffd700 1 -- 172.21.15.111:0/1599085107 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd8f0014070 con 0x7fd8f4100c50 2023-10-11T16:36:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8e00a12c0 0x7fd8e00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8dffff700 1 -- 172.21.15.111:0/4184143882 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4019050 con 0x56105a8a2b90 2023-10-11T16:36:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8dffff700 1 -- 172.21.15.111:0/4184143882 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8e401a040 con 0x56105a8a2b90 2023-10-11T16:36:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.726+0000 7fd8dffff700 1 -- 172.21.15.111:0/4184143882 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4005280 con 0x56105a8a2b90 2023-10-11T16:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 -- 172.21.15.111:0/4184143882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 msgr2=0x56105a89cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/4184143882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a89cb00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4009d10 tx=0x7fd8e4009310 comp rx=0 tx=0).stop 2023-10-11T16:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8e00a12c0 0x7fd8e00a3770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4074e50 tx=0x7fd8ec009f10 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 -- 172.21.15.111:0/4184143882 shutdown_connections 2023-10-11T16:36:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/4184143882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a89cb00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/4184143882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 0x56105a8a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/4184143882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a89c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 -- 172.21.15.111:0/4184143882 >> 172.21.15.111:0/4184143882 conn(0x56105a891f50 msgr2=0x56105a897270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 -- 172.21.15.111:0/4184143882 shutdown_connections 2023-10-11T16:36:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.727+0000 7fd90faa9b80 1 -- 172.21.15.111:0/4184143882 wait complete. 2023-10-11T16:36:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.728+0000 7fd90faa9b80 1 -- start start 2023-10-11T16:36:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.728+0000 7fd90faa9b80 1 Processor -- start 2023-10-11T16:36:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a6f86b0 0x56105a8a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a8a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a78f880 con 0x56105a8a2720 2023-10-11T16:36:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a78f520 con 0x56105a8a2b90 2023-10-11T16:36:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a6718c0 con 0x56105a6f86b0 2023-10-11T16:36:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd90faa9b80 10 client.0 _ll_register_callbacks cb 0x56105a7ad720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8fa79c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8fa79c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:36736/0 (socket says 172.21.15.111:36736) 2023-10-11T16:36:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8fa79c700 1 -- 172.21.15.111:0/3725684971 learned_addr learned my addr 172.21.15.111:0/3725684971 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:10.740 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8f979a700 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a8a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8fa79c700 1 -- 172.21.15.111:0/3725684971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a6f86b0 msgr2=0x56105a8a8a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:36:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a6f86b0 0x56105a8a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8fa79c700 1 -- 172.21.15.111:0/3725684971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 msgr2=0x56105a8a5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a8a5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8fa79c700 1 -- 172.21.15.111:0/3725684971 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e4014e20 con 0x56105a8a2b90 2023-10-11T16:36:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.729+0000 7fd8f979a700 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a8a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:36:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.730+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a6290 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec003cb0 tx=0x7fd8ec009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=427a781362d9b947 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.730+0000 7fd8dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:36:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.730+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3725684971 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec00e040 con 0x56105a8a2b90 2023-10-11T16:36:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.730+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3725684971 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd8ec023070 con 0x56105a8a2b90 2023-10-11T16:36:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.730+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3725684971 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56105a88f710 con 0x56105a8a2b90 2023-10-11T16:36:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.730+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3725684971 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8ec00be80 con 0x56105a8a2b90 2023-10-11T16:36:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.730+0000 7fd90faa9b80 10 client.26300 fetch_fsmap learned FSMap version 32 2023-10-11T16:36:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.730+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3725684971 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56105a7b4130 con 0x56105a8a2b90 2023-10-11T16:36:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3725684971 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56105a88f710 con 0x56105a8a2b90 2023-10-11T16:36:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3725684971 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 32) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd8ec026070 con 0x56105a8a2b90 2023-10-11T16:36:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd90faa9b80 10 client.26300 fetch_fsmap finished waiting for FSMap version 32 2023-10-11T16:36:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd90faa9b80 10 client.26300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd90faa9b80 20 client.26300 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56105a8b5fb0 0x56105a8b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd90faa9b80 4 client.26300 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:36:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3725684971 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56105a7b0040 con 0x56105a8b5fb0 2023-10-11T16:36:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.731+0000 7fd8f9f9b700 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56105a8b5fb0 0x56105a8b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.732+0000 7fd8f9f9b700 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56105a8b5fb0 0x56105a8b83c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=360f1bbadb8f6fce server_cookie=96697c877539bb7f in_seq=0 out_seq=0 2023-10-11T16:36:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.732+0000 7fd8dd7fa700 10 client.26300 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.732+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3725684971 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56105a7b0040 con 0x56105a8b5fb0 2023-10-11T16:36:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.732+0000 7fd8dd7fa700 10 client.26300 handle_command_reply: tid=0 2023-10-11T16:36:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.732+0000 7fd90faa9b80 1 client.26300 shutdown 2023-10-11T16:36:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3725684971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 msgr2=0x56105a8a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a6290 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec003cb0 tx=0x7fd8ec009920 comp rx=0 tx=0).stop 2023-10-11T16:36:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3725684971 shutdown_connections 2023-10-11T16:36:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56105a8b5fb0 0x56105a8b83c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a6290 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a6f86b0 0x56105a8a8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3725684971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a8a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd8dd7fa700 0 client.26300 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:10.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3725684971 >> 172.21.15.111:0/3725684971 conn(0x56105a891f50 msgr2=0x56105a8967e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3725684971 shutdown_connections 2023-10-11T16:36:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3725684971 wait complete. 2023-10-11T16:36:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.733+0000 7fd90faa9b80 20 client.26300 trim_cache size 0 max 16384 2023-10-11T16:36:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.749+0000 7fd90faa9b80 1 Processor -- start 2023-10-11T16:36:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.749+0000 7fd90faa9b80 1 -- start start 2023-10-11T16:36:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.749+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 0x56105a8ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a896f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a8a6910 con 0x56105a6f86b0 2023-10-11T16:36:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a8a2210 con 0x56105a8a2b90 2023-10-11T16:36:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a8981d0 con 0x56105a8a2720 2023-10-11T16:36:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8f979a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a896f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8f9f9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 0x56105a8ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8f979a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a896f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:53374/0 (socket says 172.21.15.111:53374) 2023-10-11T16:36:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8fa79c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8f979a700 1 -- 172.21.15.111:0/3451582154 learned_addr learned my addr 172.21.15.111:0/3451582154 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8fa79c700 1 -- 172.21.15.111:0/3451582154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 msgr2=0x56105a896f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/3451582154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a896f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8fa79c700 1 -- 172.21.15.111:0/3451582154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 msgr2=0x56105a8ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/3451582154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 0x56105a8ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8fa79c700 1 -- 172.21.15.111:0/3451582154 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56105a7b3e80 con 0x56105a8a2b90 2023-10-11T16:36:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.750+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/3451582154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a85a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00b370 tx=0x7fd8ec007ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24fc0c09ab05007b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd8e8ff9700 1 -- 172.21.15.111:0/3451582154 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec005a10 con 0x56105a8a2b90 2023-10-11T16:36:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd8e8ff9700 1 -- 172.21.15.111:0/3451582154 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8ec021070 con 0x56105a8a2b90 2023-10-11T16:36:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd8e8ff9700 1 -- 172.21.15.111:0/3451582154 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec03c430 con 0x56105a8a2b90 2023-10-11T16:36:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3451582154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 msgr2=0x56105a8a85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3451582154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a85a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00b370 tx=0x7fd8ec007ad0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3451582154 shutdown_connections 2023-10-11T16:36:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3451582154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a8a2b90 0x56105a8a85a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3451582154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a6f86b0 0x56105a8ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3451582154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2720 0x56105a896f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.751+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3451582154 >> 172.21.15.111:0/3451582154 conn(0x56105a891f50 msgr2=0x56105a89fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.752+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3451582154 shutdown_connections 2023-10-11T16:36:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.752+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3451582154 wait complete. 2023-10-11T16:36:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.752+0000 7fd90faa9b80 1 -- start start 2023-10-11T16:36:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.752+0000 7fd90faa9b80 1 Processor -- start 2023-10-11T16:36:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a6f86b0 0x56105a8ad080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a89dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd90faa9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2b90 0x56105a89e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a89e780 con 0x56105a8a2720 2023-10-11T16:36:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a89e8f0 con 0x56105a6f86b0 2023-10-11T16:36:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd90faa9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105a89eab0 con 0x56105a8a2b90 2023-10-11T16:36:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd90faa9b80 10 client.0 _ll_register_callbacks cb 0x7fd8ec0076f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a6f86b0 0x56105a8ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a6f86b0 0x56105a8ad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:36766/0 (socket says 172.21.15.111:36766) 2023-10-11T16:36:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 -- 172.21.15.111:0/3490920490 learned_addr learned my addr 172.21.15.111:0/3490920490 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2b90 0x56105a89e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f979a700 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a89dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 -- 172.21.15.111:0/3490920490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2b90 msgr2=0x56105a89e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2b90 0x56105a89e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 -- 172.21.15.111:0/3490920490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 msgr2=0x56105a89dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a89dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 -- 172.21.15.111:0/3490920490 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56105a7b4130 con 0x56105a6f86b0 2023-10-11T16:36:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8fa79c700 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2b90 0x56105a89e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8f9f9b700 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a6f86b0 0x56105a8ad080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00caa00 tx=0x7fd8f0003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c38a17a8300ab19 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.753+0000 7fd8dd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:36:10.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3490920490 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f00cae20 con 0x56105a6f86b0 2023-10-11T16:36:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3490920490 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd8f00cd020 con 0x56105a6f86b0 2023-10-11T16:36:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3490920490 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd8e401a040 con 0x56105a6f86b0 2023-10-11T16:36:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3490920490 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8f009e160 con 0x56105a6f86b0 2023-10-11T16:36:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 10 client.26306 fetch_fsmap learned FSMap version 32 2023-10-11T16:36:10.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3490920490 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56105a604c00 con 0x56105a6f86b0 2023-10-11T16:36:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3490920490 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8e401a040 con 0x56105a6f86b0 2023-10-11T16:36:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3490920490 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 32) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fd8f00cd1e0 con 0x56105a6f86b0 2023-10-11T16:36:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 10 client.26306 fetch_fsmap finished waiting for FSMap version 32 2023-10-11T16:36:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 10 client.26306 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 20 client.26306 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56105a90a5c0 0x56105a90c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 4 client.26306 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:36:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.754+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3490920490 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56105a7bc600 con 0x56105a90a5c0 2023-10-11T16:36:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.755+0000 7fd8f979a700 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56105a90a5c0 0x56105a90c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.755+0000 7fd8dd7fa700 10 client.26306 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:10.755+0000 7fd8f979a700 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56105a90a5c0 0x56105a90c9d0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51032fac0c52256a server_cookie=b81be923806158cd in_seq=0 out_seq=0 2023-10-11T16:36:11.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:11 smithi130 ceph-mon[131859]: pgmap v109: 129 pgs: 129 active+clean; 311 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 1.72k op/s 2023-10-11T16:36:11.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:11 smithi111 ceph-mon[98527]: pgmap v109: 129 pgs: 129 active+clean; 311 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 1.72k op/s 2023-10-11T16:36:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd8dd7fa700 1 -- 172.21.15.111:0/3490920490 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56105a7bc600 con 0x56105a90a5c0 2023-10-11T16:36:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd8dd7fa700 10 client.26306 handle_command_reply: tid=0 2023-10-11T16:36:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 client.26306 shutdown 2023-10-11T16:36:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3490920490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a6f86b0 msgr2=0x56105a8ad080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a6f86b0 0x56105a8ad080 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00caa00 tx=0x7fd8f0003820 comp rx=0 tx=0).stop 2023-10-11T16:36:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3490920490 shutdown_connections 2023-10-11T16:36:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56105a90a5c0 0x56105a90c9d0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56105a8a2b90 0x56105a89e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56105a6f86b0 0x56105a8ad080 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd8dd7fa700 0 client.26306 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 --2- 172.21.15.111:0/3490920490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56105a8a2720 0x56105a89dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.927+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3490920490 >> 172.21.15.111:0/3490920490 conn(0x56105a891f50 msgr2=0x56105a8a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.928+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3490920490 shutdown_connections 2023-10-11T16:36:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.928+0000 7fd90faa9b80 1 -- 172.21.15.111:0/3490920490 wait complete. 2023-10-11T16:36:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.928+0000 7fd90faa9b80 20 client.26306 trim_cache size 0 max 16384 2023-10-11T16:36:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:36:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:36:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:36:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:36:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.928+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8e00a12c0 msgr2=0x7fd8e00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.928+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8e00a12c0 0x7fd8e00a3770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4074e50 tx=0x7fd8ec009f10 comp rx=0 tx=0).stop 2023-10-11T16:36:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.928+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/1599085107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8f4100c50 msgr2=0x7fd8f40741c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.928+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8f4100c50 0x7fd8f40741c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000eb30 tx=0x7fd8f000ee40 comp rx=0 tx=0).stop 2023-10-11T16:36:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.929+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/1599085107 shutdown_connections 2023-10-11T16:36:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.929+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd8e00a12c0 0x7fd8e00a3770 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.929+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8f4105560 0x7fd8f4070270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.929+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8f4100c50 0x7fd8f40741c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.929+0000 7fd8fc1ff700 1 --2- 172.21.15.111:0/1599085107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8f41015f0 0x7fd8f4074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.929+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/1599085107 >> 172.21.15.111:0/1599085107 conn(0x7fd8f4077550 msgr2=0x7fd8f40fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.929+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/1599085107 shutdown_connections 2023-10-11T16:36:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:11.929+0000 7fd8fc1ff700 1 -- 172.21.15.111:0/1599085107 wait complete. 2023-10-11T16:36:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:11 smithi150 ceph-mon[132217]: pgmap v109: 129 pgs: 129 active+clean; 311 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 14 MiB/s wr, 1.72k op/s 2023-10-11T16:36:13.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:13 smithi130 ceph-mon[131859]: pgmap v110: 129 pgs: 129 active+clean; 311 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 48 op/s 2023-10-11T16:36:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:13 smithi111 ceph-mon[98527]: pgmap v110: 129 pgs: 129 active+clean; 311 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 48 op/s 2023-10-11T16:36:13.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:13 smithi150 ceph-mon[132217]: pgmap v110: 129 pgs: 129 active+clean; 311 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 48 op/s 2023-10-11T16:36:15.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:15 smithi130 ceph-mon[131859]: pgmap v111: 129 pgs: 129 active+clean; 287 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 139 op/s 2023-10-11T16:36:15.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:15 smithi111 ceph-mon[98527]: pgmap v111: 129 pgs: 129 active+clean; 287 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 139 op/s 2023-10-11T16:36:15.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:15 smithi150 ceph-mon[132217]: pgmap v111: 129 pgs: 129 active+clean; 287 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 139 op/s 2023-10-11T16:36:17.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:17 smithi130 ceph-mon[131859]: pgmap v112: 129 pgs: 129 active+clean; 285 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 273 op/s 2023-10-11T16:36:17.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:17 smithi111 ceph-mon[98527]: pgmap v112: 129 pgs: 129 active+clean; 285 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 273 op/s 2023-10-11T16:36:17.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:17 smithi150 ceph-mon[132217]: pgmap v112: 129 pgs: 129 active+clean; 285 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 273 op/s 2023-10-11T16:36:19.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:19 smithi130 ceph-mon[131859]: pgmap v113: 129 pgs: 129 active+clean; 285 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.6 MiB/s wr, 247 op/s 2023-10-11T16:36:19.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:19 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:36:19.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:19 smithi111 ceph-mon[98527]: pgmap v113: 129 pgs: 129 active+clean; 285 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.6 MiB/s wr, 247 op/s 2023-10-11T16:36:19.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:19 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:36:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:19 smithi150 ceph-mon[132217]: pgmap v113: 129 pgs: 129 active+clean; 285 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.6 MiB/s wr, 247 op/s 2023-10-11T16:36:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:19 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:36:21.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:21 smithi130 ceph-mon[131859]: pgmap v114: 129 pgs: 129 active+clean; 311 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.8 MiB/s wr, 319 op/s 2023-10-11T16:36:21.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:21 smithi111 ceph-mon[98527]: pgmap v114: 129 pgs: 129 active+clean; 311 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.8 MiB/s wr, 319 op/s 2023-10-11T16:36:21.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:21 smithi150 ceph-mon[132217]: pgmap v114: 129 pgs: 129 active+clean; 311 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 9.8 MiB/s wr, 319 op/s 2023-10-11T16:36:23.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:23 smithi130 ceph-mon[131859]: pgmap v115: 129 pgs: 129 active+clean; 311 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 300 op/s 2023-10-11T16:36:23.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:23 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:36:23.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:23 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:23.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:23 smithi111 ceph-mon[98527]: pgmap v115: 129 pgs: 129 active+clean; 311 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 300 op/s 2023-10-11T16:36:23.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:23 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:36:23.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:23 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:23 smithi150 ceph-mon[132217]: pgmap v115: 129 pgs: 129 active+clean; 311 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 300 op/s 2023-10-11T16:36:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:23 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:36:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:23 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 -- 172.21.15.111:0/1639156430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 msgr2=0x7f223c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 --2- 172.21.15.111:0/1639156430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 0x7f223c100910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f222c00fb20 tx=0x7f222c00fe30 comp rx=0 tx=0).stop 2023-10-11T16:36:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 -- 172.21.15.111:0/1639156430 shutdown_connections 2023-10-11T16:36:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 --2- 172.21.15.111:0/1639156430 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f223c105530 0x7f223c10d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.163 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 --2- 172.21.15.111:0/1639156430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 0x7f223c100910 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.163 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 --2- 172.21.15.111:0/1639156430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c100ee0 0x7f223c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.163 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 -- 172.21.15.111:0/1639156430 >> 172.21.15.111:0/1639156430 conn(0x7f223c078210 msgr2=0x7f223c078610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:25.163 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 -- 172.21.15.111:0/1639156430 shutdown_connections 2023-10-11T16:36:25.163 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.160+0000 7f2241bff700 1 -- 172.21.15.111:0/1639156430 wait complete. 2023-10-11T16:36:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.161+0000 7f2241bff700 1 Processor -- start 2023-10-11T16:36:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.161+0000 7f2241bff700 1 -- start start 2023-10-11T16:36:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.162+0000 7f2241bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 0x7f223c1a2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.162+0000 7f2241bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f223c100ee0 0x7f223c1a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.162+0000 7f2241bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105530 0x7f223c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.162+0000 7f2241bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c19ce70 con 0x7f223c105530 2023-10-11T16:36:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.162+0000 7f2241bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c19cfe0 con 0x7f223c100ee0 2023-10-11T16:36:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.162+0000 7f2241bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c19d1a0 con 0x7f223c100540 2023-10-11T16:36:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.162+0000 7f223affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f223c100ee0 0x7f223c1a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f223c100ee0 0x7f223c1a2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50810/0 (socket says 172.21.15.111:50810) 2023-10-11T16:36:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223affd700 1 -- 172.21.15.111:0/4095250858 learned_addr learned my addr 172.21.15.111:0/4095250858 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223b7fe700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 0x7f223c1a2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223bfff700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105530 0x7f223c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223affd700 1 -- 172.21.15.111:0/4095250858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 msgr2=0x7f223c1a2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:25.168 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223affd700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 0x7f223c1a2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.168 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223affd700 1 -- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105530 msgr2=0x7f223c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:25.168 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223affd700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105530 0x7f223c19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.168 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223affd700 1 -- 172.21.15.111:0/4095250858 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c00f750 con 0x7f223c100ee0 2023-10-11T16:36:25.168 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.163+0000 7f223b7fe700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 0x7f223c1a2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:36:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.164+0000 7f223affd700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f223c100ee0 0x7f223c1a2cd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f223000e940 tx=0x7f223000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.165+0000 7f2238ff9700 1 -- 172.21.15.111:0/4095250858 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230009730 con 0x7f223c100ee0 2023-10-11T16:36:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.165+0000 7f2238ff9700 1 -- 172.21.15.111:0/4095250858 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f223000cbd0 con 0x7f223c100ee0 2023-10-11T16:36:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.165+0000 7f2238ff9700 1 -- 172.21.15.111:0/4095250858 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230018680 con 0x7f223c100ee0 2023-10-11T16:36:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.166+0000 7f2241bff700 1 -- 172.21.15.111:0/4095250858 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223c10e1c0 con 0x7f223c100ee0 2023-10-11T16:36:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.166+0000 7f2241bff700 1 -- 172.21.15.111:0/4095250858 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c10e030 con 0x7f223c100ee0 2023-10-11T16:36:25.172 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.171+0000 7f2238ff9700 1 -- 172.21.15.111:0/4095250858 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2230010430 con 0x7f223c100ee0 2023-10-11T16:36:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.172+0000 7f2255608b80 1 Processor -- start 2023-10-11T16:36:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.172+0000 7f2255608b80 1 -- start start 2023-10-11T16:36:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f2238ff9700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f22240a1410 0x7f22240a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f2255608b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac121e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f2255608b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12720 0x55eb1ac0c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f2255608b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f2255608b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ab12000 con 0x55eb1ac12b90 2023-10-11T16:36:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f2255608b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ab64840 con 0x55eb1aa686b0 2023-10-11T16:36:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f2255608b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ab5dcd0 con 0x55eb1ac12720 2023-10-11T16:36:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f2238ff9700 1 -- 172.21.15.111:0/4095250858 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2230014070 con 0x7f223c100ee0 2023-10-11T16:36:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f223bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac121e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f223bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac121e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50814/0 (socket says 172.21.15.111:50814) 2023-10-11T16:36:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f223bfff700 1 -- 172.21.15.111:0/642101041 learned_addr learned my addr 172.21.15.111:0/642101041 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.173+0000 7f223b7fe700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f22240a1410 0x7f22240a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223affd700 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12720 0x55eb1ac0c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223bfff700 1 -- 172.21.15.111:0/642101041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12720 msgr2=0x55eb1ac0c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223bfff700 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12720 0x55eb1ac0c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223bfff700 1 -- 172.21.15.111:0/642101041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 msgr2=0x55eb1ac0cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223bfff700 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223bfff700 1 -- 172.21.15.111:0/642101041 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb1a9df5d0 con 0x55eb1aa686b0 2023-10-11T16:36:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223affd700 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12720 0x55eb1ac0c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:36:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223bfff700 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac121e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2234012640 tx=0x7f2234012950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef836ee17dd64de server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f223b7fe700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f22240a1410 0x7f22240a38c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f222c009830 tx=0x7f222c011040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.174+0000 7f2221ffb700 1 -- 172.21.15.111:0/642101041 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223400e040 con 0x55eb1aa686b0 2023-10-11T16:36:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.175+0000 7f2221ffb700 1 -- 172.21.15.111:0/642101041 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22340092d0 con 0x55eb1aa686b0 2023-10-11T16:36:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.175+0000 7f2221ffb700 1 -- 172.21.15.111:0/642101041 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234007900 con 0x55eb1aa686b0 2023-10-11T16:36:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.175+0000 7f2255608b80 1 -- 172.21.15.111:0/642101041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 msgr2=0x55eb1ac121e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.175+0000 7f2255608b80 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac121e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2234012640 tx=0x7f2234012950 comp rx=0 tx=0).stop 2023-10-11T16:36:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 -- 172.21.15.111:0/642101041 shutdown_connections 2023-10-11T16:36:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac121e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 --2- 172.21.15.111:0/642101041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12720 0x55eb1ac0c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 -- 172.21.15.111:0/642101041 >> 172.21.15.111:0/642101041 conn(0x55eb1ac01f50 msgr2=0x55eb1ac07270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 -- 172.21.15.111:0/642101041 shutdown_connections 2023-10-11T16:36:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 -- 172.21.15.111:0/642101041 wait complete. 2023-10-11T16:36:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 -- start start 2023-10-11T16:36:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.176+0000 7f2255608b80 1 Processor -- start 2023-10-11T16:36:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f2255608b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1aa686b0 0x55eb1ac18aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f2255608b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1ac12720 0x55eb1ac15d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f2255608b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 0x55eb1ac16460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f2255608b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1aaff880 con 0x55eb1ac12b90 2023-10-11T16:36:25.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f2255608b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1aaff520 con 0x55eb1ac12720 2023-10-11T16:36:25.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f2255608b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1a9e18c0 con 0x55eb1aa686b0 2023-10-11T16:36:25.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f2255608b80 10 client.0 _ll_register_callbacks cb 0x55eb1ab1d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:25.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f223b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1aa686b0 0x55eb1ac18aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f223b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1aa686b0 0x55eb1ac18aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:35878/0 (socket says 172.21.15.111:35878) 2023-10-11T16:36:25.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.177+0000 7f223b7fe700 1 -- 172.21.15.111:0/3623907720 learned_addr learned my addr 172.21.15.111:0/3623907720 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:25.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f223b7fe700 1 -- 172.21.15.111:0/3623907720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1ac12720 msgr2=0x55eb1ac15d20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:36:25.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f223b7fe700 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1ac12720 0x55eb1ac15d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f223b7fe700 1 -- 172.21.15.111:0/3623907720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 msgr2=0x55eb1ac16460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f223bfff700 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 0x55eb1ac16460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f223b7fe700 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 0x55eb1ac16460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f223b7fe700 1 -- 172.21.15.111:0/3623907720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2234012320 con 0x55eb1aa686b0 2023-10-11T16:36:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f223bfff700 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 0x55eb1ac16460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:36:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f223b7fe700 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1aa686b0 0x55eb1ac18aa0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f222c00fb20 tx=0x7f222c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aa0ba5ed1d6948e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.178+0000 7f221affd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:36:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.179+0000 7f221affd700 1 -- 172.21.15.111:0/3623907720 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c005a00 con 0x55eb1aa686b0 2023-10-11T16:36:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.179+0000 7f221affd700 1 -- 172.21.15.111:0/3623907720 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f222c003b60 con 0x55eb1aa686b0 2023-10-11T16:36:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.179+0000 7f2255608b80 1 -- 172.21.15.111:0/3623907720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb1abff710 con 0x55eb1aa686b0 2023-10-11T16:36:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.179+0000 7f221affd700 1 -- 172.21.15.111:0/3623907720 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f222c003d20 con 0x55eb1aa686b0 2023-10-11T16:36:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.179+0000 7f2255608b80 10 client.26347 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.179+0000 7f2255608b80 1 -- 172.21.15.111:0/3623907720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55eb1ab24130 con 0x55eb1aa686b0 2023-10-11T16:36:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.179+0000 7f221affd700 1 -- 172.21.15.111:0/3623907720 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eb1abff710 con 0x55eb1aa686b0 2023-10-11T16:36:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.179+0000 7f221affd700 1 -- 172.21.15.111:0/3623907720 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f222c021c40 con 0x55eb1aa686b0 2023-10-11T16:36:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f2255608b80 10 client.26347 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f2255608b80 10 client.26347 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f2255608b80 20 client.26347 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f2255608b80 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eb1ac26120 0x55eb1ac28530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f2255608b80 4 client.26347 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:36:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f2255608b80 1 -- 172.21.15.111:0/3623907720 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb1ab20040 con 0x55eb1ac26120 2023-10-11T16:36:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f223affd700 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eb1ac26120 0x55eb1ac28530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f223affd700 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eb1ac26120 0x55eb1ac28530 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a4c9169c88db075 server_cookie=96a8cbda6d1e22be in_seq=0 out_seq=0 2023-10-11T16:36:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.180+0000 7f221affd700 10 client.26347 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f221affd700 1 -- 172.21.15.111:0/3623907720 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55eb1ab20040 con 0x55eb1ac26120 2023-10-11T16:36:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f221affd700 10 client.26347 handle_command_reply: tid=0 2023-10-11T16:36:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f2255608b80 1 client.26347 shutdown 2023-10-11T16:36:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f2255608b80 1 -- 172.21.15.111:0/3623907720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1aa686b0 msgr2=0x55eb1ac18aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f2255608b80 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1aa686b0 0x55eb1ac18aa0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f222c00fb20 tx=0x7f222c009310 comp rx=0 tx=0).stop 2023-10-11T16:36:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f2255608b80 1 -- 172.21.15.111:0/3623907720 shutdown_connections 2023-10-11T16:36:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f2255608b80 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eb1ac26120 0x55eb1ac28530 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f221affd700 0 client.26347 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.191+0000 7f2255608b80 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12b90 0x55eb1ac16460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.192+0000 7f2255608b80 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1aa686b0 0x55eb1ac18aa0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.192+0000 7f2255608b80 1 --2- 172.21.15.111:0/3623907720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1ac12720 0x55eb1ac15d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.192+0000 7f2255608b80 1 -- 172.21.15.111:0/3623907720 >> 172.21.15.111:0/3623907720 conn(0x55eb1ac01f50 msgr2=0x55eb1ac067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.192+0000 7f2255608b80 1 -- 172.21.15.111:0/3623907720 shutdown_connections 2023-10-11T16:36:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.192+0000 7f2255608b80 1 -- 172.21.15.111:0/3623907720 wait complete. 2023-10-11T16:36:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.192+0000 7f2255608b80 20 client.26347 trim_cache size 0 max 16384 2023-10-11T16:36:25.211 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.209+0000 7f2255608b80 1 Processor -- start 2023-10-11T16:36:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.209+0000 7f2255608b80 1 -- start start 2023-10-11T16:36:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f2255608b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac1d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f2255608b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac18640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f2255608b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f2255608b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ac10a90 con 0x55eb1ac12720 2023-10-11T16:36:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f2255608b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ac12210 con 0x55eb1aa686b0 2023-10-11T16:36:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f2255608b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ac1d660 con 0x55eb1ac12b90 2023-10-11T16:36:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f223b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac1d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f223affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac18640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f223b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac1d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50834/0 (socket says 172.21.15.111:50834) 2023-10-11T16:36:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f223b7fe700 1 -- 172.21.15.111:0/2236794120 learned_addr learned my addr 172.21.15.111:0/2236794120 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f223bfff700 1 --2- 172.21.15.111:0/2236794120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f223affd700 1 -- 172.21.15.111:0/2236794120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 msgr2=0x55eb1ac0c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:25.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.210+0000 7f223affd700 1 --2- 172.21.15.111:0/2236794120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.211+0000 7f223affd700 1 -- 172.21.15.111:0/2236794120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 msgr2=0x55eb1ac1d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:25.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.211+0000 7f223affd700 1 --2- 172.21.15.111:0/2236794120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac1d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.211+0000 7f223affd700 1 -- 172.21.15.111:0/2236794120 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb1ab23e80 con 0x55eb1ac12720 2023-10-11T16:36:25.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.211+0000 7f223affd700 1 --2- 172.21.15.111:0/2236794120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac18640 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f22300ca650 tx=0x7f223001ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4402dfac214af814 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:25.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.211+0000 7f22227fc700 1 -- 172.21.15.111:0/2236794120 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230019260 con 0x55eb1ac12720 2023-10-11T16:36:25.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.211+0000 7f22227fc700 1 -- 172.21.15.111:0/2236794120 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22300044e0 con 0x55eb1ac12720 2023-10-11T16:36:25.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.211+0000 7f22227fc700 1 -- 172.21.15.111:0/2236794120 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230098490 con 0x55eb1ac12720 2023-10-11T16:36:25.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 -- 172.21.15.111:0/2236794120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 msgr2=0x55eb1ac18640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:25.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 --2- 172.21.15.111:0/2236794120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac18640 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f22300ca650 tx=0x7f223001ffa0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 -- 172.21.15.111:0/2236794120 shutdown_connections 2023-10-11T16:36:25.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 --2- 172.21.15.111:0/2236794120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 --2- 172.21.15.111:0/2236794120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac1d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 --2- 172.21.15.111:0/2236794120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac18640 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 -- 172.21.15.111:0/2236794120 >> 172.21.15.111:0/2236794120 conn(0x55eb1ac01f50 msgr2=0x55eb1ac22320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:25.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 -- 172.21.15.111:0/2236794120 shutdown_connections 2023-10-11T16:36:25.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.212+0000 7f2255608b80 1 -- 172.21.15.111:0/2236794120 wait complete. 2023-10-11T16:36:25.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 1 -- start start 2023-10-11T16:36:25.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 1 Processor -- start 2023-10-11T16:36:25.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac0f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac0dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ac0fa10 con 0x55eb1ac12720 2023-10-11T16:36:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ac0e710 con 0x55eb1aa686b0 2023-10-11T16:36:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb1ac0e8d0 con 0x55eb1ac12b90 2023-10-11T16:36:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.213+0000 7f2255608b80 10 client.0 _ll_register_callbacks cb 0x7f222c003880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac0f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223b7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac0f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50836/0 (socket says 172.21.15.111:50836) 2023-10-11T16:36:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223b7fe700 1 -- 172.21.15.111:0/890985121 learned_addr learned my addr 172.21.15.111:0/890985121 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223affd700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac0dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223bfff700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223b7fe700 1 -- 172.21.15.111:0/890985121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 msgr2=0x55eb1ac0e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:25.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223b7fe700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223b7fe700 1 -- 172.21.15.111:0/890985121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 msgr2=0x55eb1ac0dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:25.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223b7fe700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac0dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:25.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223b7fe700 1 -- 172.21.15.111:0/890985121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c10e030 con 0x55eb1aa686b0 2023-10-11T16:36:25.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223bfff700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:36:25.225 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.214+0000 7f223affd700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac0dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:25.225 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.215+0000 7f223b7fe700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac0f300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f222c005f50 tx=0x7f222c02d8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20efa5cfa71fc190 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:25.225 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.215+0000 7f221affd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:36:25.225 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.215+0000 7f221affd700 1 -- 172.21.15.111:0/890985121 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0040d0 con 0x55eb1aa686b0 2023-10-11T16:36:25.225 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.215+0000 7f2255608b80 1 -- 172.21.15.111:0/890985121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22340092d0 con 0x55eb1aa686b0 2023-10-11T16:36:25.226 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.215+0000 7f221affd700 1 -- 172.21.15.111:0/890985121 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f222c004230 con 0x55eb1aa686b0 2023-10-11T16:36:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.216+0000 7f221affd700 1 -- 172.21.15.111:0/890985121 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f222c02d920 con 0x55eb1aa686b0 2023-10-11T16:36:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.216+0000 7f2255608b80 10 client.26321 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.216+0000 7f2255608b80 1 -- 172.21.15.111:0/890985121 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55eb1a974c00 con 0x55eb1aa686b0 2023-10-11T16:36:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.216+0000 7f221affd700 1 -- 172.21.15.111:0/890985121 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22340092d0 con 0x55eb1aa686b0 2023-10-11T16:36:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.216+0000 7f221affd700 1 -- 172.21.15.111:0/890985121 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f222c021ac0 con 0x55eb1aa686b0 2023-10-11T16:36:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.217+0000 7f2255608b80 10 client.26321 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.217+0000 7f2255608b80 10 client.26321 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.217+0000 7f2255608b80 20 client.26321 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.217+0000 7f2255608b80 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eb1ac26120 0x55eb1ac7d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.217+0000 7f2255608b80 4 client.26321 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:36:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.217+0000 7f2255608b80 1 -- 172.21.15.111:0/890985121 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55eb1ab2c600 con 0x55eb1ac26120 2023-10-11T16:36:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.217+0000 7f223affd700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eb1ac26120 0x55eb1ac7d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.218+0000 7f221affd700 10 client.26321 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:25.218+0000 7f223affd700 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eb1ac26120 0x55eb1ac7d200 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29b1313eb96be5a1 server_cookie=cb96c0d47e43dc3f in_seq=0 out_seq=0 2023-10-11T16:36:25.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:25 smithi130 ceph-mon[131859]: pgmap v116: 129 pgs: 129 active+clean; 319 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.3 MiB/s wr, 326 op/s 2023-10-11T16:36:25.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:25 smithi111 ceph-mon[98527]: pgmap v116: 129 pgs: 129 active+clean; 319 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.3 MiB/s wr, 326 op/s 2023-10-11T16:36:25.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:25 smithi150 ceph-mon[132217]: pgmap v116: 129 pgs: 129 active+clean; 319 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.3 MiB/s wr, 326 op/s 2023-10-11T16:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.204+0000 7f221affd700 1 -- 172.21.15.111:0/890985121 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55eb1ab2c600 con 0x55eb1ac26120 2023-10-11T16:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.204+0000 7f221affd700 10 client.26321 handle_command_reply: tid=0 2023-10-11T16:36:27.206 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.204+0000 7f2255608b80 1 client.26321 shutdown 2023-10-11T16:36:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f2255608b80 1 -- 172.21.15.111:0/890985121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 msgr2=0x55eb1ac0f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f2255608b80 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac0f300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f222c005f50 tx=0x7f222c02d8b0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f2255608b80 1 -- 172.21.15.111:0/890985121 shutdown_connections 2023-10-11T16:36:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f2255608b80 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eb1ac26120 0x55eb1ac7d200 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f2255608b80 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eb1ac12b90 0x55eb1ac0e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f2255608b80 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb1aa686b0 0x55eb1ac0f300 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f221affd700 0 client.26321 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f2255608b80 1 --2- 172.21.15.111:0/890985121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eb1ac12720 0x55eb1ac0dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.205+0000 7f2255608b80 1 -- 172.21.15.111:0/890985121 >> 172.21.15.111:0/890985121 conn(0x55eb1ac01f50 msgr2=0x55eb1ac10de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.206+0000 7f2255608b80 1 -- 172.21.15.111:0/890985121 shutdown_connections 2023-10-11T16:36:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.207+0000 7f2255608b80 1 -- 172.21.15.111:0/890985121 wait complete. 2023-10-11T16:36:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.207+0000 7f2255608b80 20 client.26321 trim_cache size 0 max 16384 2023-10-11T16:36:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:36:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:36:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:36:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:36:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.208+0000 7f2241bff700 1 -- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f22240a1410 msgr2=0x7f22240a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.208+0000 7f2241bff700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f22240a1410 0x7f22240a38c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f222c009830 tx=0x7f222c011040 comp rx=0 tx=0).stop 2023-10-11T16:36:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.209+0000 7f2241bff700 1 -- 172.21.15.111:0/4095250858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f223c100ee0 msgr2=0x7f223c1a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.209+0000 7f2241bff700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f223c100ee0 0x7f223c1a2cd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f223000e940 tx=0x7f223000ed00 comp rx=0 tx=0).stop 2023-10-11T16:36:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.209+0000 7f2241bff700 1 -- 172.21.15.111:0/4095250858 shutdown_connections 2023-10-11T16:36:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.209+0000 7f2241bff700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f22240a1410 0x7f22240a38c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.209+0000 7f2241bff700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f223c105530 0x7f223c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.209+0000 7f2241bff700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f223c100540 0x7f223c1a2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.209+0000 7f2241bff700 1 --2- 172.21.15.111:0/4095250858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f223c100ee0 0x7f223c1a2cd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.209+0000 7f2241bff700 1 -- 172.21.15.111:0/4095250858 >> 172.21.15.111:0/4095250858 conn(0x7f223c078210 msgr2=0x7f223c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.210+0000 7f2241bff700 1 -- 172.21.15.111:0/4095250858 shutdown_connections 2023-10-11T16:36:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:27.210+0000 7f2241bff700 1 -- 172.21.15.111:0/4095250858 wait complete. 2023-10-11T16:36:27.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:27 smithi130 ceph-mon[131859]: pgmap v117: 129 pgs: 129 active+clean; 350 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.9 MiB/s wr, 348 op/s 2023-10-11T16:36:27.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:27 smithi111 ceph-mon[98527]: pgmap v117: 129 pgs: 129 active+clean; 350 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.9 MiB/s wr, 348 op/s 2023-10-11T16:36:27.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:27 smithi150 ceph-mon[132217]: pgmap v117: 129 pgs: 129 active+clean; 350 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.9 MiB/s wr, 348 op/s 2023-10-11T16:36:28.026 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:36:28.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.370+0000 7f05975ff700 1 -- 172.21.15.111:0/856079622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 msgr2=0x7f0590104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.370+0000 7f05975ff700 1 --2- 172.21.15.111:0/856079622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 0x7f0590104af0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f058800b0a0 tx=0x7f058800b3b0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.370+0000 7f05975ff700 1 -- 172.21.15.111:0/856079622 shutdown_connections 2023-10-11T16:36:28.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.370+0000 7f05975ff700 1 --2- 172.21.15.111:0/856079622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 0x7f0590104af0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.370+0000 7f05975ff700 1 --2- 172.21.15.111:0/856079622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900ff180 0x7f05900ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.370+0000 7f05975ff700 1 --2- 172.21.15.111:0/856079622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05900ffa90 0x7f0590104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.370+0000 7f05975ff700 1 -- 172.21.15.111:0/856079622 >> 172.21.15.111:0/856079622 conn(0x7f05900770b0 msgr2=0x7f05900774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:28.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.371+0000 7f05975ff700 1 -- 172.21.15.111:0/856079622 shutdown_connections 2023-10-11T16:36:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.371+0000 7f05975ff700 1 -- 172.21.15.111:0/856079622 wait complete. 2023-10-11T16:36:28.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.373+0000 7f05975ff700 1 Processor -- start 2023-10-11T16:36:28.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.374+0000 7f05975ff700 1 -- start start 2023-10-11T16:36:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.375+0000 7f05975ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900ff180 0x7f05901a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.375+0000 7f05975ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05900ffa90 0x7f05901a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.375+0000 7f05975ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 0x7f059019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.375+0000 7f05975ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019d090 con 0x7f05900ff180 2023-10-11T16:36:28.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.375+0000 7f05975ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019d200 con 0x7f05900ffa90 2023-10-11T16:36:28.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.375+0000 7f05975ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019d3c0 con 0x7f0590104680 2023-10-11T16:36:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.376+0000 7f0594b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05900ffa90 0x7f05901a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.376+0000 7f0594b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05900ffa90 0x7f05901a2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50866/0 (socket says 172.21.15.111:50866) 2023-10-11T16:36:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.376+0000 7f0594b9a700 1 -- 172.21.15.111:0/2254555117 learned_addr learned my addr 172.21.15.111:0/2254555117 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:28.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.376+0000 7f0595b9c700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 0x7f059019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.376+0000 7f059539b700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900ff180 0x7f05901a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.377+0000 7f0595b9c700 1 -- 172.21.15.111:0/2254555117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05900ffa90 msgr2=0x7f05901a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.377+0000 7f0595b9c700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05900ffa90 0x7f05901a2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.377+0000 7f0595b9c700 1 -- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900ff180 msgr2=0x7f05901a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.377+0000 7f0595b9c700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900ff180 0x7f05901a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.377+0000 7f0595b9c700 1 -- 172.21.15.111:0/2254555117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0588009d00 con 0x7f0590104680 2023-10-11T16:36:28.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.377+0000 7f059539b700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900ff180 0x7f05901a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:36:28.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.378+0000 7f0595b9c700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 0x7f059019ca20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0588009fd0 tx=0x7f058800ce50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.378+0000 7f05867fc700 1 -- 172.21.15.111:0/2254555117 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058800e040 con 0x7f0590104680 2023-10-11T16:36:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.379+0000 7f05867fc700 1 -- 172.21.15.111:0/2254555117 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058801d940 con 0x7f0590104680 2023-10-11T16:36:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.379+0000 7f05867fc700 1 -- 172.21.15.111:0/2254555117 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588005660 con 0x7f0590104680 2023-10-11T16:36:28.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.379+0000 7f05975ff700 1 -- 172.21.15.111:0/2254555117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05901169c0 con 0x7f0590104680 2023-10-11T16:36:28.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.379+0000 7f05975ff700 1 -- 172.21.15.111:0/2254555117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0590116830 con 0x7f0590104680 2023-10-11T16:36:28.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.384+0000 7f05867fc700 1 -- 172.21.15.111:0/2254555117 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f05880057c0 con 0x7f0590104680 2023-10-11T16:36:28.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.386+0000 7f05867fc700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f057c0a1340 0x7f057c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.387+0000 7f05867fc700 1 -- 172.21.15.111:0/2254555117 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f05880ce9b0 con 0x7f0590104680 2023-10-11T16:36:28.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.387+0000 7f059539b700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f057c0a1340 0x7f057c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.388+0000 7f059539b700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f057c0a1340 0x7f057c0a37f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f058c005fd0 tx=0x7f058c005ee0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:28.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.389+0000 7f05aaf65b80 1 Processor -- start 2023-10-11T16:36:28.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.390+0000 7f05aaf65b80 1 -- start start 2023-10-11T16:36:28.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.390+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e68519f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.390+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e6851e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.390+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851e8b0 0x564e6851ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.390+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6841f060 con 0x564e6851a470 2023-10-11T16:36:28.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.390+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e684718a0 con 0x564e68519b60 2023-10-11T16:36:28.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.390+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6846ad30 con 0x564e6851e8b0 2023-10-11T16:36:28.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f059539b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e68519f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f0594b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e6851e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f059539b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e68519f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50880/0 (socket says 172.21.15.111:50880) 2023-10-11T16:36:28.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f059539b700 1 -- 172.21.15.111:0/3691732967 learned_addr learned my addr 172.21.15.111:0/3691732967 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:28.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f0595b9c700 1 --2- 172.21.15.111:0/3691732967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851e8b0 0x564e6851ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f059539b700 1 -- 172.21.15.111:0/3691732967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851e8b0 msgr2=0x564e6851ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:28.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f059539b700 1 --2- 172.21.15.111:0/3691732967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851e8b0 0x564e6851ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f059539b700 1 -- 172.21.15.111:0/3691732967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 msgr2=0x564e6851e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:28.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f059539b700 1 --2- 172.21.15.111:0/3691732967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e6851e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.391+0000 7f059539b700 1 -- 172.21.15.111:0/3691732967 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e682ec630 con 0x564e68519b60 2023-10-11T16:36:28.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.392+0000 7f059539b700 1 --2- 172.21.15.111:0/3691732967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e68519f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f058c00edd0 tx=0x7f058c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87f15b52cd81f455 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:28.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.393+0000 7f057b7fe700 1 -- 172.21.15.111:0/3691732967 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c022070 con 0x564e68519b60 2023-10-11T16:36:28.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.393+0000 7f057b7fe700 1 -- 172.21.15.111:0/3691732967 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c00ee00 con 0x564e68519b60 2023-10-11T16:36:28.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.394+0000 7f057b7fe700 1 -- 172.21.15.111:0/3691732967 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c005410 con 0x564e68519b60 2023-10-11T16:36:28.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.394+0000 7f05aaf65b80 1 -- 172.21.15.111:0/3691732967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 msgr2=0x564e68519f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.394+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/3691732967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e68519f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f058c00edd0 tx=0x7f058c009f90 comp rx=0 tx=0).stop 2023-10-11T16:36:28.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.395+0000 7f05aaf65b80 1 -- 172.21.15.111:0/3691732967 shutdown_connections 2023-10-11T16:36:28.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.395+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/3691732967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851e8b0 0x564e6851ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.395+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/3691732967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e68519f30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.395+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/3691732967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e6851e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.395+0000 7f05aaf65b80 1 -- 172.21.15.111:0/3691732967 >> 172.21.15.111:0/3691732967 conn(0x564e6850f360 msgr2=0x564e68514b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:28.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.395+0000 7f05aaf65b80 1 -- 172.21.15.111:0/3691732967 shutdown_connections 2023-10-11T16:36:28.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.395+0000 7f05aaf65b80 1 -- 172.21.15.111:0/3691732967 wait complete. 2023-10-11T16:36:28.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.396+0000 7f05aaf65b80 1 -- start start 2023-10-11T16:36:28.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.396+0000 7f05aaf65b80 1 Processor -- start 2023-10-11T16:36:28.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.398+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68526010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.398+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68523300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.398+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 0x564e68523980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.398+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6840c8e0 con 0x564e6851a470 2023-10-11T16:36:28.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.398+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6840c580 con 0x564e6851e8b0 2023-10-11T16:36:28.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.398+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e682ee920 con 0x564e68519b60 2023-10-11T16:36:28.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.398+0000 7f05aaf65b80 10 client.0 _ll_register_callbacks cb 0x564e6842a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:28.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.399+0000 7f0594b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68523300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.399+0000 7f0594b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68523300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55986/0 (socket says 172.21.15.111:55986) 2023-10-11T16:36:28.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.399+0000 7f0594b9a700 1 -- 172.21.15.111:0/1929247374 learned_addr learned my addr 172.21.15.111:0/1929247374 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:28.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.399+0000 7f059539b700 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68526010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.400+0000 7f0594b9a700 1 -- 172.21.15.111:0/1929247374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 msgr2=0x564e68526010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:28.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.400+0000 7f0594b9a700 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68526010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.400+0000 7f0594b9a700 1 -- 172.21.15.111:0/1929247374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 msgr2=0x564e68523980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:36:28.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.400+0000 7f0594b9a700 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 0x564e68523980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.400+0000 7f0594b9a700 1 -- 172.21.15.111:0/1929247374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c00eab0 con 0x564e6851a470 2023-10-11T16:36:28.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.400+0000 7f059539b700 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68526010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:36:28.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.400+0000 7f0594b9a700 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68523300 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0580011680 tx=0x7f0580011a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0f7ff716fa249f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:28.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.400+0000 7f0578ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:36:28.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.401+0000 7f0578ff9700 1 -- 172.21.15.111:0/1929247374 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580013670 con 0x564e6851a470 2023-10-11T16:36:28.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.401+0000 7f0578ff9700 1 -- 172.21.15.111:0/1929247374 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f0580018070 con 0x564e6851a470 2023-10-11T16:36:28.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.401+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1929247374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e6850cb20 con 0x564e6851a470 2023-10-11T16:36:28.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.402+0000 7f0578ff9700 1 -- 172.21.15.111:0/1929247374 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05800137d0 con 0x564e6851a470 2023-10-11T16:36:28.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.402+0000 7f05aaf65b80 10 client.16251 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:28.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.402+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1929247374 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564e68431190 con 0x564e6851a470 2023-10-11T16:36:28.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.402+0000 7f0578ff9700 1 -- 172.21.15.111:0/1929247374 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564e6850cb20 con 0x564e6851a470 2023-10-11T16:36:28.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.403+0000 7f0578ff9700 1 -- 172.21.15.111:0/1929247374 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f058001dbd0 con 0x564e6851a470 2023-10-11T16:36:28.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.403+0000 7f05aaf65b80 10 client.16251 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:28.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.403+0000 7f05aaf65b80 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:28.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.403+0000 7f05aaf65b80 20 client.16251 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:28.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.403+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564e68533ad0 0x564e68535ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.403+0000 7f05aaf65b80 4 client.16251 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:36:28.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.403+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1929247374 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e6842d0a0 con 0x564e68533ad0 2023-10-11T16:36:28.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.403+0000 7f059539b700 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564e68533ad0 0x564e68535ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.404+0000 7f0578ff9700 10 client.16251 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:28.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.404+0000 7f059539b700 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564e68533ad0 0x564e68535ee0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecbde1751945cfcb server_cookie=ccfec49e2045126f in_seq=0 out_seq=0 2023-10-11T16:36:28.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.407+0000 7f0578ff9700 1 -- 172.21.15.111:0/1929247374 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564e6842d0a0 con 0x564e68533ad0 2023-10-11T16:36:28.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.407+0000 7f0578ff9700 10 client.16251 handle_command_reply: tid=0 2023-10-11T16:36:28.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.407+0000 7f05aaf65b80 1 client.16251 shutdown 2023-10-11T16:36:28.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1929247374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 msgr2=0x564e68523300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68523300 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0580011680 tx=0x7f0580011a40 comp rx=0 tx=0).stop 2023-10-11T16:36:28.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1929247374 shutdown_connections 2023-10-11T16:36:28.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564e68533ad0 0x564e68535ee0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 0x564e68523980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68526010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f0578ff9700 0 client.16251 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:28.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1929247374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68523300 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1929247374 >> 172.21.15.111:0/1929247374 conn(0x564e6850f360 msgr2=0x564e68513de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:28.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1929247374 shutdown_connections 2023-10-11T16:36:28.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.408+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1929247374 wait complete. 2023-10-11T16:36:28.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.409+0000 7f05aaf65b80 20 client.16251 trim_cache size 0 max 16384 2023-10-11T16:36:28.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.438+0000 7f05aaf65b80 1 Processor -- start 2023-10-11T16:36:28.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.439+0000 7f05aaf65b80 1 -- start start 2023-10-11T16:36:28.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.439+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e6852aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.439+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851a470 0x564e6851da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.439+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851e8b0 0x564e6851df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.439+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6851abc0 con 0x564e6851e8b0 2023-10-11T16:36:28.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.439+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6850f170 con 0x564e68519b60 2023-10-11T16:36:28.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.439+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6850c660 con 0x564e6851a470 2023-10-11T16:36:28.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.440+0000 7f0595b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851e8b0 0x564e6851df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.440+0000 7f0594b9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851a470 0x564e6851da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.440+0000 7f0594b9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851a470 0x564e6851da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:35926/0 (socket says 172.21.15.111:35926) 2023-10-11T16:36:28.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.440+0000 7f0594b9a700 1 -- 172.21.15.111:0/196623264 learned_addr learned my addr 172.21.15.111:0/196623264 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:28.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.440+0000 7f059539b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e6852aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.441+0000 7f059539b700 1 -- 172.21.15.111:0/196623264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851a470 msgr2=0x564e6851da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:28.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.441+0000 7f059539b700 1 --2- 172.21.15.111:0/196623264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851a470 0x564e6851da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.441+0000 7f059539b700 1 -- 172.21.15.111:0/196623264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851e8b0 msgr2=0x564e6851df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:28.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.441+0000 7f059539b700 1 --2- 172.21.15.111:0/196623264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851e8b0 0x564e6851df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.441+0000 7f059539b700 1 -- 172.21.15.111:0/196623264 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e68430ee0 con 0x564e68519b60 2023-10-11T16:36:28.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.441+0000 7f0595b9c700 1 --2- 172.21.15.111:0/196623264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851e8b0 0x564e6851df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:28.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.442+0000 7f059539b700 1 --2- 172.21.15.111:0/196623264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e6852aa40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f058c013040 tx=0x7f058c011be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=890faf8ea56102d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:28.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.442+0000 7f057bfff700 1 -- 172.21.15.111:0/196623264 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c022040 con 0x564e68519b60 2023-10-11T16:36:28.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.442+0000 7f057bfff700 1 -- 172.21.15.111:0/196623264 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c015070 con 0x564e68519b60 2023-10-11T16:36:28.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.442+0000 7f057bfff700 1 -- 172.21.15.111:0/196623264 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c005560 con 0x564e68519b60 2023-10-11T16:36:28.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.443+0000 7f05aaf65b80 1 -- 172.21.15.111:0/196623264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 msgr2=0x564e6852aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.443+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/196623264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e6852aa40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f058c013040 tx=0x7f058c011be0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.443+0000 7f05aaf65b80 1 -- 172.21.15.111:0/196623264 shutdown_connections 2023-10-11T16:36:28.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.443+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/196623264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851e8b0 0x564e6851df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.443+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/196623264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e68519b60 0x564e6852aa40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.443+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/196623264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e6851a470 0x564e6851da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.444+0000 7f05aaf65b80 1 -- 172.21.15.111:0/196623264 >> 172.21.15.111:0/196623264 conn(0x564e6850f360 msgr2=0x564e6851ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:28.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.444+0000 7f05aaf65b80 1 -- 172.21.15.111:0/196623264 shutdown_connections 2023-10-11T16:36:28.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.444+0000 7f05aaf65b80 1 -- 172.21.15.111:0/196623264 wait complete. 2023-10-11T16:36:28.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.444+0000 7f05aaf65b80 1 -- start start 2023-10-11T16:36:28.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.445+0000 7f05aaf65b80 1 Processor -- start 2023-10-11T16:36:28.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.446+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68520ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.446+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68521020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.446+0000 7f05aaf65b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 0x564e6851f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.447+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6851fc90 con 0x564e6851a470 2023-10-11T16:36:28.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.447+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6851fe00 con 0x564e6851e8b0 2023-10-11T16:36:28.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.447+0000 7f0595b9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 0x564e6851f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.447+0000 7f05aaf65b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6851ffc0 con 0x564e68519b60 2023-10-11T16:36:28.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.447+0000 7f05aaf65b80 10 client.0 _ll_register_callbacks cb 0x7f0580009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:28.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.447+0000 7f0595b9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 0x564e6851f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50906/0 (socket says 172.21.15.111:50906) 2023-10-11T16:36:28.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.447+0000 7f0595b9c700 1 -- 172.21.15.111:0/1981890956 learned_addr learned my addr 172.21.15.111:0/1981890956 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:28.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.447+0000 7f0594b9a700 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68521020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.448+0000 7f059539b700 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68520ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.448+0000 7f0594b9a700 1 -- 172.21.15.111:0/1981890956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 msgr2=0x564e68520ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:28.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.448+0000 7f0594b9a700 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68520ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.448+0000 7f0594b9a700 1 -- 172.21.15.111:0/1981890956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 msgr2=0x564e6851f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:28.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.448+0000 7f0594b9a700 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 0x564e6851f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.448+0000 7f0594b9a700 1 -- 172.21.15.111:0/1981890956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0590116830 con 0x564e6851a470 2023-10-11T16:36:28.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.449+0000 7f0594b9a700 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68521020 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0580001eb0 tx=0x7f0580013880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b3463aa2c5fa29 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:28.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.449+0000 7f0578ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:36:28.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.449+0000 7f0578ff9700 1 -- 172.21.15.111:0/1981890956 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580013d20 con 0x564e6851a470 2023-10-11T16:36:28.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.449+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1981890956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f058c00ee00 con 0x564e6851a470 2023-10-11T16:36:28.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.449+0000 7f0578ff9700 1 -- 172.21.15.111:0/1981890956 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f0580018040 con 0x564e6851a470 2023-10-11T16:36:28.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.450+0000 7f0578ff9700 1 -- 172.21.15.111:0/1981890956 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0580013e80 con 0x564e6851a470 2023-10-11T16:36:28.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.450+0000 7f05aaf65b80 10 client.16257 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:28.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.450+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1981890956 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564e682ec630 con 0x564e6851a470 2023-10-11T16:36:28.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.451+0000 7f0578ff9700 1 -- 172.21.15.111:0/1981890956 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c00ee00 con 0x564e6851a470 2023-10-11T16:36:28.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.452+0000 7f0578ff9700 1 -- 172.21.15.111:0/1981890956 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0580031080 con 0x564e6851a470 2023-10-11T16:36:28.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.452+0000 7f05aaf65b80 10 client.16257 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:28.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.452+0000 7f05aaf65b80 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:28.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.452+0000 7f05aaf65b80 20 client.16257 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:28.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.452+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564e68588970 0x564e6858ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:28.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.452+0000 7f05aaf65b80 4 client.16257 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:36:28.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.452+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1981890956 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564e68439660 con 0x564e68588970 2023-10-11T16:36:28.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.452+0000 7f059539b700 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564e68588970 0x564e6858ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:28.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.454+0000 7f059539b700 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564e68588970 0x564e6858ad80 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21305bf1d707cda6 server_cookie=bb7c6eafde9edc61 in_seq=0 out_seq=0 2023-10-11T16:36:28.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.454+0000 7f0578ff9700 10 client.16257 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.460+0000 7f0578ff9700 1 -- 172.21.15.111:0/1981890956 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564e68439660 con 0x564e68588970 2023-10-11T16:36:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.460+0000 7f0578ff9700 10 client.16257 handle_command_reply: tid=0 2023-10-11T16:36:28.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.460+0000 7f05aaf65b80 1 client.16257 shutdown 2023-10-11T16:36:28.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1981890956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 msgr2=0x564e68521020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68521020 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0580001eb0 tx=0x7f0580013880 comp rx=0 tx=0).stop 2023-10-11T16:36:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1981890956 shutdown_connections 2023-10-11T16:36:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564e68588970 0x564e6858ad80 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e6851e8b0 0x564e6851f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564e68519b60 0x564e68520ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f0578ff9700 0 client.16257 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:28.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 --2- 172.21.15.111:0/1981890956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564e6851a470 0x564e68521020 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1981890956 >> 172.21.15.111:0/1981890956 conn(0x564e6850f360 msgr2=0x564e6851da10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:28.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.461+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1981890956 shutdown_connections 2023-10-11T16:36:28.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.462+0000 7f05aaf65b80 1 -- 172.21.15.111:0/1981890956 wait complete. 2023-10-11T16:36:28.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.462+0000 7f05aaf65b80 20 client.16257 trim_cache size 0 max 16384 2023-10-11T16:36:28.466 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:36:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:36:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:36:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "e4b353b6-bcd8-4549-a24b-3247f2c5740c": { 2023-10-11T16:36:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:36:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "e4b353b6-bcd8-4549-a24b-3247f2c5740c", 2023-10-11T16:36:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:36:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:36:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:36:28.469 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:36:28.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 -- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f057c0a1340 msgr2=0x7f057c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f057c0a1340 0x7f057c0a37f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f058c005fd0 tx=0x7f058c005ee0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 -- 172.21.15.111:0/2254555117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 msgr2=0x7f059019ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:28.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 0x7f059019ca20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0588009fd0 tx=0x7f058800ce50 comp rx=0 tx=0).stop 2023-10-11T16:36:28.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 -- 172.21.15.111:0/2254555117 shutdown_connections 2023-10-11T16:36:28.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f057c0a1340 0x7f057c0a37f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0590104680 0x7f059019ca20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900ff180 0x7f05901a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 --2- 172.21.15.111:0/2254555117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05900ffa90 0x7f05901a2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:28.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.463+0000 7f05975ff700 1 -- 172.21.15.111:0/2254555117 >> 172.21.15.111:0/2254555117 conn(0x7f05900770b0 msgr2=0x7f05901020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:28.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.464+0000 7f05975ff700 1 -- 172.21.15.111:0/2254555117 shutdown_connections 2023-10-11T16:36:28.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:28.464+0000 7f05975ff700 1 -- 172.21.15.111:0/2254555117 wait complete. 2023-10-11T16:36:28.500 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b353b6-bcd8-4549-a24b-3247f2c5740c - {'path': '/', 'tag': 'e4b353b6-bcd8-4549-a24b-3247f2c5740c', 'options': 'recursive,force'} 2023-10-11T16:36:29.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:29 smithi130 ceph-mon[131859]: pgmap v118: 129 pgs: 129 active+clean; 350 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 210 op/s 2023-10-11T16:36:29.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:29 smithi111 ceph-mon[98527]: pgmap v118: 129 pgs: 129 active+clean; 350 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 210 op/s 2023-10-11T16:36:29.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:29 smithi150 ceph-mon[132217]: pgmap v118: 129 pgs: 129 active+clean; 350 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 210 op/s 2023-10-11T16:36:31.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:31 smithi130 ceph-mon[131859]: pgmap v119: 129 pgs: 129 active+clean; 357 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.8 MiB/s wr, 732 op/s 2023-10-11T16:36:31.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:31 smithi111 ceph-mon[98527]: pgmap v119: 129 pgs: 129 active+clean; 357 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.8 MiB/s wr, 732 op/s 2023-10-11T16:36:31.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:31 smithi150 ceph-mon[132217]: pgmap v119: 129 pgs: 129 active+clean; 357 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.8 MiB/s wr, 732 op/s 2023-10-11T16:36:33.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:33 smithi130 ceph-mon[131859]: pgmap v120: 129 pgs: 129 active+clean; 357 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.6 MiB/s wr, 660 op/s 2023-10-11T16:36:33.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:33 smithi111 ceph-mon[98527]: pgmap v120: 129 pgs: 129 active+clean; 357 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.6 MiB/s wr, 660 op/s 2023-10-11T16:36:33.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:33 smithi150 ceph-mon[132217]: pgmap v120: 129 pgs: 129 active+clean; 357 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.6 MiB/s wr, 660 op/s 2023-10-11T16:36:35.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:35 smithi130 ceph-mon[131859]: pgmap v121: 129 pgs: 129 active+clean; 370 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.8 MiB/s wr, 706 op/s 2023-10-11T16:36:35.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:35 smithi111 ceph-mon[98527]: pgmap v121: 129 pgs: 129 active+clean; 370 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.8 MiB/s wr, 706 op/s 2023-10-11T16:36:35.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:35 smithi150 ceph-mon[132217]: pgmap v121: 129 pgs: 129 active+clean; 370 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.8 MiB/s wr, 706 op/s 2023-10-11T16:36:37.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:37 smithi111 ceph-mon[98527]: pgmap v122: 129 pgs: 129 active+clean; 391 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 854 op/s 2023-10-11T16:36:37.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:37 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:36:37.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:37 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:37 smithi150 ceph-mon[132217]: pgmap v122: 129 pgs: 129 active+clean; 391 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 854 op/s 2023-10-11T16:36:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:37 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:36:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:37 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:38.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:37 smithi130 ceph-mon[131859]: pgmap v122: 129 pgs: 129 active+clean; 391 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 8.8 MiB/s wr, 854 op/s 2023-10-11T16:36:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:37 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:36:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:37 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:39 smithi111 ceph-mon[98527]: pgmap v123: 129 pgs: 129 active+clean; 391 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.3 MiB/s wr, 740 op/s 2023-10-11T16:36:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:39 smithi150 ceph-mon[132217]: pgmap v123: 129 pgs: 129 active+clean; 391 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.3 MiB/s wr, 740 op/s 2023-10-11T16:36:40.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:39 smithi130 ceph-mon[131859]: pgmap v123: 129 pgs: 129 active+clean; 391 MiB data, 941 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.3 MiB/s wr, 740 op/s 2023-10-11T16:36:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.493+0000 7f5953fff700 1 -- 172.21.15.111:0/2975304803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 msgr2=0x7f5954110b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.493+0000 7f5953fff700 1 --2- 172.21.15.111:0/2975304803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 0x7f5954110b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f593c009990 tx=0x7f593c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.494+0000 7f5953fff700 1 -- 172.21.15.111:0/2975304803 shutdown_connections 2023-10-11T16:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.494+0000 7f5953fff700 1 --2- 172.21.15.111:0/2975304803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5954111080 0x7f5954113460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.494+0000 7f5953fff700 1 --2- 172.21.15.111:0/2975304803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954072e90 0x7f5954073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.494+0000 7f5953fff700 1 --2- 172.21.15.111:0/2975304803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 0x7f5954110b40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.494+0000 7f5953fff700 1 -- 172.21.15.111:0/2975304803 >> 172.21.15.111:0/2975304803 conn(0x7f59540fbb70 msgr2=0x7f59540fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.494+0000 7f5953fff700 1 -- 172.21.15.111:0/2975304803 shutdown_connections 2023-10-11T16:36:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.495+0000 7f5953fff700 1 -- 172.21.15.111:0/2975304803 wait complete. 2023-10-11T16:36:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.497+0000 7f5953fff700 1 Processor -- start 2023-10-11T16:36:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.498+0000 7f5953fff700 1 -- start start 2023-10-11T16:36:40.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.499+0000 7f5953fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5954072e90 0x7f5954110410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.499+0000 7f5953fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 0x7f5954110990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.499+0000 7f5953fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954111080 0x7f59541094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.500+0000 7f5953fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5954109c70 con 0x7f5954111080 2023-10-11T16:36:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.500+0000 7f594a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 0x7f5954110990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.500+0000 7f594a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 0x7f5954110990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36724/0 (socket says 172.21.15.111:36724) 2023-10-11T16:36:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.500+0000 7f594a7fc700 1 -- 172.21.15.111:0/1873857741 learned_addr learned my addr 172.21.15.111:0/1873857741 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.500+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5954109de0 con 0x7f5954072e90 2023-10-11T16:36:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.500+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5954109fa0 con 0x7f59540737a0 2023-10-11T16:36:40.503 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.500+0000 7f5952ffd700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5954072e90 0x7f5954110410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.504 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.501+0000 7f594a7fc700 1 -- 172.21.15.111:0/1873857741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5954072e90 msgr2=0x7f5954110410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:40.504 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.501+0000 7f594a7fc700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5954072e90 0x7f5954110410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.501+0000 7f594a7fc700 1 -- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954111080 msgr2=0x7f59541094a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:40.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.501+0000 7f594a7fc700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954111080 0x7f59541094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.501+0000 7f594a7fc700 1 -- 172.21.15.111:0/1873857741 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0095f0 con 0x7f59540737a0 2023-10-11T16:36:40.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.501+0000 7f594a7fc700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 0x7f5954110990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f593c00c010 tx=0x7f593c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:40.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.502+0000 7f5950ff9700 1 -- 172.21.15.111:0/1873857741 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c01d070 con 0x7f59540737a0 2023-10-11T16:36:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.502+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5954116770 con 0x7f59540737a0 2023-10-11T16:36:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.502+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59541165e0 con 0x7f59540737a0 2023-10-11T16:36:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.503+0000 7f5950ff9700 1 -- 172.21.15.111:0/1873857741 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f593c021410 con 0x7f59540737a0 2023-10-11T16:36:40.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.503+0000 7f5950ff9700 1 -- 172.21.15.111:0/1873857741 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c017650 con 0x7f59540737a0 2023-10-11T16:36:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.515+0000 7f596d092b80 1 Processor -- start 2023-10-11T16:36:40.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.516+0000 7f5950ff9700 1 -- 172.21.15.111:0/1873857741 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f593c00fb30 con 0x7f59540737a0 2023-10-11T16:36:40.519 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.517+0000 7f596d092b80 1 -- start start 2023-10-11T16:36:40.520 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.518+0000 7f5950ff9700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f59400a1380 0x7f59400a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.518+0000 7f5950ff9700 1 -- 172.21.15.111:0/1873857741 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f593c0ce190 con 0x7f59540737a0 2023-10-11T16:36:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.519+0000 7f59537fe700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f59400a1380 0x7f59400a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.521+0000 7f59537fe700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f59400a1380 0x7f59400a3830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f595406a0b0 tx=0x7f594c009620 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.522+0000 7f596d092b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fe8f16b0 0x5618fea9b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.522+0000 7f596d092b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea957b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.522+0000 7f596d092b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fea95cf0 0x5618fea96120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.523+0000 7f5952ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fe8f16b0 0x5618fea9b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.527 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.523+0000 7f5952ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fe8f16b0 0x5618fea9b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33126/0 (socket says 172.21.15.111:33126) 2023-10-11T16:36:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.523+0000 7f5952ffd700 1 -- 172.21.15.111:0/2373497982 learned_addr learned my addr 172.21.15.111:0/2373497982 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.524+0000 7f596d092b80 1 -- 172.21.15.111:0/2373497982 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fe99b000 con 0x5618fea95cf0 2023-10-11T16:36:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.524+0000 7f596d092b80 1 -- 172.21.15.111:0/2373497982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fe9ed840 con 0x5618fe8f16b0 2023-10-11T16:36:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.524+0000 7f596d092b80 1 -- 172.21.15.111:0/2373497982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fe9e6cd0 con 0x5618fea9b990 2023-10-11T16:36:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.523+0000 7f59537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fea95cf0 0x5618fea96120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.524+0000 7f59537fe700 1 -- 172.21.15.111:0/2373497982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 msgr2=0x5618fea957b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.524+0000 7f59537fe700 1 --2- 172.21.15.111:0/2373497982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.524+0000 7f59537fe700 1 -- 172.21.15.111:0/2373497982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fe8f16b0 msgr2=0x5618fea9b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.524+0000 7f59537fe700 1 --2- 172.21.15.111:0/2373497982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fe8f16b0 0x5618fea9b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.525+0000 7f59537fe700 1 -- 172.21.15.111:0/2373497982 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618fe8685d0 con 0x5618fea95cf0 2023-10-11T16:36:40.531 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.526+0000 7f59537fe700 1 --2- 172.21.15.111:0/2373497982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fea95cf0 0x5618fea96120 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f594c00e9e0 tx=0x7f594c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fee02fdc67c554 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:40.531 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.527+0000 7f59497fa700 1 -- 172.21.15.111:0/2373497982 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c004a30 con 0x5618fea95cf0 2023-10-11T16:36:40.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.527+0000 7f59497fa700 1 -- 172.21.15.111:0/2373497982 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f594c0054e0 con 0x5618fea95cf0 2023-10-11T16:36:40.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.527+0000 7f59497fa700 1 -- 172.21.15.111:0/2373497982 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c010a50 con 0x5618fea95cf0 2023-10-11T16:36:40.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.528+0000 7f596d092b80 1 -- 172.21.15.111:0/2373497982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fea95cf0 msgr2=0x5618fea96120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:40.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.528+0000 7f596d092b80 1 --2- 172.21.15.111:0/2373497982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fea95cf0 0x5618fea96120 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f594c00e9e0 tx=0x7f594c011000 comp rx=0 tx=0).stop 2023-10-11T16:36:40.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.529+0000 7f596d092b80 1 -- 172.21.15.111:0/2373497982 shutdown_connections 2023-10-11T16:36:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.529+0000 7f596d092b80 1 --2- 172.21.15.111:0/2373497982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fea95cf0 0x5618fea96120 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.529+0000 7f596d092b80 1 --2- 172.21.15.111:0/2373497982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fe8f16b0 0x5618fea9b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.529+0000 7f596d092b80 1 --2- 172.21.15.111:0/2373497982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.529+0000 7f596d092b80 1 -- 172.21.15.111:0/2373497982 >> 172.21.15.111:0/2373497982 conn(0x5618fea8b1a0 msgr2=0x5618fea903e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.529+0000 7f596d092b80 1 -- 172.21.15.111:0/2373497982 shutdown_connections 2023-10-11T16:36:40.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.529+0000 7f596d092b80 1 -- 172.21.15.111:0/2373497982 wait complete. 2023-10-11T16:36:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.530+0000 7f596d092b80 1 -- start start 2023-10-11T16:36:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.530+0000 7f596d092b80 1 Processor -- start 2023-10-11T16:36:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.532+0000 7f596d092b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618fea9ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.532+0000 7f596d092b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618fea9f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.532+0000 7f596d092b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618fea9f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.532+0000 7f596d092b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fe988880 con 0x5618fe8f16b0 2023-10-11T16:36:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.532+0000 7f596d092b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fe988520 con 0x5618fea9b990 2023-10-11T16:36:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.532+0000 7f596d092b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fe86a8c0 con 0x5618fea95cf0 2023-10-11T16:36:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.532+0000 7f596d092b80 10 client.0 _ll_register_callbacks cb 0x5618fe9a6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.533+0000 7f594a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618fea9f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.533+0000 7f594a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618fea9f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36752/0 (socket says 172.21.15.111:36752) 2023-10-11T16:36:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.533+0000 7f594a7fc700 1 -- 172.21.15.111:0/4095596056 learned_addr learned my addr 172.21.15.111:0/4095596056 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.533+0000 7f59537fe700 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618fea9f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.534+0000 7f594a7fc700 1 -- 172.21.15.111:0/4095596056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 msgr2=0x5618fea9f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.534+0000 7f594a7fc700 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618fea9f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.534+0000 7f594a7fc700 1 -- 172.21.15.111:0/4095596056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 msgr2=0x5618fea9ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.534+0000 7f594a7fc700 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618fea9ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.534+0000 7f594a7fc700 1 -- 172.21.15.111:0/4095596056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c00e6c0 con 0x5618fea95cf0 2023-10-11T16:36:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.534+0000 7f594a7fc700 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618fea9f480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f593c097440 tx=0x7f593c09af40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e971c528b2dbe8f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.535+0000 7f593a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:36:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.535+0000 7f593a7fc700 1 -- 172.21.15.111:0/4095596056 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c0ce020 con 0x5618fea95cf0 2023-10-11T16:36:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.535+0000 7f593a7fc700 1 -- 172.21.15.111:0/4095596056 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f593c09a430 con 0x5618fea95cf0 2023-10-11T16:36:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.536+0000 7f596d092b80 1 -- 172.21.15.111:0/4095596056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5618fea88960 con 0x5618fea95cf0 2023-10-11T16:36:40.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.537+0000 7f593a7fc700 1 -- 172.21.15.111:0/4095596056 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f593c09a5f0 con 0x5618fea95cf0 2023-10-11T16:36:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.537+0000 7f596d092b80 10 client.26380 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.537+0000 7f596d092b80 1 -- 172.21.15.111:0/4095596056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5618fe9ad130 con 0x5618fea95cf0 2023-10-11T16:36:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.537+0000 7f593a7fc700 1 -- 172.21.15.111:0/4095596056 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5618fea88960 con 0x5618fea95cf0 2023-10-11T16:36:40.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.538+0000 7f593a7fc700 1 -- 172.21.15.111:0/4095596056 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f593c095b50 con 0x5618fea95cf0 2023-10-11T16:36:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.538+0000 7f596d092b80 10 client.26380 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.539+0000 7f596d092b80 10 client.26380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.539+0000 7f596d092b80 20 client.26380 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.539+0000 7f596d092b80 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5618feaaef60 0x5618feab1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.539+0000 7f596d092b80 4 client.26380 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:36:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.539+0000 7f596d092b80 1 -- 172.21.15.111:0/4095596056 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618fe9a9040 con 0x5618feaaef60 2023-10-11T16:36:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.540+0000 7f5952ffd700 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5618feaaef60 0x5618feab1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.543+0000 7f5952ffd700 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5618feaaef60 0x5618feab1370 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=126394ec45fe6661 server_cookie=773122ac18f3cc39 in_seq=0 out_seq=0 2023-10-11T16:36:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.543+0000 7f593a7fc700 10 client.26380 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.544+0000 7f593a7fc700 1 -- 172.21.15.111:0/4095596056 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5618fe9a9040 con 0x5618feaaef60 2023-10-11T16:36:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.544+0000 7f593a7fc700 10 client.26380 handle_command_reply: tid=0 2023-10-11T16:36:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.545+0000 7f596d092b80 1 client.26380 shutdown 2023-10-11T16:36:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.548+0000 7f596d092b80 1 -- 172.21.15.111:0/4095596056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 msgr2=0x5618fea9f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.548+0000 7f596d092b80 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618fea9f480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f593c097440 tx=0x7f593c09af40 comp rx=0 tx=0).stop 2023-10-11T16:36:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.548+0000 7f596d092b80 1 -- 172.21.15.111:0/4095596056 shutdown_connections 2023-10-11T16:36:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.548+0000 7f596d092b80 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5618feaaef60 0x5618feab1370 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.548+0000 7f596d092b80 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618fea9f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.548+0000 7f596d092b80 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618fea9ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.549+0000 7f593a7fc700 0 client.26380 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.549+0000 7f596d092b80 1 --2- 172.21.15.111:0/4095596056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618fea9f480 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.549+0000 7f596d092b80 1 -- 172.21.15.111:0/4095596056 >> 172.21.15.111:0/4095596056 conn(0x5618fea8b1a0 msgr2=0x5618fea8fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.549+0000 7f596d092b80 1 -- 172.21.15.111:0/4095596056 shutdown_connections 2023-10-11T16:36:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.550+0000 7f596d092b80 1 -- 172.21.15.111:0/4095596056 wait complete. 2023-10-11T16:36:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.550+0000 7f596d092b80 20 client.26380 trim_cache size 0 max 16384 2023-10-11T16:36:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.595+0000 7f596d092b80 1 Processor -- start 2023-10-11T16:36:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f596d092b80 1 -- start start 2023-10-11T16:36:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f596d092b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618fea957b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f596d092b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea95cf0 0x5618fea9fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f596d092b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea90180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f596d092b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fea92280 con 0x5618fe8f16b0 2023-10-11T16:36:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f596d092b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fea88810 con 0x5618fea95cf0 2023-10-11T16:36:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f596d092b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618fea8cb10 con 0x5618fea9b990 2023-10-11T16:36:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f59537fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea90180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f59537fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea90180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36766/0 (socket says 172.21.15.111:36766) 2023-10-11T16:36:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f59537fe700 1 -- 172.21.15.111:0/2534518095 learned_addr learned my addr 172.21.15.111:0/2534518095 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.596+0000 7f594a7fc700 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea95cf0 0x5618fea9fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f5952ffd700 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618fea957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f59537fe700 1 -- 172.21.15.111:0/2534518095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea95cf0 msgr2=0x5618fea9fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f59537fe700 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea95cf0 0x5618fea9fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f59537fe700 1 -- 172.21.15.111:0/2534518095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 msgr2=0x5618fea957b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f59537fe700 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618fea957b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f59537fe700 1 -- 172.21.15.111:0/2534518095 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618feab5f60 con 0x5618fea9b990 2023-10-11T16:36:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f594a7fc700 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea95cf0 0x5618fea9fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f5952ffd700 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618fea957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:36:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.597+0000 7f59537fe700 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea90180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f594c0059d0 tx=0x7f594c005670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e87e8053f03f100 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.599+0000 7f5949ffb700 1 -- 172.21.15.111:0/2534518095 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c0107b0 con 0x5618fea9b990 2023-10-11T16:36:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.599+0000 7f5949ffb700 1 -- 172.21.15.111:0/2534518095 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f594c029040 con 0x5618fea9b990 2023-10-11T16:36:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.600+0000 7f5949ffb700 1 -- 172.21.15.111:0/2534518095 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c02ed70 con 0x5618fea9b990 2023-10-11T16:36:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.600+0000 7f596d092b80 1 -- 172.21.15.111:0/2534518095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 msgr2=0x5618fea90180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.600+0000 7f596d092b80 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea90180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f594c0059d0 tx=0x7f594c005670 comp rx=0 tx=0).stop 2023-10-11T16:36:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.600+0000 7f596d092b80 1 -- 172.21.15.111:0/2534518095 shutdown_connections 2023-10-11T16:36:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.600+0000 7f596d092b80 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea9b990 0x5618fea90180 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.600+0000 7f596d092b80 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618fea957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.600+0000 7f596d092b80 1 --2- 172.21.15.111:0/2534518095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea95cf0 0x5618fea9fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.600+0000 7f596d092b80 1 -- 172.21.15.111:0/2534518095 >> 172.21.15.111:0/2534518095 conn(0x5618fea8b1a0 msgr2=0x5618fea9f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.601+0000 7f596d092b80 1 -- 172.21.15.111:0/2534518095 shutdown_connections 2023-10-11T16:36:40.609 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.601+0000 7f596d092b80 1 -- 172.21.15.111:0/2534518095 wait complete. 2023-10-11T16:36:40.609 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.601+0000 7f596d092b80 1 -- start start 2023-10-11T16:36:40.609 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f596d092b80 1 Processor -- start 2023-10-11T16:36:40.609 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f596d092b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618feaa5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f596d092b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618feaa5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f596d092b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618feb01de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f596d092b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618feaa6360 con 0x5618fe8f16b0 2023-10-11T16:36:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f596d092b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618feb02320 con 0x5618fea9b990 2023-10-11T16:36:40.610 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f596d092b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618feb02460 con 0x5618fea95cf0 2023-10-11T16:36:40.611 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f596d092b80 10 client.0 _ll_register_callbacks cb 0x5618fe9a6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:40.611 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f59537fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618feb01de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.611 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f59537fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618feb01de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33156/0 (socket says 172.21.15.111:33156) 2023-10-11T16:36:40.611 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.602+0000 7f59537fe700 1 -- 172.21.15.111:0/2680027176 learned_addr learned my addr 172.21.15.111:0/2680027176 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:40.611 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.603+0000 7f59537fe700 1 -- 172.21.15.111:0/2680027176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 msgr2=0x5618feaa5a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:36:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.603+0000 7f59537fe700 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618feaa5a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.603+0000 7f59537fe700 1 -- 172.21.15.111:0/2680027176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 msgr2=0x5618feaa5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.603+0000 7f59537fe700 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618feaa5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.603+0000 7f59537fe700 1 -- 172.21.15.111:0/2680027176 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618fe9ad130 con 0x5618fea9b990 2023-10-11T16:36:40.612 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.603+0000 7f59537fe700 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618feb01de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f594c02ec30 tx=0x7f594c0094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e311753b120ccf1b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.603+0000 7f593a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:36:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.603+0000 7f593a7fc700 1 -- 172.21.15.111:0/2680027176 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c010980 con 0x5618fea9b990 2023-10-11T16:36:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.604+0000 7f593a7fc700 1 -- 172.21.15.111:0/2680027176 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f594c010ae0 con 0x5618fea9b990 2023-10-11T16:36:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.604+0000 7f596d092b80 1 -- 172.21.15.111:0/2680027176 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f594c0054e0 con 0x5618fea9b990 2023-10-11T16:36:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.604+0000 7f593a7fc700 1 -- 172.21.15.111:0/2680027176 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f594c010ca0 con 0x5618fea9b990 2023-10-11T16:36:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.604+0000 7f596d092b80 10 client.26351 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.604+0000 7f596d092b80 1 -- 172.21.15.111:0/2680027176 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5618fe8685d0 con 0x5618fea9b990 2023-10-11T16:36:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.604+0000 7f593a7fc700 1 -- 172.21.15.111:0/2680027176 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f594c0054e0 con 0x5618fea9b990 2023-10-11T16:36:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f593a7fc700 1 -- 172.21.15.111:0/2680027176 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f594c035070 con 0x5618fea9b990 2023-10-11T16:36:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f596d092b80 10 client.26351 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f596d092b80 10 client.26351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f596d092b80 20 client.26351 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f596d092b80 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5618feaaef60 0x5618feb055d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f596d092b80 4 client.26351 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:36:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f596d092b80 1 -- 172.21.15.111:0/2680027176 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5618fe9b5600 con 0x5618feaaef60 2023-10-11T16:36:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f5952ffd700 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5618feaaef60 0x5618feb055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:40.617 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f5952ffd700 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5618feaaef60 0x5618feb055d0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1c88bc3e3812abe server_cookie=20c10720be7255ca in_seq=0 out_seq=0 2023-10-11T16:36:40.617 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:40.605+0000 7f593a7fc700 10 client.26351 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.305+0000 7f593a7fc700 1 -- 172.21.15.111:0/2680027176 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5618fe9b5600 con 0x5618feaaef60 2023-10-11T16:36:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.305+0000 7f593a7fc700 10 client.26351 handle_command_reply: tid=0 2023-10-11T16:36:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.305+0000 7f596d092b80 1 client.26351 shutdown 2023-10-11T16:36:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.305+0000 7f596d092b80 1 -- 172.21.15.111:0/2680027176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 msgr2=0x5618feb01de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618feb01de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f594c02ec30 tx=0x7f594c0094b0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 -- 172.21.15.111:0/2680027176 shutdown_connections 2023-10-11T16:36:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5618feaaef60 0x5618feb055d0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5618fea9b990 0x5618feb01de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5618fe8f16b0 0x5618feaa5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 --2- 172.21.15.111:0/2680027176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5618fea95cf0 0x5618feaa5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 -- 172.21.15.111:0/2680027176 >> 172.21.15.111:0/2680027176 conn(0x5618fea8b1a0 msgr2=0x5618fea9a040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f593a7fc700 0 client.26351 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 -- 172.21.15.111:0/2680027176 shutdown_connections 2023-10-11T16:36:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 1 -- 172.21.15.111:0/2680027176 wait complete. 2023-10-11T16:36:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.306+0000 7f596d092b80 20 client.26351 trim_cache size 0 max 16384 2023-10-11T16:36:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:36:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:36:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:36:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:36:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.307+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f59400a1380 msgr2=0x7f59400a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.307+0000 7f5953fff700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f59400a1380 0x7f59400a3830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f595406a0b0 tx=0x7f594c009620 comp rx=0 tx=0).stop 2023-10-11T16:36:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.307+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 msgr2=0x7f5954110990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.307+0000 7f5953fff700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 0x7f5954110990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f593c00c010 tx=0x7f593c0057f0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.309+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 shutdown_connections 2023-10-11T16:36:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.309+0000 7f5953fff700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f59400a1380 0x7f59400a3830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.309+0000 7f5953fff700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5954111080 0x7f59541094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.309+0000 7f5953fff700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5954072e90 0x7f5954110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.309+0000 7f5953fff700 1 --2- 172.21.15.111:0/1873857741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59540737a0 0x7f5954110990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.309+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 >> 172.21.15.111:0/1873857741 conn(0x7f59540fbb70 msgr2=0x7f59540fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.309+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 shutdown_connections 2023-10-11T16:36:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:41.309+0000 7f5953fff700 1 -- 172.21.15.111:0/1873857741 wait complete. 2023-10-11T16:36:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:41 smithi111 ceph-mon[98527]: pgmap v124: 129 pgs: 129 active+clean; 416 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.3 MiB/s wr, 939 op/s 2023-10-11T16:36:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:41 smithi150 ceph-mon[132217]: pgmap v124: 129 pgs: 129 active+clean; 416 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.3 MiB/s wr, 939 op/s 2023-10-11T16:36:42.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:41 smithi130 ceph-mon[131859]: pgmap v124: 129 pgs: 129 active+clean; 416 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.3 MiB/s wr, 939 op/s 2023-10-11T16:36:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:43 smithi111 ceph-mon[98527]: pgmap v125: 129 pgs: 129 active+clean; 416 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 417 op/s 2023-10-11T16:36:43.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:43 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:36:43.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:43 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:43.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:43 smithi150 ceph-mon[132217]: pgmap v125: 129 pgs: 129 active+clean; 416 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 417 op/s 2023-10-11T16:36:43.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:43 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:36:43.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:43 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:44.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:43 smithi130 ceph-mon[131859]: pgmap v125: 129 pgs: 129 active+clean; 416 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.9 MiB/s wr, 417 op/s 2023-10-11T16:36:44.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:43 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:36:44.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:43 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:45 smithi111 ceph-mon[98527]: pgmap v126: 129 pgs: 129 active+clean; 412 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 505 op/s 2023-10-11T16:36:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:45 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:36:45.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:45 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:45 smithi150 ceph-mon[132217]: pgmap v126: 129 pgs: 129 active+clean; 412 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 505 op/s 2023-10-11T16:36:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:45 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:36:45.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:45 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:46.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:45 smithi130 ceph-mon[131859]: pgmap v126: 129 pgs: 129 active+clean; 412 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 505 op/s 2023-10-11T16:36:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:45 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:36:46.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:45 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:47.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:47 smithi111 ceph-mon[98527]: pgmap v127: 129 pgs: 129 active+clean; 418 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.3 MiB/s wr, 831 op/s 2023-10-11T16:36:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:47 smithi150 ceph-mon[132217]: pgmap v127: 129 pgs: 129 active+clean; 418 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.3 MiB/s wr, 831 op/s 2023-10-11T16:36:48.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:47 smithi130 ceph-mon[131859]: pgmap v127: 129 pgs: 129 active+clean; 418 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.3 MiB/s wr, 831 op/s 2023-10-11T16:36:49.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:49 smithi111 ceph-mon[98527]: pgmap v128: 129 pgs: 129 active+clean; 418 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 657 op/s 2023-10-11T16:36:49.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:49 smithi150 ceph-mon[132217]: pgmap v128: 129 pgs: 129 active+clean; 418 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 657 op/s 2023-10-11T16:36:50.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:49 smithi130 ceph-mon[131859]: pgmap v128: 129 pgs: 129 active+clean; 418 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 657 op/s 2023-10-11T16:36:50.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:50 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:36:50.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:50 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:51.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:50 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:36:51.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:50 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:51.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:50 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:36:51.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:50 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:51.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:51 smithi150 ceph-mon[132217]: pgmap v129: 129 pgs: 129 active+clean; 446 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 868 op/s 2023-10-11T16:36:52.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:51 smithi130 ceph-mon[131859]: pgmap v129: 129 pgs: 129 active+clean; 446 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 868 op/s 2023-10-11T16:36:52.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:51 smithi111 ceph-mon[98527]: pgmap v129: 129 pgs: 129 active+clean; 446 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 868 op/s 2023-10-11T16:36:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:53 smithi150 ceph-mon[132217]: pgmap v130: 129 pgs: 129 active+clean; 446 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 670 op/s 2023-10-11T16:36:54.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:53 smithi130 ceph-mon[131859]: pgmap v130: 129 pgs: 129 active+clean; 446 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 670 op/s 2023-10-11T16:36:54.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:53 smithi111 ceph-mon[98527]: pgmap v130: 129 pgs: 129 active+clean; 446 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 670 op/s 2023-10-11T16:36:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 -- 172.21.15.111:0/771626648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc1015f0 msgr2=0x7faecc104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 --2- 172.21.15.111:0/771626648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc1015f0 0x7faecc104e80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faec0009990 tx=0x7faec0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 -- 172.21.15.111:0/771626648 shutdown_connections 2023-10-11T16:36:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 --2- 172.21.15.111:0/771626648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc105560 0x7faecc10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 --2- 172.21.15.111:0/771626648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faecc100c50 0x7faecc101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 --2- 172.21.15.111:0/771626648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc1015f0 0x7faecc104e80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 -- 172.21.15.111:0/771626648 >> 172.21.15.111:0/771626648 conn(0x7faecc077550 msgr2=0x7faecc077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 -- 172.21.15.111:0/771626648 shutdown_connections 2023-10-11T16:36:54.489 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.486+0000 7faed1bff700 1 -- 172.21.15.111:0/771626648 wait complete. 2023-10-11T16:36:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.487+0000 7faed1bff700 1 Processor -- start 2023-10-11T16:36:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faed1bff700 1 -- start start 2023-10-11T16:36:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faed1bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faecc100c50 0x7faecc196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faed1bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc1015f0 0x7faecc1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faed1bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc105560 0x7faecc19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.491 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faed1bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc19b370 con 0x7faecc1015f0 2023-10-11T16:36:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faed1bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc19b4e0 con 0x7faecc105560 2023-10-11T16:36:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faed1bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc19b6a0 con 0x7faecc100c50 2023-10-11T16:36:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faecb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faecc100c50 0x7faecc196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faecb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faecc100c50 0x7faecc196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:57520/0 (socket says 172.21.15.111:57520) 2023-10-11T16:36:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faecb7fe700 1 -- 172.21.15.111:0/2573301494 learned_addr learned my addr 172.21.15.111:0/2573301494 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faecaffd700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc1015f0 0x7faecc1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.488+0000 7faecbfff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc105560 0x7faecc19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faecbfff700 1 -- 172.21.15.111:0/2573301494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faecc100c50 msgr2=0x7faecc196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faecbfff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faecc100c50 0x7faecc196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faecbfff700 1 -- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc1015f0 msgr2=0x7faecc1968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:54.493 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faecbfff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc1015f0 0x7faecc1968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faecbfff700 1 -- 172.21.15.111:0/2573301494 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec00095f0 con 0x7faecc105560 2023-10-11T16:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faecc100c50 0x7faecc196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faecbfff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc105560 0x7faecc19ad30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faec400efc0 tx=0x7faec400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faec8ff9700 1 -- 172.21.15.111:0/2573301494 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4018070 con 0x7faecc105560 2023-10-11T16:36:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faec8ff9700 1 -- 172.21.15.111:0/2573301494 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faec400f040 con 0x7faecc105560 2023-10-11T16:36:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faed1bff700 1 -- 172.21.15.111:0/2573301494 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc110320 con 0x7faecc105560 2023-10-11T16:36:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.489+0000 7faec8ff9700 1 -- 172.21.15.111:0/2573301494 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec400cd40 con 0x7faecc105560 2023-10-11T16:36:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.490+0000 7faed1bff700 1 -- 172.21.15.111:0/2573301494 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc110190 con 0x7faecc105560 2023-10-11T16:36:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.494+0000 7faec8ff9700 1 -- 172.21.15.111:0/2573301494 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7faec4003a00 con 0x7faecc105560 2023-10-11T16:36:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.495+0000 7faee55a6b80 1 Processor -- start 2023-10-11T16:36:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faee55a6b80 1 -- start start 2023-10-11T16:36:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d44056b0 0x55a5d45af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45a9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faec8ff9700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7faeb40a1340 0x7faeb40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faec8ff9700 1 -- 172.21.15.111:0/2573301494 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7faec40ccbb0 con 0x7faecc105560 2023-10-11T16:36:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faecb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faecaffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d44056b0 0x55a5d45af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faecb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45a9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:57522/0 (socket says 172.21.15.111:57522) 2023-10-11T16:36:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faecaffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d44056b0 0x55a5d45af1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52768/0 (socket says 172.21.15.111:52768) 2023-10-11T16:36:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.496+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 0x55a5d45a9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faecb7fe700 1 -- 172.21.15.111:0/2301886860 learned_addr learned my addr 172.21.15.111:0/2301886860 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faee55a6b80 1 -- 172.21.15.111:0/2301886860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d44af000 con 0x55a5d45afb90 2023-10-11T16:36:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faee55a6b80 1 -- 172.21.15.111:0/2301886860 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d4501840 con 0x55a5d44056b0 2023-10-11T16:36:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faee55a6b80 1 -- 172.21.15.111:0/2301886860 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d44facd0 con 0x55a5d45af720 2023-10-11T16:36:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faecaffd700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7faeb40a1340 0x7faeb40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faecb7fe700 1 -- 172.21.15.111:0/2301886860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d44056b0 msgr2=0x55a5d45af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2301886860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d44056b0 0x55a5d45af1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faecb7fe700 1 -- 172.21.15.111:0/2301886860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 msgr2=0x55a5d45a9b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:36:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2301886860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 0x55a5d45a9b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faecb7fe700 1 -- 172.21.15.111:0/2301886860 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5d437c5d0 con 0x55a5d45af720 2023-10-11T16:36:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.497+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2301886860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45a9560 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faebc0117e0 tx=0x7faebc011af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5468ff34212a225 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.498+0000 7faeb1ffb700 1 -- 172.21.15.111:0/2301886860 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc00f480 con 0x55a5d45af720 2023-10-11T16:36:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.498+0000 7faeb1ffb700 1 -- 172.21.15.111:0/2301886860 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faebc00faa0 con 0x55a5d45af720 2023-10-11T16:36:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.498+0000 7faeb1ffb700 1 -- 172.21.15.111:0/2301886860 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc019a20 con 0x55a5d45af720 2023-10-11T16:36:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.498+0000 7faee55a6b80 1 -- 172.21.15.111:0/2301886860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 msgr2=0x55a5d45a9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.498+0000 7faecaffd700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7faeb40a1340 0x7faeb40a37f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faec000b2e0 tx=0x7faec0005fd0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.498+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2301886860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45a9560 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faebc0117e0 tx=0x7faebc011af0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.499+0000 7faee55a6b80 1 -- 172.21.15.111:0/2301886860 shutdown_connections 2023-10-11T16:36:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.499+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2301886860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 0x55a5d45a9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.499+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2301886860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d44056b0 0x55a5d45af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.499+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2301886860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45a9560 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.499+0000 7faee55a6b80 1 -- 172.21.15.111:0/2301886860 >> 172.21.15.111:0/2301886860 conn(0x55a5d459ef50 msgr2=0x55a5d45a4270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.499+0000 7faee55a6b80 1 -- 172.21.15.111:0/2301886860 shutdown_connections 2023-10-11T16:36:54.507 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.499+0000 7faee55a6b80 1 -- 172.21.15.111:0/2301886860 wait complete. 2023-10-11T16:36:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.499+0000 7faee55a6b80 1 -- start start 2023-10-11T16:36:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.500+0000 7faee55a6b80 1 Processor -- start 2023-10-11T16:36:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.500+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.500+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.500+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.500+0000 7faee55a6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d449c880 con 0x55a5d44056b0 2023-10-11T16:36:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.500+0000 7faee55a6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d449c520 con 0x55a5d45afb90 2023-10-11T16:36:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.500+0000 7faee55a6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d437e8c0 con 0x55a5d45af720 2023-10-11T16:36:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.500+0000 7faee55a6b80 10 client.0 _ll_register_callbacks cb 0x55a5d44ba720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.509 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52778/0 (socket says 172.21.15.111:52778) 2023-10-11T16:36:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 -- 172.21.15.111:0/2614724739 learned_addr learned my addr 172.21.15.111:0/2614724739 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 -- 172.21.15.111:0/2614724739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 msgr2=0x55a5d45b3290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:36:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45b3290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.510 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 -- 172.21.15.111:0/2614724739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 msgr2=0x55a5d45b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 -- 172.21.15.111:0/2614724739 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc011440 con 0x55a5d45afb90 2023-10-11T16:36:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faecbfff700 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45b37d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faec400ef90 tx=0x7faec4095d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba9cd72e3375cb8d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.501+0000 7faeaaffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:36:54.511 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.502+0000 7faeaaffd700 1 -- 172.21.15.111:0/2614724739 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec40b7ea0 con 0x55a5d45afb90 2023-10-11T16:36:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.502+0000 7faeaaffd700 1 -- 172.21.15.111:0/2614724739 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7faec4004ad0 con 0x55a5d45afb90 2023-10-11T16:36:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.502+0000 7faee55a6b80 1 -- 172.21.15.111:0/2614724739 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5d459c710 con 0x55a5d45afb90 2023-10-11T16:36:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.502+0000 7faeaaffd700 1 -- 172.21.15.111:0/2614724739 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faec4099900 con 0x55a5d45afb90 2023-10-11T16:36:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.502+0000 7faee55a6b80 10 client.26363 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:54.512 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.502+0000 7faee55a6b80 1 -- 172.21.15.111:0/2614724739 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a5d44c1130 con 0x55a5d45afb90 2023-10-11T16:36:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.502+0000 7faeaaffd700 1 -- 172.21.15.111:0/2614724739 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a5d459c710 con 0x55a5d45afb90 2023-10-11T16:36:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.502+0000 7faeaaffd700 1 -- 172.21.15.111:0/2614724739 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7faec409db90 con 0x55a5d45afb90 2023-10-11T16:36:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.503+0000 7faee55a6b80 10 client.26363 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.503+0000 7faee55a6b80 10 client.26363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.503+0000 7faee55a6b80 20 client.26363 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:54.513 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.503+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5d45c30b0 0x55a5d45c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.503+0000 7faee55a6b80 4 client.26363 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:36:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.503+0000 7faee55a6b80 1 -- 172.21.15.111:0/2614724739 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5d44bd040 con 0x55a5d45c30b0 2023-10-11T16:36:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.503+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5d45c30b0 0x55a5d45c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.504+0000 7faeaaffd700 10 client.26363 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:54.514 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.504+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5d45c30b0 0x55a5d45c54c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=323ff2793f90fa77 server_cookie=b61bd2ec60fcc5da in_seq=0 out_seq=0 2023-10-11T16:36:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faeaaffd700 1 -- 172.21.15.111:0/2614724739 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a5d44bd040 con 0x55a5d45c30b0 2023-10-11T16:36:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faeaaffd700 10 client.26363 handle_command_reply: tid=0 2023-10-11T16:36:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 client.26363 shutdown 2023-10-11T16:36:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 -- 172.21.15.111:0/2614724739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 msgr2=0x55a5d45b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45b37d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faec400ef90 tx=0x7faec4095d60 comp rx=0 tx=0).stop 2023-10-11T16:36:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 -- 172.21.15.111:0/2614724739 shutdown_connections 2023-10-11T16:36:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5d45c30b0 0x55a5d45c54c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45b37d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.517 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2614724739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faeaaffd700 0 client.26363 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 -- 172.21.15.111:0/2614724739 >> 172.21.15.111:0/2614724739 conn(0x55a5d459ef50 msgr2=0x55a5d45a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 -- 172.21.15.111:0/2614724739 shutdown_connections 2023-10-11T16:36:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.506+0000 7faee55a6b80 1 -- 172.21.15.111:0/2614724739 wait complete. 2023-10-11T16:36:54.518 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.508+0000 7faee55a6b80 20 client.26363 trim_cache size 0 max 16384 2023-10-11T16:36:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.528+0000 7faee55a6b80 1 Processor -- start 2023-10-11T16:36:54.529 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.528+0000 7faee55a6b80 1 -- start start 2023-10-11T16:36:54.530 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d44056b0 0x55a5d45a9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45af720 0x55a5d45ba120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 0x55a5d45a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faee55a6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d45c06e0 con 0x55a5d45afb90 2023-10-11T16:36:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faee55a6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d45ad9e0 con 0x55a5d45af720 2023-10-11T16:36:54.531 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faee55a6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d459c570 con 0x55a5d44056b0 2023-10-11T16:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d44056b0 0x55a5d45a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d44056b0 0x55a5d45a9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:57532/0 (socket says 172.21.15.111:57532) 2023-10-11T16:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecb7fe700 1 -- 172.21.15.111:0/2903845685 learned_addr learned my addr 172.21.15.111:0/2903845685 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecbfff700 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 0x55a5d45a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecaffd700 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45af720 0x55a5d45ba120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecb7fe700 1 -- 172.21.15.111:0/2903845685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45af720 msgr2=0x55a5d45ba120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45af720 0x55a5d45ba120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecb7fe700 1 -- 172.21.15.111:0/2903845685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 msgr2=0x55a5d45a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 0x55a5d45a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.529+0000 7faecb7fe700 1 -- 172.21.15.111:0/2903845685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5d44c0e80 con 0x55a5d44056b0 2023-10-11T16:36:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.530+0000 7faecb7fe700 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d44056b0 0x55a5d45a9560 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faebc010010 tx=0x7faebc005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16516ea11f26b615 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.530+0000 7faeb27fc700 1 -- 172.21.15.111:0/2903845685 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc00fd30 con 0x55a5d44056b0 2023-10-11T16:36:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.530+0000 7faeb27fc700 1 -- 172.21.15.111:0/2903845685 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faebc0050b0 con 0x55a5d44056b0 2023-10-11T16:36:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.530+0000 7faeb27fc700 1 -- 172.21.15.111:0/2903845685 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc0038e0 con 0x55a5d44056b0 2023-10-11T16:36:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 -- 172.21.15.111:0/2903845685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d44056b0 msgr2=0x55a5d45a9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d44056b0 0x55a5d45a9560 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faebc010010 tx=0x7faebc005040 comp rx=0 tx=0).stop 2023-10-11T16:36:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 -- 172.21.15.111:0/2903845685 shutdown_connections 2023-10-11T16:36:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d45afb90 0x55a5d45a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d44056b0 0x55a5d45a9560 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 --2- 172.21.15.111:0/2903845685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45af720 0x55a5d45ba120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 -- 172.21.15.111:0/2903845685 >> 172.21.15.111:0/2903845685 conn(0x55a5d459ef50 msgr2=0x55a5d45acea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 -- 172.21.15.111:0/2903845685 shutdown_connections 2023-10-11T16:36:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.531+0000 7faee55a6b80 1 -- 172.21.15.111:0/2903845685 wait complete. 2023-10-11T16:36:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.532+0000 7faee55a6b80 1 -- start start 2023-10-11T16:36:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.532+0000 7faee55a6b80 1 Processor -- start 2023-10-11T16:36:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.532+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45abce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45ac220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.539 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faee55a6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faee55a6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d45ac820 con 0x55a5d44056b0 2023-10-11T16:36:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faee55a6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d45ac990 con 0x55a5d45afb90 2023-10-11T16:36:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faee55a6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5d45ab1a0 con 0x55a5d45af720 2023-10-11T16:36:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faee55a6b80 10 client.0 _ll_register_callbacks cb 0x7faec4095c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faecb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faecb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45abce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53052/0 (socket says 172.21.15.111:53052) 2023-10-11T16:36:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faecb7fe700 1 -- 172.21.15.111:0/4048973118 learned_addr learned my addr 172.21.15.111:0/4048973118 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faecaffd700 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faecb7fe700 1 -- 172.21.15.111:0/4048973118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 msgr2=0x55a5d45ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faecb7fe700 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45ac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.533+0000 7faecb7fe700 1 -- 172.21.15.111:0/4048973118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 msgr2=0x55a5d45aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.534+0000 7faecb7fe700 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.534+0000 7faecb7fe700 1 -- 172.21.15.111:0/4048973118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5d437c5d0 con 0x55a5d44056b0 2023-10-11T16:36:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.534+0000 7faecaffd700 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.534+0000 7faecb7fe700 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45abce0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7faebc009880 tx=0x7faebc00fdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7ff74fac8e8fc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.534+0000 7faeaaffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:36:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.534+0000 7faeaaffd700 1 -- 172.21.15.111:0/4048973118 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc005220 con 0x55a5d44056b0 2023-10-11T16:36:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.535+0000 7faeaaffd700 1 -- 172.21.15.111:0/4048973118 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7faebc005380 con 0x55a5d44056b0 2023-10-11T16:36:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.535+0000 7faee55a6b80 1 -- 172.21.15.111:0/4048973118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faebc00faa0 con 0x55a5d44056b0 2023-10-11T16:36:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.535+0000 7faeaaffd700 1 -- 172.21.15.111:0/4048973118 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7faebc005560 con 0x55a5d44056b0 2023-10-11T16:36:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.535+0000 7faee55a6b80 10 client.16275 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.535+0000 7faee55a6b80 1 -- 172.21.15.111:0/4048973118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a5d4311c00 con 0x55a5d44056b0 2023-10-11T16:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.535+0000 7faeaaffd700 1 -- 172.21.15.111:0/4048973118 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faebc00faa0 con 0x55a5d44056b0 2023-10-11T16:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.536+0000 7faeaaffd700 1 -- 172.21.15.111:0/4048973118 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7faebc034070 con 0x55a5d44056b0 2023-10-11T16:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.536+0000 7faee55a6b80 10 client.16275 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.536+0000 7faee55a6b80 10 client.16275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.536+0000 7faee55a6b80 20 client.16275 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.536+0000 7faee55a6b80 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5d4617800 0x55a5d4619c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.536+0000 7faee55a6b80 4 client.16275 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:36:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.536+0000 7faee55a6b80 1 -- 172.21.15.111:0/4048973118 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a5d44c9600 con 0x55a5d4617800 2023-10-11T16:36:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.536+0000 7faecaffd700 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5d4617800 0x55a5d4619c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.537+0000 7faeaaffd700 10 client.16275 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:54.537+0000 7faecaffd700 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5d4617800 0x55a5d4619c10 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f92f5cbd465b34 server_cookie=d2d513bcfab4d462 in_seq=0 out_seq=0 2023-10-11T16:36:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.475+0000 7faeaaffd700 1 -- 172.21.15.111:0/4048973118 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a5d44c9600 con 0x55a5d4617800 2023-10-11T16:36:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.475+0000 7faeaaffd700 10 client.16275 handle_command_reply: tid=0 2023-10-11T16:36:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.475+0000 7faee55a6b80 1 client.16275 shutdown 2023-10-11T16:36:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 -- 172.21.15.111:0/4048973118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 msgr2=0x55a5d45abce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45abce0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7faebc009880 tx=0x7faebc00fdb0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 -- 172.21.15.111:0/4048973118 shutdown_connections 2023-10-11T16:36:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a5d4617800 0x55a5d4619c10 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a5d45afb90 0x55a5d45aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a5d44056b0 0x55a5d45abce0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 --2- 172.21.15.111:0/4048973118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a5d45af720 0x55a5d45ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 -- 172.21.15.111:0/4048973118 >> 172.21.15.111:0/4048973118 conn(0x55a5d459ef50 msgr2=0x55a5d45b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faeaaffd700 0 client.16275 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 -- 172.21.15.111:0/4048973118 shutdown_connections 2023-10-11T16:36:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.476+0000 7faee55a6b80 1 -- 172.21.15.111:0/4048973118 wait complete. 2023-10-11T16:36:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.477+0000 7faee55a6b80 20 client.16275 trim_cache size 0 max 16384 2023-10-11T16:36:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:36:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:36:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:36:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:36:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 -- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7faeb40a1340 msgr2=0x7faeb40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7faeb40a1340 0x7faeb40a37f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faec000b2e0 tx=0x7faec0005fd0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 -- 172.21.15.111:0/2573301494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc105560 msgr2=0x7faecc19ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc105560 0x7faecc19ad30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faec400efc0 tx=0x7faec400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.482 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 -- 172.21.15.111:0/2573301494 shutdown_connections 2023-10-11T16:36:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7faeb40a1340 0x7faeb40a37f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faecc105560 0x7faecc19ad30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faecc100c50 0x7faecc196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 --2- 172.21.15.111:0/2573301494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faecc1015f0 0x7faecc1968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 -- 172.21.15.111:0/2573301494 >> 172.21.15.111:0/2573301494 conn(0x7faecc077550 msgr2=0x7faecc0fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 -- 172.21.15.111:0/2573301494 shutdown_connections 2023-10-11T16:36:55.484 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:36:55.478+0000 7faed1bff700 1 -- 172.21.15.111:0/2573301494 wait complete. 2023-10-11T16:36:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:55 smithi150 ceph-mon[132217]: pgmap v131: 129 pgs: 129 active+clean; 453 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.4 MiB/s wr, 717 op/s 2023-10-11T16:36:56.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:55 smithi130 ceph-mon[131859]: pgmap v131: 129 pgs: 129 active+clean; 453 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.4 MiB/s wr, 717 op/s 2023-10-11T16:36:56.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:55 smithi111 ceph-mon[98527]: pgmap v131: 129 pgs: 129 active+clean; 453 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.4 MiB/s wr, 717 op/s 2023-10-11T16:36:57.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:57 smithi150 ceph-mon[132217]: pgmap v132: 129 pgs: 129 active+clean; 464 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.2 MiB/s wr, 802 op/s 2023-10-11T16:36:57.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:57 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:36:57.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:57 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:58.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:57 smithi130 ceph-mon[131859]: pgmap v132: 129 pgs: 129 active+clean; 464 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.2 MiB/s wr, 802 op/s 2023-10-11T16:36:58.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:57 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:36:58.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:57 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:58.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:57 smithi111 ceph-mon[98527]: pgmap v132: 129 pgs: 129 active+clean; 464 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.2 MiB/s wr, 802 op/s 2023-10-11T16:36:58.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:57 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:36:58.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:57 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:36:58.501 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:36:58.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 -- 172.21.15.111:0/1960513752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2838100a50 msgr2=0x7f2838100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 --2- 172.21.15.111:0/1960513752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2838100a50 0x7f2838100ec0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f282c0098f0 tx=0x7f282c009c00 comp rx=0 tx=0).stop 2023-10-11T16:36:58.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 -- 172.21.15.111:0/1960513752 shutdown_connections 2023-10-11T16:36:58.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 --2- 172.21.15.111:0/1960513752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838101400 0x7f283810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 --2- 172.21.15.111:0/1960513752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28381063f0 0x7f28381067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 --2- 172.21.15.111:0/1960513752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2838100a50 0x7f2838100ec0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 -- 172.21.15.111:0/1960513752 >> 172.21.15.111:0/1960513752 conn(0x7f2838077590 msgr2=0x7f2838077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:58.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 -- 172.21.15.111:0/1960513752 shutdown_connections 2023-10-11T16:36:58.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.823+0000 7f283d9ff700 1 -- 172.21.15.111:0/1960513752 wait complete. 2023-10-11T16:36:58.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.824+0000 7f283d9ff700 1 Processor -- start 2023-10-11T16:36:58.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f283d9ff700 1 -- start start 2023-10-11T16:36:58.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f283d9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838100a50 0x7f2838198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f283d9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101400 0x7f2838198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f283d9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28381063f0 0x7f283819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f283d9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283819d620 con 0x7f28381063f0 2023-10-11T16:36:58.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f283d9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283819d790 con 0x7f2838101400 2023-10-11T16:36:58.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f283d9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283819d950 con 0x7f2838100a50 2023-10-11T16:36:58.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f2836ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838100a50 0x7f2838198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f28377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28381063f0 0x7f283819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f2836ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838100a50 0x7f2838198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:53872/0 (socket says 172.21.15.111:53872) 2023-10-11T16:36:58.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f2836ffd700 1 -- 172.21.15.111:0/2010248543 learned_addr learned my addr 172.21.15.111:0/2010248543 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:58.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.825+0000 7f28367fc700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101400 0x7f2838198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f2836ffd700 1 -- 172.21.15.111:0/2010248543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101400 msgr2=0x7f2838198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f2836ffd700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101400 0x7f2838198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f2836ffd700 1 -- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28381063f0 msgr2=0x7f283819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f2836ffd700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28381063f0 0x7f283819cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f2836ffd700 1 -- 172.21.15.111:0/2010248543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c009520 con 0x7f2838100a50 2023-10-11T16:36:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f28377fe700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28381063f0 0x7f283819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f2836ffd700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838100a50 0x7f2838198640 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f282800e9c0 tx=0x7f282800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f283c9fd700 1 -- 172.21.15.111:0/2010248543 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2828004a30 con 0x7f2838100a50 2023-10-11T16:36:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f283c9fd700 1 -- 172.21.15.111:0/2010248543 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f282800cc80 con 0x7f2838100a50 2023-10-11T16:36:58.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f283d9ff700 1 -- 172.21.15.111:0/2010248543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2838110410 con 0x7f2838100a50 2023-10-11T16:36:58.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f283c9fd700 1 -- 172.21.15.111:0/2010248543 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28280054e0 con 0x7f2838100a50 2023-10-11T16:36:58.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.826+0000 7f283d9ff700 1 -- 172.21.15.111:0/2010248543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2838110280 con 0x7f2838100a50 2023-10-11T16:36:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.829+0000 7f283c9fd700 1 -- 172.21.15.111:0/2010248543 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2828004b90 con 0x7f2838100a50 2023-10-11T16:36:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.830+0000 7f283c9fd700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f28200a1340 0x7f28200a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f285138ab80 1 Processor -- start 2023-10-11T16:36:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f283c9fd700 1 -- 172.21.15.111:0/2010248543 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2828014070 con 0x7f2838100a50 2023-10-11T16:36:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f28367fc700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f28200a1340 0x7f28200a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f285138ab80 1 -- start start 2023-10-11T16:36:58.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f285138ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100b8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f285138ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100bd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f285138ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100bdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f285138ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0ffbe060 con 0x564a100b9470 2023-10-11T16:36:58.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f285138ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a100108a0 con 0x564a100b8b60 2023-10-11T16:36:58.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f285138ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a10009d30 con 0x564a100bd8b0 2023-10-11T16:36:58.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f2836ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100b8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f2836ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100b8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46892/0 (socket says 172.21.15.111:46892) 2023-10-11T16:36:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.831+0000 7f2836ffd700 1 -- 172.21.15.111:0/454390312 learned_addr learned my addr 172.21.15.111:0/454390312 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28377fe700 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100bdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28367fc700 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100bd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28367fc700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f28200a1340 0x7f28200a37f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f282c005e90 tx=0x7f282c019040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28377fe700 1 -- 172.21.15.111:0/454390312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 msgr2=0x564a100b8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28377fe700 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100b8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28377fe700 1 -- 172.21.15.111:0/454390312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 msgr2=0x564a100bd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28377fe700 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100bd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28377fe700 1 -- 172.21.15.111:0/454390312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a0fe8b630 con 0x564a100bd8b0 2023-10-11T16:36:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f2836ffd700 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100b8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f28377fe700 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100bdd00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2830003db0 tx=0x7f2830009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c3e68dd19b50dbf server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f281dffb700 1 -- 172.21.15.111:0/454390312 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830004bb0 con 0x564a100bd8b0 2023-10-11T16:36:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f281dffb700 1 -- 172.21.15.111:0/454390312 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f283000f040 con 0x564a100bd8b0 2023-10-11T16:36:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.832+0000 7f281dffb700 1 -- 172.21.15.111:0/454390312 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830007c50 con 0x564a100bd8b0 2023-10-11T16:36:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 -- 172.21.15.111:0/454390312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 msgr2=0x564a100bdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100bdd00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2830003db0 tx=0x7f2830009920 comp rx=0 tx=0).stop 2023-10-11T16:36:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 -- 172.21.15.111:0/454390312 shutdown_connections 2023-10-11T16:36:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100bdd00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100b8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 --2- 172.21.15.111:0/454390312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 -- 172.21.15.111:0/454390312 >> 172.21.15.111:0/454390312 conn(0x564a100ae360 msgr2=0x564a100b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:58.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 -- 172.21.15.111:0/454390312 shutdown_connections 2023-10-11T16:36:58.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.833+0000 7f285138ab80 1 -- 172.21.15.111:0/454390312 wait complete. 2023-10-11T16:36:58.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.834+0000 7f285138ab80 1 -- start start 2023-10-11T16:36:58.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.834+0000 7f285138ab80 1 Processor -- start 2023-10-11T16:36:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.834+0000 7f285138ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100c5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.834+0000 7f285138ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100c2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.834+0000 7f285138ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.834+0000 7f285138ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0ffab8e0 con 0x564a100b9470 2023-10-11T16:36:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.834+0000 7f285138ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0ffab580 con 0x564a100b8b60 2023-10-11T16:36:58.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.834+0000 7f285138ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0fe8d920 con 0x564a100bd8b0 2023-10-11T16:36:58.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f285138ab80 10 client.0 _ll_register_callbacks cb 0x564a0ffc9780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:58.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100c2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39434/0 (socket says 172.21.15.111:39434) 2023-10-11T16:36:58.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 -- 172.21.15.111:0/1658613981 learned_addr learned my addr 172.21.15.111:0/1658613981 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28377fe700 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f2836ffd700 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 -- 172.21.15.111:0/1658613981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 msgr2=0x564a100c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 -- 172.21.15.111:0/1658613981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 msgr2=0x564a100c5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100c5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 -- 172.21.15.111:0/1658613981 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2830003a60 con 0x564a100b9470 2023-10-11T16:36:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f28367fc700 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100c2350 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f282c000c00 tx=0x7f282c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a4afe1259e0646 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.835+0000 7f2816ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:36:58.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f2816ffd700 1 -- 172.21.15.111:0/1658613981 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c02a050 con 0x564a100b9470 2023-10-11T16:36:58.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f285138ab80 1 -- 172.21.15.111:0/1658613981 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564a100abb20 con 0x564a100b9470 2023-10-11T16:36:58.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f2816ffd700 1 -- 172.21.15.111:0/1658613981 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f282c005240 con 0x564a100b9470 2023-10-11T16:36:58.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f2816ffd700 1 -- 172.21.15.111:0/1658613981 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f282c005400 con 0x564a100b9470 2023-10-11T16:36:58.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f285138ab80 10 client.16287 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:58.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f285138ab80 1 -- 172.21.15.111:0/1658613981 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564a0ffd0190 con 0x564a100b9470 2023-10-11T16:36:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f2816ffd700 1 -- 172.21.15.111:0/1658613981 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564a100abb20 con 0x564a100b9470 2023-10-11T16:36:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f2816ffd700 1 -- 172.21.15.111:0/1658613981 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f282c037070 con 0x564a100b9470 2023-10-11T16:36:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f285138ab80 10 client.16287 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f285138ab80 10 client.16287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:58.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f285138ab80 20 client.16287 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.836+0000 7f285138ab80 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564a100d2870 0x564a100d4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.837+0000 7f285138ab80 4 client.16287 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:36:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.837+0000 7f285138ab80 1 -- 172.21.15.111:0/1658613981 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a0ffcc0a0 con 0x564a100d2870 2023-10-11T16:36:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.837+0000 7f2836ffd700 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564a100d2870 0x564a100d4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.837+0000 7f2816ffd700 10 client.16287 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:58.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.837+0000 7f2836ffd700 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564a100d2870 0x564a100d4c80 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c034f833aa556f08 server_cookie=1ee06f99c5fc71d7 in_seq=0 out_seq=0 2023-10-11T16:36:58.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f2816ffd700 1 -- 172.21.15.111:0/1658613981 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564a0ffcc0a0 con 0x564a100d2870 2023-10-11T16:36:58.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f2816ffd700 10 client.16287 handle_command_reply: tid=0 2023-10-11T16:36:58.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 client.16287 shutdown 2023-10-11T16:36:58.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 -- 172.21.15.111:0/1658613981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 msgr2=0x564a100c2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100c2350 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f282c000c00 tx=0x7f282c009c00 comp rx=0 tx=0).stop 2023-10-11T16:36:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 -- 172.21.15.111:0/1658613981 shutdown_connections 2023-10-11T16:36:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564a100d2870 0x564a100d4c80 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b8b60 0x564a100c5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 --2- 172.21.15.111:0/1658613981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b9470 0x564a100c2350 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f2816ffd700 0 client.16287 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 -- 172.21.15.111:0/1658613981 >> 172.21.15.111:0/1658613981 conn(0x564a100ae360 msgr2=0x564a100b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 -- 172.21.15.111:0/1658613981 shutdown_connections 2023-10-11T16:36:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 1 -- 172.21.15.111:0/1658613981 wait complete. 2023-10-11T16:36:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.838+0000 7f285138ab80 20 client.16287 trim_cache size 0 max 16384 2023-10-11T16:36:58.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.866+0000 7f285138ab80 1 Processor -- start 2023-10-11T16:36:58.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.866+0000 7f285138ab80 1 -- start start 2023-10-11T16:36:58.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.866+0000 7f285138ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.866+0000 7f285138ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b9470 0x564a100bcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.867+0000 7f285138ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.867+0000 7f285138ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a100bc480 con 0x564a100b8b60 2023-10-11T16:36:58.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.867+0000 7f285138ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a100c37e0 con 0x564a100b9470 2023-10-11T16:36:58.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.867+0000 7f285138ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a100c9ee0 con 0x564a100bd8b0 2023-10-11T16:36:58.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.867+0000 7f28367fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b9470 0x564a100bcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.867+0000 7f28367fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b9470 0x564a100bcfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:46910/0 (socket says 172.21.15.111:46910) 2023-10-11T16:36:58.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.867+0000 7f28367fc700 1 -- 172.21.15.111:0/4293852914 learned_addr learned my addr 172.21.15.111:0/4293852914 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:58.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.867+0000 7f2836ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.868+0000 7f28377fe700 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.868+0000 7f28367fc700 1 -- 172.21.15.111:0/4293852914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 msgr2=0x564a100c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:58.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.868+0000 7f28367fc700 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.868+0000 7f28367fc700 1 -- 172.21.15.111:0/4293852914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 msgr2=0x564a100c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:58.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.868+0000 7f28367fc700 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.868+0000 7f28367fc700 1 -- 172.21.15.111:0/4293852914 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a0ffcfee0 con 0x564a100b9470 2023-10-11T16:36:58.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.868+0000 7f2836ffd700 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:58.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.868+0000 7f28367fc700 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b9470 0x564a100bcfa0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f282c006010 tx=0x7f282c009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2041c416fa5ad663 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:58.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.869+0000 7f281e7fc700 1 -- 172.21.15.111:0/4293852914 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c02a070 con 0x564a100b9470 2023-10-11T16:36:58.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.869+0000 7f281e7fc700 1 -- 172.21.15.111:0/4293852914 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f282c02b040 con 0x564a100b9470 2023-10-11T16:36:58.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.869+0000 7f281e7fc700 1 -- 172.21.15.111:0/4293852914 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c005720 con 0x564a100b9470 2023-10-11T16:36:58.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.870+0000 7f285138ab80 1 -- 172.21.15.111:0/4293852914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b9470 msgr2=0x564a100bcfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.870+0000 7f285138ab80 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b9470 0x564a100bcfa0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f282c006010 tx=0x7f282c009230 comp rx=0 tx=0).stop 2023-10-11T16:36:58.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.870+0000 7f285138ab80 1 -- 172.21.15.111:0/4293852914 shutdown_connections 2023-10-11T16:36:58.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.870+0000 7f285138ab80 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100bd8b0 0x564a100c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.870+0000 7f285138ab80 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.870+0000 7f285138ab80 1 --2- 172.21.15.111:0/4293852914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100b9470 0x564a100bcfa0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.870+0000 7f285138ab80 1 -- 172.21.15.111:0/4293852914 >> 172.21.15.111:0/4293852914 conn(0x564a100ae360 msgr2=0x564a100bc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:58.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.871+0000 7f285138ab80 1 -- 172.21.15.111:0/4293852914 shutdown_connections 2023-10-11T16:36:58.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.871+0000 7f285138ab80 1 -- 172.21.15.111:0/4293852914 wait complete. 2023-10-11T16:36:58.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.871+0000 7f285138ab80 1 -- start start 2023-10-11T16:36:58.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.872+0000 7f285138ab80 1 Processor -- start 2023-10-11T16:36:58.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.872+0000 7f285138ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.872+0000 7f285138ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100b9470 0x564a100c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.872+0000 7f285138ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100bd8b0 0x564a100c0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.873+0000 7f285138ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a100c1010 con 0x564a100b8b60 2023-10-11T16:36:58.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.873+0000 7f285138ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a100c1180 con 0x564a100bd8b0 2023-10-11T16:36:58.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.873+0000 7f285138ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a100c1340 con 0x564a100b9470 2023-10-11T16:36:58.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.873+0000 7f285138ab80 10 client.0 _ll_register_callbacks cb 0x7f282c00b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:36:58.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.873+0000 7f28367fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100b9470 0x564a100c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.873+0000 7f28367fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100b9470 0x564a100c2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:53920/0 (socket says 172.21.15.111:53920) 2023-10-11T16:36:58.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.873+0000 7f28367fc700 1 -- 172.21.15.111:0/2736401692 learned_addr learned my addr 172.21.15.111:0/2736401692 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:36:58.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.874+0000 7f2836ffd700 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.874+0000 7f28377fe700 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100bd8b0 0x564a100c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.877+0000 7f28377fe700 1 -- 172.21.15.111:0/2736401692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100b9470 msgr2=0x564a100c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:58.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.877+0000 7f28377fe700 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100b9470 0x564a100c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.877+0000 7f28377fe700 1 -- 172.21.15.111:0/2736401692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 msgr2=0x564a100c1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:36:58.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.877+0000 7f28377fe700 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.877+0000 7f28377fe700 1 -- 172.21.15.111:0/2736401692 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2838110410 con 0x564a100bd8b0 2023-10-11T16:36:58.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.877+0000 7f2836ffd700 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:36:58.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.878+0000 7f28377fe700 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100bd8b0 0x564a100c0900 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2830003d80 tx=0x7f2830004c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b96629f9df05dba1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:36:58.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.879+0000 7f2816ffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:36:58.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.882+0000 7f2816ffd700 1 -- 172.21.15.111:0/2736401692 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830007710 con 0x564a100bd8b0 2023-10-11T16:36:58.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.882+0000 7f2816ffd700 1 -- 172.21.15.111:0/2736401692 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2830007870 con 0x564a100bd8b0 2023-10-11T16:36:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.883+0000 7f285138ab80 1 -- 172.21.15.111:0/2736401692 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f283000f040 con 0x564a100bd8b0 2023-10-11T16:36:58.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.884+0000 7f2816ffd700 1 -- 172.21.15.111:0/2736401692 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2830007a30 con 0x564a100bd8b0 2023-10-11T16:36:58.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.885+0000 7f285138ab80 10 client.26384 fetch_fsmap learned FSMap version 33 2023-10-11T16:36:58.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.886+0000 7f285138ab80 1 -- 172.21.15.111:0/2736401692 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564a0ffd0190 con 0x564a100bd8b0 2023-10-11T16:36:58.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f2816ffd700 1 -- 172.21.15.111:0/2736401692 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2830007b90 con 0x564a100bd8b0 2023-10-11T16:36:58.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f2816ffd700 1 -- 172.21.15.111:0/2736401692 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f2830031470 con 0x564a100bd8b0 2023-10-11T16:36:58.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f285138ab80 10 client.26384 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:36:58.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f285138ab80 10 client.26384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:36:58.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f285138ab80 20 client.26384 populate_metadata read hostname 'smithi111' 2023-10-11T16:36:58.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f285138ab80 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564a10140fa0 0x564a101433b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:36:58.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f285138ab80 4 client.26384 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:36:58.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f285138ab80 1 -- 172.21.15.111:0/2736401692 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f283000f040 con 0x564a10140fa0 2023-10-11T16:36:58.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.887+0000 7f2836ffd700 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564a10140fa0 0x564a101433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:36:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.888+0000 7f2836ffd700 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564a10140fa0 0x564a101433b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a19894b3cbc5d7 server_cookie=86008a3782e4af8a in_seq=0 out_seq=0 2023-10-11T16:36:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.888+0000 7f2816ffd700 10 client.26384 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.889+0000 7f2816ffd700 1 -- 172.21.15.111:0/2736401692 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f283000f040 con 0x564a10140fa0 2023-10-11T16:36:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.889+0000 7f2816ffd700 10 client.26384 handle_command_reply: tid=0 2023-10-11T16:36:58.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.889+0000 7f285138ab80 1 client.26384 shutdown 2023-10-11T16:36:58.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.892+0000 7f285138ab80 1 -- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100bd8b0 msgr2=0x564a100c0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.892+0000 7f285138ab80 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100bd8b0 0x564a100c0900 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2830003d80 tx=0x7f2830004c20 comp rx=0 tx=0).stop 2023-10-11T16:36:58.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.892+0000 7f285138ab80 1 -- 172.21.15.111:0/2736401692 shutdown_connections 2023-10-11T16:36:58.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.892+0000 7f285138ab80 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564a10140fa0 0x564a101433b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.893+0000 7f285138ab80 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564a100bd8b0 0x564a100c0900 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.893+0000 7f285138ab80 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564a100b8b60 0x564a100c1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.893+0000 7f285138ab80 1 --2- 172.21.15.111:0/2736401692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564a100b9470 0x564a100c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.893+0000 7f285138ab80 1 -- 172.21.15.111:0/2736401692 >> 172.21.15.111:0/2736401692 conn(0x564a100ae360 msgr2=0x564a100bc890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:58.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.893+0000 7f2816ffd700 0 client.26384 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:36:58.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.894+0000 7f285138ab80 1 -- 172.21.15.111:0/2736401692 shutdown_connections 2023-10-11T16:36:58.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.894+0000 7f285138ab80 1 -- 172.21.15.111:0/2736401692 wait complete. 2023-10-11T16:36:58.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.894+0000 7f285138ab80 20 client.26384 trim_cache size 0 max 16384 2023-10-11T16:36:58.896 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:36:58.897 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:36:58.897 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:36:58.897 INFO:teuthology.orchestra.run.smithi111.stdout: "e4b353b6-bcd8-4549-a24b-3247f2c5740c": { 2023-10-11T16:36:58.897 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:36:58.897 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "e4b353b6-bcd8-4549-a24b-3247f2c5740c", 2023-10-11T16:36:58.898 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:36:58.898 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:36:58.898 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:36:58.898 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:36:58.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.898+0000 7f283d9ff700 1 -- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f28200a1340 msgr2=0x7f28200a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.898+0000 7f283d9ff700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f28200a1340 0x7f28200a37f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f282c005e90 tx=0x7f282c019040 comp rx=0 tx=0).stop 2023-10-11T16:36:58.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.898+0000 7f283d9ff700 1 -- 172.21.15.111:0/2010248543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838100a50 msgr2=0x7f2838198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:36:58.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.898+0000 7f283d9ff700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838100a50 0x7f2838198640 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f282800e9c0 tx=0x7f282800ecd0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.899+0000 7f283d9ff700 1 -- 172.21.15.111:0/2010248543 shutdown_connections 2023-10-11T16:36:58.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.900+0000 7f283d9ff700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f28200a1340 0x7f28200a37f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.900+0000 7f283d9ff700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28381063f0 0x7f283819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.900+0000 7f283d9ff700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2838100a50 0x7f2838198640 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.900+0000 7f283d9ff700 1 --2- 172.21.15.111:0/2010248543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2838101400 0x7f2838198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:36:58.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.900+0000 7f283d9ff700 1 -- 172.21.15.111:0/2010248543 >> 172.21.15.111:0/2010248543 conn(0x7f2838077590 msgr2=0x7f283810b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:36:58.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.900+0000 7f283d9ff700 1 -- 172.21.15.111:0/2010248543 shutdown_connections 2023-10-11T16:36:58.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:36:58.900+0000 7f283d9ff700 1 -- 172.21.15.111:0/2010248543 wait complete. 2023-10-11T16:36:58.947 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b353b6-bcd8-4549-a24b-3247f2c5740c - {'path': '/', 'tag': 'e4b353b6-bcd8-4549-a24b-3247f2c5740c', 'options': 'recursive,force'} 2023-10-11T16:36:59.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:36:59 smithi150 ceph-mon[132217]: pgmap v133: 129 pgs: 129 active+clean; 464 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 431 op/s 2023-10-11T16:37:00.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:36:59 smithi130 ceph-mon[131859]: pgmap v133: 129 pgs: 129 active+clean; 464 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 431 op/s 2023-10-11T16:37:00.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:36:59 smithi111 ceph-mon[98527]: pgmap v133: 129 pgs: 129 active+clean; 464 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 431 op/s 2023-10-11T16:37:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:01 smithi150 ceph-mon[132217]: pgmap v134: 129 pgs: 129 active+clean; 468 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.0 MiB/s wr, 864 op/s 2023-10-11T16:37:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:01 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:37:01.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:01 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:01 smithi130 ceph-mon[131859]: pgmap v134: 129 pgs: 129 active+clean; 468 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.0 MiB/s wr, 864 op/s 2023-10-11T16:37:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:01 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:37:02.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:01 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:02.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:01 smithi111 ceph-mon[98527]: pgmap v134: 129 pgs: 129 active+clean; 468 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.0 MiB/s wr, 864 op/s 2023-10-11T16:37:02.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:01 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:37:02.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:01 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:02.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:02 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:37:02.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:02 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:03.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:02 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:37:03.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:02 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:03.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:02 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:37:03.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:02 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:03 smithi150 ceph-mon[132217]: pgmap v135: 129 pgs: 129 active+clean; 468 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 653 op/s 2023-10-11T16:37:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:03 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:37:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:03 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:03 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:37:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:03 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:03 smithi130 ceph-mon[131859]: pgmap v135: 129 pgs: 129 active+clean; 468 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 653 op/s 2023-10-11T16:37:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:03 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:37:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:03 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:04.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:03 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:37:04.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:03 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:04.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:03 smithi111 ceph-mon[98527]: pgmap v135: 129 pgs: 129 active+clean; 468 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 653 op/s 2023-10-11T16:37:04.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:03 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:37:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:03 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:03 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:37:04.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:03 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:04.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:04 smithi150 ceph-mon[132217]: pgmap v136: 129 pgs: 129 active+clean; 472 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 685 op/s 2023-10-11T16:37:05.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:04 smithi130 ceph-mon[131859]: pgmap v136: 129 pgs: 129 active+clean; 472 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 685 op/s 2023-10-11T16:37:05.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:04 smithi111 ceph-mon[98527]: pgmap v136: 129 pgs: 129 active+clean; 472 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 685 op/s 2023-10-11T16:37:06.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:05 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:37:06.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:05 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:37:06.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:05 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:37:07.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:06 smithi130 ceph-mon[131859]: pgmap v137: 129 pgs: 129 active+clean; 491 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.0 MiB/s wr, 795 op/s 2023-10-11T16:37:07.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:06 smithi111 ceph-mon[98527]: pgmap v137: 129 pgs: 129 active+clean; 491 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.0 MiB/s wr, 795 op/s 2023-10-11T16:37:07.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:06 smithi150 ceph-mon[132217]: pgmap v137: 129 pgs: 129 active+clean; 491 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.0 MiB/s wr, 795 op/s 2023-10-11T16:37:08.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:07 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:37:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:07 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:37:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:07 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:37:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:07 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:37:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:07 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:37:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:07 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:37:08.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:07 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:37:08.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:07 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:37:08.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:07 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:37:08.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:07 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:37:08.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:07 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:37:08.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:07 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:37:08.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:07 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:37:08.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:07 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:37:08.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:07 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:37:08.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:07 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:37:08.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:07 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:37:08.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:07 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:37:08.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:07 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:37:08.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:07 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:37:08.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:07 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:37:08.690 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.689+0000 7fa1965ff700 1 -- 172.21.15.111:0/3081380914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ff230 msgr2=0x7fa1900ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:08.690 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.689+0000 7fa1965ff700 1 --2- 172.21.15.111:0/3081380914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ff230 0x7fa1900ff600 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa180009990 tx=0x7fa180009ca0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.690+0000 7fa1965ff700 1 -- 172.21.15.111:0/3081380914 shutdown_connections 2023-10-11T16:37:08.691 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.690+0000 7fa1965ff700 1 --2- 172.21.15.111:0/3081380914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa190104630 0x7fa190104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.690+0000 7fa1965ff700 1 --2- 172.21.15.111:0/3081380914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ff230 0x7fa1900ff600 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.690+0000 7fa1965ff700 1 --2- 172.21.15.111:0/3081380914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1900ffb40 0x7fa190104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.690+0000 7fa1965ff700 1 -- 172.21.15.111:0/3081380914 >> 172.21.15.111:0/3081380914 conn(0x7fa1900763a0 msgr2=0x7fa1900767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:08.692 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.690+0000 7fa1965ff700 1 -- 172.21.15.111:0/3081380914 shutdown_connections 2023-10-11T16:37:08.693 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.690+0000 7fa1965ff700 1 -- 172.21.15.111:0/3081380914 wait complete. 2023-10-11T16:37:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.691+0000 7fa1965ff700 1 Processor -- start 2023-10-11T16:37:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.691+0000 7fa1965ff700 1 -- start start 2023-10-11T16:37:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa1965ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1900ff230 0x7fa19019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.695 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa1965ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ffb40 0x7fa19019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa1965ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa190104630 0x7fa1901a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa1965ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901a1910 con 0x7fa1900ff230 2023-10-11T16:37:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa1965ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901a1a80 con 0x7fa1900ffb40 2023-10-11T16:37:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa1965ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901a1c40 con 0x7fa190104630 2023-10-11T16:37:08.696 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ffb40 0x7fa19019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1900ff230 0x7fa19019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ffb40 0x7fa19019ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:41596/0 (socket says 172.21.15.111:41596) 2023-10-11T16:37:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1900ff230 0x7fa19019c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56098/0 (socket says 172.21.15.111:56098) 2023-10-11T16:37:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18f7fe700 1 -- 172.21.15.111:0/84522213 learned_addr learned my addr 172.21.15.111:0/84522213 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa194b9c700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa190104630 0x7fa1901a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18f7fe700 1 -- 172.21.15.111:0/84522213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa190104630 msgr2=0x7fa1901a1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa190104630 0x7fa1901a1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18f7fe700 1 -- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1900ff230 msgr2=0x7fa19019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.692+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1900ff230 0x7fa19019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.693+0000 7fa18f7fe700 1 -- 172.21.15.111:0/84522213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa184009520 con 0x7fa1900ffb40 2023-10-11T16:37:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.693+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ffb40 0x7fa19019ceb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa18400eb30 tx=0x7fa18400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.693+0000 7fa18d7fa700 1 -- 172.21.15.111:0/84522213 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa184009880 con 0x7fa1900ffb40 2023-10-11T16:37:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.693+0000 7fa18d7fa700 1 -- 172.21.15.111:0/84522213 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1840044e0 con 0x7fa1900ffb40 2023-10-11T16:37:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.693+0000 7fa18d7fa700 1 -- 172.21.15.111:0/84522213 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa184003d50 con 0x7fa1900ffb40 2023-10-11T16:37:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.693+0000 7fa1965ff700 1 -- 172.21.15.111:0/84522213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1800095f0 con 0x7fa1900ffb40 2023-10-11T16:37:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.693+0000 7fa1965ff700 1 -- 172.21.15.111:0/84522213 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1901168c0 con 0x7fa1900ffb40 2023-10-11T16:37:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.698+0000 7fa18d7fa700 1 -- 172.21.15.111:0/84522213 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fa184003ef0 con 0x7fa1900ffb40 2023-10-11T16:37:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.699+0000 7fa1a9fcdb80 1 Processor -- start 2023-10-11T16:37:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.701+0000 7fa18d7fa700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1780a1340 0x7fa1780a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.701+0000 7fa18d7fa700 1 -- 172.21.15.111:0/84522213 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa184014070 con 0x7fa1900ffb40 2023-10-11T16:37:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa1a9fcdb80 1 -- start start 2023-10-11T16:37:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa194b9c700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1780a1340 0x7fa1780a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f661e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f60560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66b90 0x560ec3f60b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3e66000 con 0x560ec3dbc6b0 2023-10-11T16:37:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3eb8840 con 0x560ec3f66b90 2023-10-11T16:37:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3eb1cd0 con 0x560ec3f66720 2023-10-11T16:37:08.705 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f60560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f60560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:49176/0 (socket says 172.21.15.111:49176) 2023-10-11T16:37:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.702+0000 7fa18f7fe700 1 -- 172.21.15.111:0/2343956244 learned_addr learned my addr 172.21.15.111:0/2343956244 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:08.706 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa194b9c700 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66b90 0x560ec3f60b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa18f7fe700 1 -- 172.21.15.111:0/2343956244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66b90 msgr2=0x560ec3f60b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66b90 0x560ec3f60b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa18f7fe700 1 -- 172.21.15.111:0/2343956244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 msgr2=0x560ec3f661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa18f7fe700 1 -- 172.21.15.111:0/2343956244 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ec3d335d0 con 0x560ec3f66720 2023-10-11T16:37:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa18ffff700 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:37:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f60560 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa1840a2190 tx=0x7fa18409a170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bafd63593718515e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa1767fc700 1 -- 172.21.15.111:0/2343956244 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18409a270 con 0x560ec3f66720 2023-10-11T16:37:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa1767fc700 1 -- 172.21.15.111:0/2343956244 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa18409e2a0 con 0x560ec3f66720 2023-10-11T16:37:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.703+0000 7fa1767fc700 1 -- 172.21.15.111:0/2343956244 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1840cadb0 con 0x560ec3f66720 2023-10-11T16:37:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.704+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/2343956244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 msgr2=0x560ec3f60560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.704+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f60560 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa1840a2190 tx=0x7fa18409a170 comp rx=0 tx=0).stop 2023-10-11T16:37:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.704+0000 7fa194b9c700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1780a1340 0x7fa1780a37f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa18800f880 tx=0x7fa18800fc40 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.704+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/2343956244 shutdown_connections 2023-10-11T16:37:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.704+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66b90 0x560ec3f60b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.704+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.704+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/2343956244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f60560 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.704+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/2343956244 >> 172.21.15.111:0/2343956244 conn(0x560ec3f55f50 msgr2=0x560ec3f5b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.705+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/2343956244 shutdown_connections 2023-10-11T16:37:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.705+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/2343956244 wait complete. 2023-10-11T16:37:08.714 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.705+0000 7fa1a9fcdb80 1 -- start start 2023-10-11T16:37:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.706+0000 7fa1a9fcdb80 1 Processor -- start 2023-10-11T16:37:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.706+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3dbc6b0 0x560ec3f6fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.706+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f70010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.706+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f69d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.706+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3e53880 con 0x560ec3f66b90 2023-10-11T16:37:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.706+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3e53520 con 0x560ec3f66720 2023-10-11T16:37:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.706+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3d358c0 con 0x560ec3dbc6b0 2023-10-11T16:37:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.706+0000 7fa1a9fcdb80 10 client.0 _ll_register_callbacks cb 0x560ec3e71720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3dbc6b0 0x560ec3f6fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3dbc6b0 0x560ec3f6fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:49182/0 (socket says 172.21.15.111:49182) 2023-10-11T16:37:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 -- 172.21.15.111:0/912783599 learned_addr learned my addr 172.21.15.111:0/912783599 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f70010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 -- 172.21.15.111:0/912783599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 msgr2=0x560ec3f70010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f70010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 -- 172.21.15.111:0/912783599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 msgr2=0x560ec3f69d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa194b9c700 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f69d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f69d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 -- 172.21.15.111:0/912783599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1901168c0 con 0x560ec3dbc6b0 2023-10-11T16:37:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa194b9c700 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f69d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:37:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa18ffff700 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3dbc6b0 0x560ec3f6fad0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa18000bf60 tx=0x7fa18000bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e8a13d9dc472d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.707+0000 7fa16f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:37:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.708+0000 7fa16f7fe700 1 -- 172.21.15.111:0/912783599 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18001d070 con 0x560ec3dbc6b0 2023-10-11T16:37:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.708+0000 7fa16f7fe700 1 -- 172.21.15.111:0/912783599 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa180004700 con 0x560ec3dbc6b0 2023-10-11T16:37:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.708+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/912783599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ec3f53710 con 0x560ec3dbc6b0 2023-10-11T16:37:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.708+0000 7fa16f7fe700 1 -- 172.21.15.111:0/912783599 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1800056e0 con 0x560ec3dbc6b0 2023-10-11T16:37:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.708+0000 7fa1a9fcdb80 10 client.26428 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.708+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/912783599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560ec3e78130 con 0x560ec3dbc6b0 2023-10-11T16:37:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.708+0000 7fa16f7fe700 1 -- 172.21.15.111:0/912783599 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560ec3f53710 con 0x560ec3dbc6b0 2023-10-11T16:37:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa16f7fe700 1 -- 172.21.15.111:0/912783599 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa18000f8d0 con 0x560ec3dbc6b0 2023-10-11T16:37:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa1a9fcdb80 10 client.26428 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa1a9fcdb80 10 client.26428 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa1a9fcdb80 20 client.26428 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560ec3f79f90 0x560ec3f7c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa1a9fcdb80 4 client.26428 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:37:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/912783599 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ec3e74040 con 0x560ec3f79f90 2023-10-11T16:37:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560ec3f79f90 0x560ec3f7c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560ec3f79f90 0x560ec3f7c3a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53b5b142ac3ba12f server_cookie=48c2816710726e01 in_seq=0 out_seq=0 2023-10-11T16:37:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.709+0000 7fa16f7fe700 10 client.26428 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa16f7fe700 1 -- 172.21.15.111:0/912783599 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560ec3e74040 con 0x560ec3f79f90 2023-10-11T16:37:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa16f7fe700 10 client.26428 handle_command_reply: tid=0 2023-10-11T16:37:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 client.26428 shutdown 2023-10-11T16:37:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/912783599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3dbc6b0 msgr2=0x560ec3f6fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3dbc6b0 0x560ec3f6fad0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa18000bf60 tx=0x7fa18000bf90 comp rx=0 tx=0).stop 2023-10-11T16:37:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/912783599 shutdown_connections 2023-10-11T16:37:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560ec3f79f90 0x560ec3f7c3a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f69d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3dbc6b0 0x560ec3f6fad0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/912783599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f70010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/912783599 >> 172.21.15.111:0/912783599 conn(0x560ec3f55f50 msgr2=0x560ec3f5a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa16f7fe700 0 client.26428 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/912783599 shutdown_connections 2023-10-11T16:37:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.711+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/912783599 wait complete. 2023-10-11T16:37:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.712+0000 7fa1a9fcdb80 20 client.26428 trim_cache size 0 max 16384 2023-10-11T16:37:08.744 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.744+0000 7fa1a9fcdb80 1 Processor -- start 2023-10-11T16:37:08.745 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.744+0000 7fa1a9fcdb80 1 -- start start 2023-10-11T16:37:08.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.744+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f60560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.744+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f60aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.744+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66b90 0x560ec3f694e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.744+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3f60fe0 con 0x560ec3dbc6b0 2023-10-11T16:37:08.746 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.745+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3f68280 con 0x560ec3f66720 2023-10-11T16:37:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.745+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3f649b0 con 0x560ec3f66b90 2023-10-11T16:37:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.745+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f60560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.745+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f60560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56136/0 (socket says 172.21.15.111:56136) 2023-10-11T16:37:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.745+0000 7fa18ffff700 1 -- 172.21.15.111:0/1419769492 learned_addr learned my addr 172.21.15.111:0/1419769492 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:08.747 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.745+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f60aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa18ffff700 1 -- 172.21.15.111:0/1419769492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66b90 msgr2=0x560ec3f694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa18ffff700 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66b90 0x560ec3f694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa18ffff700 1 -- 172.21.15.111:0/1419769492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 msgr2=0x560ec3f60aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:08.749 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa18ffff700 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f60aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa18ffff700 1 -- 172.21.15.111:0/1419769492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ec3e77e80 con 0x560ec3dbc6b0 2023-10-11T16:37:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f60aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:37:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa18ffff700 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f60560 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa180024040 tx=0x7fa18000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d4287f6811dcd server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:08.750 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa176ffd700 1 -- 172.21.15.111:0/1419769492 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18001d030 con 0x560ec3dbc6b0 2023-10-11T16:37:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa176ffd700 1 -- 172.21.15.111:0/1419769492 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa180038070 con 0x560ec3dbc6b0 2023-10-11T16:37:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.746+0000 7fa176ffd700 1 -- 172.21.15.111:0/1419769492 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180016a80 con 0x560ec3dbc6b0 2023-10-11T16:37:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.747+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/1419769492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 msgr2=0x560ec3f60560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.747+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f60560 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa180024040 tx=0x7fa18000b390 comp rx=0 tx=0).stop 2023-10-11T16:37:08.751 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.747+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/1419769492 shutdown_connections 2023-10-11T16:37:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.747+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66b90 0x560ec3f694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.747+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3dbc6b0 0x560ec3f60560 secure :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa180024040 tx=0x7fa18000b390 comp rx=0 tx=0).stop 2023-10-11T16:37:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.747+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/1419769492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f66720 0x560ec3f60aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.752 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.747+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/1419769492 >> 172.21.15.111:0/1419769492 conn(0x560ec3f55f50 msgr2=0x560ec3f6a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.747+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/1419769492 shutdown_connections 2023-10-11T16:37:08.753 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.748+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/1419769492 wait complete. 2023-10-11T16:37:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.748+0000 7fa1a9fcdb80 1 -- start start 2023-10-11T16:37:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.748+0000 7fa1a9fcdb80 1 Processor -- start 2023-10-11T16:37:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f6d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f6db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.756 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa1a9fcdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f6e0c0 0x560ec3fcce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3fcd350 con 0x560ec3f66b90 2023-10-11T16:37:08.757 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3fcd490 con 0x560ec3f6e0c0 2023-10-11T16:37:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa1a9fcdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ec3fcd5d0 con 0x560ec3f66720 2023-10-11T16:37:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa1a9fcdb80 10 client.0 _ll_register_callbacks cb 0x7fa180009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:08.758 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f6d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f6d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:49196/0 (socket says 172.21.15.111:49196) 2023-10-11T16:37:08.759 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa18ffff700 1 -- 172.21.15.111:0/3904802736 learned_addr learned my addr 172.21.15.111:0/3904802736 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.749+0000 7fa194b9c700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f6e0c0 0x560ec3fcce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa18ffff700 1 -- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f6e0c0 msgr2=0x560ec3fcce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f6db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa18ffff700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f6e0c0 0x560ec3fcce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa18ffff700 1 -- 172.21.15.111:0/3904802736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 msgr2=0x560ec3f6db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa18ffff700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f6db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa18ffff700 1 -- 172.21.15.111:0/3904802736 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ec3d335d0 con 0x560ec3f66720 2023-10-11T16:37:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa194b9c700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f6e0c0 0x560ec3fcce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:37:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f6db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:37:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa18ffff700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f6d640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa18000b4c0 tx=0x7fa180005cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=649a14476cebe0f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.750+0000 7fa16f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:37:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.751+0000 7fa16f7fe700 1 -- 172.21.15.111:0/3904802736 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18000f460 con 0x560ec3f66720 2023-10-11T16:37:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.751+0000 7fa16f7fe700 1 -- 172.21.15.111:0/3904802736 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa180000e40 con 0x560ec3f66720 2023-10-11T16:37:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.751+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/3904802736 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa18409e2a0 con 0x560ec3f66720 2023-10-11T16:37:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.752+0000 7fa16f7fe700 1 -- 172.21.15.111:0/3904802736 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa18000f5c0 con 0x560ec3f66720 2023-10-11T16:37:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.752+0000 7fa1a9fcdb80 10 client.26431 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.752+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/3904802736 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560ec3e78130 con 0x560ec3f66720 2023-10-11T16:37:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.752+0000 7fa16f7fe700 1 -- 172.21.15.111:0/3904802736 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa18409e2a0 con 0x560ec3f66720 2023-10-11T16:37:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.754+0000 7fa16f7fe700 1 -- 172.21.15.111:0/3904802736 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa180047070 con 0x560ec3f66720 2023-10-11T16:37:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.754+0000 7fa1a9fcdb80 10 client.26431 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.754+0000 7fa1a9fcdb80 10 client.26431 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.754+0000 7fa1a9fcdb80 20 client.26431 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.754+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560ec3fce760 0x560ec3fd0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.754+0000 7fa1a9fcdb80 4 client.26431 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:37:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.754+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/3904802736 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560ec3e80600 con 0x560ec3fce760 2023-10-11T16:37:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.755+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560ec3fce760 0x560ec3fd0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.756+0000 7fa18f7fe700 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560ec3fce760 0x560ec3fd0b40 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a0b920233103b10 server_cookie=74db87629dbe4a15 in_seq=0 out_seq=0 2023-10-11T16:37:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:08.756+0000 7fa16f7fe700 10 client.26431 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:09.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:08 smithi130 ceph-mon[131859]: pgmap v138: 129 pgs: 129 active+clean; 491 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 623 op/s 2023-10-11T16:37:09.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:08 smithi111 ceph-mon[98527]: pgmap v138: 129 pgs: 129 active+clean; 491 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 623 op/s 2023-10-11T16:37:09.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:08 smithi150 ceph-mon[132217]: pgmap v138: 129 pgs: 129 active+clean; 491 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 623 op/s 2023-10-11T16:37:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.670+0000 7fa16f7fe700 1 -- 172.21.15.111:0/3904802736 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560ec3e80600 con 0x560ec3fce760 2023-10-11T16:37:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.670+0000 7fa16f7fe700 10 client.26431 handle_command_reply: tid=0 2023-10-11T16:37:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.670+0000 7fa1a9fcdb80 1 client.26431 shutdown 2023-10-11T16:37:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.672+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/3904802736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 msgr2=0x560ec3f6d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.672+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f6d640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa18000b4c0 tx=0x7fa180005cc0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/3904802736 shutdown_connections 2023-10-11T16:37:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560ec3fce760 0x560ec3fd0b40 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560ec3f6e0c0 0x560ec3fcce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560ec3f66720 0x560ec3f6d640 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa1a9fcdb80 1 --2- 172.21.15.111:0/3904802736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560ec3f66b90 0x560ec3f6db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.676 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa16f7fe700 0 client.26431 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/3904802736 >> 172.21.15.111:0/3904802736 conn(0x560ec3f55f50 msgr2=0x560ec3f66f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/3904802736 shutdown_connections 2023-10-11T16:37:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.674+0000 7fa1a9fcdb80 1 -- 172.21.15.111:0/3904802736 wait complete. 2023-10-11T16:37:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:37:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:37:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:37:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:37:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.675+0000 7fa1a9fcdb80 20 client.26431 trim_cache size 0 max 16384 2023-10-11T16:37:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.678+0000 7fa1965ff700 1 -- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1780a1340 msgr2=0x7fa1780a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.678+0000 7fa1965ff700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1780a1340 0x7fa1780a37f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa18800f880 tx=0x7fa18800fc40 comp rx=0 tx=0).stop 2023-10-11T16:37:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.678+0000 7fa1965ff700 1 -- 172.21.15.111:0/84522213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ffb40 msgr2=0x7fa19019ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.678+0000 7fa1965ff700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ffb40 0x7fa19019ceb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa18400eb30 tx=0x7fa18400ee40 comp rx=0 tx=0).stop 2023-10-11T16:37:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.681+0000 7fa1965ff700 1 -- 172.21.15.111:0/84522213 shutdown_connections 2023-10-11T16:37:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.681+0000 7fa1965ff700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa1780a1340 0x7fa1780a37f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.681+0000 7fa1965ff700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa190104630 0x7fa1901a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.681+0000 7fa1965ff700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1900ff230 0x7fa19019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.683 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.681+0000 7fa1965ff700 1 --2- 172.21.15.111:0/84522213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa1900ffb40 0x7fa19019ceb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:09.684 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.681+0000 7fa1965ff700 1 -- 172.21.15.111:0/84522213 >> 172.21.15.111:0/84522213 conn(0x7fa1900763a0 msgr2=0x7fa190102050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.681+0000 7fa1965ff700 1 -- 172.21.15.111:0/84522213 shutdown_connections 2023-10-11T16:37:09.685 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:09.682+0000 7fa1965ff700 1 -- 172.21.15.111:0/84522213 wait complete. 2023-10-11T16:37:11.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:11 smithi150 ceph-mon[132217]: pgmap v139: 129 pgs: 129 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 825 op/s 2023-10-11T16:37:11.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:11 smithi130 ceph-mon[131859]: pgmap v139: 129 pgs: 129 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 825 op/s 2023-10-11T16:37:11.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:11 smithi111 ceph-mon[98527]: pgmap v139: 129 pgs: 129 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.0 MiB/s wr, 825 op/s 2023-10-11T16:37:13.454 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:13 smithi150 ceph-mon[132217]: pgmap v140: 129 pgs: 129 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 392 op/s 2023-10-11T16:37:13.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:13 smithi130 ceph-mon[131859]: pgmap v140: 129 pgs: 129 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 392 op/s 2023-10-11T16:37:13.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:13 smithi111 ceph-mon[98527]: pgmap v140: 129 pgs: 129 active+clean; 514 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 392 op/s 2023-10-11T16:37:15.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:15 smithi150 ceph-mon[132217]: pgmap v141: 129 pgs: 129 active+clean; 513 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.5 MiB/s wr, 475 op/s 2023-10-11T16:37:15.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:15 smithi130 ceph-mon[131859]: pgmap v141: 129 pgs: 129 active+clean; 513 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.5 MiB/s wr, 475 op/s 2023-10-11T16:37:15.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:15 smithi111 ceph-mon[98527]: pgmap v141: 129 pgs: 129 active+clean; 513 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.5 MiB/s wr, 475 op/s 2023-10-11T16:37:17.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:17 smithi150 ceph-mon[132217]: pgmap v142: 129 pgs: 129 active+clean; 521 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 737 op/s 2023-10-11T16:37:17.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:17 smithi130 ceph-mon[131859]: pgmap v142: 129 pgs: 129 active+clean; 521 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 737 op/s 2023-10-11T16:37:17.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:17 smithi111 ceph-mon[98527]: pgmap v142: 129 pgs: 129 active+clean; 521 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 737 op/s 2023-10-11T16:37:19.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:19 smithi130 ceph-mon[131859]: pgmap v143: 129 pgs: 129 active+clean; 521 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 579 op/s 2023-10-11T16:37:19.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:19 smithi111 ceph-mon[98527]: pgmap v143: 129 pgs: 129 active+clean; 521 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 579 op/s 2023-10-11T16:37:19.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:19 smithi150 ceph-mon[132217]: pgmap v143: 129 pgs: 129 active+clean; 521 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 579 op/s 2023-10-11T16:37:21.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:21 smithi130 ceph-mon[131859]: pgmap v144: 129 pgs: 129 active+clean; 539 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 731 op/s 2023-10-11T16:37:21.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:21 smithi111 ceph-mon[98527]: pgmap v144: 129 pgs: 129 active+clean; 539 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 731 op/s 2023-10-11T16:37:21.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:21 smithi150 ceph-mon[132217]: pgmap v144: 129 pgs: 129 active+clean; 539 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 731 op/s 2023-10-11T16:37:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.908+0000 7f52323ff700 1 -- 172.21.15.111:0/1678180963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f522c101440 msgr2=0x7f522c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.908+0000 7f52323ff700 1 --2- 172.21.15.111:0/1678180963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f522c101440 0x7f522c101810 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f521c009990 tx=0x7f521c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.910+0000 7f52323ff700 1 -- 172.21.15.111:0/1678180963 shutdown_connections 2023-10-11T16:37:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.910+0000 7f52323ff700 1 --2- 172.21.15.111:0/1678180963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f522c10a810 0x7f522c10cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.910+0000 7f52323ff700 1 --2- 172.21.15.111:0/1678180963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f522c101440 0x7f522c101810 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.910+0000 7f52323ff700 1 --2- 172.21.15.111:0/1678180963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522c101d50 0x7f522c10a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.910+0000 7f52323ff700 1 -- 172.21.15.111:0/1678180963 >> 172.21.15.111:0/1678180963 conn(0x7f522c0763a0 msgr2=0x7f522c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.911+0000 7f52323ff700 1 -- 172.21.15.111:0/1678180963 shutdown_connections 2023-10-11T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.911+0000 7f52323ff700 1 -- 172.21.15.111:0/1678180963 wait complete. 2023-10-11T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.912+0000 7f52323ff700 1 Processor -- start 2023-10-11T16:37:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.913+0000 7f52323ff700 1 -- start start 2023-10-11T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.913+0000 7f52323ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522c101440 0x7f522c19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.913+0000 7f52323ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f522c101d50 0x7f522c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.913+0000 7f52323ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f522c10a810 0x7f522c1a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.913+0000 7f52323ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1a1920 con 0x7f522c101440 2023-10-11T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.914+0000 7f52323ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1a1a90 con 0x7f522c10a810 2023-10-11T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.914+0000 7f52323ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1a1c50 con 0x7f522c101d50 2023-10-11T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.914+0000 7f522bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522c101440 0x7f522c19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.914+0000 7f522bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522c101440 0x7f522c19c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33926/0 (socket says 172.21.15.111:33926) 2023-10-11T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.914+0000 7f522bfff700 1 -- 172.21.15.111:0/1488491599 learned_addr learned my addr 172.21.15.111:0/1488491599 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.914+0000 7f523099c700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f522c10a810 0x7f522c1a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.914+0000 7f522bfff700 1 -- 172.21.15.111:0/1488491599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f522c101d50 msgr2=0x7f522c19cec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.914+0000 7f522bfff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f522c101d50 0x7f522c19cec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.915+0000 7f522bfff700 1 -- 172.21.15.111:0/1488491599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f522c10a810 msgr2=0x7f522c1a1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.915+0000 7f522bfff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f522c10a810 0x7f522c1a1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.915+0000 7f522bfff700 1 -- 172.21.15.111:0/1488491599 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c0095f0 con 0x7f522c101440 2023-10-11T16:37:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.915+0000 7f522bfff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522c101440 0x7f522c19c980 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f521c0095c0 tx=0x7f521c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.915+0000 7f52297fa700 1 -- 172.21.15.111:0/1488491599 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c01d070 con 0x7f522c101440 2023-10-11T16:37:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.916+0000 7f52297fa700 1 -- 172.21.15.111:0/1488491599 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f521c0054f0 con 0x7f522c101440 2023-10-11T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.916+0000 7f52297fa700 1 -- 172.21.15.111:0/1488491599 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c01d070 con 0x7f522c101440 2023-10-11T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.918+0000 7f52323ff700 1 -- 172.21.15.111:0/1488491599 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c10e170 con 0x7f522c101440 2023-10-11T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.918+0000 7f52323ff700 1 -- 172.21.15.111:0/1488491599 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c10dfe0 con 0x7f522c101440 2023-10-11T16:37:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.925+0000 7f5245d38b80 1 Processor -- start 2023-10-11T16:37:22.927 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.926+0000 7f5245d38b80 1 -- start start 2023-10-11T16:37:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b731e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b6d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.928 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 0x563523b6db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f52297fa700 1 -- 172.21.15.111:0/1488491599 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f521c00fb80 con 0x7f522c101440 2023-10-11T16:37:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f523099c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f523099c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33308/0 (socket says 172.21.15.111:33308) 2023-10-11T16:37:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f5245d38b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523a73000 con 0x563523b73b90 2023-10-11T16:37:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f5245d38b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523ac5840 con 0x5635239c96b0 2023-10-11T16:37:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f5245d38b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523abecd0 con 0x563523b73720 2023-10-11T16:37:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.927+0000 7f523099c700 1 -- 172.21.15.111:0/1657002520 learned_addr learned my addr 172.21.15.111:0/1657002520 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.928+0000 7f522bfff700 1 --2- 172.21.15.111:0/1657002520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b6d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f523099c700 1 -- 172.21.15.111:0/1657002520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 msgr2=0x563523b6d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f523099c700 1 --2- 172.21.15.111:0/1657002520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b6d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f523099c700 1 -- 172.21.15.111:0/1657002520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 msgr2=0x563523b6db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f523099c700 1 --2- 172.21.15.111:0/1657002520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 0x563523b6db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f523099c700 1 -- 172.21.15.111:0/1657002520 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635239405d0 con 0x5635239c96b0 2023-10-11T16:37:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f52297fa700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f52140a1420 0x7f52140a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f523099c700 1 --2- 172.21.15.111:0/1657002520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b731e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f522400a430 tx=0x7f5224009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6f35f6a0c302f34 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f52297fa700 1 -- 172.21.15.111:0/1488491599 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f521c0cd860 con 0x7f522c101440 2023-10-11T16:37:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f52127fc700 1 -- 172.21.15.111:0/1657002520 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224010040 con 0x5635239c96b0 2023-10-11T16:37:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f52127fc700 1 -- 172.21.15.111:0/1657002520 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5224014ae0 con 0x5635239c96b0 2023-10-11T16:37:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.929+0000 7f52127fc700 1 -- 172.21.15.111:0/1657002520 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224013ad0 con 0x5635239c96b0 2023-10-11T16:37:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.930+0000 7f522bfff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f52140a1420 0x7f52140a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.930+0000 7f5245d38b80 1 -- 172.21.15.111:0/1657002520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 msgr2=0x563523b731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.930+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1657002520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b731e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f522400a430 tx=0x7f5224009650 comp rx=0 tx=0).stop 2023-10-11T16:37:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.930+0000 7f5245d38b80 1 -- 172.21.15.111:0/1657002520 shutdown_connections 2023-10-11T16:37:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.930+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1657002520 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 0x563523b6db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.930+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1657002520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b731e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.935 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.930+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1657002520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b6d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.930+0000 7f5245d38b80 1 -- 172.21.15.111:0/1657002520 >> 172.21.15.111:0/1657002520 conn(0x563523b62f50 msgr2=0x563523b68270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.931+0000 7f5245d38b80 1 -- 172.21.15.111:0/1657002520 shutdown_connections 2023-10-11T16:37:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.931+0000 7f5245d38b80 1 -- 172.21.15.111:0/1657002520 wait complete. 2023-10-11T16:37:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.931+0000 7f5245d38b80 1 -- start start 2023-10-11T16:37:22.936 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.932+0000 7f5245d38b80 1 Processor -- start 2023-10-11T16:37:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.932+0000 7f522bfff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f52140a1420 0x7f52140a38d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f521c0a2440 tx=0x7f521c0d3530 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:22.937 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.934+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b79a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.934+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b76cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.934+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 0x563523b77300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.934+0000 7f5245d38b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523a60880 con 0x563523b73b90 2023-10-11T16:37:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.934+0000 7f5245d38b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523a60520 con 0x5635239c96b0 2023-10-11T16:37:22.938 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.934+0000 7f5245d38b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635239428c0 con 0x563523b73720 2023-10-11T16:37:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.934+0000 7f5245d38b80 10 client.0 _ll_register_callbacks cb 0x563523a7e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.934+0000 7f523099c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b79a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.935+0000 7f522b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b76cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.935+0000 7f523099c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b79a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33310/0 (socket says 172.21.15.111:33310) 2023-10-11T16:37:22.939 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.935+0000 7f523099c700 1 -- 172.21.15.111:0/2677862007 learned_addr learned my addr 172.21.15.111:0/2677862007 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.935+0000 7f523099c700 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 0x563523b77300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.937+0000 7f522b7fe700 1 -- 172.21.15.111:0/2677862007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 msgr2=0x563523b79a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.937+0000 7f522b7fe700 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b79a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.941 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.937+0000 7f522b7fe700 1 -- 172.21.15.111:0/2677862007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 msgr2=0x563523b77300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:22.941 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.937+0000 7f522b7fe700 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 0x563523b77300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.941 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.937+0000 7f522b7fe700 1 -- 172.21.15.111:0/2677862007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5224009c60 con 0x563523b73720 2023-10-11T16:37:22.941 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.938+0000 7f522b7fe700 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b76cf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f522c19df90 tx=0x7f522000a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f8380b766e38ab5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.938+0000 7f520b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:37:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.940+0000 7f520b7fe700 1 -- 172.21.15.111:0/2677862007 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220015600 con 0x563523b73720 2023-10-11T16:37:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.940+0000 7f520b7fe700 1 -- 172.21.15.111:0/2677862007 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f5220015760 con 0x563523b73720 2023-10-11T16:37:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.940+0000 7f5245d38b80 1 -- 172.21.15.111:0/2677862007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563523b60710 con 0x563523b73720 2023-10-11T16:37:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.941+0000 7f520b7fe700 1 -- 172.21.15.111:0/2677862007 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5220015920 con 0x563523b73720 2023-10-11T16:37:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.941+0000 7f5245d38b80 10 client.26437 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.941+0000 7f5245d38b80 1 -- 172.21.15.111:0/2677862007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563523a85130 con 0x563523b73720 2023-10-11T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.942+0000 7f520b7fe700 1 -- 172.21.15.111:0/2677862007 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563523b60710 con 0x563523b73720 2023-10-11T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.942+0000 7f520b7fe700 1 -- 172.21.15.111:0/2677862007 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5220014be0 con 0x563523b73720 2023-10-11T16:37:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.942+0000 7f5245d38b80 10 client.26437 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.942+0000 7f5245d38b80 10 client.26437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.942+0000 7f5245d38b80 20 client.26437 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.942+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563523b86fe0 0x563523b893f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.942+0000 7f5245d38b80 4 client.26437 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.942+0000 7f5245d38b80 1 -- 172.21.15.111:0/2677862007 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563523a81040 con 0x563523b86fe0 2023-10-11T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.944+0000 7f523099c700 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563523b86fe0 0x563523b893f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.944+0000 7f520b7fe700 10 client.26437 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.944+0000 7f523099c700 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563523b86fe0 0x563523b893f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a32e1a2d42bb4c94 server_cookie=4e8cf5cf568993a7 in_seq=0 out_seq=0 2023-10-11T16:37:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.964+0000 7f520b7fe700 1 -- 172.21.15.111:0/2677862007 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563523a81040 con 0x563523b86fe0 2023-10-11T16:37:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.964+0000 7f520b7fe700 10 client.26437 handle_command_reply: tid=0 2023-10-11T16:37:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.964+0000 7f5245d38b80 1 client.26437 shutdown 2023-10-11T16:37:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 -- 172.21.15.111:0/2677862007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 msgr2=0x563523b76cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b76cf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f522c19df90 tx=0x7f522000a5b0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 -- 172.21.15.111:0/2677862007 shutdown_connections 2023-10-11T16:37:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563523b86fe0 0x563523b893f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73b90 0x563523b77300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b79a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2677862007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73720 0x563523b76cf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 -- 172.21.15.111:0/2677862007 >> 172.21.15.111:0/2677862007 conn(0x563523b62f50 msgr2=0x563523b677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f520b7fe700 0 client.26437 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.965+0000 7f5245d38b80 1 -- 172.21.15.111:0/2677862007 shutdown_connections 2023-10-11T16:37:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.966+0000 7f5245d38b80 1 -- 172.21.15.111:0/2677862007 wait complete. 2023-10-11T16:37:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.966+0000 7f5245d38b80 20 client.26437 trim_cache size 0 max 16384 2023-10-11T16:37:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.997+0000 7f5245d38b80 1 Processor -- start 2023-10-11T16:37:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.998+0000 7f5245d38b80 1 -- start start 2023-10-11T16:37:23.000 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.998+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b7e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.998+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73720 0x563523b6d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.998+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b67f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.998+0000 7f5245d38b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523b845c0 con 0x563523b73720 2023-10-11T16:37:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.998+0000 7f5245d38b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523b687e0 con 0x5635239c96b0 2023-10-11T16:37:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.998+0000 7f5245d38b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523b62b70 con 0x563523b73b90 2023-10-11T16:37:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.999+0000 7f523099c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b7e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.999+0000 7f523099c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b7e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33326/0 (socket says 172.21.15.111:33326) 2023-10-11T16:37:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.999+0000 7f523099c700 1 -- 172.21.15.111:0/2799966645 learned_addr learned my addr 172.21.15.111:0/2799966645 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.999+0000 7f523099c700 1 --2- 172.21.15.111:0/2799966645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b67f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.999+0000 7f523099c700 1 -- 172.21.15.111:0/2799966645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 msgr2=0x563523b67f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:22.999+0000 7f523099c700 1 --2- 172.21.15.111:0/2799966645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b67f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.000+0000 7f523099c700 1 -- 172.21.15.111:0/2799966645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73720 msgr2=0x563523b6d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.000+0000 7f523099c700 1 --2- 172.21.15.111:0/2799966645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73720 0x563523b6d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.000+0000 7f523099c700 1 -- 172.21.15.111:0/2799966645 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563523b646f0 con 0x5635239c96b0 2023-10-11T16:37:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.000+0000 7f523099c700 1 --2- 172.21.15.111:0/2799966645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b7e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x563523b89cc0 tx=0x7f5224014f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d3bd92fb371ce85 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.000+0000 7f5212ffd700 1 -- 172.21.15.111:0/2799966645 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522402a480 con 0x5635239c96b0 2023-10-11T16:37:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.000+0000 7f5212ffd700 1 -- 172.21.15.111:0/2799966645 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5224030070 con 0x5635239c96b0 2023-10-11T16:37:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.000+0000 7f5212ffd700 1 -- 172.21.15.111:0/2799966645 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52240135e0 con 0x5635239c96b0 2023-10-11T16:37:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.001+0000 7f5245d38b80 1 -- 172.21.15.111:0/2799966645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 msgr2=0x563523b7e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:23.006 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.001+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2799966645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b7e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x563523b89cc0 tx=0x7f5224014f90 comp rx=0 tx=0).stop 2023-10-11T16:37:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.002+0000 7f5245d38b80 1 -- 172.21.15.111:0/2799966645 shutdown_connections 2023-10-11T16:37:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.002+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2799966645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b67f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.002+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2799966645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5635239c96b0 0x563523b7e090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.002+0000 7f5245d38b80 1 --2- 172.21.15.111:0/2799966645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563523b73720 0x563523b6d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.002+0000 7f5245d38b80 1 -- 172.21.15.111:0/2799966645 >> 172.21.15.111:0/2799966645 conn(0x563523b62f50 msgr2=0x563523b79630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.002+0000 7f5245d38b80 1 -- 172.21.15.111:0/2799966645 shutdown_connections 2023-10-11T16:37:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.002+0000 7f5245d38b80 1 -- 172.21.15.111:0/2799966645 wait complete. 2023-10-11T16:37:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.003+0000 7f5245d38b80 1 -- start start 2023-10-11T16:37:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.003+0000 7f5245d38b80 1 Processor -- start 2023-10-11T16:37:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.006+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635239c96b0 0x563523b6fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.006+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563523b73720 0x563523b70370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.006+0000 7f5245d38b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b6ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.006+0000 7f5245d38b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523b7e560 con 0x5635239c96b0 2023-10-11T16:37:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.006+0000 7f5245d38b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523b7e6d0 con 0x563523b73720 2023-10-11T16:37:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.007+0000 7f523099c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635239c96b0 0x563523b6fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.007+0000 7f523099c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635239c96b0 0x563523b6fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33964/0 (socket says 172.21.15.111:33964) 2023-10-11T16:37:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.007+0000 7f523099c700 1 -- 172.21.15.111:0/1329104858 learned_addr learned my addr 172.21.15.111:0/1329104858 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563523b6f220 con 0x563523b73b90 2023-10-11T16:37:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f5245d38b80 10 client.0 _ll_register_callbacks cb 0x7f522000d920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f522b7fe700 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563523b73720 0x563523b70370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:23.012 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f523099c700 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b6ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f522b7fe700 1 -- 172.21.15.111:0/1329104858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 msgr2=0x563523b6ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f522b7fe700 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b6ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f522b7fe700 1 -- 172.21.15.111:0/1329104858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635239c96b0 msgr2=0x563523b6fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f522b7fe700 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635239c96b0 0x563523b6fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f522b7fe700 1 -- 172.21.15.111:0/1329104858 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522402a040 con 0x563523b73720 2023-10-11T16:37:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.008+0000 7f523099c700 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b6ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:37:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.009+0000 7f522b7fe700 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563523b73720 0x563523b70370 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x563523b62ea0 tx=0x7f522001da10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5ab1f09145c61bb server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.009+0000 7f520b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:37:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.009+0000 7f520b7fe700 1 -- 172.21.15.111:0/1329104858 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220005ee0 con 0x563523b73720 2023-10-11T16:37:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.009+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5224014ae0 con 0x563523b73720 2023-10-11T16:37:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.009+0000 7f520b7fe700 1 -- 172.21.15.111:0/1329104858 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f5220000e40 con 0x563523b73720 2023-10-11T16:37:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.010+0000 7f520b7fe700 1 -- 172.21.15.111:0/1329104858 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5220005ee0 con 0x563523b73720 2023-10-11T16:37:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.010+0000 7f5245d38b80 10 client.26417 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.010+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5635238ecd80 con 0x563523b73720 2023-10-11T16:37:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.010+0000 7f520b7fe700 1 -- 172.21.15.111:0/1329104858 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5224014ae0 con 0x563523b73720 2023-10-11T16:37:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.011+0000 7f520b7fe700 1 -- 172.21.15.111:0/1329104858 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f522001c720 con 0x563523b73720 2023-10-11T16:37:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.011+0000 7f5245d38b80 10 client.26417 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.011+0000 7f5245d38b80 10 client.26417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.011+0000 7f5245d38b80 20 client.26417 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.011+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563523becb00 0x563523beef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.011+0000 7f5245d38b80 4 client.26417 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:37:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.011+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563523a8d600 con 0x563523becb00 2023-10-11T16:37:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.011+0000 7f523099c700 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563523becb00 0x563523beef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.012+0000 7f523099c700 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563523becb00 0x563523beef10 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f055551525dde5 server_cookie=ec2108c9290fee7 in_seq=0 out_seq=0 2023-10-11T16:37:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:23.012+0000 7f520b7fe700 10 client.26417 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:23.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:23 smithi130 ceph-mon[131859]: pgmap v145: 129 pgs: 129 active+clean; 539 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 529 op/s 2023-10-11T16:37:23.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:23 smithi111 ceph-mon[98527]: pgmap v145: 129 pgs: 129 active+clean; 539 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 529 op/s 2023-10-11T16:37:23.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:23 smithi150 ceph-mon[132217]: pgmap v145: 129 pgs: 129 active+clean; 539 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 529 op/s 2023-10-11T16:37:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.032+0000 7f520b7fe700 1 -- 172.21.15.111:0/1329104858 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563523a8d600 con 0x563523becb00 2023-10-11T16:37:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.033+0000 7f520b7fe700 10 client.26417 handle_command_reply: tid=0 2023-10-11T16:37:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.033+0000 7f5245d38b80 1 client.26417 shutdown 2023-10-11T16:37:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.033+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563523b73720 msgr2=0x563523b70370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.033+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563523b73720 0x563523b70370 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x563523b62ea0 tx=0x7f522001da10 comp rx=0 tx=0).stop 2023-10-11T16:37:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.034+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 shutdown_connections 2023-10-11T16:37:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.034+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563523becb00 0x563523beef10 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.034+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563523b73b90 0x563523b6ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.034+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5635239c96b0 0x563523b6fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.034+0000 7f5245d38b80 1 --2- 172.21.15.111:0/1329104858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563523b73720 0x563523b70370 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.034+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 >> 172.21.15.111:0/1329104858 conn(0x563523b62f50 msgr2=0x563523b71e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.034+0000 7f520b7fe700 0 client.26417 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.036+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 shutdown_connections 2023-10-11T16:37:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.036+0000 7f5245d38b80 1 -- 172.21.15.111:0/1329104858 wait complete. 2023-10-11T16:37:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.037+0000 7f5245d38b80 20 client.26417 trim_cache size 0 max 16384 2023-10-11T16:37:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:37:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:37:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:37:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:37:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.038+0000 7f52323ff700 1 -- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f52140a1420 msgr2=0x7f52140a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.038+0000 7f52323ff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f52140a1420 0x7f52140a38d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f521c0a2440 tx=0x7f521c0d3530 comp rx=0 tx=0).stop 2023-10-11T16:37:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.038+0000 7f52323ff700 1 -- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522c101440 msgr2=0x7f522c19c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.038+0000 7f52323ff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522c101440 0x7f522c19c980 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f521c0095c0 tx=0x7f521c00f690 comp rx=0 tx=0).stop 2023-10-11T16:37:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.039+0000 7f52323ff700 1 -- 172.21.15.111:0/1488491599 shutdown_connections 2023-10-11T16:37:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.039+0000 7f52323ff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f52140a1420 0x7f52140a38d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.039+0000 7f52323ff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f522c10a810 0x7f522c1a1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.039+0000 7f52323ff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f522c101440 0x7f522c19c980 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.039+0000 7f52323ff700 1 --2- 172.21.15.111:0/1488491599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f522c101d50 0x7f522c19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.039+0000 7f52323ff700 1 -- 172.21.15.111:0/1488491599 >> 172.21.15.111:0/1488491599 conn(0x7f522c0763a0 msgr2=0x7f522c1021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.039+0000 7f52323ff700 1 -- 172.21.15.111:0/1488491599 shutdown_connections 2023-10-11T16:37:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:24.039+0000 7f52323ff700 1 -- 172.21.15.111:0/1488491599 wait complete. 2023-10-11T16:37:25.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:25 smithi130 ceph-mon[131859]: pgmap v146: 129 pgs: 129 active+clean; 549 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 583 op/s 2023-10-11T16:37:25.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:25 smithi111 ceph-mon[98527]: pgmap v146: 129 pgs: 129 active+clean; 549 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 583 op/s 2023-10-11T16:37:25.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:25 smithi150 ceph-mon[132217]: pgmap v146: 129 pgs: 129 active+clean; 549 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 583 op/s 2023-10-11T16:37:27.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:27 smithi130 ceph-mon[131859]: pgmap v147: 129 pgs: 129 active+clean; 564 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 714 op/s 2023-10-11T16:37:27.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:27 smithi111 ceph-mon[98527]: pgmap v147: 129 pgs: 129 active+clean; 564 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 714 op/s 2023-10-11T16:37:27.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:27 smithi150 ceph-mon[132217]: pgmap v147: 129 pgs: 129 active+clean; 564 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 714 op/s 2023-10-11T16:37:28.949 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:37:29.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.205+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/4278018722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8101660 msgr2=0x7ff7a8104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.205+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/4278018722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8101660 0x7ff7a8104ef0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff79c009990 tx=0x7ff79c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.205+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/4278018722 shutdown_connections 2023-10-11T16:37:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.205+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/4278018722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7a8105620 0x7ff7a810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.205+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/4278018722 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7a8100cc0 0x7ff7a8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.205+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/4278018722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8101660 0x7ff7a8104ef0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.205+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/4278018722 >> 172.21.15.111:0/4278018722 conn(0x7ff7a8077560 msgr2=0x7ff7a8077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.206+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/4278018722 shutdown_connections 2023-10-11T16:37:29.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.206+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/4278018722 wait complete. 2023-10-11T16:37:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7ae9ff700 1 Processor -- start 2023-10-11T16:37:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7ae9ff700 1 -- start start 2023-10-11T16:37:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7ae9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8100cc0 0x7ff7a819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7ae9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7a8101660 0x7ff7a819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7ae9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7a8105620 0x7ff7a8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7ae9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8198be0 con 0x7ff7a8101660 2023-10-11T16:37:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7ae9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8198d50 con 0x7ff7a8105620 2023-10-11T16:37:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7ae9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8198f10 con 0x7ff7a8100cc0 2023-10-11T16:37:29.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7a7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8100cc0 0x7ff7a819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7a7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8100cc0 0x7ff7a819e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:51560/0 (socket says 172.21.15.111:51560) 2023-10-11T16:37:29.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.207+0000 7ff7a7fff700 1 -- 172.21.15.111:0/3181573783 learned_addr learned my addr 172.21.15.111:0/3181573783 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:29.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7a8105620 0x7ff7a8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7a8101660 0x7ff7a819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7acf9c700 1 -- 172.21.15.111:0/3181573783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8100cc0 msgr2=0x7ff7a819e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8100cc0 0x7ff7a819e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7acf9c700 1 -- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7a8101660 msgr2=0x7ff7a819ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7a8101660 0x7ff7a819ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7acf9c700 1 -- 172.21.15.111:0/3181573783 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c0095f0 con 0x7ff7a8105620 2023-10-11T16:37:29.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8100cc0 0x7ff7a819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:37:29.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7a8101660 0x7ff7a819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:37:29.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.208+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7a8105620 0x7ff7a8198570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000efc0 tx=0x7ff7a000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:29.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.209+0000 7ff7a57fa700 1 -- 172.21.15.111:0/3181573783 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0018070 con 0x7ff7a8105620 2023-10-11T16:37:29.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.209+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/3181573783 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a8110370 con 0x7ff7a8105620 2023-10-11T16:37:29.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.209+0000 7ff7a57fa700 1 -- 172.21.15.111:0/3181573783 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7a000f040 con 0x7ff7a8105620 2023-10-11T16:37:29.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.209+0000 7ff7a57fa700 1 -- 172.21.15.111:0/3181573783 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a000cd40 con 0x7ff7a8105620 2023-10-11T16:37:29.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.209+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/3181573783 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a81101e0 con 0x7ff7a8105620 2023-10-11T16:37:29.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.212+0000 7ff7a57fa700 1 -- 172.21.15.111:0/3181573783 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7ff7a0003a00 con 0x7ff7a8105620 2023-10-11T16:37:29.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.214+0000 7ff7c22ceb80 1 Processor -- start 2023-10-11T16:37:29.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.215+0000 7ff7c22ceb80 1 -- start start 2023-10-11T16:37:29.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.215+0000 7ff7a57fa700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff7900a12c0 0x7ff7900a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79152f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79157370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79157d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b79058060 con 0x563b79153470 2023-10-11T16:37:29.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b790aa8a0 con 0x563b791578b0 2023-10-11T16:37:29.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b790a3d30 con 0x563b79152b60 2023-10-11T16:37:29.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7a57fa700 1 -- 172.21.15.111:0/3181573783 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff7a00ccc20 con 0x7ff7a8105620 2023-10-11T16:37:29.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff7900a12c0 0x7ff7900a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7a7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7a7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:51572/0 (socket says 172.21.15.111:51572) 2023-10-11T16:37:29.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7a7fff700 1 -- 172.21.15.111:0/1884053063 learned_addr learned my addr 172.21.15.111:0/1884053063 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:29.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7acf9c700 1 -- 172.21.15.111:0/1884053063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 msgr2=0x563b79152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:29.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.216+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff7acf9c700 1 -- 172.21.15.111:0/1884053063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 msgr2=0x563b79157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:29.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff7acf9c700 1 -- 172.21.15.111:0/1884053063 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b78f25630 con 0x563b791578b0 2023-10-11T16:37:29.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:37:29.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79157d00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00c9f60 tx=0x7ff7a0095d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b243646c06fd0d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:29.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff7900a12c0 0x7ff7900a3770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff79c000c00 tx=0x7ff79c005ee0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:29.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff78e7fc700 1 -- 172.21.15.111:0/1884053063 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a00992d0 con 0x563b791578b0 2023-10-11T16:37:29.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff78e7fc700 1 -- 172.21.15.111:0/1884053063 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7a0022200 con 0x563b791578b0 2023-10-11T16:37:29.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.217+0000 7ff78e7fc700 1 -- 172.21.15.111:0/1884053063 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0082d80 con 0x563b791578b0 2023-10-11T16:37:29.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1884053063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 msgr2=0x563b79157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79157d00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff7a00c9f60 tx=0x7ff7a0095d30 comp rx=0 tx=0).stop 2023-10-11T16:37:29.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1884053063 shutdown_connections 2023-10-11T16:37:29.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79157d00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/1884053063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1884053063 >> 172.21.15.111:0/1884053063 conn(0x563b79148360 msgr2=0x563b7914db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:29.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1884053063 shutdown_connections 2023-10-11T16:37:29.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.218+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1884053063 wait complete. 2023-10-11T16:37:29.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.219+0000 7ff7c22ceb80 1 -- start start 2023-10-11T16:37:29.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.219+0000 7ff7c22ceb80 1 Processor -- start 2023-10-11T16:37:29.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b7915f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b79153470 0x563b7915c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b791578b0 0x563b7915caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.229 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b790458e0 con 0x563b791578b0 2023-10-11T16:37:29.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b79045580 con 0x563b79153470 2023-10-11T16:37:29.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b78f27920 con 0x563b79152b60 2023-10-11T16:37:29.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7c22ceb80 10 client.0 _ll_register_callbacks cb 0x563b79063780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:29.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b79153470 0x563b7915c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b7915f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b79153470 0x563b7915c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:38048/0 (socket says 172.21.15.111:38048) 2023-10-11T16:37:29.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a77fe700 1 -- 172.21.15.111:0/262378966 learned_addr learned my addr 172.21.15.111:0/262378966 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:29.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b791578b0 0x563b7915caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a77fe700 1 -- 172.21.15.111:0/262378966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 msgr2=0x563b7915f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:29.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b7915f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a77fe700 1 -- 172.21.15.111:0/262378966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b791578b0 msgr2=0x563b7915caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:29.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b791578b0 0x563b7915caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.220+0000 7ff7a77fe700 1 -- 172.21.15.111:0/262378966 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a81101e0 con 0x563b79153470 2023-10-11T16:37:29.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b79153470 0x563b7915c360 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff79c005f50 tx=0x7ff79c0058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b72fb62e5e1037c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:29.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7877fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:37:29.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7877fe700 1 -- 172.21.15.111:0/262378966 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c02a9f0 con 0x563b79153470 2023-10-11T16:37:29.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7877fe700 1 -- 172.21.15.111:0/262378966 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff79c02ab50 con 0x563b79153470 2023-10-11T16:37:29.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/262378966 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563b79145b20 con 0x563b79153470 2023-10-11T16:37:29.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7877fe700 1 -- 172.21.15.111:0/262378966 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff79c02ad10 con 0x563b79153470 2023-10-11T16:37:29.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7c22ceb80 10 client.26429 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:29.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/262378966 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563b7906a190 con 0x563b79153470 2023-10-11T16:37:29.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7877fe700 1 -- 172.21.15.111:0/262378966 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563b79145b20 con 0x563b79153470 2023-10-11T16:37:29.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.221+0000 7ff7877fe700 1 -- 172.21.15.111:0/262378966 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff79c034c90 con 0x563b79153470 2023-10-11T16:37:29.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7c22ceb80 10 client.26429 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:29.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7c22ceb80 10 client.26429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:29.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7c22ceb80 20 client.26429 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:29.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563b7916c980 0x563b7916ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7c22ceb80 4 client.26429 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:37:29.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/262378966 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b790660a0 con 0x563b7916c980 2023-10-11T16:37:29.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563b7916c980 0x563b7916ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563b7916c980 0x563b7916ed90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ecc16c176a44bbc server_cookie=25907d4f4be44faa in_seq=0 out_seq=0 2023-10-11T16:37:29.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.222+0000 7ff7877fe700 10 client.26429 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:29.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7877fe700 1 -- 172.21.15.111:0/262378966 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563b790660a0 con 0x563b7916c980 2023-10-11T16:37:29.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7877fe700 10 client.26429 handle_command_reply: tid=0 2023-10-11T16:37:29.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 client.26429 shutdown 2023-10-11T16:37:29.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/262378966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b79153470 msgr2=0x563b7915c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b79153470 0x563b7915c360 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff79c005f50 tx=0x7ff79c0058e0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/262378966 shutdown_connections 2023-10-11T16:37:29.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563b7916c980 0x563b7916ed90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b791578b0 0x563b7915caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7877fe700 0 client.26429 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:29.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b7915f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/262378966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b79153470 0x563b7915c360 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/262378966 >> 172.21.15.111:0/262378966 conn(0x563b79148360 msgr2=0x563b79158d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:29.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/262378966 shutdown_connections 2023-10-11T16:37:29.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/262378966 wait complete. 2023-10-11T16:37:29.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.224+0000 7ff7c22ceb80 20 client.26429 trim_cache size 0 max 16384 2023-10-11T16:37:29.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.263+0000 7ff7c22ceb80 1 Processor -- start 2023-10-11T16:37:29.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.263+0000 7ff7c22ceb80 1 -- start start 2023-10-11T16:37:29.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.263+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79152b60 0x563b791639b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79153470 0x563b7915ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79155930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b79156490 con 0x563b79152b60 2023-10-11T16:37:29.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b79153900 con 0x563b791578b0 2023-10-11T16:37:29.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b79163ef0 con 0x563b79153470 2023-10-11T16:37:29.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79153470 0x563b7915ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79153470 0x563b7915ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:51590/0 (socket says 172.21.15.111:51590) 2023-10-11T16:37:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a77fe700 1 -- 172.21.15.111:0/1305665196 learned_addr learned my addr 172.21.15.111:0/1305665196 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79152b60 0x563b791639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a77fe700 1 -- 172.21.15.111:0/1305665196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 msgr2=0x563b79155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:29.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a77fe700 1 -- 172.21.15.111:0/1305665196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79152b60 msgr2=0x563b791639b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:29.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79152b60 0x563b791639b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.264+0000 7ff7a77fe700 1 -- 172.21.15.111:0/1305665196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b79149b00 con 0x563b79153470 2023-10-11T16:37:29.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.266+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79153470 0x563b7915ec80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff79c005eb0 tx=0x7ff79c0174e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39c308c3ffdd9334 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:29.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.266+0000 7ff78effd700 1 -- 172.21.15.111:0/1305665196 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c009290 con 0x563b79153470 2023-10-11T16:37:29.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.266+0000 7ff78effd700 1 -- 172.21.15.111:0/1305665196 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff79c017680 con 0x563b79153470 2023-10-11T16:37:29.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.266+0000 7ff78effd700 1 -- 172.21.15.111:0/1305665196 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c017ab0 con 0x563b79153470 2023-10-11T16:37:29.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.267+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1305665196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79153470 msgr2=0x563b7915ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.267+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79153470 0x563b7915ec80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff79c005eb0 tx=0x7ff79c0174e0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.268+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1305665196 shutdown_connections 2023-10-11T16:37:29.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.268+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.268+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79152b60 0x563b791639b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.268+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/1305665196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79153470 0x563b7915ec80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.268+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1305665196 >> 172.21.15.111:0/1305665196 conn(0x563b79148360 msgr2=0x563b7914cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:29.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.268+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1305665196 shutdown_connections 2023-10-11T16:37:29.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.269+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/1305665196 wait complete. 2023-10-11T16:37:29.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.270+0000 7ff7c22ceb80 1 -- start start 2023-10-11T16:37:29.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.270+0000 7ff7c22ceb80 1 Processor -- start 2023-10-11T16:37:29.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.271+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79161990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.271+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79161ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.271+0000 7ff7c22ceb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79162410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.271+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b791c8050 con 0x563b79153470 2023-10-11T16:37:29.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.271+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b791c8190 con 0x563b791578b0 2023-10-11T16:37:29.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.271+0000 7ff7c22ceb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b791c82d0 con 0x563b79152b60 2023-10-11T16:37:29.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.271+0000 7ff7c22ceb80 10 client.0 _ll_register_callbacks cb 0x7ff79c00f840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:29.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.272+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79161ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.272+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79161ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57962/0 (socket says 172.21.15.111:57962) 2023-10-11T16:37:29.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.272+0000 7ff7a77fe700 1 -- 172.21.15.111:0/3417098668 learned_addr learned my addr 172.21.15.111:0/3417098668 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:29.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.272+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.272+0000 7ff7acf9c700 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79162410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.273+0000 7ff7a7fff700 1 -- 172.21.15.111:0/3417098668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 msgr2=0x563b79162410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:29.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.273+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79162410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.273+0000 7ff7a7fff700 1 -- 172.21.15.111:0/3417098668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 msgr2=0x563b79161ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:29.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.273+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79161ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.273+0000 7ff7a7fff700 1 -- 172.21.15.111:0/3417098668 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b7906a190 con 0x563b79152b60 2023-10-11T16:37:29.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.273+0000 7ff7a7fff700 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79161990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x563b7916cd50 tx=0x7ff798021d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33a18f7b192022d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:29.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.274+0000 7ff7877fe700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:37:29.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.275+0000 7ff7877fe700 1 -- 172.21.15.111:0/3417098668 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798017040 con 0x563b79152b60 2023-10-11T16:37:29.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.275+0000 7ff7877fe700 1 -- 172.21.15.111:0/3417098668 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff798021070 con 0x563b79152b60 2023-10-11T16:37:29.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.276+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/3417098668 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff7a0022200 con 0x563b79152b60 2023-10-11T16:37:29.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.276+0000 7ff7877fe700 1 -- 172.21.15.111:0/3417098668 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7980122d0 con 0x563b79152b60 2023-10-11T16:37:29.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.277+0000 7ff7c22ceb80 10 client.26461 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:29.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.277+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/3417098668 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563b78ed1de0 con 0x563b79152b60 2023-10-11T16:37:29.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.278+0000 7ff7877fe700 1 -- 172.21.15.111:0/3417098668 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7a0022200 con 0x563b79152b60 2023-10-11T16:37:29.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.278+0000 7ff7877fe700 1 -- 172.21.15.111:0/3417098668 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff798005020 con 0x563b79152b60 2023-10-11T16:37:29.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.278+0000 7ff7c22ceb80 10 client.26461 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:29.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.278+0000 7ff7c22ceb80 10 client.26461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:29.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.278+0000 7ff7c22ceb80 20 client.26461 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:29.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.278+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563b791c99d0 0x563b791cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:29.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.278+0000 7ff7c22ceb80 4 client.26461 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:37:29.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.278+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/3417098668 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563b79072660 con 0x563b791c99d0 2023-10-11T16:37:29.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.280+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563b791c99d0 0x563b791cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:29.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.283+0000 7ff7877fe700 10 client.26461 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:29.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.283+0000 7ff7a77fe700 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563b791c99d0 0x563b791cbde0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52fe0d39fe35dc7e server_cookie=9c3e3e1483c4db3c in_seq=0 out_seq=0 2023-10-11T16:37:29.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7877fe700 1 -- 172.21.15.111:0/3417098668 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563b79072660 con 0x563b791c99d0 2023-10-11T16:37:29.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7877fe700 10 client.26461 handle_command_reply: tid=0 2023-10-11T16:37:29.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 client.26461 shutdown 2023-10-11T16:37:29.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/3417098668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 msgr2=0x563b79161990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79161990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x563b7916cd50 tx=0x7ff798021d30 comp rx=0 tx=0).stop 2023-10-11T16:37:29.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/3417098668 shutdown_connections 2023-10-11T16:37:29.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563b791c99d0 0x563b791cbde0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563b791578b0 0x563b79162410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563b79152b60 0x563b79161990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 --2- 172.21.15.111:0/3417098668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563b79153470 0x563b79161ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7877fe700 0 client.26461 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:29.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/3417098668 >> 172.21.15.111:0/3417098668 conn(0x563b79148360 msgr2=0x563b79155930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:29.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.284+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/3417098668 shutdown_connections 2023-10-11T16:37:29.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.285+0000 7ff7c22ceb80 1 -- 172.21.15.111:0/3417098668 wait complete. 2023-10-11T16:37:29.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.285+0000 7ff7c22ceb80 20 client.26461 trim_cache size 0 max 16384 2023-10-11T16:37:29.289 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:37:29.290 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:37:29.290 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:37:29.290 INFO:teuthology.orchestra.run.smithi111.stdout: "e4b353b6-bcd8-4549-a24b-3247f2c5740c": { 2023-10-11T16:37:29.290 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:37:29.290 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "e4b353b6-bcd8-4549-a24b-3247f2c5740c", 2023-10-11T16:37:29.291 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:37:29.291 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:37:29.291 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:37:29.291 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:37:29.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.288+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff7900a12c0 msgr2=0x7ff7900a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.288+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff7900a12c0 0x7ff7900a3770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff79c000c00 tx=0x7ff79c005ee0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.288+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/3181573783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7a8105620 msgr2=0x7ff7a8198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:29.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.288+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7a8105620 0x7ff7a8198570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000efc0 tx=0x7ff7a000c3a0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.289+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/3181573783 shutdown_connections 2023-10-11T16:37:29.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.289+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff7900a12c0 0x7ff7900a3770 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.289+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7a8105620 0x7ff7a8198570 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.289+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7a8100cc0 0x7ff7a819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.289+0000 7ff7ae9ff700 1 --2- 172.21.15.111:0/3181573783 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7a8101660 0x7ff7a819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:29.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.290+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/3181573783 >> 172.21.15.111:0/3181573783 conn(0x7ff7a8077560 msgr2=0x7ff7a80feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:29.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.290+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/3181573783 shutdown_connections 2023-10-11T16:37:29.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:29.290+0000 7ff7ae9ff700 1 -- 172.21.15.111:0/3181573783 wait complete. 2023-10-11T16:37:29.326 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:29 smithi111 ceph-mon[98527]: pgmap v148: 129 pgs: 129 active+clean; 564 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 419 op/s 2023-10-11T16:37:29.332 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b353b6-bcd8-4549-a24b-3247f2c5740c - {'path': '/', 'tag': 'e4b353b6-bcd8-4549-a24b-3247f2c5740c', 'options': 'recursive,force'} 2023-10-11T16:37:29.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:29 smithi130 ceph-mon[131859]: pgmap v148: 129 pgs: 129 active+clean; 564 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 419 op/s 2023-10-11T16:37:29.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:29 smithi150 ceph-mon[132217]: pgmap v148: 129 pgs: 129 active+clean; 564 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 419 op/s 2023-10-11T16:37:31.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:31 smithi130 ceph-mon[131859]: pgmap v149: 129 pgs: 129 active+clean; 573 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.5 MiB/s wr, 668 op/s 2023-10-11T16:37:31.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:31 smithi111 ceph-mon[98527]: pgmap v149: 129 pgs: 129 active+clean; 573 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.5 MiB/s wr, 668 op/s 2023-10-11T16:37:31.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:31 smithi150 ceph-mon[132217]: pgmap v149: 129 pgs: 129 active+clean; 573 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.5 MiB/s wr, 668 op/s 2023-10-11T16:37:33.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:33 smithi130 ceph-mon[131859]: pgmap v150: 129 pgs: 129 active+clean; 573 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 516 op/s 2023-10-11T16:37:33.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:33 smithi111 ceph-mon[98527]: pgmap v150: 129 pgs: 129 active+clean; 573 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 516 op/s 2023-10-11T16:37:33.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:33 smithi150 ceph-mon[132217]: pgmap v150: 129 pgs: 129 active+clean; 573 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 516 op/s 2023-10-11T16:37:35.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:35 smithi130 ceph-mon[131859]: pgmap v151: 129 pgs: 129 active+clean; 578 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 546 op/s 2023-10-11T16:37:35.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:35 smithi111 ceph-mon[98527]: pgmap v151: 129 pgs: 129 active+clean; 578 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 546 op/s 2023-10-11T16:37:35.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:35 smithi150 ceph-mon[132217]: pgmap v151: 129 pgs: 129 active+clean; 578 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 546 op/s 2023-10-11T16:37:37.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 -- 172.21.15.111:0/1444490797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101c70 msgr2=0x7efce410a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:37.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 --2- 172.21.15.111:0/1444490797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101c70 0x7efce410a340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009990 tx=0x7efcd8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 -- 172.21.15.111:0/1444490797 shutdown_connections 2023-10-11T16:37:37.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 --2- 172.21.15.111:0/1444490797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efce410a880 0x7efce410cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 --2- 172.21.15.111:0/1444490797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce4101360 0x7efce4101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 --2- 172.21.15.111:0/1444490797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101c70 0x7efce410a340 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 -- 172.21.15.111:0/1444490797 >> 172.21.15.111:0/1444490797 conn(0x7efce4075fa0 msgr2=0x7efce40763a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:37.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 -- 172.21.15.111:0/1444490797 shutdown_connections 2023-10-11T16:37:37.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.277+0000 7efce8dff700 1 -- 172.21.15.111:0/1444490797 wait complete. 2023-10-11T16:37:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.278+0000 7efce8dff700 1 Processor -- start 2023-10-11T16:37:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.278+0000 7efce8dff700 1 -- start start 2023-10-11T16:37:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.279+0000 7efce8dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101360 0x7efce41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.283 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.279+0000 7efce8dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efce4101c70 0x7efce4198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.279+0000 7efce8dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce410a880 0x7efce419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.279+0000 7efce8dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce419d5a0 con 0x7efce4101c70 2023-10-11T16:37:37.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.279+0000 7efce8dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce419d710 con 0x7efce410a880 2023-10-11T16:37:37.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.279+0000 7efce8dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce419d8d0 con 0x7efce4101360 2023-10-11T16:37:37.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce410a880 0x7efce419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efce4101c70 0x7efce4198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce410a880 0x7efce419cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54914/0 (socket says 172.21.15.111:54914) 2023-10-11T16:37:37.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 -- 172.21.15.111:0/404562544 learned_addr learned my addr 172.21.15.111:0/404562544 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:37.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce259c700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101360 0x7efce41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 -- 172.21.15.111:0/404562544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101360 msgr2=0x7efce41985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:37.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101360 0x7efce41985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 -- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efce4101c70 msgr2=0x7efce4198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:37.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efce4101c70 0x7efce4198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 -- 172.21.15.111:0/404562544 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd80095f0 con 0x7efce410a880 2023-10-11T16:37:37.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce259c700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101360 0x7efce41985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:37:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.280+0000 7efce2d9d700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce410a880 0x7efce419cf60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00efc0 tx=0x7efcdc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.281+0000 7efcd37fe700 1 -- 172.21.15.111:0/404562544 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc018070 con 0x7efce410a880 2023-10-11T16:37:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.281+0000 7efce8dff700 1 -- 172.21.15.111:0/404562544 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce4110300 con 0x7efce410a880 2023-10-11T16:37:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.281+0000 7efcd37fe700 1 -- 172.21.15.111:0/404562544 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcdc00f040 con 0x7efce410a880 2023-10-11T16:37:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.281+0000 7efcd37fe700 1 -- 172.21.15.111:0/404562544 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc00cdb0 con 0x7efce410a880 2023-10-11T16:37:37.288 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.281+0000 7efce8dff700 1 -- 172.21.15.111:0/404562544 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce4110170 con 0x7efce410a880 2023-10-11T16:37:37.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.285+0000 7efcd37fe700 1 -- 172.21.15.111:0/404562544 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7efcdc003a00 con 0x7efce410a880 2023-10-11T16:37:37.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.286+0000 7efcfc730b80 1 Processor -- start 2023-10-11T16:37:37.289 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.286+0000 7efcd37fe700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efccc0a1340 0x7efccc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.290 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efcfc730b80 1 -- start start 2023-10-11T16:37:37.290 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efcd37fe700 1 -- 172.21.15.111:0/404562544 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7efcdc0cca60 con 0x7efce410a880 2023-10-11T16:37:37.290 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c68031e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.290 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c67fd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c67fdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efcfc730b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c6703000 con 0x5593c66596b0 2023-10-11T16:37:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efcfc730b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c6755840 con 0x5593c6803b90 2023-10-11T16:37:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efcfc730b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c674ecd0 con 0x5593c6803720 2023-10-11T16:37:37.291 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efce1d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c67fd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efce2d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c67fdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efce1d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c67fd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36626/0 (socket says 172.21.15.111:36626) 2023-10-11T16:37:37.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efce1d9b700 1 -- 172.21.15.111:0/2019992231 learned_addr learned my addr 172.21.15.111:0/2019992231 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:37.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.287+0000 7efce2d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c67fdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54926/0 (socket says 172.21.15.111:54926) 2023-10-11T16:37:37.292 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efce259c700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efccc0a1340 0x7efccc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efce2d9d700 1 -- 172.21.15.111:0/2019992231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 msgr2=0x5593c67fd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:37.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efce2d9d700 1 --2- 172.21.15.111:0/2019992231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c67fd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efce2d9d700 1 -- 172.21.15.111:0/2019992231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 msgr2=0x5593c68031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:37.297 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efce2d9d700 1 --2- 172.21.15.111:0/2019992231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c68031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efce2d9d700 1 -- 172.21.15.111:0/2019992231 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593c65d05d0 con 0x5593c6803b90 2023-10-11T16:37:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efce2d9d700 1 --2- 172.21.15.111:0/2019992231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c67fdb00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0c9e00 tx=0x7efcdc009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a40cadffe8b6bfd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efcd0ff9700 1 -- 172.21.15.111:0/2019992231 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc099770 con 0x5593c6803b90 2023-10-11T16:37:37.298 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efce259c700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efccc0a1340 0x7efccc0a37f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009d40 tx=0x7efcd4009630 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:37.299 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efcd0ff9700 1 -- 172.21.15.111:0/2019992231 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcdc09d7a0 con 0x5593c6803b90 2023-10-11T16:37:37.299 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.288+0000 7efcd0ff9700 1 -- 172.21.15.111:0/2019992231 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc096010 con 0x5593c6803b90 2023-10-11T16:37:37.299 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.289+0000 7efcfc730b80 1 -- 172.21.15.111:0/2019992231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 msgr2=0x5593c67fdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:37.299 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.289+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2019992231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c67fdb00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0c9e00 tx=0x7efcdc009f90 comp rx=0 tx=0).stop 2023-10-11T16:37:37.300 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 -- 172.21.15.111:0/2019992231 shutdown_connections 2023-10-11T16:37:37.300 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2019992231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c67fdb00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.300 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2019992231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c68031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.300 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2019992231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c67fd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.301 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 -- 172.21.15.111:0/2019992231 >> 172.21.15.111:0/2019992231 conn(0x5593c67f2f50 msgr2=0x5593c67f8270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:37.301 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 -- 172.21.15.111:0/2019992231 shutdown_connections 2023-10-11T16:37:37.301 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 -- 172.21.15.111:0/2019992231 wait complete. 2023-10-11T16:37:37.301 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 -- start start 2023-10-11T16:37:37.302 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.290+0000 7efcfc730b80 1 Processor -- start 2023-10-11T16:37:37.302 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c66596b0 0x5593c6809b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.302 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c6803720 0x5593c6806d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.302 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c6807480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efcfc730b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c66f0880 con 0x5593c6803720 2023-10-11T16:37:37.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efcfc730b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c66f0520 con 0x5593c66596b0 2023-10-11T16:37:37.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efcfc730b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c65d28c0 con 0x5593c6803b90 2023-10-11T16:37:37.303 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efcfc730b80 10 client.0 _ll_register_callbacks cb 0x5593c670e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:37.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efce259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c66596b0 0x5593c6809b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efce259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c66596b0 0x5593c6809b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54940/0 (socket says 172.21.15.111:54940) 2023-10-11T16:37:37.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efce259c700 1 -- 172.21.15.111:0/2201188662 learned_addr learned my addr 172.21.15.111:0/2201188662 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:37.304 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.291+0000 7efce1d9b700 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c6803720 0x5593c6806d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efce2d9d700 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c6807480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efce259c700 1 -- 172.21.15.111:0/2201188662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 msgr2=0x5593c6807480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:37.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efce259c700 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c6807480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efce259c700 1 -- 172.21.15.111:0/2201188662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c6803720 msgr2=0x5593c6806d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:37.305 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efce259c700 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c6803720 0x5593c6806d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efce259c700 1 -- 172.21.15.111:0/2201188662 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce4110170 con 0x5593c66596b0 2023-10-11T16:37:37.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efce259c700 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c66596b0 0x5593c6809b30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efcd40047c0 tx=0x7efcd4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e24ca4f3a2940326 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:37.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efcc9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:37:37.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efcc9ffb700 1 -- 172.21.15.111:0/2201188662 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4010970 con 0x5593c66596b0 2023-10-11T16:37:37.306 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.292+0000 7efcc9ffb700 1 -- 172.21.15.111:0/2201188662 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7efcd4010ad0 con 0x5593c66596b0 2023-10-11T16:37:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.293+0000 7efcfc730b80 1 -- 172.21.15.111:0/2201188662 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593c67f0710 con 0x5593c66596b0 2023-10-11T16:37:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.293+0000 7efcc9ffb700 1 -- 172.21.15.111:0/2201188662 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcd4010c90 con 0x5593c66596b0 2023-10-11T16:37:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.293+0000 7efcfc730b80 10 client.26444 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.293+0000 7efcfc730b80 1 -- 172.21.15.111:0/2201188662 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5593c6715130 con 0x5593c66596b0 2023-10-11T16:37:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.293+0000 7efcc9ffb700 1 -- 172.21.15.111:0/2201188662 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5593c67f0710 con 0x5593c66596b0 2023-10-11T16:37:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.294+0000 7efcc9ffb700 1 -- 172.21.15.111:0/2201188662 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7efcd401c400 con 0x5593c66596b0 2023-10-11T16:37:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.294+0000 7efcfc730b80 10 client.26444 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.294+0000 7efcfc730b80 10 client.26444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.294+0000 7efcfc730b80 20 client.26444 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.294+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593c681f930 0x5593c6821d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.294+0000 7efcfc730b80 4 client.26444 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:37:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.294+0000 7efcfc730b80 1 -- 172.21.15.111:0/2201188662 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5593c6711040 con 0x5593c681f930 2023-10-11T16:37:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.294+0000 7efce1d9b700 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593c681f930 0x5593c6821d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.295+0000 7efcc9ffb700 10 client.26444 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.295+0000 7efce1d9b700 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593c681f930 0x5593c6821d40 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc7b35ac081797f server_cookie=dbb1a4dd68ba725e in_seq=0 out_seq=0 2023-10-11T16:37:37.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.297+0000 7efcc9ffb700 1 -- 172.21.15.111:0/2201188662 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5593c6711040 con 0x5593c681f930 2023-10-11T16:37:37.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.297+0000 7efcc9ffb700 10 client.26444 handle_command_reply: tid=0 2023-10-11T16:37:37.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.297+0000 7efcfc730b80 1 client.26444 shutdown 2023-10-11T16:37:37.311 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.297+0000 7efcfc730b80 1 -- 172.21.15.111:0/2201188662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c66596b0 msgr2=0x5593c6809b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:37.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.297+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c66596b0 0x5593c6809b30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efcd40047c0 tx=0x7efcd4009260 comp rx=0 tx=0).stop 2023-10-11T16:37:37.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 1 -- 172.21.15.111:0/2201188662 shutdown_connections 2023-10-11T16:37:37.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593c681f930 0x5593c6821d40 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c6807480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c66596b0 0x5593c6809b30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 1 --2- 172.21.15.111:0/2201188662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c6803720 0x5593c6806d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 1 -- 172.21.15.111:0/2201188662 >> 172.21.15.111:0/2201188662 conn(0x5593c67f2f50 msgr2=0x5593c67f77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcc9ffb700 0 client.26444 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 1 -- 172.21.15.111:0/2201188662 shutdown_connections 2023-10-11T16:37:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 1 -- 172.21.15.111:0/2201188662 wait complete. 2023-10-11T16:37:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.298+0000 7efcfc730b80 20 client.26444 trim_cache size 0 max 16384 2023-10-11T16:37:37.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.317+0000 7efcfc730b80 1 Processor -- start 2023-10-11T16:37:37.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.317+0000 7efcfc730b80 1 -- start start 2023-10-11T16:37:37.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c67f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803720 0x5593c6809660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c67ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efcfc730b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c67f4fd0 con 0x5593c66596b0 2023-10-11T16:37:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efcfc730b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c6807a80 con 0x5593c6803720 2023-10-11T16:37:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efcfc730b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c67f9ba0 con 0x5593c6803b90 2023-10-11T16:37:37.322 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c67f7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.322 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce1d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803720 0x5593c6809660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce2d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c67ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce1d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803720 0x5593c6809660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54946/0 (socket says 172.21.15.111:54946) 2023-10-11T16:37:37.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c67f7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59178/0 (socket says 172.21.15.111:59178) 2023-10-11T16:37:37.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce1d9b700 1 -- 172.21.15.111:0/1827305760 learned_addr learned my addr 172.21.15.111:0/1827305760 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:37.323 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce2d9d700 1 -- 172.21.15.111:0/1827305760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803720 msgr2=0x5593c6809660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce2d9d700 1 --2- 172.21.15.111:0/1827305760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803720 0x5593c6809660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce2d9d700 1 -- 172.21.15.111:0/1827305760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 msgr2=0x5593c67f7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce2d9d700 1 --2- 172.21.15.111:0/1827305760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c67f7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce2d9d700 1 -- 172.21.15.111:0/1827305760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593c6714e80 con 0x5593c6803b90 2023-10-11T16:37:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.318+0000 7efce1d9b700 1 --2- 172.21.15.111:0/1827305760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803720 0x5593c6809660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:37:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.319+0000 7efce2d9d700 1 --2- 172.21.15.111:0/1827305760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c67ffc70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efcdc003f90 tx=0x7efcdc009dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0177cc4aba6043d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.319+0000 7efcd17fa700 1 -- 172.21.15.111:0/1827305760 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc099770 con 0x5593c6803b90 2023-10-11T16:37:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.319+0000 7efcd17fa700 1 -- 172.21.15.111:0/1827305760 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcdc0a1b30 con 0x5593c6803b90 2023-10-11T16:37:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.319+0000 7efcd17fa700 1 -- 172.21.15.111:0/1827305760 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc0999d0 con 0x5593c6803b90 2023-10-11T16:37:37.326 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 -- 172.21.15.111:0/1827305760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 msgr2=0x5593c67ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:37.326 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 --2- 172.21.15.111:0/1827305760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c67ffc70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efcdc003f90 tx=0x7efcdc009dc0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.326 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 -- 172.21.15.111:0/1827305760 shutdown_connections 2023-10-11T16:37:37.326 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 --2- 172.21.15.111:0/1827305760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803b90 0x5593c67ffc70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 --2- 172.21.15.111:0/1827305760 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c67f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 --2- 172.21.15.111:0/1827305760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803720 0x5593c6809660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 -- 172.21.15.111:0/1827305760 >> 172.21.15.111:0/1827305760 conn(0x5593c67f2f50 msgr2=0x5593c6800e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:37.327 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 -- 172.21.15.111:0/1827305760 shutdown_connections 2023-10-11T16:37:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 -- 172.21.15.111:0/1827305760 wait complete. 2023-10-11T16:37:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.320+0000 7efcfc730b80 1 -- start start 2023-10-11T16:37:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efcfc730b80 1 Processor -- start 2023-10-11T16:37:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c6816940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c6814920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efcfc730b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c6814e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.332 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efcfc730b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c68153a0 con 0x5593c66596b0 2023-10-11T16:37:37.332 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efcfc730b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c6815510 con 0x5593c6803b90 2023-10-11T16:37:37.332 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efcfc730b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593c68156d0 con 0x5593c6803720 2023-10-11T16:37:37.332 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efcfc730b80 10 client.0 _ll_register_callbacks cb 0x7efcd4005420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efce1d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c6814920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efce1d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c6814920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36648/0 (socket says 172.21.15.111:36648) 2023-10-11T16:37:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.321+0000 7efce1d9b700 1 -- 172.21.15.111:0/4246835398 learned_addr learned my addr 172.21.15.111:0/4246835398 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.322+0000 7efce2d9d700 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c6814e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.333 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.322+0000 7efce259c700 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c6816940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.322+0000 7efce1d9b700 1 -- 172.21.15.111:0/4246835398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 msgr2=0x5593c6814e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.322+0000 7efce1d9b700 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c6814e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.334 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.322+0000 7efce1d9b700 1 -- 172.21.15.111:0/4246835398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 msgr2=0x5593c6816940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:37.335 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.322+0000 7efce1d9b700 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c6816940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:37.335 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.322+0000 7efce1d9b700 1 -- 172.21.15.111:0/4246835398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce410d9b0 con 0x5593c6803720 2023-10-11T16:37:37.335 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.322+0000 7efce259c700 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c6816940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:37:37.335 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.323+0000 7efce1d9b700 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c6814920 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efcd800b2e0 tx=0x7efcd800bc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cefe55a67b5c5a71 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.323+0000 7efcc9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:37:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.323+0000 7efcc9ffb700 1 -- 172.21.15.111:0/4246835398 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd801d040 con 0x5593c6803720 2023-10-11T16:37:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.323+0000 7efcc9ffb700 1 -- 172.21.15.111:0/4246835398 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7efcd800be70 con 0x5593c6803720 2023-10-11T16:37:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.323+0000 7efcfc730b80 1 -- 172.21.15.111:0/4246835398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efcdc09d7a0 con 0x5593c6803720 2023-10-11T16:37:37.336 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.323+0000 7efcc9ffb700 1 -- 172.21.15.111:0/4246835398 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcd80055a0 con 0x5593c6803720 2023-10-11T16:37:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.323+0000 7efcfc730b80 10 client.26479 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.323+0000 7efcfc730b80 1 -- 172.21.15.111:0/4246835398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5593c65d05d0 con 0x5593c6803720 2023-10-11T16:37:37.337 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efcc9ffb700 1 -- 172.21.15.111:0/4246835398 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcdc09d7a0 con 0x5593c6803720 2023-10-11T16:37:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efcc9ffb700 1 -- 172.21.15.111:0/4246835398 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7efcd8004ab0 con 0x5593c6803720 2023-10-11T16:37:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efcfc730b80 10 client.26479 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efcfc730b80 10 client.26479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efcfc730b80 20 client.26479 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efcfc730b80 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593c6873ca0 0x5593c68760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efcfc730b80 4 client.26479 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:37:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efcfc730b80 1 -- 172.21.15.111:0/4246835398 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5593c671d600 con 0x5593c6873ca0 2023-10-11T16:37:37.339 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.324+0000 7efce259c700 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593c6873ca0 0x5593c68760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.325+0000 7efce259c700 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593c6873ca0 0x5593c68760b0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c35ddad9803316c server_cookie=c846ee48307316bc in_seq=0 out_seq=0 2023-10-11T16:37:37.340 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:37.325+0000 7efcc9ffb700 10 client.26479 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:37.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:37 smithi130 ceph-mon[131859]: pgmap v152: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 626 op/s 2023-10-11T16:37:37.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:37 smithi111 ceph-mon[98527]: pgmap v152: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 626 op/s 2023-10-11T16:37:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:37 smithi150 ceph-mon[132217]: pgmap v152: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 626 op/s 2023-10-11T16:37:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcc9ffb700 1 -- 172.21.15.111:0/4246835398 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5593c671d600 con 0x5593c6873ca0 2023-10-11T16:37:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcc9ffb700 10 client.26479 handle_command_reply: tid=0 2023-10-11T16:37:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 client.26479 shutdown 2023-10-11T16:37:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 -- 172.21.15.111:0/4246835398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 msgr2=0x5593c6814920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c6814920 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efcd800b2e0 tx=0x7efcd800bc90 comp rx=0 tx=0).stop 2023-10-11T16:37:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 -- 172.21.15.111:0/4246835398 shutdown_connections 2023-10-11T16:37:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593c6873ca0 0x5593c68760b0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593c6803b90 0x5593c6814e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593c66596b0 0x5593c6816940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 --2- 172.21.15.111:0/4246835398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593c6803720 0x5593c6814920 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.078+0000 7efcfc730b80 1 -- 172.21.15.111:0/4246835398 >> 172.21.15.111:0/4246835398 conn(0x5593c67f2f50 msgr2=0x5593c6801de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.079+0000 7efcc9ffb700 0 client.26479 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.079+0000 7efcfc730b80 1 -- 172.21.15.111:0/4246835398 shutdown_connections 2023-10-11T16:37:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.079+0000 7efcfc730b80 1 -- 172.21.15.111:0/4246835398 wait complete. 2023-10-11T16:37:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.080+0000 7efcfc730b80 20 client.26479 trim_cache size 0 max 16384 2023-10-11T16:37:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:37:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:37:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:37:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:37:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.081+0000 7efce8dff700 1 -- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efccc0a1340 msgr2=0x7efccc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.081+0000 7efce8dff700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efccc0a1340 0x7efccc0a37f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009d40 tx=0x7efcd4009630 comp rx=0 tx=0).stop 2023-10-11T16:37:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.081+0000 7efce8dff700 1 -- 172.21.15.111:0/404562544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce410a880 msgr2=0x7efce419cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.081+0000 7efce8dff700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce410a880 0x7efce419cf60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00efc0 tx=0x7efcdc00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.082+0000 7efce8dff700 1 -- 172.21.15.111:0/404562544 shutdown_connections 2023-10-11T16:37:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.082+0000 7efce8dff700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efccc0a1340 0x7efccc0a37f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.082+0000 7efce8dff700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce410a880 0x7efce419cf60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.082+0000 7efce8dff700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efce4101360 0x7efce41985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.082+0000 7efce8dff700 1 --2- 172.21.15.111:0/404562544 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efce4101c70 0x7efce4198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.082+0000 7efce8dff700 1 -- 172.21.15.111:0/404562544 >> 172.21.15.111:0/404562544 conn(0x7efce4075fa0 msgr2=0x7efce40ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.082+0000 7efce8dff700 1 -- 172.21.15.111:0/404562544 shutdown_connections 2023-10-11T16:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:38.082+0000 7efce8dff700 1 -- 172.21.15.111:0/404562544 wait complete. 2023-10-11T16:37:39.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:39 smithi130 ceph-mon[131859]: pgmap v153: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 412 op/s 2023-10-11T16:37:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:39 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:37:39.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:39 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:39.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:39 smithi111 ceph-mon[98527]: pgmap v153: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 412 op/s 2023-10-11T16:37:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:39 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:37:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:39 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:39 smithi150 ceph-mon[132217]: pgmap v153: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 412 op/s 2023-10-11T16:37:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:39 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:37:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:39 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:40.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:40 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:37:40.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:40 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:40.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:40 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:37:40.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:40 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:40.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:40 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:37:40.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:40 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:41 smithi130 ceph-mon[131859]: pgmap v154: 129 pgs: 129 active+clean; 604 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.2 MiB/s wr, 1.01k op/s 2023-10-11T16:37:41.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:41 smithi111 ceph-mon[98527]: pgmap v154: 129 pgs: 129 active+clean; 604 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.2 MiB/s wr, 1.01k op/s 2023-10-11T16:37:41.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:41 smithi150 ceph-mon[132217]: pgmap v154: 129 pgs: 129 active+clean; 604 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.2 MiB/s wr, 1.01k op/s 2023-10-11T16:37:43.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:43 smithi130 ceph-mon[131859]: pgmap v155: 129 pgs: 129 active+clean; 604 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 765 op/s 2023-10-11T16:37:43.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:43 smithi111 ceph-mon[98527]: pgmap v155: 129 pgs: 129 active+clean; 604 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 765 op/s 2023-10-11T16:37:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:43 smithi150 ceph-mon[132217]: pgmap v155: 129 pgs: 129 active+clean; 604 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.4 MiB/s wr, 765 op/s 2023-10-11T16:37:44.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:44 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:37:44.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:44 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:37:44.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:44 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:37:45.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:45 smithi130 ceph-mon[131859]: pgmap v156: 129 pgs: 129 active+clean; 606 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 821 op/s 2023-10-11T16:37:45.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:45 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:45 smithi111 ceph-mon[98527]: pgmap v156: 129 pgs: 129 active+clean; 606 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 821 op/s 2023-10-11T16:37:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:45 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:45 smithi150 ceph-mon[132217]: pgmap v156: 129 pgs: 129 active+clean; 606 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 821 op/s 2023-10-11T16:37:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:45 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:47.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:47 smithi130 ceph-mon[131859]: pgmap v157: 129 pgs: 129 active+clean; 622 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 1.06k op/s 2023-10-11T16:37:47.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:47 smithi111 ceph-mon[98527]: pgmap v157: 129 pgs: 129 active+clean; 622 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 1.06k op/s 2023-10-11T16:37:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:47 smithi150 ceph-mon[132217]: pgmap v157: 129 pgs: 129 active+clean; 622 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 1.06k op/s 2023-10-11T16:37:49.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:49 smithi130 ceph-mon[131859]: pgmap v158: 129 pgs: 129 active+clean; 622 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 924 op/s 2023-10-11T16:37:49.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:49 smithi111 ceph-mon[98527]: pgmap v158: 129 pgs: 129 active+clean; 622 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 924 op/s 2023-10-11T16:37:49.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:49 smithi150 ceph-mon[132217]: pgmap v158: 129 pgs: 129 active+clean; 622 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 924 op/s 2023-10-11T16:37:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.352+0000 7fd7c3fff700 1 -- 172.21.15.111:0/3114596062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 msgr2=0x7fd7bc10eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.352+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/3114596062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 0x7fd7bc10eb50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80098f0 tx=0x7fd7b8009c00 comp rx=0 tx=0).stop 2023-10-11T16:37:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.354+0000 7fd7c3fff700 1 -- 172.21.15.111:0/3114596062 shutdown_connections 2023-10-11T16:37:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.354+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/3114596062 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7bc119a40 0x7fd7bc11be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.354+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/3114596062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 0x7fd7bc10eb50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.354+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/3114596062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7bc107ec0 0x7fd7bc108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.354+0000 7fd7c3fff700 1 -- 172.21.15.111:0/3114596062 >> 172.21.15.111:0/3114596062 conn(0x7fd7bc06c510 msgr2=0x7fd7bc0716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.355+0000 7fd7c3fff700 1 -- 172.21.15.111:0/3114596062 shutdown_connections 2023-10-11T16:37:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.355+0000 7fd7c3fff700 1 -- 172.21.15.111:0/3114596062 wait complete. 2023-10-11T16:37:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.357+0000 7fd7c3fff700 1 Processor -- start 2023-10-11T16:37:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.358+0000 7fd7c3fff700 1 -- start start 2023-10-11T16:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.361+0000 7fd7c3fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 0x7fd7bc1149c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.361+0000 7fd7c3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7bc107ec0 0x7fd7bc114f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.361+0000 7fd7c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7bc119a40 0x7fd7bc10fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.362+0000 7fd7c159a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7bc107ec0 0x7fd7bc114f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.362+0000 7fd7c159a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7bc107ec0 0x7fd7bc114f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40230/0 (socket says 172.21.15.111:40230) 2023-10-11T16:37:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.362+0000 7fd7c159a700 1 -- 172.21.15.111:0/1386532344 learned_addr learned my addr 172.21.15.111:0/1386532344 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.361+0000 7fd7c1d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 0x7fd7bc1149c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.362+0000 7fd7c259c700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7bc119a40 0x7fd7bc10fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.362+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc110180 con 0x7fd7bc119a40 2023-10-11T16:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.362+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc1102f0 con 0x7fd7bc107af0 2023-10-11T16:37:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.362+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7bc1104b0 con 0x7fd7bc107ec0 2023-10-11T16:37:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.363+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/1386532344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7bc107ec0 msgr2=0x7fd7bc114f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.363+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7bc107ec0 0x7fd7bc114f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.363+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7bc119a40 msgr2=0x7fd7bc10fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.363+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7bc119a40 0x7fd7bc10fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.363+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/1386532344 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b8009520 con 0x7fd7bc107af0 2023-10-11T16:37:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.364+0000 7fd7c159a700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7bc107ec0 0x7fd7bc114f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:37:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.364+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 0x7fd7bc1149c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8000c00 tx=0x7fd7b800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.365+0000 7fd7b2ffd700 1 -- 172.21.15.111:0/1386532344 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8005240 con 0x7fd7bc107af0 2023-10-11T16:37:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.365+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7bc11f040 con 0x7fd7bc107af0 2023-10-11T16:37:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.366+0000 7fd7b2ffd700 1 -- 172.21.15.111:0/1386532344 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7b8003680 con 0x7fd7bc107af0 2023-10-11T16:37:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.366+0000 7fd7b2ffd700 1 -- 172.21.15.111:0/1386532344 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8005240 con 0x7fd7bc107af0 2023-10-11T16:37:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.366+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7bc11eeb0 con 0x7fd7bc107af0 2023-10-11T16:37:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.375+0000 7fd7b2ffd700 1 -- 172.21.15.111:0/1386532344 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fd7b80053a0 con 0x7fd7bc107af0 2023-10-11T16:37:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.381+0000 7fd7b2ffd700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7a80a13a0 0x7fd7a80a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.382+0000 7fd7b2ffd700 1 -- 172.21.15.111:0/1386532344 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd7b80cdf90 con 0x7fd7bc107af0 2023-10-11T16:37:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7d7a05b80 1 Processor -- start 2023-10-11T16:37:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7d7a05b80 1 -- start start 2023-10-11T16:37:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7c159a700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7a80a13a0 0x7fd7a80a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f30207e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207e720 0x55f302078560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207eb90 0x55f302078b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f301f7e000 con 0x55f30207e720 2023-10-11T16:37:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f301fd0840 con 0x55f301ed46b0 2023-10-11T16:37:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.383+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f301fc9cd0 con 0x55f30207eb90 2023-10-11T16:37:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207eb90 0x55f302078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207eb90 0x55f302078b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40232/0 (socket says 172.21.15.111:40232) 2023-10-11T16:37:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c259c700 1 -- 172.21.15.111:0/665795479 learned_addr learned my addr 172.21.15.111:0/665795479 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/665795479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f30207e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c259c700 1 -- 172.21.15.111:0/665795479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 msgr2=0x55f30207e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c259c700 1 --2- 172.21.15.111:0/665795479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f30207e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c259c700 1 -- 172.21.15.111:0/665795479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207e720 msgr2=0x55f302078560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:37:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c259c700 1 --2- 172.21.15.111:0/665795479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207e720 0x55f302078560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.384+0000 7fd7c259c700 1 -- 172.21.15.111:0/665795479 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f301e4b5d0 con 0x55f30207eb90 2023-10-11T16:37:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.385+0000 7fd7c259c700 1 --2- 172.21.15.111:0/665795479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207eb90 0x55f302078b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400f410 tx=0x7fd7b400f720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94cc5d3f96ccd707 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.385+0000 7fd7a7fff700 1 -- 172.21.15.111:0/665795479 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4004bb0 con 0x55f30207eb90 2023-10-11T16:37:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.385+0000 7fd7a7fff700 1 -- 172.21.15.111:0/665795479 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7b4004d10 con 0x55f30207eb90 2023-10-11T16:37:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.385+0000 7fd7a7fff700 1 -- 172.21.15.111:0/665795479 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4005020 con 0x55f30207eb90 2023-10-11T16:37:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.387+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/665795479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207eb90 msgr2=0x55f302078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.387+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/665795479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207eb90 0x55f302078b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400f410 tx=0x7fd7b400f720 comp rx=0 tx=0).stop 2023-10-11T16:37:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.387+0000 7fd7c159a700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7a80a13a0 0x7fd7a80a3850 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc118cd0 tx=0x7fd7ac00b4e0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.387+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/665795479 shutdown_connections 2023-10-11T16:37:51.390 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.387+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/665795479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207eb90 0x55f302078b00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.387+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/665795479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f30207e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.387+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/665795479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207e720 0x55f302078560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.387+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/665795479 >> 172.21.15.111:0/665795479 conn(0x55f30206df50 msgr2=0x55f302073270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.388+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/665795479 shutdown_connections 2023-10-11T16:37:51.391 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.388+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/665795479 wait complete. 2023-10-11T16:37:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.388+0000 7fd7d7a05b80 1 -- start start 2023-10-11T16:37:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.389+0000 7fd7d7a05b80 1 Processor -- start 2023-10-11T16:37:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f302084ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f302081ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.393 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 0x55f302082420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f301f6b880 con 0x55f30207eb90 2023-10-11T16:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f301f6b520 con 0x55f301ed46b0 2023-10-11T16:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f301e4d8c0 con 0x55f30207e720 2023-10-11T16:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7d7a05b80 10 client.0 _ll_register_callbacks cb 0x55f301f89720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:51.394 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7c1d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f302084ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7c1d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f302084ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:34218/0 (socket says 172.21.15.111:34218) 2023-10-11T16:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.390+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/3490398763 learned_addr learned my addr 172.21.15.111:0/3490398763 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.391+0000 7fd7c159a700 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f302081ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.391+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/3490398763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 msgr2=0x55f302081ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:51.395 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.391+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f302081ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.391+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/3490398763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 msgr2=0x55f302082420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.391+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 0x55f302082420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.391+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/3490398763 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b4007e50 con 0x55f301ed46b0 2023-10-11T16:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.392+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f302084ad0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80cb3e0 tx=0x7fd7b809ae70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=276670654f20d7e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:51.396 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.392+0000 7fd7a57fa700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.392+0000 7fd7a57fa700 1 -- 172.21.15.111:0/3490398763 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b809afa0 con 0x55f301ed46b0 2023-10-11T16:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.392+0000 7fd7a57fa700 1 -- 172.21.15.111:0/3490398763 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd7b809b100 con 0x55f301ed46b0 2023-10-11T16:37:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.392+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/3490398763 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f30206b710 con 0x55f301ed46b0 2023-10-11T16:37:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.392+0000 7fd7a57fa700 1 -- 172.21.15.111:0/3490398763 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7b80b3260 con 0x55f301ed46b0 2023-10-11T16:37:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.394+0000 7fd7d7a05b80 10 client.26462 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.394+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/3490398763 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f301f90130 con 0x55f301ed46b0 2023-10-11T16:37:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.394+0000 7fd7a57fa700 1 -- 172.21.15.111:0/3490398763 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7b80b9080 con 0x55f301ed46b0 2023-10-11T16:37:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.395+0000 7fd7a57fa700 1 -- 172.21.15.111:0/3490398763 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fd7b80a2c70 con 0x55f301ed46b0 2023-10-11T16:37:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.395+0000 7fd7d7a05b80 10 client.26462 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.395+0000 7fd7d7a05b80 10 client.26462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.395+0000 7fd7d7a05b80 20 client.26462 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.395+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f302091c40 0x55f302094050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.395+0000 7fd7d7a05b80 4 client.26462 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.395+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/3490398763 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f301f8c040 con 0x55f302091c40 2023-10-11T16:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.396+0000 7fd7c159a700 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f302091c40 0x55f302094050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.396+0000 7fd7c159a700 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f302091c40 0x55f302094050 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4988036fdb49f75 server_cookie=12f4cc68bf5d2673 in_seq=0 out_seq=0 2023-10-11T16:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.396+0000 7fd7a57fa700 10 client.26462 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.397+0000 7fd7a57fa700 1 -- 172.21.15.111:0/3490398763 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f301f8c040 con 0x55f302091c40 2023-10-11T16:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.397+0000 7fd7a57fa700 10 client.26462 handle_command_reply: tid=0 2023-10-11T16:37:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.397+0000 7fd7d7a05b80 1 client.26462 shutdown 2023-10-11T16:37:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.398+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/3490398763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 msgr2=0x55f302084ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.398+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f302084ad0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80cb3e0 tx=0x7fd7b809ae70 comp rx=0 tx=0).stop 2023-10-11T16:37:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.398+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/3490398763 shutdown_connections 2023-10-11T16:37:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.398+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f302091c40 0x55f302094050 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.398+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 0x55f302082420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.398+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f301ed46b0 0x55f302084ad0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.403 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.398+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/3490398763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f302081ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.398+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/3490398763 >> 172.21.15.111:0/3490398763 conn(0x55f30206df50 msgr2=0x55f3020727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.399+0000 7fd7a57fa700 0 client.26462 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.399+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/3490398763 shutdown_connections 2023-10-11T16:37:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.400+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/3490398763 wait complete. 2023-10-11T16:37:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.402+0000 7fd7d7a05b80 20 client.26462 trim_cache size 0 max 16384 2023-10-11T16:37:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7d7a05b80 1 Processor -- start 2023-10-11T16:37:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7d7a05b80 1 -- start start 2023-10-11T16:37:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207e720 0x55f302089090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 0x55f302072f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f302084600 0x55f302084a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f302074000 con 0x55f30207eb90 2023-10-11T16:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f302079330 con 0x55f30207e720 2023-10-11T16:37:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f30206b300 con 0x55f302084600 2023-10-11T16:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7c1d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207e720 0x55f302089090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7c159a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 0x55f302072f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7c259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f302084600 0x55f302084a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7c1d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207e720 0x55f302089090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:34224/0 (socket says 172.21.15.111:34224) 2023-10-11T16:37:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.432+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/2946623492 learned_addr learned my addr 172.21.15.111:0/2946623492 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/2946623492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f302084600 msgr2=0x55f302084a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/2946623492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f302084600 0x55f302084a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/2946623492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 msgr2=0x55f302072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/2946623492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 0x55f302072f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.438 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7c1d9b700 1 -- 172.21.15.111:0/2946623492 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f30206f6f0 con 0x55f30207e720 2023-10-11T16:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/2946623492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207e720 0x55f302089090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8004fd0 tx=0x7fd7b80992f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c33064f63318fa server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7b0ff9700 1 -- 172.21.15.111:0/2946623492 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b80b9080 con 0x55f30207e720 2023-10-11T16:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7b0ff9700 1 -- 172.21.15.111:0/2946623492 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f30206b710 con 0x55f30207e720 2023-10-11T16:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.433+0000 7fd7b0ff9700 1 -- 172.21.15.111:0/2946623492 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b80a2c70 con 0x55f30207e720 2023-10-11T16:37:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.434+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/2946623492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207e720 msgr2=0x55f302089090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.434+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/2946623492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207e720 0x55f302089090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8004fd0 tx=0x7fd7b80992f0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.434+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/2946623492 shutdown_connections 2023-10-11T16:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.434+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/2946623492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f302084600 0x55f302084a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.434+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/2946623492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207e720 0x55f302089090 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.440 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.434+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/2946623492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f30207eb90 0x55f302072f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.434+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/2946623492 >> 172.21.15.111:0/2946623492 conn(0x55f30206df50 msgr2=0x55f302093c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.435+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/2946623492 shutdown_connections 2023-10-11T16:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.435+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/2946623492 wait complete. 2023-10-11T16:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.435+0000 7fd7d7a05b80 1 -- start start 2023-10-11T16:37:51.441 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.435+0000 7fd7d7a05b80 1 Processor -- start 2023-10-11T16:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f3020819f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207eb90 0x55f302088a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7d7a05b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f302084600 0x55f302088f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3020894d0 con 0x55f302084600 2023-10-11T16:37:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f302089610 con 0x55f30207eb90 2023-10-11T16:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7d7a05b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f302080240 con 0x55f30207e720 2023-10-11T16:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7d7a05b80 10 client.0 _ll_register_callbacks cb 0x7fd7b809a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7c159a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207eb90 0x55f302088a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7c259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f302084600 0x55f302088f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7c159a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207eb90 0x55f302088a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:34230/0 (socket says 172.21.15.111:34230) 2023-10-11T16:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.436+0000 7fd7c159a700 1 -- 172.21.15.111:0/1703594381 learned_addr learned my addr 172.21.15.111:0/1703594381 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f3020819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7c159a700 1 -- 172.21.15.111:0/1703594381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 msgr2=0x55f3020819f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:51.444 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7c159a700 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f3020819f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7c159a700 1 -- 172.21.15.111:0/1703594381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f302084600 msgr2=0x55f302088f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7c159a700 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f302084600 0x55f302088f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7c159a700 1 -- 172.21.15.111:0/1703594381 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f301f90130 con 0x55f30207eb90 2023-10-11T16:37:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f3020819f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:37:51.445 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7c159a700 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207eb90 0x55f302088a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac010ce0 tx=0x7fd7ac013070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=995fac5a4adc8a87 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:51.446 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7a57fa700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:37:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.437+0000 7fd7a57fa700 1 -- 172.21.15.111:0/1703594381 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac025cc0 con 0x55f30207eb90 2023-10-11T16:37:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7a57fa700 1 -- 172.21.15.111:0/1703594381 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fd7ac017070 con 0x55f30207eb90 2023-10-11T16:37:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/1703594381 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7b4004d10 con 0x55f30207eb90 2023-10-11T16:37:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7a57fa700 1 -- 172.21.15.111:0/1703594381 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7ac025e20 con 0x55f30207eb90 2023-10-11T16:37:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 10 client.26468 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/1703594381 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f301df7d80 con 0x55f30207eb90 2023-10-11T16:37:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7a57fa700 1 -- 172.21.15.111:0/1703594381 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7b4004d10 con 0x55f30207eb90 2023-10-11T16:37:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7a57fa700 1 -- 172.21.15.111:0/1703594381 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fd7ac02e8b0 con 0x55f30207eb90 2023-10-11T16:37:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 10 client.26468 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 10 client.26468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 20 client.26468 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f3020e6310 0x55f3020e8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 4 client.26468 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:37:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.438+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/1703594381 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f301f98600 con 0x55f3020e6310 2023-10-11T16:37:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.439+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f3020e6310 0x55f3020e8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.439+0000 7fd7a57fa700 10 client.26468 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:51.439+0000 7fd7c1d9b700 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f3020e6310 0x55f3020e8720 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae07d336cf5af29a server_cookie=895745175b99e84 in_seq=0 out_seq=0 2023-10-11T16:37:51.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:51 smithi130 ceph-mon[131859]: pgmap v159: 129 pgs: 129 active+clean; 636 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 1.34k op/s 2023-10-11T16:37:51.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:51 smithi111 ceph-mon[98527]: pgmap v159: 129 pgs: 129 active+clean; 636 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 1.34k op/s 2023-10-11T16:37:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:51 smithi150 ceph-mon[132217]: pgmap v159: 129 pgs: 129 active+clean; 636 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 1.34k op/s 2023-10-11T16:37:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.246+0000 7fd7a57fa700 1 -- 172.21.15.111:0/1703594381 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f301f98600 con 0x55f3020e6310 2023-10-11T16:37:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.246+0000 7fd7a57fa700 10 client.26468 handle_command_reply: tid=0 2023-10-11T16:37:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.246+0000 7fd7d7a05b80 1 client.26468 shutdown 2023-10-11T16:37:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/1703594381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207eb90 msgr2=0x55f302088a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207eb90 0x55f302088a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac010ce0 tx=0x7fd7ac013070 comp rx=0 tx=0).stop 2023-10-11T16:37:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/1703594381 shutdown_connections 2023-10-11T16:37:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f3020e6310 0x55f3020e8720 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f302084600 0x55f302088f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f30207e720 0x55f3020819f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7d7a05b80 1 --2- 172.21.15.111:0/1703594381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f30207eb90 0x55f302088a00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/1703594381 >> 172.21.15.111:0/1703594381 conn(0x55f30206df50 msgr2=0x55f30207bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.247+0000 7fd7a57fa700 0 client.26468 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.248+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/1703594381 shutdown_connections 2023-10-11T16:37:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.248+0000 7fd7d7a05b80 1 -- 172.21.15.111:0/1703594381 wait complete. 2023-10-11T16:37:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.248+0000 7fd7d7a05b80 20 client.26468 trim_cache size 0 max 16384 2023-10-11T16:37:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:37:53.251 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:37:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:37:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:37:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.254+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7a80a13a0 msgr2=0x7fd7a80a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.254+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7a80a13a0 0x7fd7a80a3850 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc118cd0 tx=0x7fd7ac00b4e0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.254+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 msgr2=0x7fd7bc1149c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.254+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 0x7fd7bc1149c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8000c00 tx=0x7fd7b800b8c0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.255+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 shutdown_connections 2023-10-11T16:37:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.255+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd7a80a13a0 0x7fd7a80a3850 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.255+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7bc119a40 0x7fd7bc10fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.255+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7bc107af0 0x7fd7bc1149c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.255+0000 7fd7c3fff700 1 --2- 172.21.15.111:0/1386532344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7bc107ec0 0x7fd7bc114f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.255+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 >> 172.21.15.111:0/1386532344 conn(0x7fd7bc06c510 msgr2=0x7fd7bc11a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.256+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 shutdown_connections 2023-10-11T16:37:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:37:53.256+0000 7fd7c3fff700 1 -- 172.21.15.111:0/1386532344 wait complete. 2023-10-11T16:37:53.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:53 smithi111 ceph-mon[98527]: pgmap v160: 129 pgs: 129 active+clean; 636 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 737 op/s 2023-10-11T16:37:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:53 smithi150 ceph-mon[132217]: pgmap v160: 129 pgs: 129 active+clean; 636 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 737 op/s 2023-10-11T16:37:53.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:53 smithi130 ceph-mon[131859]: pgmap v160: 129 pgs: 129 active+clean; 636 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 737 op/s 2023-10-11T16:37:55.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:55 smithi111 ceph-mon[98527]: pgmap v161: 129 pgs: 129 active+clean; 640 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 815 op/s 2023-10-11T16:37:55.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:55 smithi150 ceph-mon[132217]: pgmap v161: 129 pgs: 129 active+clean; 640 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 815 op/s 2023-10-11T16:37:55.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:55 smithi130 ceph-mon[131859]: pgmap v161: 129 pgs: 129 active+clean; 640 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 815 op/s 2023-10-11T16:37:57.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:57 smithi111 ceph-mon[98527]: pgmap v162: 129 pgs: 129 active+clean; 640 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.12k op/s 2023-10-11T16:37:57.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:57 smithi150 ceph-mon[132217]: pgmap v162: 129 pgs: 129 active+clean; 640 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.12k op/s 2023-10-11T16:37:57.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:57 smithi130 ceph-mon[131859]: pgmap v162: 129 pgs: 129 active+clean; 640 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.12k op/s 2023-10-11T16:37:58.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:58 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:37:58.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:58 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:37:58.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:58 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:37:59.333 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:37:59.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.596+0000 7fa21cdff700 1 -- 172.21.15.111:0/1712117462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa218107090 msgr2=0x7fa2181136e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:59.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.596+0000 7fa21cdff700 1 --2- 172.21.15.111:0/1712117462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa218107090 0x7fa2181136e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa21000b0a0 tx=0x7fa21000b3b0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.597+0000 7fa21cdff700 1 -- 172.21.15.111:0/1712117462 shutdown_connections 2023-10-11T16:37:59.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.597+0000 7fa21cdff700 1 --2- 172.21.15.111:0/1712117462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa218107090 0x7fa2181136e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.597+0000 7fa21cdff700 1 --2- 172.21.15.111:0/1712117462 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa21810c920 0x7fa21810ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.597+0000 7fa21cdff700 1 --2- 172.21.15.111:0/1712117462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa21810d230 0x7fa218106960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.597+0000 7fa21cdff700 1 -- 172.21.15.111:0/1712117462 >> 172.21.15.111:0/1712117462 conn(0x7fa2180738c0 msgr2=0x7fa218075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:59.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.597+0000 7fa21cdff700 1 -- 172.21.15.111:0/1712117462 shutdown_connections 2023-10-11T16:37:59.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.597+0000 7fa21cdff700 1 -- 172.21.15.111:0/1712117462 wait complete. 2023-10-11T16:37:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.599+0000 7fa21cdff700 1 Processor -- start 2023-10-11T16:37:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.599+0000 7fa21cdff700 1 -- start start 2023-10-11T16:37:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.600+0000 7fa21cdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa218107090 0x7fa21806fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.600+0000 7fa21cdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21810c920 0x7fa218070300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.601+0000 7fa21cdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa21810d230 0x7fa21806af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.601+0000 7fa21cdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21806b5f0 con 0x7fa218107090 2023-10-11T16:37:59.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.601+0000 7fa21cdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21806b760 con 0x7fa21810d230 2023-10-11T16:37:59.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.601+0000 7fa21cdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21806b920 con 0x7fa21810c920 2023-10-11T16:37:59.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.601+0000 7fa21659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa218107090 0x7fa21806fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.601+0000 7fa21659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa218107090 0x7fa21806fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58878/0 (socket says 172.21.15.111:58878) 2023-10-11T16:37:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.601+0000 7fa21659c700 1 -- 172.21.15.111:0/2049127481 learned_addr learned my addr 172.21.15.111:0/2049127481 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.602+0000 7fa216d9d700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa21810d230 0x7fa21806af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.602+0000 7fa21659c700 1 -- 172.21.15.111:0/2049127481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21810c920 msgr2=0x7fa218070300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-11T16:37:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.602+0000 7fa21659c700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21810c920 0x7fa218070300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.602+0000 7fa21659c700 1 -- 172.21.15.111:0/2049127481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa21810d230 msgr2=0x7fa21806af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.602+0000 7fa21659c700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa21810d230 0x7fa21806af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.602+0000 7fa21659c700 1 -- 172.21.15.111:0/2049127481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa210009d00 con 0x7fa218107090 2023-10-11T16:37:59.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.603+0000 7fa21659c700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa218107090 0x7fa21806fdc0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa20800d980 tx=0x7fa20800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:59.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.603+0000 7fa20f7fe700 1 -- 172.21.15.111:0/2049127481 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208004220 con 0x7fa218107090 2023-10-11T16:37:59.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.604+0000 7fa20f7fe700 1 -- 172.21.15.111:0/2049127481 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa208004cf0 con 0x7fa218107090 2023-10-11T16:37:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.604+0000 7fa20f7fe700 1 -- 172.21.15.111:0/2049127481 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208005720 con 0x7fa218107090 2023-10-11T16:37:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.604+0000 7fa21cdff700 1 -- 172.21.15.111:0/2049127481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa218116980 con 0x7fa218107090 2023-10-11T16:37:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.604+0000 7fa21cdff700 1 -- 172.21.15.111:0/2049127481 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2181167f0 con 0x7fa218107090 2023-10-11T16:37:59.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.615+0000 7fa20f7fe700 1 -- 172.21.15.111:0/2049127481 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fa208004410 con 0x7fa218107090 2023-10-11T16:37:59.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.617+0000 7fa230787b80 1 Processor -- start 2023-10-11T16:37:59.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.619+0000 7fa230787b80 1 -- start start 2023-10-11T16:37:59.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.619+0000 7fa20f7fe700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2040a1400 0x7fa2040a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.619+0000 7fa230787b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a87139bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.620+0000 7fa230787b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 0x55a8713a0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.620+0000 7fa230787b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a8713a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.620+0000 7fa230787b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8712a1060 con 0x55a87139c470 2023-10-11T16:37:59.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.620+0000 7fa230787b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8712f38a0 con 0x55a8713a08b0 2023-10-11T16:37:59.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.620+0000 7fa230787b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8712ecd30 con 0x55a87139bb60 2023-10-11T16:37:59.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.620+0000 7fa20f7fe700 1 -- 172.21.15.111:0/2049127481 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa2080cced0 con 0x7fa218107090 2023-10-11T16:37:59.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.620+0000 7fa216d9d700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2040a1400 0x7fa2040a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.621+0000 7fa21659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a87139bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.622+0000 7fa216d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a8713a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.622+0000 7fa21659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a87139bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48316/0 (socket says 172.21.15.111:48316) 2023-10-11T16:37:59.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.622+0000 7fa21659c700 1 -- 172.21.15.111:0/3802947971 learned_addr learned my addr 172.21.15.111:0/3802947971 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:59.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.622+0000 7fa20ffff700 1 --2- 172.21.15.111:0/3802947971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 0x55a8713a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.623+0000 7fa20ffff700 1 -- 172.21.15.111:0/3802947971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 msgr2=0x55a87139bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:59.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.623+0000 7fa20ffff700 1 --2- 172.21.15.111:0/3802947971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a87139bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.623+0000 7fa20ffff700 1 -- 172.21.15.111:0/3802947971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 msgr2=0x55a8713a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:59.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.623+0000 7fa20ffff700 1 --2- 172.21.15.111:0/3802947971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a8713a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.623+0000 7fa20ffff700 1 -- 172.21.15.111:0/3802947971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a87116e630 con 0x55a87139c470 2023-10-11T16:37:59.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.624+0000 7fa20ffff700 1 --2- 172.21.15.111:0/3802947971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 0x55a8713a0370 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa21806c730 tx=0x7fa20000aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2540f6c4030e4a2b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.625+0000 7fa20cff9700 1 -- 172.21.15.111:0/3802947971 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200004220 con 0x55a87139c470 2023-10-11T16:37:59.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.625+0000 7fa20cff9700 1 -- 172.21.15.111:0/3802947971 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa200004380 con 0x55a87139c470 2023-10-11T16:37:59.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.625+0000 7fa20cff9700 1 -- 172.21.15.111:0/3802947971 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200003b60 con 0x55a87139c470 2023-10-11T16:37:59.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.626+0000 7fa230787b80 1 -- 172.21.15.111:0/3802947971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 msgr2=0x55a8713a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:59.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.626+0000 7fa216d9d700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2040a1400 0x7fa2040a38b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa210009cd0 tx=0x7fa21000b7d0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:59.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.626+0000 7fa230787b80 1 --2- 172.21.15.111:0/3802947971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 0x55a8713a0370 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa21806c730 tx=0x7fa20000aba0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.627+0000 7fa230787b80 1 -- 172.21.15.111:0/3802947971 shutdown_connections 2023-10-11T16:37:59.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.627+0000 7fa230787b80 1 --2- 172.21.15.111:0/3802947971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a8713a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.627+0000 7fa230787b80 1 --2- 172.21.15.111:0/3802947971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a87139bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.627+0000 7fa230787b80 1 --2- 172.21.15.111:0/3802947971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 0x55a8713a0370 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.627+0000 7fa230787b80 1 -- 172.21.15.111:0/3802947971 >> 172.21.15.111:0/3802947971 conn(0x55a871391360 msgr2=0x55a871396b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.628+0000 7fa230787b80 1 -- 172.21.15.111:0/3802947971 shutdown_connections 2023-10-11T16:37:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.628+0000 7fa230787b80 1 -- 172.21.15.111:0/3802947971 wait complete. 2023-10-11T16:37:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.629+0000 7fa230787b80 1 -- start start 2023-10-11T16:37:59.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.629+0000 7fa230787b80 1 Processor -- start 2023-10-11T16:37:59.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.631+0000 7fa230787b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139bb60 0x55a8713a7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.631+0000 7fa230787b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713a5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.631+0000 7fa230787b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a8713a08b0 0x55a8713a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.631+0000 7fa230787b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a87128e8e0 con 0x55a87139bb60 2023-10-11T16:37:59.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.631+0000 7fa230787b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a87128e580 con 0x55a87139c470 2023-10-11T16:37:59.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.631+0000 7fa230787b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a871170920 con 0x55a8713a08b0 2023-10-11T16:37:59.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.631+0000 7fa230787b80 10 client.0 _ll_register_callbacks cb 0x55a8712ac780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:59.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.632+0000 7fa20ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.632+0000 7fa20ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713a5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:34766/0 (socket says 172.21.15.111:34766) 2023-10-11T16:37:59.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.632+0000 7fa20ffff700 1 -- 172.21.15.111:0/4071589460 learned_addr learned my addr 172.21.15.111:0/4071589460 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:59.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.633+0000 7fa20ffff700 1 -- 172.21.15.111:0/4071589460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a8713a08b0 msgr2=0x55a8713a57d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:37:59.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.633+0000 7fa20ffff700 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a8713a08b0 0x55a8713a57d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.633+0000 7fa20ffff700 1 -- 172.21.15.111:0/4071589460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139bb60 msgr2=0x55a8713a7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.633+0000 7fa20ffff700 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139bb60 0x55a8713a7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.633+0000 7fa20ffff700 1 -- 172.21.15.111:0/4071589460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20000a520 con 0x55a87139c470 2023-10-11T16:37:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.634+0000 7fa1fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:37:59.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.634+0000 7fa20ffff700 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713a5210 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa20000a4b0 tx=0x7fa20000ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4a5135e50a40d5e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.634+0000 7fa1fdffb700 1 -- 172.21.15.111:0/4071589460 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20001f070 con 0x55a87139c470 2023-10-11T16:37:59.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.634+0000 7fa1fdffb700 1 -- 172.21.15.111:0/4071589460 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa200004830 con 0x55a87139c470 2023-10-11T16:37:59.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:59 smithi111 ceph-mon[98527]: pgmap v163: 129 pgs: 129 active+clean; 640 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 851 op/s 2023-10-11T16:37:59.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:37:59 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:59.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.635+0000 7fa230787b80 1 -- 172.21.15.111:0/4071589460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a87138eb20 con 0x55a87139c470 2023-10-11T16:37:59.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.636+0000 7fa1fdffb700 1 -- 172.21.15.111:0/4071589460 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa20001f070 con 0x55a87139c470 2023-10-11T16:37:59.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.637+0000 7fa230787b80 10 client.26477 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:59.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.637+0000 7fa230787b80 1 -- 172.21.15.111:0/4071589460 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a8712b3190 con 0x55a87139c470 2023-10-11T16:37:59.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.637+0000 7fa1fdffb700 1 -- 172.21.15.111:0/4071589460 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa200005020 con 0x55a87139c470 2023-10-11T16:37:59.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.637+0000 7fa1fdffb700 1 -- 172.21.15.111:0/4071589460 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa200016030 con 0x55a87139c470 2023-10-11T16:37:59.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.638+0000 7fa230787b80 10 client.26477 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:59.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.638+0000 7fa230787b80 10 client.26477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:59.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.638+0000 7fa230787b80 20 client.26477 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:59.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.638+0000 7fa230787b80 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a8713b5750 0x55a8713b7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.638+0000 7fa230787b80 4 client.26477 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:37:59.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.638+0000 7fa230787b80 1 -- 172.21.15.111:0/4071589460 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a87138eb20 con 0x55a8713b5750 2023-10-11T16:37:59.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.639+0000 7fa21659c700 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a8713b5750 0x55a8713b7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.640+0000 7fa1fdffb700 10 client.26477 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:59.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.640+0000 7fa21659c700 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a8713b5750 0x55a8713b7ba0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=144a6d26735e9718 server_cookie=5df11a0088012c9a in_seq=0 out_seq=0 2023-10-11T16:37:59.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.644+0000 7fa1fdffb700 1 -- 172.21.15.111:0/4071589460 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a87138eb20 con 0x55a8713b5750 2023-10-11T16:37:59.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.644+0000 7fa1fdffb700 10 client.26477 handle_command_reply: tid=0 2023-10-11T16:37:59.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.645+0000 7fa230787b80 1 client.26477 shutdown 2023-10-11T16:37:59.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.646+0000 7fa230787b80 1 -- 172.21.15.111:0/4071589460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 msgr2=0x55a8713a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:59.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.646+0000 7fa230787b80 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713a5210 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa20000a4b0 tx=0x7fa20000ca40 comp rx=0 tx=0).stop 2023-10-11T16:37:59.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.646+0000 7fa230787b80 1 -- 172.21.15.111:0/4071589460 shutdown_connections 2023-10-11T16:37:59.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.646+0000 7fa230787b80 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a8713b5750 0x55a8713b7ba0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.648+0000 7fa230787b80 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a8713a08b0 0x55a8713a57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.648+0000 7fa230787b80 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139bb60 0x55a8713a7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.648+0000 7fa230787b80 1 --2- 172.21.15.111:0/4071589460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713a5210 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.648+0000 7fa1fdffb700 0 client.26477 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:59.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.648+0000 7fa230787b80 1 -- 172.21.15.111:0/4071589460 >> 172.21.15.111:0/4071589460 conn(0x55a871391360 msgr2=0x55a8713a1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:59.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.648+0000 7fa230787b80 1 -- 172.21.15.111:0/4071589460 shutdown_connections 2023-10-11T16:37:59.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.648+0000 7fa230787b80 1 -- 172.21.15.111:0/4071589460 wait complete. 2023-10-11T16:37:59.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.649+0000 7fa230787b80 20 client.26477 trim_cache size 0 max 16384 2023-10-11T16:37:59.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.692+0000 7fa230787b80 1 Processor -- start 2023-10-11T16:37:59.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.692+0000 7fa230787b80 1 -- start start 2023-10-11T16:37:59.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.692+0000 7fa230787b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a8713ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.692+0000 7fa230787b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 0x55a8713a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.692+0000 7fa230787b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a87139e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.692+0000 7fa230787b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8713a67f0 con 0x55a87139c470 2023-10-11T16:37:59.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.692+0000 7fa230787b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8713a8730 con 0x55a8713a08b0 2023-10-11T16:37:59.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.692+0000 7fa230787b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a871390ee0 con 0x55a87139bb60 2023-10-11T16:37:59.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.693+0000 7fa216d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a87139e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.693+0000 7fa216d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a87139e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:34778/0 (socket says 172.21.15.111:34778) 2023-10-11T16:37:59.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.693+0000 7fa216d9d700 1 -- 172.21.15.111:0/1078864415 learned_addr learned my addr 172.21.15.111:0/1078864415 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:59.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.694+0000 7fa216d9d700 1 -- 172.21.15.111:0/1078864415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 msgr2=0x55a8713ac9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:37:59.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.694+0000 7fa216d9d700 1 --2- 172.21.15.111:0/1078864415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a8713ac9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.694+0000 7fa216d9d700 1 -- 172.21.15.111:0/1078864415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 msgr2=0x55a8713a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:59.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.694+0000 7fa216d9d700 1 --2- 172.21.15.111:0/1078864415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 0x55a8713a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.694+0000 7fa216d9d700 1 -- 172.21.15.111:0/1078864415 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8712b2ee0 con 0x55a8713a08b0 2023-10-11T16:37:59.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.695+0000 7fa216d9d700 1 --2- 172.21.15.111:0/1078864415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a87139e8f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa210009470 tx=0x7fa21000e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4598ad9b3ef9cdca server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:59.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.696+0000 7fa20d7fa700 1 -- 172.21.15.111:0/1078864415 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21000e070 con 0x55a8713a08b0 2023-10-11T16:37:59.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.696+0000 7fa20d7fa700 1 -- 172.21.15.111:0/1078864415 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2100079a0 con 0x55a8713a08b0 2023-10-11T16:37:59.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.696+0000 7fa20d7fa700 1 -- 172.21.15.111:0/1078864415 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210005040 con 0x55a8713a08b0 2023-10-11T16:37:59.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.698+0000 7fa230787b80 1 -- 172.21.15.111:0/1078864415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 msgr2=0x55a87139e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:59.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.698+0000 7fa230787b80 1 --2- 172.21.15.111:0/1078864415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a87139e8f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa210009470 tx=0x7fa21000e000 comp rx=0 tx=0).stop 2023-10-11T16:37:59.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.699+0000 7fa230787b80 1 -- 172.21.15.111:0/1078864415 shutdown_connections 2023-10-11T16:37:59.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.699+0000 7fa230787b80 1 --2- 172.21.15.111:0/1078864415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a8713a08b0 0x55a87139e8f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.699+0000 7fa230787b80 1 --2- 172.21.15.111:0/1078864415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a8713ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.699+0000 7fa230787b80 1 --2- 172.21.15.111:0/1078864415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a87139c470 0x55a8713a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.699+0000 7fa230787b80 1 -- 172.21.15.111:0/1078864415 >> 172.21.15.111:0/1078864415 conn(0x55a871391360 msgr2=0x55a8713b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:59.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.700+0000 7fa230787b80 1 -- 172.21.15.111:0/1078864415 shutdown_connections 2023-10-11T16:37:59.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.700+0000 7fa230787b80 1 -- 172.21.15.111:0/1078864415 wait complete. 2023-10-11T16:37:59.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.701+0000 7fa230787b80 1 -- start start 2023-10-11T16:37:59.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.701+0000 7fa230787b80 1 Processor -- start 2023-10-11T16:37:59.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:59 smithi150 ceph-mon[132217]: pgmap v163: 129 pgs: 129 active+clean; 640 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 851 op/s 2023-10-11T16:37:59.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:37:59 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:37:59.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.702+0000 7fa230787b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a8713aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.702+0000 7fa230787b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.702+0000 7fa230787b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a8713a08b0 0x55a871408650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.702+0000 7fa230787b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8713ab0e0 con 0x55a8713a08b0 2023-10-11T16:37:59.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.702+0000 7fa230787b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8713ab250 con 0x55a87139c470 2023-10-11T16:37:59.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.702+0000 7fa230787b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a871408c10 con 0x55a87139bb60 2023-10-11T16:37:59.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.703+0000 7fa230787b80 10 client.0 _ll_register_callbacks cb 0x7fa208004e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:37:59.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.703+0000 7fa216d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a8713a08b0 0x55a871408650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.703+0000 7fa216d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a8713a08b0 0x55a871408650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58916/0 (socket says 172.21.15.111:58916) 2023-10-11T16:37:59.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.703+0000 7fa216d9d700 1 -- 172.21.15.111:0/3364572603 learned_addr learned my addr 172.21.15.111:0/3364572603 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:37:59.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.703+0000 7fa20ffff700 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.704+0000 7fa216d9d700 1 -- 172.21.15.111:0/3364572603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 msgr2=0x55a8713aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:59.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.704+0000 7fa216d9d700 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a8713aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.704+0000 7fa216d9d700 1 -- 172.21.15.111:0/3364572603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 msgr2=0x55a8713aa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:37:59.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.704+0000 7fa216d9d700 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713aa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.704+0000 7fa216d9d700 1 -- 172.21.15.111:0/3364572603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa218113fa0 con 0x55a8713a08b0 2023-10-11T16:37:59.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.705+0000 7fa216d9d700 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a8713a08b0 0x55a871408650 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa21000cfd0 tx=0x7fa210020a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700a74bc3cc3ad83 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:37:59.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.705+0000 7fa1fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:37:59.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.705+0000 7fa1fdffb700 1 -- 172.21.15.111:0/3364572603 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210020860 con 0x55a8713a08b0 2023-10-11T16:37:59.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.706+0000 7fa230787b80 1 -- 172.21.15.111:0/3364572603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa200004380 con 0x55a8713a08b0 2023-10-11T16:37:59.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.706+0000 7fa1fdffb700 1 -- 172.21.15.111:0/3364572603 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa210021e90 con 0x55a8713a08b0 2023-10-11T16:37:59.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.707+0000 7fa1fdffb700 1 -- 172.21.15.111:0/3364572603 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa210003a00 con 0x55a8713a08b0 2023-10-11T16:37:59.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.708+0000 7fa230787b80 10 client.16368 fetch_fsmap learned FSMap version 33 2023-10-11T16:37:59.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.708+0000 7fa230787b80 1 -- 172.21.15.111:0/3364572603 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a871103c60 con 0x55a8713a08b0 2023-10-11T16:37:59.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.708+0000 7fa1fdffb700 1 -- 172.21.15.111:0/3364572603 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa200004380 con 0x55a8713a08b0 2023-10-11T16:37:59.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.709+0000 7fa1fdffb700 1 -- 172.21.15.111:0/3364572603 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa21004e070 con 0x55a8713a08b0 2023-10-11T16:37:59.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.709+0000 7fa230787b80 10 client.16368 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:37:59.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.709+0000 7fa230787b80 10 client.16368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:37:59.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.709+0000 7fa230787b80 20 client.16368 populate_metadata read hostname 'smithi111' 2023-10-11T16:37:59.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.709+0000 7fa230787b80 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a87140a2c0 0x55a87140c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:37:59.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.709+0000 7fa230787b80 4 client.16368 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:37:59.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.709+0000 7fa230787b80 1 -- 172.21.15.111:0/3364572603 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a8712af0a0 con 0x55a87140a2c0 2023-10-11T16:37:59.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.710+0000 7fa21659c700 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a87140a2c0 0x55a87140c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:37:59.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.712+0000 7fa21659c700 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a87140a2c0 0x55a87140c6d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaa2a9b38fdfab33 server_cookie=2d9ddb2df6ca9763 in_seq=0 out_seq=0 2023-10-11T16:37:59.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.712+0000 7fa1fdffb700 10 client.16368 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:59.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.713+0000 7fa1fdffb700 1 -- 172.21.15.111:0/3364572603 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a8712af0a0 con 0x55a87140a2c0 2023-10-11T16:37:59.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.713+0000 7fa1fdffb700 10 client.16368 handle_command_reply: tid=0 2023-10-11T16:37:59.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.713+0000 7fa230787b80 1 client.16368 shutdown 2023-10-11T16:37:59.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.713+0000 7fa230787b80 1 -- 172.21.15.111:0/3364572603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a8713a08b0 msgr2=0x55a871408650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:59.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.713+0000 7fa230787b80 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a8713a08b0 0x55a871408650 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa21000cfd0 tx=0x7fa210020a20 comp rx=0 tx=0).stop 2023-10-11T16:37:59.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa230787b80 1 -- 172.21.15.111:0/3364572603 shutdown_connections 2023-10-11T16:37:59.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa230787b80 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55a87140a2c0 0x55a87140c6d0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa230787b80 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a8713a08b0 0x55a871408650 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa230787b80 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a87139bb60 0x55a8713aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa230787b80 1 --2- 172.21.15.111:0/3364572603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a87139c470 0x55a8713aa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa230787b80 1 -- 172.21.15.111:0/3364572603 >> 172.21.15.111:0/3364572603 conn(0x55a871391360 msgr2=0x55a87139e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:59.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa1fdffb700 0 client.16368 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:37:59.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa230787b80 1 -- 172.21.15.111:0/3364572603 shutdown_connections 2023-10-11T16:37:59.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.714+0000 7fa230787b80 1 -- 172.21.15.111:0/3364572603 wait complete. 2023-10-11T16:37:59.717 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:37:59.718 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:37:59.718 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:37:59.718 INFO:teuthology.orchestra.run.smithi111.stdout: "e4b353b6-bcd8-4549-a24b-3247f2c5740c": { 2023-10-11T16:37:59.718 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:37:59.719 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "e4b353b6-bcd8-4549-a24b-3247f2c5740c", 2023-10-11T16:37:59.719 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:37:59.719 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:37:59.719 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:37:59.719 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:37:59.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.715+0000 7fa230787b80 20 client.16368 trim_cache size 0 max 16384 2023-10-11T16:37:59.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.720+0000 7fa21cdff700 1 -- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2040a1400 msgr2=0x7fa2040a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:59.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.720+0000 7fa21cdff700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2040a1400 0x7fa2040a38b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa210009cd0 tx=0x7fa21000b7d0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.720+0000 7fa21cdff700 1 -- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa218107090 msgr2=0x7fa21806fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:37:59.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.720+0000 7fa21cdff700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa218107090 0x7fa21806fdc0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa20800d980 tx=0x7fa20800dc90 comp rx=0 tx=0).stop 2023-10-11T16:37:59.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.721+0000 7fa21cdff700 1 -- 172.21.15.111:0/2049127481 shutdown_connections 2023-10-11T16:37:59.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.721+0000 7fa21cdff700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2040a1400 0x7fa2040a38b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.721+0000 7fa21cdff700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa21810d230 0x7fa21806af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.721+0000 7fa21cdff700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa218107090 0x7fa21806fdc0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.721+0000 7fa21cdff700 1 --2- 172.21.15.111:0/2049127481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa21810c920 0x7fa218070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:37:59.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.721+0000 7fa21cdff700 1 -- 172.21.15.111:0/2049127481 >> 172.21.15.111:0/2049127481 conn(0x7fa2180738c0 msgr2=0x7fa218074f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:37:59.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.721+0000 7fa21cdff700 1 -- 172.21.15.111:0/2049127481 shutdown_connections 2023-10-11T16:37:59.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:37:59.722+0000 7fa21cdff700 1 -- 172.21.15.111:0/2049127481 wait complete. 2023-10-11T16:37:59.772 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b353b6-bcd8-4549-a24b-3247f2c5740c - {'path': '/', 'tag': 'e4b353b6-bcd8-4549-a24b-3247f2c5740c', 'options': 'recursive,force'} 2023-10-11T16:37:59.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:59 smithi130 ceph-mon[131859]: pgmap v163: 129 pgs: 129 active+clean; 640 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 851 op/s 2023-10-11T16:37:59.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:37:59 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:00.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:00 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:00.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:00 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:00.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:00 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:00.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:00 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:00.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:00 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:00.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:00 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:01.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:01 smithi111 ceph-mon[98527]: pgmap v164: 129 pgs: 129 active+clean; 653 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 1.12k op/s 2023-10-11T16:38:01.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:01 smithi150 ceph-mon[132217]: pgmap v164: 129 pgs: 129 active+clean; 653 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 1.12k op/s 2023-10-11T16:38:01.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:01 smithi130 ceph-mon[131859]: pgmap v164: 129 pgs: 129 active+clean; 653 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 1.12k op/s 2023-10-11T16:38:03.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:03 smithi111 ceph-mon[98527]: pgmap v165: 129 pgs: 129 active+clean; 653 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 706 op/s 2023-10-11T16:38:03.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:03 smithi150 ceph-mon[132217]: pgmap v165: 129 pgs: 129 active+clean; 653 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 706 op/s 2023-10-11T16:38:03.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:03 smithi130 ceph-mon[131859]: pgmap v165: 129 pgs: 129 active+clean; 653 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 706 op/s 2023-10-11T16:38:05.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:05 smithi111 ceph-mon[98527]: pgmap v166: 129 pgs: 129 active+clean; 657 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 791 op/s 2023-10-11T16:38:05.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:05 smithi150 ceph-mon[132217]: pgmap v166: 129 pgs: 129 active+clean; 657 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 791 op/s 2023-10-11T16:38:05.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:05 smithi130 ceph-mon[131859]: pgmap v166: 129 pgs: 129 active+clean; 657 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 791 op/s 2023-10-11T16:38:06.546 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.546+0000 7f2dc339d700 1 -- 172.21.15.111:0/1959052798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db4099330 msgr2=0x7f2db4099700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.546+0000 7f2dc339d700 1 --2- 172.21.15.111:0/1959052798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db4099330 0x7f2db4099700 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2dac0099c0 tx=0x7f2dac009cd0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.547+0000 7f2dc339d700 1 -- 172.21.15.111:0/1959052798 shutdown_connections 2023-10-11T16:38:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.547+0000 7f2dc339d700 1 --2- 172.21.15.111:0/1959052798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2db409aa20 0x7f2db40a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.547+0000 7f2dc339d700 1 --2- 172.21.15.111:0/1959052798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db4099330 0x7f2db4099700 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.547+0000 7f2dc339d700 1 --2- 172.21.15.111:0/1959052798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db409a070 0x7f2db409a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.547+0000 7f2dc339d700 1 -- 172.21.15.111:0/1959052798 >> 172.21.15.111:0/1959052798 conn(0x7f2db400a4a0 msgr2=0x7f2db400a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:06.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.548+0000 7f2dc339d700 1 -- 172.21.15.111:0/1959052798 shutdown_connections 2023-10-11T16:38:06.549 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.548+0000 7f2dc339d700 1 -- 172.21.15.111:0/1959052798 wait complete. 2023-10-11T16:38:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.549+0000 7f2dc339d700 1 Processor -- start 2023-10-11T16:38:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.550+0000 7f2dc339d700 1 -- start start 2023-10-11T16:38:06.551 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.550+0000 7f2dc339d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db409a070 0x7f2db40a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.550+0000 7f2dc339d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2db409aa20 0x7f2db40a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.550+0000 7f2dc339d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40a27f0 0x7f2db4139450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.551+0000 7f2dc339d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db4139b30 con 0x7f2db40a27f0 2023-10-11T16:38:06.552 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.551+0000 7f2dc339d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db4139ca0 con 0x7f2db409aa20 2023-10-11T16:38:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.551+0000 7f2dc339d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db4139e60 con 0x7f2db409a070 2023-10-11T16:38:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.551+0000 7f2dc1b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2db409aa20 0x7f2db40a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.551+0000 7f2dc1b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2db409aa20 0x7f2db40a22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:47184/0 (socket says 172.21.15.111:47184) 2023-10-11T16:38:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.551+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/2354156222 learned_addr learned my addr 172.21.15.111:0/2354156222 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.551+0000 7f2dc239b700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db409a070 0x7f2db40a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.551+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40a27f0 0x7f2db4139450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.552+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/2354156222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db409a070 msgr2=0x7f2db40a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.552+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db409a070 0x7f2db40a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.552+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40a27f0 msgr2=0x7f2db4139450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.552+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40a27f0 0x7f2db4139450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.552+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/2354156222 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac0095f0 con 0x7f2db409aa20 2023-10-11T16:38:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.552+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2db409aa20 0x7f2db40a22b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2db800c9b0 tx=0x7f2db800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.553+0000 7f2db37fe700 1 -- 172.21.15.111:0/2354156222 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8007b00 con 0x7f2db409aa20 2023-10-11T16:38:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.553+0000 7f2db37fe700 1 -- 172.21.15.111:0/2354156222 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2db8007c60 con 0x7f2db409aa20 2023-10-11T16:38:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.553+0000 7f2dc339d700 1 -- 172.21.15.111:0/2354156222 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db40a54f0 con 0x7f2db409aa20 2023-10-11T16:38:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.553+0000 7f2db37fe700 1 -- 172.21.15.111:0/2354156222 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8018660 con 0x7f2db409aa20 2023-10-11T16:38:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.553+0000 7f2dc339d700 1 -- 172.21.15.111:0/2354156222 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db40a5360 con 0x7f2db409aa20 2023-10-11T16:38:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.560+0000 7f2db37fe700 1 -- 172.21.15.111:0/2354156222 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2db800f450 con 0x7f2db409aa20 2023-10-11T16:38:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.564+0000 7f2dd7e4bb80 1 Processor -- start 2023-10-11T16:38:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.564+0000 7f2db37fe700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2da80a1380 0x7f2da80a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.565+0000 7f2db37fe700 1 -- 172.21.15.111:0/2354156222 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2db80cd050 con 0x7f2db409aa20 2023-10-11T16:38:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.565+0000 7f2dd7e4bb80 1 -- start start 2023-10-11T16:38:06.568 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.565+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca15550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.565+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.565+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca0e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.565+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2c916000 con 0x564c2c86c6b0 2023-10-11T16:38:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.565+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2c968840 con 0x564c2ca0dcd0 2023-10-11T16:38:06.569 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.565+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2c961cd0 con 0x564c2ca15a90 2023-10-11T16:38:06.570 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.566+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2da80a1380 0x7f2da80a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.570 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc1b9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.570 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc1b9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41600/0 (socket says 172.21.15.111:41600) 2023-10-11T16:38:06.570 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/1238920040 learned_addr learned my addr 172.21.15.111:0/1238920040 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:06.570 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca0e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.571 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc239b700 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca15550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.572 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/1238920040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 msgr2=0x564c2ca0e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca0e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/1238920040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 msgr2=0x564c2ca15550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.567+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca15550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.568+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/1238920040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c2c7e35d0 con 0x564c2ca15a90 2023-10-11T16:38:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.568+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0d790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2db8096270 tx=0x7f2db80996d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1a7e66885737b40 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.569+0000 7f2db0bfe700 1 -- 172.21.15.111:0/1238920040 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db80a21c0 con 0x564c2ca15a90 2023-10-11T16:38:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.569+0000 7f2db0bfe700 1 -- 172.21.15.111:0/1238920040 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2db800de00 con 0x564c2ca15a90 2023-10-11T16:38:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.569+0000 7f2db0bfe700 1 -- 172.21.15.111:0/1238920040 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db800f960 con 0x564c2ca15a90 2023-10-11T16:38:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.570+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1238920040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 msgr2=0x564c2ca0d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.570+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0d790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2db8096270 tx=0x7f2db80996d0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.572+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1238920040 shutdown_connections 2023-10-11T16:38:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.572+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca0e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.572+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca15550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.572+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1238920040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0d790 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.572+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1238920040 >> 172.21.15.111:0/1238920040 conn(0x564c2ca061a0 msgr2=0x564c2ca0b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.572+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2da80a1380 0x7f2da80a3830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc04f620 tx=0x7f2dbc04ef60 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.572+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1238920040 shutdown_connections 2023-10-11T16:38:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.573+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1238920040 wait complete. 2023-10-11T16:38:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.574+0000 7f2dd7e4bb80 1 -- start start 2023-10-11T16:38:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.577+0000 7f2dd7e4bb80 1 Processor -- start 2023-10-11T16:38:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca1eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.578 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca0dcd0 0x564c2ca1f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca15a90 0x564c2ca19e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2c903880 con 0x564c2c86c6b0 2023-10-11T16:38:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2c903520 con 0x564c2ca15a90 2023-10-11T16:38:06.579 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2c7e58c0 con 0x564c2ca0dcd0 2023-10-11T16:38:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dd7e4bb80 10 client.0 _ll_register_callbacks cb 0x564c2c921720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dc1b9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca0dcd0 0x564c2ca1f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dc1b9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca0dcd0 0x564c2ca1f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41610/0 (socket says 172.21.15.111:41610) 2023-10-11T16:38:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/1522235622 learned_addr learned my addr 172.21.15.111:0/1522235622 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.578+0000 7f2dc239b700 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca1eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.579+0000 7f2dc239b700 1 -- 172.21.15.111:0/1522235622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca0dcd0 msgr2=0x564c2ca1f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.579+0000 7f2dc239b700 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca0dcd0 0x564c2ca1f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.579+0000 7f2dc239b700 1 -- 172.21.15.111:0/1522235622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca15a90 msgr2=0x564c2ca19e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:38:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.579+0000 7f2dc239b700 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca15a90 0x564c2ca19e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.579+0000 7f2dc239b700 1 -- 172.21.15.111:0/1522235622 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db40a5360 con 0x564c2c86c6b0 2023-10-11T16:38:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.580+0000 7f2dc239b700 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca1eba0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00b580 tx=0x7f2dac00ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52765fc069388a54 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.580+0000 7f2da5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:38:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.581+0000 7f2da5ffb700 1 -- 172.21.15.111:0/1522235622 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac01c030 con 0x564c2c86c6b0 2023-10-11T16:38:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.581+0000 7f2da5ffb700 1 -- 172.21.15.111:0/1522235622 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2dac01d070 con 0x564c2c86c6b0 2023-10-11T16:38:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.581+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1522235622 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564c2ca03960 con 0x564c2c86c6b0 2023-10-11T16:38:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.581+0000 7f2da5ffb700 1 -- 172.21.15.111:0/1522235622 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dac00bea0 con 0x564c2c86c6b0 2023-10-11T16:38:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.581+0000 7f2dd7e4bb80 10 client.16377 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.582+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1522235622 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564c2c928130 con 0x564c2c86c6b0 2023-10-11T16:38:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.583+0000 7f2da5ffb700 1 -- 172.21.15.111:0/1522235622 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564c2ca03960 con 0x564c2c86c6b0 2023-10-11T16:38:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.585+0000 7f2da5ffb700 1 -- 172.21.15.111:0/1522235622 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f2dac022070 con 0x564c2c86c6b0 2023-10-11T16:38:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.585+0000 7f2dd7e4bb80 10 client.16377 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.586+0000 7f2dd7e4bb80 10 client.16377 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.586+0000 7f2dd7e4bb80 20 client.16377 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.586+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564c2ca2a090 0x564c2ca2c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.586+0000 7f2dd7e4bb80 4 client.16377 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:38:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.586+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1522235622 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c2c924040 con 0x564c2ca2a090 2023-10-11T16:38:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.587+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564c2ca2a090 0x564c2ca2c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.588+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564c2ca2a090 0x564c2ca2c4a0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9da36b5a312ec4e7 server_cookie=ab23387dfe198b28 in_seq=0 out_seq=0 2023-10-11T16:38:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.588+0000 7f2da5ffb700 10 client.16377 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.591+0000 7f2da5ffb700 1 -- 172.21.15.111:0/1522235622 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564c2c924040 con 0x564c2ca2a090 2023-10-11T16:38:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.591+0000 7f2da5ffb700 10 client.16377 handle_command_reply: tid=0 2023-10-11T16:38:06.592 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.591+0000 7f2dd7e4bb80 1 client.16377 shutdown 2023-10-11T16:38:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.592+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1522235622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 msgr2=0x564c2ca1eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.592+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca1eba0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00b580 tx=0x7f2dac00ba70 comp rx=0 tx=0).stop 2023-10-11T16:38:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1522235622 shutdown_connections 2023-10-11T16:38:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564c2ca2a090 0x564c2ca2c4a0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca15a90 0x564c2ca19e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2c86c6b0 0x564c2ca1eba0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/1522235622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca0dcd0 0x564c2ca1f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2da5ffb700 0 client.16377 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1522235622 >> 172.21.15.111:0/1522235622 conn(0x564c2ca061a0 msgr2=0x564c2ca0aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1522235622 shutdown_connections 2023-10-11T16:38:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/1522235622 wait complete. 2023-10-11T16:38:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.594+0000 7f2dd7e4bb80 20 client.16377 trim_cache size 0 max 16384 2023-10-11T16:38:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.628+0000 7f2dd7e4bb80 1 Processor -- start 2023-10-11T16:38:06.628 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.628+0000 7f2dd7e4bb80 1 -- start start 2023-10-11T16:38:06.629 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.629+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2c86c6b0 0x564c2ca2c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.629+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca0dcd0 0x564c2ca0e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.629+0000 7f2dc239b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2c86c6b0 0x564c2ca2c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.629+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.629+0000 7f2dc239b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2c86c6b0 0x564c2ca2c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:47204/0 (socket says 172.21.15.111:47204) 2023-10-11T16:38:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.629+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2ca0c1b0 con 0x564c2ca0dcd0 2023-10-11T16:38:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.629+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2ca0edc0 con 0x564c2c86c6b0 2023-10-11T16:38:06.631 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.629+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2ca1b4d0 con 0x564c2ca15a90 2023-10-11T16:38:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.630+0000 7f2dc2b9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.631+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/912752225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca0dcd0 0x564c2ca0e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.630+0000 7f2dc239b700 1 -- 172.21.15.111:0/912752225 learned_addr learned my addr 172.21.15.111:0/912752225 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.631+0000 7f2dc2b9c700 1 -- 172.21.15.111:0/912752225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2c86c6b0 msgr2=0x564c2ca2c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.631+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/912752225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2c86c6b0 0x564c2ca2c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.631+0000 7f2dc2b9c700 1 -- 172.21.15.111:0/912752225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca0dcd0 msgr2=0x564c2ca0e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.631+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/912752225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca0dcd0 0x564c2ca0e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.631+0000 7f2dc2b9c700 1 -- 172.21.15.111:0/912752225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c2c927e80 con 0x564c2ca15a90 2023-10-11T16:38:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.633+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/912752225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0b180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc04fde0 tx=0x7f2dbc066da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41219b85064d4cda server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.634+0000 7f2db13ff700 1 -- 172.21.15.111:0/912752225 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc073ea0 con 0x564c2ca15a90 2023-10-11T16:38:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.634+0000 7f2db13ff700 1 -- 172.21.15.111:0/912752225 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dbc0aac60 con 0x564c2ca15a90 2023-10-11T16:38:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.634+0000 7f2db13ff700 1 -- 172.21.15.111:0/912752225 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0b2670 con 0x564c2ca15a90 2023-10-11T16:38:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.636+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/912752225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 msgr2=0x564c2ca0b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.636+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/912752225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0b180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc04fde0 tx=0x7f2dbc066da0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.636+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/912752225 shutdown_connections 2023-10-11T16:38:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.636+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/912752225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2ca15a90 0x564c2ca0b180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.636+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/912752225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2c86c6b0 0x564c2ca2c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.636+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/912752225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca0dcd0 0x564c2ca0e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.636+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/912752225 >> 172.21.15.111:0/912752225 conn(0x564c2ca061a0 msgr2=0x564c2ca32070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.636+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/912752225 shutdown_connections 2023-10-11T16:38:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.637+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/912752225 wait complete. 2023-10-11T16:38:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.637+0000 7f2dd7e4bb80 1 -- start start 2023-10-11T16:38:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.638+0000 7f2dd7e4bb80 1 Processor -- start 2023-10-11T16:38:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.639+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2c86c6b0 0x564c2ca236f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.639+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca104a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.639+0000 7f2dd7e4bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca15a90 0x564c2ca109e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.640+0000 7f2dc1b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca104a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.640+0000 7f2dc1b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca104a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:47206/0 (socket says 172.21.15.111:47206) 2023-10-11T16:38:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.640+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2ca10f90 con 0x564c2ca15a90 2023-10-11T16:38:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.640+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2ca11100 con 0x564c2ca0dcd0 2023-10-11T16:38:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.640+0000 7f2dd7e4bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c2ca112c0 con 0x564c2c86c6b0 2023-10-11T16:38:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.640+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/3219733489 learned_addr learned my addr 172.21.15.111:0/3219733489 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.640+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca15a90 0x564c2ca109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.640+0000 7f2dd7e4bb80 10 client.0 _ll_register_callbacks cb 0x7f2dac005bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/3219733489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2c86c6b0 msgr2=0x564c2ca236f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2c86c6b0 0x564c2ca236f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/3219733489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca15a90 msgr2=0x564c2ca109e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca15a90 0x564c2ca109e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2dc1b9a700 1 -- 172.21.15.111:0/3219733489 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dbc09f050 con 0x564c2ca0dcd0 2023-10-11T16:38:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2dc2b9c700 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca15a90 0x564c2ca109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:38:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2dc1b9a700 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca104a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2db800dfd0 tx=0x7f2db800ff30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=368d62329e1ef55a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2da5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:38:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2da5ffb700 1 -- 172.21.15.111:0/3219733489 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db8098110 con 0x564c2ca0dcd0 2023-10-11T16:38:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2da5ffb700 1 -- 172.21.15.111:0/3219733489 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f2db8096480 con 0x564c2ca0dcd0 2023-10-11T16:38:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.641+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/3219733489 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2db800de00 con 0x564c2ca0dcd0 2023-10-11T16:38:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.642+0000 7f2da5ffb700 1 -- 172.21.15.111:0/3219733489 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2db8096640 con 0x564c2ca0dcd0 2023-10-11T16:38:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.642+0000 7f2dd7e4bb80 10 client.26498 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:06.646 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.642+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/3219733489 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564c2c778c00 con 0x564c2ca0dcd0 2023-10-11T16:38:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.642+0000 7f2da5ffb700 1 -- 172.21.15.111:0/3219733489 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2db800de00 con 0x564c2ca0dcd0 2023-10-11T16:38:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.643+0000 7f2da5ffb700 1 -- 172.21.15.111:0/3219733489 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f2db80993a0 con 0x564c2ca0dcd0 2023-10-11T16:38:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.644+0000 7f2dd7e4bb80 10 client.26498 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.644+0000 7f2dd7e4bb80 10 client.26498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.644+0000 7f2dd7e4bb80 20 client.26498 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.644+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564c2ca7e350 0x564c2ca80760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.644+0000 7f2dd7e4bb80 4 client.26498 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:38:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.644+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/3219733489 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564c2c930600 con 0x564c2ca7e350 2023-10-11T16:38:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.645+0000 7f2dc239b700 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564c2ca7e350 0x564c2ca80760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.650+0000 7f2dc239b700 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564c2ca7e350 0x564c2ca80760 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d10676423a655bf9 server_cookie=21103c07dd69d5e1 in_seq=0 out_seq=0 2023-10-11T16:38:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:06.650+0000 7f2da5ffb700 10 client.26498 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.038+0000 7f2da5ffb700 1 -- 172.21.15.111:0/3219733489 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564c2c930600 con 0x564c2ca7e350 2023-10-11T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.038+0000 7f2da5ffb700 10 client.26498 handle_command_reply: tid=0 2023-10-11T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.038+0000 7f2dd7e4bb80 1 client.26498 shutdown 2023-10-11T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/3219733489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 msgr2=0x564c2ca104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca104a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2db800dfd0 tx=0x7f2db800ff30 comp rx=0 tx=0).stop 2023-10-11T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/3219733489 shutdown_connections 2023-10-11T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564c2ca7e350 0x564c2ca80760 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564c2ca15a90 0x564c2ca109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564c2c86c6b0 0x564c2ca236f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 --2- 172.21.15.111:0/3219733489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c2ca0dcd0 0x564c2ca104a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/3219733489 >> 172.21.15.111:0/3219733489 conn(0x564c2ca061a0 msgr2=0x564c2ca25f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2da5ffb700 0 client.26498 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/3219733489 shutdown_connections 2023-10-11T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.039+0000 7f2dd7e4bb80 1 -- 172.21.15.111:0/3219733489 wait complete. 2023-10-11T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.040+0000 7f2dd7e4bb80 20 client.26498 trim_cache size 0 max 16384 2023-10-11T16:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:38:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:38:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.042+0000 7f2dc339d700 1 -- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2da80a1380 msgr2=0x7f2da80a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.042+0000 7f2dc339d700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2da80a1380 0x7f2da80a3830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc04f620 tx=0x7f2dbc04ef60 comp rx=0 tx=0).stop 2023-10-11T16:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.042+0000 7f2dc339d700 1 -- 172.21.15.111:0/2354156222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2db409aa20 msgr2=0x7f2db40a22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.042+0000 7f2dc339d700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2db409aa20 0x7f2db40a22b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2db800c9b0 tx=0x7f2db800ccc0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.044+0000 7f2dc339d700 1 -- 172.21.15.111:0/2354156222 shutdown_connections 2023-10-11T16:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.044+0000 7f2dc339d700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2da80a1380 0x7f2da80a3830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.044+0000 7f2dc339d700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db40a27f0 0x7f2db4139450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.044+0000 7f2dc339d700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db409a070 0x7f2db40a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.044+0000 7f2dc339d700 1 --2- 172.21.15.111:0/2354156222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2db409aa20 0x7f2db40a22b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:07.047 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.044+0000 7f2dc339d700 1 -- 172.21.15.111:0/2354156222 >> 172.21.15.111:0/2354156222 conn(0x7f2db400a4a0 msgr2=0x7f2db40a0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.045+0000 7f2dc339d700 1 -- 172.21.15.111:0/2354156222 shutdown_connections 2023-10-11T16:38:07.048 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:07.045+0000 7f2dc339d700 1 -- 172.21.15.111:0/2354156222 wait complete. 2023-10-11T16:38:07.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:07 smithi111 ceph-mon[98527]: pgmap v167: 129 pgs: 129 active+clean; 665 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 885 op/s 2023-10-11T16:38:07.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:07 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:38:07.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:07 smithi150 ceph-mon[132217]: pgmap v167: 129 pgs: 129 active+clean; 665 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 885 op/s 2023-10-11T16:38:07.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:07 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:38:07.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:07 smithi130 ceph-mon[131859]: pgmap v167: 129 pgs: 129 active+clean; 665 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 885 op/s 2023-10-11T16:38:07.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:07 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:38:08.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:08 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:08.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:08 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:08.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:08 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:08.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:08 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:08.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:08 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:08.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:08 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:09.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:09 smithi150 ceph-mon[132217]: pgmap v168: 129 pgs: 129 active+clean; 665 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 527 op/s 2023-10-11T16:38:09.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:09 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:38:09.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:09 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:38:09.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:09 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:38:09.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:09 smithi130 ceph-mon[131859]: pgmap v168: 129 pgs: 129 active+clean; 665 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 527 op/s 2023-10-11T16:38:09.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:09 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:38:09.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:09 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:38:09.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:09 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:38:09.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:09 smithi111 ceph-mon[98527]: pgmap v168: 129 pgs: 129 active+clean; 665 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 527 op/s 2023-10-11T16:38:09.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:09 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:38:09.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:09 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:38:09.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:09 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:38:10.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:10 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:38:10.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:10 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:38:10.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:10 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:38:10.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:10 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:38:10.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:10 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:10.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:10 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:10.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:10 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:10.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:10 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:10.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:10 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:38:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:10 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:38:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:10 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:38:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:10 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:38:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:10 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:10 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:10.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:10 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:10.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:10 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:10.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:10 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:38:10.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:10 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:38:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:10 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:38:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:10 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:38:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:10 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:10 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:10 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:10.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:10 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:11.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:11 smithi150 ceph-mon[132217]: pgmap v169: 129 pgs: 129 active+clean; 663 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 879 op/s 2023-10-11T16:38:11.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:11 smithi130 ceph-mon[131859]: pgmap v169: 129 pgs: 129 active+clean; 663 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 879 op/s 2023-10-11T16:38:11.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:11 smithi111 ceph-mon[98527]: pgmap v169: 129 pgs: 129 active+clean; 663 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 879 op/s 2023-10-11T16:38:13.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:13 smithi150 ceph-mon[132217]: pgmap v170: 129 pgs: 129 active+clean; 663 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 608 op/s 2023-10-11T16:38:13.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:13 smithi130 ceph-mon[131859]: pgmap v170: 129 pgs: 129 active+clean; 663 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 608 op/s 2023-10-11T16:38:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:13 smithi111 ceph-mon[98527]: pgmap v170: 129 pgs: 129 active+clean; 663 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 608 op/s 2023-10-11T16:38:14.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:14 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:14.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:14 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:14 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:14.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:14 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:14 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:14.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:14 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:14 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:14.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:14 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:14 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:14.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:14 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:14 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:14.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:14 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:14 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:14.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:14 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:14 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:14.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:14 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:14 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:14.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:14 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:14 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:14.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:14 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:14 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:14.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:14 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:14.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:14 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:14.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:14 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:15.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:15 smithi150 ceph-mon[132217]: pgmap v171: 129 pgs: 129 active+clean; 667 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 717 op/s 2023-10-11T16:38:15.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:15 smithi130 ceph-mon[131859]: pgmap v171: 129 pgs: 129 active+clean; 667 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 717 op/s 2023-10-11T16:38:15.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:15 smithi111 ceph-mon[98527]: pgmap v171: 129 pgs: 129 active+clean; 667 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 717 op/s 2023-10-11T16:38:17.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:17 smithi150 ceph-mon[132217]: pgmap v172: 129 pgs: 129 active+clean; 668 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 814 op/s 2023-10-11T16:38:17.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:17 smithi130 ceph-mon[131859]: pgmap v172: 129 pgs: 129 active+clean; 668 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 814 op/s 2023-10-11T16:38:17.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:17 smithi111 ceph-mon[98527]: pgmap v172: 129 pgs: 129 active+clean; 668 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 814 op/s 2023-10-11T16:38:18.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:18 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:18.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:18 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:18.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:18 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:18.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:18 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:18.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:18 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:18.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:18 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:19.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:19 smithi150 ceph-mon[132217]: pgmap v173: 129 pgs: 129 active+clean; 668 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 642 op/s 2023-10-11T16:38:19.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:19 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:19.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:19 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:19.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:19 smithi130 ceph-mon[131859]: pgmap v173: 129 pgs: 129 active+clean; 668 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 642 op/s 2023-10-11T16:38:19.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:19 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:19.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:19 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:19.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:19 smithi111 ceph-mon[98527]: pgmap v173: 129 pgs: 129 active+clean; 668 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 642 op/s 2023-10-11T16:38:19.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:19 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:19.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:19 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 -- 172.21.15.111:0/822118565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 msgr2=0x7f8418101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 --2- 172.21.15.111:0/822118565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 0x7f8418101020 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f84140099c0 tx=0x7f8414009cd0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 -- 172.21.15.111:0/822118565 shutdown_connections 2023-10-11T16:38:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 --2- 172.21.15.111:0/822118565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8418105560 0x7f841810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 --2- 172.21.15.111:0/822118565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 0x7f8418101020 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 --2- 172.21.15.111:0/822118565 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84181015f0 0x7f8418104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 -- 172.21.15.111:0/822118565 >> 172.21.15.111:0/822118565 conn(0x7f8418077550 msgr2=0x7f8418077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 -- 172.21.15.111:0/822118565 shutdown_connections 2023-10-11T16:38:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.218+0000 7f841f7ff700 1 -- 172.21.15.111:0/822118565 wait complete. 2023-10-11T16:38:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.220+0000 7f841f7ff700 1 Processor -- start 2023-10-11T16:38:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.220+0000 7f841f7ff700 1 -- start start 2023-10-11T16:38:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.220+0000 7f841f7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 0x7f84181963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.220+0000 7f841f7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84181015f0 0x7f84181968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.220+0000 7f841f7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8418105560 0x7f841819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.220+0000 7f841f7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841819b390 con 0x7f8418100c50 2023-10-11T16:38:20.233 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.220+0000 7f841f7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841819b500 con 0x7f84181015f0 2023-10-11T16:38:20.233 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.220+0000 7f841f7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841819b6c0 con 0x7f8418105560 2023-10-11T16:38:20.233 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 0x7f84181963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.233 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 0x7f84181963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54454/0 (socket says 172.21.15.111:54454) 2023-10-11T16:38:20.234 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841dd9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8418105560 0x7f841819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.234 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 -- 172.21.15.111:0/1408129587 learned_addr learned my addr 172.21.15.111:0/1408129587 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:20.234 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841cd9a700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84181015f0 0x7f84181968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.234 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 -- 172.21.15.111:0/1408129587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8418105560 msgr2=0x7f841819ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.235 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8418105560 0x7f841819ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.235 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 -- 172.21.15.111:0/1408129587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84181015f0 msgr2=0x7f84181968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.235 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84181015f0 0x7f84181968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 -- 172.21.15.111:0/1408129587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84140095f0 con 0x7f8418100c50 2023-10-11T16:38:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f841d59b700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 0x7f84181963b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f841400b580 tx=0x7f841400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.221+0000 7f840e7fc700 1 -- 172.21.15.111:0/1408129587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841401d070 con 0x7f8418100c50 2023-10-11T16:38:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.222+0000 7f840e7fc700 1 -- 172.21.15.111:0/1408129587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8414005050 con 0x7f8418100c50 2023-10-11T16:38:20.236 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.222+0000 7f841f7ff700 1 -- 172.21.15.111:0/1408129587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84181103b0 con 0x7f8418100c50 2023-10-11T16:38:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.222+0000 7f840e7fc700 1 -- 172.21.15.111:0/1408129587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8414004a40 con 0x7f8418100c50 2023-10-11T16:38:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.222+0000 7f841f7ff700 1 -- 172.21.15.111:0/1408129587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8418110220 con 0x7f8418100c50 2023-10-11T16:38:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.225+0000 7f840e7fc700 1 -- 172.21.15.111:0/1408129587 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f84140051c0 con 0x7f8418100c50 2023-10-11T16:38:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.226+0000 7f8433186b80 1 Processor -- start 2023-10-11T16:38:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.226+0000 7f840e7fc700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84040a1340 0x7f84040a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.226+0000 7f8433186b80 1 -- start start 2023-10-11T16:38:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f840e7fc700 1 -- 172.21.15.111:0/1408129587 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f84140cddc0 con 0x7f8418100c50 2023-10-11T16:38:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f8433186b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbb6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f8433186b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbb29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f8433186b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f8433186b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bab8000 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f8433186b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bb0a840 con 0x55cc9bbb2560 2023-10-11T16:38:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f8433186b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bb03cd0 con 0x55cc9bbb2f10 2023-10-11T16:38:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f841dd9c700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84040a1340 0x7f84040a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f841d59b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbb6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f841cd9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbb29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f841d59b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbb6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54470/0 (socket says 172.21.15.111:54470) 2023-10-11T16:38:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f841d59b700 1 -- 172.21.15.111:0/3806924566 learned_addr learned my addr 172.21.15.111:0/3806924566 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.227+0000 7f841dd9c700 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f841cd9a700 1 -- 172.21.15.111:0/3806924566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 msgr2=0x55cc9bbc71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f841cd9a700 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f841cd9a700 1 -- 172.21.15.111:0/3806924566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 msgr2=0x55cc9bbb6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f841cd9a700 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbb6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f841cd9a700 1 -- 172.21.15.111:0/3806924566 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc9b9855d0 con 0x55cc9bbb2560 2023-10-11T16:38:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f841d59b700 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbb6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:38:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f841dd9c700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84040a1340 0x7f84040a37f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f841000f870 tx=0x7f841000fc30 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f841cd9a700 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbb29d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f841806a260 tx=0x7f8408009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64c4cb0b54cb01c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f84037fe700 1 -- 172.21.15.111:0/3806924566 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408011c80 con 0x55cc9bbb2560 2023-10-11T16:38:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f84037fe700 1 -- 172.21.15.111:0/3806924566 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8408011de0 con 0x55cc9bbb2560 2023-10-11T16:38:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.228+0000 7f84037fe700 1 -- 172.21.15.111:0/3806924566 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84080193e0 con 0x55cc9bbb2560 2023-10-11T16:38:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 -- 172.21.15.111:0/3806924566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 msgr2=0x55cc9bbb29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbb29d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f841806a260 tx=0x7f8408009f60 comp rx=0 tx=0).stop 2023-10-11T16:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 -- 172.21.15.111:0/3806924566 shutdown_connections 2023-10-11T16:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbb6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 --2- 172.21.15.111:0/3806924566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbb29d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 -- 172.21.15.111:0/3806924566 >> 172.21.15.111:0/3806924566 conn(0x55cc9bba7f50 msgr2=0x55cc9bbad270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 -- 172.21.15.111:0/3806924566 shutdown_connections 2023-10-11T16:38:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.229+0000 7f8433186b80 1 -- 172.21.15.111:0/3806924566 wait complete. 2023-10-11T16:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.230+0000 7f8433186b80 1 -- start start 2023-10-11T16:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.230+0000 7f8433186b80 1 Processor -- start 2023-10-11T16:38:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.230+0000 7f8433186b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.230+0000 7f8433186b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2560 0x55cc9bbc4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.230+0000 7f8433186b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.230+0000 7f8433186b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9baa5880 con 0x55cc9bbb2f10 2023-10-11T16:38:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.230+0000 7f8433186b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9baa5520 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.230+0000 7f8433186b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9b9878c0 con 0x55cc9bbb2560 2023-10-11T16:38:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f8433186b80 10 client.0 _ll_register_callbacks cb 0x55cc9bac3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2560 0x55cc9bbc4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2560 0x55cc9bbc4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:54432/0 (socket says 172.21.15.111:54432) 2023-10-11T16:38:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 -- 172.21.15.111:0/1159781785 learned_addr learned my addr 172.21.15.111:0/1159781785 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841d59b700 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841dd9c700 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 -- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9ba0e6b0 msgr2=0x55cc9bbc7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 -- 172.21.15.111:0/1159781785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9bbb2f10 msgr2=0x55cc9bbc47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 -- 172.21.15.111:0/1159781785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84080098e0 con 0x55cc9bbb2560 2023-10-11T16:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841d59b700 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f841cd9a700 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2560 0x55cc9bbc4260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8408003b10 tx=0x7f8408003860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2109936c73358667 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.231+0000 7f8400ff9700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:38:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.232+0000 7f8400ff9700 1 -- 172.21.15.111:0/1159781785 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408005240 con 0x55cc9bbb2560 2023-10-11T16:38:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.232+0000 7f8433186b80 1 -- 172.21.15.111:0/1159781785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc9bba5710 con 0x55cc9bbb2560 2023-10-11T16:38:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.232+0000 7f8400ff9700 1 -- 172.21.15.111:0/1159781785 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f84080053a0 con 0x55cc9bbb2560 2023-10-11T16:38:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.232+0000 7f8400ff9700 1 -- 172.21.15.111:0/1159781785 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8408005560 con 0x55cc9bbb2560 2023-10-11T16:38:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.232+0000 7f8433186b80 10 client.26527 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.232+0000 7f8433186b80 1 -- 172.21.15.111:0/1159781785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cc9baca130 con 0x55cc9bbb2560 2023-10-11T16:38:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.232+0000 7f8400ff9700 1 -- 172.21.15.111:0/1159781785 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cc9bba5710 con 0x55cc9bbb2560 2023-10-11T16:38:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.232+0000 7f8400ff9700 1 -- 172.21.15.111:0/1159781785 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8408004070 con 0x55cc9bbb2560 2023-10-11T16:38:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f8433186b80 10 client.26527 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f8433186b80 10 client.26527 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f8433186b80 20 client.26527 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f8433186b80 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cc9bbd46d0 0x55cc9bbd6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f8433186b80 4 client.26527 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:38:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f8433186b80 1 -- 172.21.15.111:0/1159781785 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc9bac6040 con 0x55cc9bbd46d0 2023-10-11T16:38:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f841d59b700 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cc9bbd46d0 0x55cc9bbd6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f841d59b700 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cc9bbd46d0 0x55cc9bbd6ae0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77178748f7093881 server_cookie=2227b815b2aedd0c in_seq=0 out_seq=0 2023-10-11T16:38:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.233+0000 7f8400ff9700 10 client.26527 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8400ff9700 1 -- 172.21.15.111:0/1159781785 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cc9bac6040 con 0x55cc9bbd46d0 2023-10-11T16:38:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8400ff9700 10 client.26527 handle_command_reply: tid=0 2023-10-11T16:38:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 client.26527 shutdown 2023-10-11T16:38:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 -- 172.21.15.111:0/1159781785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2560 msgr2=0x55cc9bbc4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2560 0x55cc9bbc4260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8408003b10 tx=0x7f8408003860 comp rx=0 tx=0).stop 2023-10-11T16:38:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 -- 172.21.15.111:0/1159781785 shutdown_connections 2023-10-11T16:38:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cc9bbd46d0 0x55cc9bbd6ae0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 --2- 172.21.15.111:0/1159781785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2560 0x55cc9bbc4260 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8400ff9700 0 client.26527 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.234+0000 7f8433186b80 1 -- 172.21.15.111:0/1159781785 >> 172.21.15.111:0/1159781785 conn(0x55cc9bba7f50 msgr2=0x55cc9bbabf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.235+0000 7f8433186b80 1 -- 172.21.15.111:0/1159781785 shutdown_connections 2023-10-11T16:38:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.235+0000 7f8433186b80 1 -- 172.21.15.111:0/1159781785 wait complete. 2023-10-11T16:38:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.235+0000 7f8433186b80 20 client.26527 trim_cache size 0 max 16384 2023-10-11T16:38:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.251+0000 7f8433186b80 1 Processor -- start 2023-10-11T16:38:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.251+0000 7f8433186b80 1 -- start start 2023-10-11T16:38:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f8433186b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f8433186b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbacf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f8433186b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbbf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f8433186b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bbd10e0 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f8433186b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bac6040 con 0x55cc9bbb2560 2023-10-11T16:38:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f8433186b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bbb5ed0 con 0x55cc9bbb2f10 2023-10-11T16:38:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841dd9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbbf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841dd9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbbf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:54436/0 (socket says 172.21.15.111:54436) 2023-10-11T16:38:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841dd9c700 1 -- 172.21.15.111:0/574950155 learned_addr learned my addr 172.21.15.111:0/574950155 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841d59b700 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.274 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841cd9a700 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbacf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841dd9c700 1 -- 172.21.15.111:0/574950155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 msgr2=0x55cc9bbacf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841dd9c700 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbacf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841dd9c700 1 -- 172.21.15.111:0/574950155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 msgr2=0x55cc9bbc6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841dd9c700 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.275 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841dd9c700 1 -- 172.21.15.111:0/574950155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc9bba96f0 con 0x55cc9bbb2f10 2023-10-11T16:38:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.252+0000 7f841d59b700 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:38:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.253+0000 7f841dd9c700 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbbf540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f84100090b0 tx=0x7f841001f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e2daddc592de0e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.253+0000 7f8403fff700 1 -- 172.21.15.111:0/574950155 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841000e040 con 0x55cc9bbb2f10 2023-10-11T16:38:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.253+0000 7f8403fff700 1 -- 172.21.15.111:0/574950155 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f841000f040 con 0x55cc9bbb2f10 2023-10-11T16:38:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.253+0000 7f8403fff700 1 -- 172.21.15.111:0/574950155 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410008d20 con 0x55cc9bbb2f10 2023-10-11T16:38:20.276 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.253+0000 7f8433186b80 1 -- 172.21.15.111:0/574950155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 msgr2=0x55cc9bbbf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.253+0000 7f8433186b80 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbbf540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f84100090b0 tx=0x7f841001f000 comp rx=0 tx=0).stop 2023-10-11T16:38:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 -- 172.21.15.111:0/574950155 shutdown_connections 2023-10-11T16:38:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbbf540 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbc6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 --2- 172.21.15.111:0/574950155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbacf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 -- 172.21.15.111:0/574950155 >> 172.21.15.111:0/574950155 conn(0x55cc9bba7f50 msgr2=0x55cc9bbd09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 -- 172.21.15.111:0/574950155 shutdown_connections 2023-10-11T16:38:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 -- 172.21.15.111:0/574950155 wait complete. 2023-10-11T16:38:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 -- start start 2023-10-11T16:38:20.278 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.254+0000 7f8433186b80 1 Processor -- start 2023-10-11T16:38:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f8433186b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbcb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f8433186b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbcb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f8433186b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f8433186b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bbc9d50 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f8433186b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bbc9ec0 con 0x55cc9bbb2560 2023-10-11T16:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f8433186b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9bbca080 con 0x55cc9bbb2f10 2023-10-11T16:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f8433186b80 10 client.0 _ll_register_callbacks cb 0x7f8414096e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841d59b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbcb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841d59b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbcb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54488/0 (socket says 172.21.15.111:54488) 2023-10-11T16:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841d59b700 1 -- 172.21.15.111:0/675193333 learned_addr learned my addr 172.21.15.111:0/675193333 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841cd9a700 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbcb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841dd9c700 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841d59b700 1 -- 172.21.15.111:0/675193333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 msgr2=0x55cc9bbc9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841d59b700 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841d59b700 1 -- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 msgr2=0x55cc9bbcb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841d59b700 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbcb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841d59b700 1 -- 172.21.15.111:0/675193333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84181103b0 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.255+0000 7f841cd9a700 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbcb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f841d59b700 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbcb2c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f841409aaa0 tx=0x7f84140a0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c4e0ae3fe294e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f8400ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:38:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f8400ff9700 1 -- 172.21.15.111:0/675193333 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84140056a0 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f8400ff9700 1 -- 172.21.15.111:0/675193333 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f841409ad70 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f8433186b80 1 -- 172.21.15.111:0/675193333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8408011de0 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f8400ff9700 1 -- 172.21.15.111:0/675193333 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84140b3250 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f8433186b80 10 client.16401 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f8433186b80 1 -- 172.21.15.111:0/675193333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f841000fdb0 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.256+0000 7f8400ff9700 1 -- 172.21.15.111:0/675193333 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8408011de0 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f8400ff9700 1 -- 172.21.15.111:0/675193333 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f84140c6030 con 0x55cc9ba0e6b0 2023-10-11T16:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f8433186b80 10 client.16401 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f8433186b80 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f8433186b80 20 client.16401 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f8433186b80 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cc9bc293f0 0x55cc9bc2b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f8433186b80 4 client.16401 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f8433186b80 1 -- 172.21.15.111:0/675193333 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cc9bad2600 con 0x55cc9bc293f0 2023-10-11T16:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f841cd9a700 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cc9bc293f0 0x55cc9bc2b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f841cd9a700 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cc9bc293f0 0x55cc9bc2b800 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ecfdc4dc5d406fe server_cookie=5446ee48a87429b in_seq=0 out_seq=0 2023-10-11T16:38:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.257+0000 7f8400ff9700 10 client.16401 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.648+0000 7f8400ff9700 1 -- 172.21.15.111:0/675193333 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cc9bad2600 con 0x55cc9bc293f0 2023-10-11T16:38:20.651 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.648+0000 7f8400ff9700 10 client.16401 handle_command_reply: tid=0 2023-10-11T16:38:20.651 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.649+0000 7f8433186b80 1 client.16401 shutdown 2023-10-11T16:38:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.649+0000 7f8433186b80 1 -- 172.21.15.111:0/675193333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 msgr2=0x55cc9bbcb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.649+0000 7f8433186b80 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbcb2c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f841409aaa0 tx=0x7f84140a0040 comp rx=0 tx=0).stop 2023-10-11T16:38:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8433186b80 1 -- 172.21.15.111:0/675193333 shutdown_connections 2023-10-11T16:38:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8433186b80 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cc9bc293f0 0x55cc9bc2b800 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8433186b80 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc9bbb2f10 0x55cc9bbc9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8433186b80 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc9ba0e6b0 0x55cc9bbcb2c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8433186b80 1 --2- 172.21.15.111:0/675193333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc9bbb2560 0x55cc9bbcb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8433186b80 1 -- 172.21.15.111:0/675193333 >> 172.21.15.111:0/675193333 conn(0x55cc9bba7f50 msgr2=0x55cc9bbbf540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8400ff9700 0 client.16401 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8433186b80 1 -- 172.21.15.111:0/675193333 shutdown_connections 2023-10-11T16:38:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.650+0000 7f8433186b80 1 -- 172.21.15.111:0/675193333 wait complete. 2023-10-11T16:38:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.651+0000 7f8433186b80 20 client.16401 trim_cache size 0 max 16384 2023-10-11T16:38:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:38:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:38:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:38:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:38:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 -- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84040a1340 msgr2=0x7f84040a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84040a1340 0x7f84040a37f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f841000f870 tx=0x7f841000fc30 comp rx=0 tx=0).stop 2023-10-11T16:38:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 -- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 msgr2=0x7f84181963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 0x7f84181963b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f841400b580 tx=0x7f841400bfd0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 -- 172.21.15.111:0/1408129587 shutdown_connections 2023-10-11T16:38:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f84040a1340 0x7f84040a37f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8418105560 0x7f841819ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8418100c50 0x7f84181963b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 --2- 172.21.15.111:0/1408129587 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84181015f0 0x7f84181968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.652+0000 7f841f7ff700 1 -- 172.21.15.111:0/1408129587 >> 172.21.15.111:0/1408129587 conn(0x7f8418077550 msgr2=0x7f84180fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.653+0000 7f841f7ff700 1 -- 172.21.15.111:0/1408129587 shutdown_connections 2023-10-11T16:38:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:20.654+0000 7f841f7ff700 1 -- 172.21.15.111:0/1408129587 wait complete. 2023-10-11T16:38:21.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:21 smithi150 ceph-mon[132217]: pgmap v174: 129 pgs: 129 active+clean; 666 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 848 op/s 2023-10-11T16:38:21.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:21 smithi130 ceph-mon[131859]: pgmap v174: 129 pgs: 129 active+clean; 666 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 848 op/s 2023-10-11T16:38:21.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:21 smithi111 ceph-mon[98527]: pgmap v174: 129 pgs: 129 active+clean; 666 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 848 op/s 2023-10-11T16:38:22.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:22 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:22.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:22 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:22.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:22 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:22.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:22 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:22.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:22 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:22.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:22 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:23.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:23 smithi150 ceph-mon[132217]: pgmap v175: 129 pgs: 129 active+clean; 666 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 497 op/s 2023-10-11T16:38:23.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:23 smithi130 ceph-mon[131859]: pgmap v175: 129 pgs: 129 active+clean; 666 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 497 op/s 2023-10-11T16:38:23.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:23 smithi111 ceph-mon[98527]: pgmap v175: 129 pgs: 129 active+clean; 666 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 497 op/s 2023-10-11T16:38:24.851 INFO:tasks.workunit.client.0.smithi111.stdout:cleaned up hierarchy 2023-10-11T16:38:25.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:25 smithi150 ceph-mon[132217]: pgmap v176: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 571 op/s 2023-10-11T16:38:25.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:25 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:38:25.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:25 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:25.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:25 smithi130 ceph-mon[131859]: pgmap v176: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 571 op/s 2023-10-11T16:38:25.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:25 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:38:25.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:25 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:25.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:25 smithi111 ceph-mon[98527]: pgmap v176: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 571 op/s 2023-10-11T16:38:25.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:25 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:38:25.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:25 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:26.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:26 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:26.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:26 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:26.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:26 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:38:27.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:27 smithi150 ceph-mon[132217]: pgmap v177: 129 pgs: 129 active+clean; 671 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 651 op/s 2023-10-11T16:38:27.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:27 smithi130 ceph-mon[131859]: pgmap v177: 129 pgs: 129 active+clean; 671 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 651 op/s 2023-10-11T16:38:27.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:27 smithi111 ceph-mon[98527]: pgmap v177: 129 pgs: 129 active+clean; 671 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 651 op/s 2023-10-11T16:38:29.183 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-11T16:38:29.184 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-11T16:38:29.264 INFO:tasks.workunit:Running workunit fs/misc/filelock_deadlock.py... 2023-10-11T16:38:29.264 DEBUG:teuthology.orchestra.run.smithi111:workunit test fs/misc/filelock_deadlock.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_deadlock.py 2023-10-11T16:38:29.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:29 smithi111 ceph-mon[98527]: pgmap v178: 129 pgs: 129 active+clean; 671 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 469 op/s 2023-10-11T16:38:29.774 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:38:29.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:29 smithi130 ceph-mon[131859]: pgmap v178: 129 pgs: 129 active+clean; 671 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 469 op/s 2023-10-11T16:38:29.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:29 smithi150 ceph-mon[132217]: pgmap v178: 129 pgs: 129 active+clean; 671 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 469 op/s 2023-10-11T16:38:29.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.974+0000 7f05543ff700 1 -- 172.21.15.111:0/4100402911 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c100cc0 msgr2=0x7f054c101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:29.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.974+0000 7f05543ff700 1 --2- 172.21.15.111:0/4100402911 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c100cc0 0x7f054c101090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0548009920 tx=0x7f0548009c30 comp rx=0 tx=0).stop 2023-10-11T16:38:29.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.974+0000 7f05543ff700 1 -- 172.21.15.111:0/4100402911 shutdown_connections 2023-10-11T16:38:29.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.974+0000 7f05543ff700 1 --2- 172.21.15.111:0/4100402911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054c105620 0x7f054c10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.974+0000 7f05543ff700 1 --2- 172.21.15.111:0/4100402911 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c100cc0 0x7f054c101090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.974+0000 7f05543ff700 1 --2- 172.21.15.111:0/4100402911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f054c101660 0x7f054c104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.974+0000 7f05543ff700 1 -- 172.21.15.111:0/4100402911 >> 172.21.15.111:0/4100402911 conn(0x7f054c077560 msgr2=0x7f054c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:29.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.974+0000 7f05543ff700 1 -- 172.21.15.111:0/4100402911 shutdown_connections 2023-10-11T16:38:29.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.975+0000 7f05543ff700 1 -- 172.21.15.111:0/4100402911 wait complete. 2023-10-11T16:38:29.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.975+0000 7f05543ff700 1 Processor -- start 2023-10-11T16:38:29.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f05543ff700 1 -- start start 2023-10-11T16:38:29.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f05543ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f054c100cc0 0x7f054c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:29.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f05543ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054c101660 0x7f054c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:29.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f05543ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c105620 0x7f054c1a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:29.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f05543ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c073b60 con 0x7f054c100cc0 2023-10-11T16:38:29.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f05543ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1a4c50 con 0x7f054c105620 2023-10-11T16:38:29.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f05543ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1a4d90 con 0x7f054c101660 2023-10-11T16:38:29.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f055219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f054c100cc0 0x7f054c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:29.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f055299c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c105620 0x7f054c1a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:29.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f055219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f054c100cc0 0x7f054c072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34882/0 (socket says 172.21.15.111:34882) 2023-10-11T16:38:29.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f055219b700 1 -- 172.21.15.111:0/2972007011 learned_addr learned my addr 172.21.15.111:0/2972007011 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:29.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.976+0000 7f055199a700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054c101660 0x7f054c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:29.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f055219b700 1 -- 172.21.15.111:0/2972007011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054c101660 msgr2=0x7f054c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:29.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f055219b700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054c101660 0x7f054c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f055219b700 1 -- 172.21.15.111:0/2972007011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c105620 msgr2=0x7f054c1a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:29.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f055219b700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c105620 0x7f054c1a4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f055219b700 1 -- 172.21.15.111:0/2972007011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c0095f0 con 0x7f054c100cc0 2023-10-11T16:38:29.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f055299c700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c105620 0x7f054c1a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:38:29.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f055219b700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f054c100cc0 0x7f054c072df0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0548009840 tx=0x7f054800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:29.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f05437fe700 1 -- 172.21.15.111:0/2972007011 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054801c070 con 0x7f054c100cc0 2023-10-11T16:38:29.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f05543ff700 1 -- 172.21.15.111:0/2972007011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0548009520 con 0x7f054c100cc0 2023-10-11T16:38:29.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f05437fe700 1 -- 172.21.15.111:0/2972007011 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f054800bba0 con 0x7f054c100cc0 2023-10-11T16:38:29.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f05437fe700 1 -- 172.21.15.111:0/2972007011 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05480038c0 con 0x7f054c100cc0 2023-10-11T16:38:29.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.977+0000 7f05543ff700 1 -- 172.21.15.111:0/2972007011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c1103d0 con 0x7f054c100cc0 2023-10-11T16:38:29.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.980+0000 7f05437fe700 1 -- 172.21.15.111:0/2972007011 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f0548020030 con 0x7f054c100cc0 2023-10-11T16:38:29.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.981+0000 7f0567dbdb80 1 Processor -- start 2023-10-11T16:38:29.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.981+0000 7f05437fe700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05380a12c0 0x7f05380a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:29.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f0567dbdb80 1 -- start start 2023-10-11T16:38:29.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f05437fe700 1 -- 172.21.15.111:0/2972007011 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f05480ce180 con 0x7f054c100cc0 2023-10-11T16:38:29.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f055299c700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05380a12c0 0x7f05380a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:29.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757daef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:29.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757db3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:29.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757db38b0 0x558757db3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:29.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757cb4060 con 0x558757db38b0 2023-10-11T16:38:29.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757d068a0 con 0x558757daf470 2023-10-11T16:38:29.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757cffd30 con 0x558757daeb60 2023-10-11T16:38:29.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f055199a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757db3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:29.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f055299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757db38b0 0x558757db3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:29.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f055199a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757db3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:44522/0 (socket says 172.21.15.111:44522) 2023-10-11T16:38:29.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f055199a700 1 -- 172.21.15.111:0/623167142 learned_addr learned my addr 172.21.15.111:0/623167142 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:29.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.982+0000 7f055219b700 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757daef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:29.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055199a700 1 -- 172.21.15.111:0/623167142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 msgr2=0x558757daef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:29.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055199a700 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757daef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055199a700 1 -- 172.21.15.111:0/623167142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757db38b0 msgr2=0x558757db3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:29.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055199a700 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757db38b0 0x558757db3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055199a700 1 -- 172.21.15.111:0/623167142 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558757b81630 con 0x558757daf470 2023-10-11T16:38:29.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055219b700 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757daef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:38:29.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055299c700 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757db38b0 0x558757db3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:38:29.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055299c700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05380a12c0 0x7f05380a3770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0544009550 tx=0x7f0544007e30 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:29.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f055199a700 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757db3370 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f053c009f20 tx=0x7f053c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147f32a103ad5c37 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:29.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f0540bfe700 1 -- 172.21.15.111:0/623167142 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c0042f0 con 0x558757daf470 2023-10-11T16:38:29.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f0540bfe700 1 -- 172.21.15.111:0/623167142 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f053c004450 con 0x558757daf470 2023-10-11T16:38:29.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.983+0000 7f0540bfe700 1 -- 172.21.15.111:0/623167142 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c019870 con 0x558757daf470 2023-10-11T16:38:29.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 -- 172.21.15.111:0/623167142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 msgr2=0x558757db3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:29.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757db3370 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f053c009f20 tx=0x7f053c011720 comp rx=0 tx=0).stop 2023-10-11T16:38:29.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 -- 172.21.15.111:0/623167142 shutdown_connections 2023-10-11T16:38:29.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757db38b0 0x558757db3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757daef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/623167142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757db3370 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:29.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 -- 172.21.15.111:0/623167142 >> 172.21.15.111:0/623167142 conn(0x558757da4360 msgr2=0x558757da9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:29.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 -- 172.21.15.111:0/623167142 shutdown_connections 2023-10-11T16:38:29.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 -- 172.21.15.111:0/623167142 wait complete. 2023-10-11T16:38:30.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.984+0000 7f0567dbdb80 1 -- start start 2023-10-11T16:38:30.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f0567dbdb80 1 Processor -- start 2023-10-11T16:38:30.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 0x558757db82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757db38b0 0x558757db8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757ca18e0 con 0x558757daeb60 2023-10-11T16:38:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757ca1580 con 0x558757db38b0 2023-10-11T16:38:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757b83920 con 0x558757daf470 2023-10-11T16:38:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f0567dbdb80 10 client.0 _ll_register_callbacks cb 0x558757cbf780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:30.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f055219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f055219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34906/0 (socket says 172.21.15.111:34906) 2023-10-11T16:38:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f055219b700 1 -- 172.21.15.111:0/3205706786 learned_addr learned my addr 172.21.15.111:0/3205706786 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.985+0000 7f055199a700 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 0x558757db82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f055299c700 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757db38b0 0x558757db8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f055219b700 1 -- 172.21.15.111:0/3205706786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 msgr2=0x558757db82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:30.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f055219b700 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 0x558757db82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f055219b700 1 -- 172.21.15.111:0/3205706786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757db38b0 msgr2=0x558757db8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:30.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f055219b700 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757db38b0 0x558757db8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f055219b700 1 -- 172.21.15.111:0/3205706786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c009b80 con 0x558757daeb60 2023-10-11T16:38:30.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f055219b700 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbaff0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0548097940 tx=0x7f054800fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c7efd533891cb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:30.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f0535ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:38:30.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f0535ffb700 1 -- 172.21.15.111:0/3205706786 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054809ef00 con 0x558757daeb60 2023-10-11T16:38:30.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f0535ffb700 1 -- 172.21.15.111:0/3205706786 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f05480a3290 con 0x558757daeb60 2023-10-11T16:38:30.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f0567dbdb80 1 -- 172.21.15.111:0/3205706786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558757da1b20 con 0x558757daeb60 2023-10-11T16:38:30.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f0535ffb700 1 -- 172.21.15.111:0/3205706786 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05480051b0 con 0x558757daeb60 2023-10-11T16:38:30.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f0567dbdb80 10 client.16413 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:30.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.986+0000 7f0567dbdb80 1 -- 172.21.15.111:0/3205706786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558757cc6190 con 0x558757daeb60 2023-10-11T16:38:30.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f0535ffb700 1 -- 172.21.15.111:0/3205706786 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558757da1b20 con 0x558757daeb60 2023-10-11T16:38:30.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f0535ffb700 1 -- 172.21.15.111:0/3205706786 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f054809b130 con 0x558757daeb60 2023-10-11T16:38:30.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f0567dbdb80 10 client.16413 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:30.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f0567dbdb80 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:30.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f0567dbdb80 20 client.16413 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:30.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558757dc8380 0x558757dca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f0567dbdb80 4 client.16413 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:38:30.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f0567dbdb80 1 -- 172.21.15.111:0/3205706786 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558757cc20a0 con 0x558757dc8380 2023-10-11T16:38:30.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.987+0000 7f055199a700 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558757dc8380 0x558757dca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.988+0000 7f0535ffb700 10 client.16413 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:30.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.988+0000 7f055199a700 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558757dc8380 0x558757dca760 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a85d8ce68a276805 server_cookie=a8a969edf23022fe in_seq=0 out_seq=0 2023-10-11T16:38:30.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.988+0000 7f0535ffb700 1 -- 172.21.15.111:0/3205706786 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558757cc20a0 con 0x558757dc8380 2023-10-11T16:38:30.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.988+0000 7f0535ffb700 10 client.16413 handle_command_reply: tid=0 2023-10-11T16:38:30.007 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.988+0000 7f0567dbdb80 1 client.16413 shutdown 2023-10-11T16:38:30.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 -- 172.21.15.111:0/3205706786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 msgr2=0x558757dbaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:30.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbaff0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0548097940 tx=0x7f054800fa70 comp rx=0 tx=0).stop 2023-10-11T16:38:30.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 -- 172.21.15.111:0/3205706786 shutdown_connections 2023-10-11T16:38:30.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558757dc8380 0x558757dca760 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757db38b0 0x558757db8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbaff0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0535ffb700 0 client.16413 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/3205706786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 0x558757db82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 -- 172.21.15.111:0/3205706786 >> 172.21.15.111:0/3205706786 conn(0x558757da4360 msgr2=0x558757db4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:30.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 -- 172.21.15.111:0/3205706786 shutdown_connections 2023-10-11T16:38:30.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 1 -- 172.21.15.111:0/3205706786 wait complete. 2023-10-11T16:38:30.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:29.989+0000 7f0567dbdb80 20 client.16413 trim_cache size 0 max 16384 2023-10-11T16:38:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.006+0000 7f0567dbdb80 1 Processor -- start 2023-10-11T16:38:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.006+0000 7f0567dbdb80 1 -- start start 2023-10-11T16:38:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.006+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757da4360 0x558757dbf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.006+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.006+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 0x558757db19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.006+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757db6650 con 0x558757daeb60 2023-10-11T16:38:30.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.006+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757db2480 con 0x558757da4360 2023-10-11T16:38:30.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.006+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757dbfef0 con 0x558757daf470 2023-10-11T16:38:30.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055199a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055219b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757da4360 0x558757dbf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055219b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757da4360 0x558757dbf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:44526/0 (socket says 172.21.15.111:44526) 2023-10-11T16:38:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055219b700 1 -- 172.21.15.111:0/2882526955 learned_addr learned my addr 172.21.15.111:0/2882526955 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055299c700 1 --2- 172.21.15.111:0/2882526955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 0x558757db19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055199a700 1 -- 172.21.15.111:0/2882526955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 msgr2=0x558757db19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055199a700 1 --2- 172.21.15.111:0/2882526955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 0x558757db19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055199a700 1 -- 172.21.15.111:0/2882526955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757da4360 msgr2=0x558757dbf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055199a700 1 --2- 172.21.15.111:0/2882526955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757da4360 0x558757dbf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055199a700 1 -- 172.21.15.111:0/2882526955 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558757dab490 con 0x558757daeb60 2023-10-11T16:38:30.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f055199a700 1 --2- 172.21.15.111:0/2882526955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbab90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f053c006010 tx=0x7f053c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2220668171d24a97 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:30.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f05413ff700 1 -- 172.21.15.111:0/2882526955 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c003680 con 0x558757daeb60 2023-10-11T16:38:30.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f05413ff700 1 -- 172.21.15.111:0/2882526955 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f053c003ca0 con 0x558757daeb60 2023-10-11T16:38:30.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.007+0000 7f05413ff700 1 -- 172.21.15.111:0/2882526955 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c0045c0 con 0x558757daeb60 2023-10-11T16:38:30.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 -- 172.21.15.111:0/2882526955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 msgr2=0x558757dbab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:30.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/2882526955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbab90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f053c006010 tx=0x7f053c005e80 comp rx=0 tx=0).stop 2023-10-11T16:38:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 -- 172.21.15.111:0/2882526955 shutdown_connections 2023-10-11T16:38:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/2882526955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daf470 0x558757db19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/2882526955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757da4360 0x558757dbf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/2882526955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757daeb60 0x558757dbab90 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 -- 172.21.15.111:0/2882526955 >> 172.21.15.111:0/2882526955 conn(0x558757da9190 msgr2=0x558757da8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 -- 172.21.15.111:0/2882526955 shutdown_connections 2023-10-11T16:38:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 -- 172.21.15.111:0/2882526955 wait complete. 2023-10-11T16:38:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.008+0000 7f0567dbdb80 1 -- start start 2023-10-11T16:38:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f0567dbdb80 1 Processor -- start 2023-10-11T16:38:30.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757da4360 0x558757dbf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757dbd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f0567dbdb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757dbded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757dbe410 con 0x558757da4360 2023-10-11T16:38:30.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757dbe550 con 0x558757daf470 2023-10-11T16:38:30.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f0567dbdb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558757dbe710 con 0x558757daeb60 2023-10-11T16:38:30.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f0567dbdb80 10 client.0 _ll_register_callbacks cb 0x7f054809aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:30.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f055219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757da4360 0x558757dbf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f055219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757da4360 0x558757dbf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34928/0 (socket says 172.21.15.111:34928) 2023-10-11T16:38:30.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.009+0000 7f055219b700 1 -- 172.21.15.111:0/600019364 learned_addr learned my addr 172.21.15.111:0/600019364 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:30.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f055299c700 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757dbded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f055199a700 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757dbd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f055219b700 1 -- 172.21.15.111:0/600019364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 msgr2=0x558757dbd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:30.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f055219b700 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757dbd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f055219b700 1 -- 172.21.15.111:0/600019364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 msgr2=0x558757dbded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:30.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f055219b700 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757dbded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f055219b700 1 -- 172.21.15.111:0/600019364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c01e040 con 0x558757da4360 2023-10-11T16:38:30.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f055219b700 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757da4360 0x558757dbf630 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0548099220 tx=0x7f054809ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340213ad98ac314c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:30.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f0535ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:38:30.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f0535ffb700 1 -- 172.21.15.111:0/600019364 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05480cb950 con 0x558757da4360 2023-10-11T16:38:30.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f0535ffb700 1 -- 172.21.15.111:0/600019364 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f05480a3290 con 0x558757da4360 2023-10-11T16:38:30.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f0567dbdb80 1 -- 172.21.15.111:0/600019364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f053c004450 con 0x558757da4360 2023-10-11T16:38:30.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f0535ffb700 1 -- 172.21.15.111:0/600019364 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f054809ef00 con 0x558757da4360 2023-10-11T16:38:30.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f0567dbdb80 10 client.16419 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:30.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.010+0000 7f0567dbdb80 1 -- 172.21.15.111:0/600019364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558757b2dde0 con 0x558757da4360 2023-10-11T16:38:30.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f0535ffb700 1 -- 172.21.15.111:0/600019364 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f054807d020 con 0x558757da4360 2023-10-11T16:38:30.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f0535ffb700 1 -- 172.21.15.111:0/600019364 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f05480051b0 con 0x558757da4360 2023-10-11T16:38:30.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f0567dbdb80 10 client.16419 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:30.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f0567dbdb80 10 client.16419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:30.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f0567dbdb80 20 client.16419 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:30.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558757db38b0 0x558757e1f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:30.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f0567dbdb80 4 client.16419 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:38:30.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f0567dbdb80 1 -- 172.21.15.111:0/600019364 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558757cce660 con 0x558757db38b0 2023-10-11T16:38:30.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.011+0000 7f055199a700 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558757db38b0 0x558757e1f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:30.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0535ffb700 10 client.16419 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:30.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f055199a700 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558757db38b0 0x558757e1f190 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=530d8178eb29b3c4 server_cookie=c6387b89338e2c6a in_seq=0 out_seq=0 2023-10-11T16:38:30.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0535ffb700 1 -- 172.21.15.111:0/600019364 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558757cce660 con 0x558757db38b0 2023-10-11T16:38:30.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0535ffb700 10 client.16419 handle_command_reply: tid=0 2023-10-11T16:38:30.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 client.16419 shutdown 2023-10-11T16:38:30.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 -- 172.21.15.111:0/600019364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757da4360 msgr2=0x558757dbf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:30.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757da4360 0x558757dbf630 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0548099220 tx=0x7f054809ab40 comp rx=0 tx=0).stop 2023-10-11T16:38:30.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 -- 172.21.15.111:0/600019364 shutdown_connections 2023-10-11T16:38:30.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558757db38b0 0x558757e1f190 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558757daf470 0x558757dbded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558757da4360 0x558757dbf630 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0535ffb700 0 client.16419 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:30.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 --2- 172.21.15.111:0/600019364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558757daeb60 0x558757dbd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 -- 172.21.15.111:0/600019364 >> 172.21.15.111:0/600019364 conn(0x558757da9190 msgr2=0x558757db19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:30.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 -- 172.21.15.111:0/600019364 shutdown_connections 2023-10-11T16:38:30.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 1 -- 172.21.15.111:0/600019364 wait complete. 2023-10-11T16:38:30.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.012+0000 7f0567dbdb80 20 client.16419 trim_cache size 0 max 16384 2023-10-11T16:38:30.033 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:38:30.034 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:38:30.034 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:38:30.034 INFO:teuthology.orchestra.run.smithi111.stdout: "e4b353b6-bcd8-4549-a24b-3247f2c5740c": { 2023-10-11T16:38:30.035 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:38:30.035 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "e4b353b6-bcd8-4549-a24b-3247f2c5740c", 2023-10-11T16:38:30.035 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:38:30.035 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:38:30.035 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:38:30.036 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:38:30.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 -- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05380a12c0 msgr2=0x7f05380a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:30.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05380a12c0 0x7f05380a3770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0544009550 tx=0x7f0544007e30 comp rx=0 tx=0).stop 2023-10-11T16:38:30.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 -- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f054c100cc0 msgr2=0x7f054c072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:30.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f054c100cc0 0x7f054c072df0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0548009840 tx=0x7f054800fa00 comp rx=0 tx=0).stop 2023-10-11T16:38:30.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 -- 172.21.15.111:0/2972007011 shutdown_connections 2023-10-11T16:38:30.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05380a12c0 0x7f05380a3770 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f054c105620 0x7f054c1a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f054c100cc0 0x7f054c072df0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 --2- 172.21.15.111:0/2972007011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054c101660 0x7f054c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:30.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 -- 172.21.15.111:0/2972007011 >> 172.21.15.111:0/2972007011 conn(0x7f054c077560 msgr2=0x7f054c0ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:30.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 -- 172.21.15.111:0/2972007011 shutdown_connections 2023-10-11T16:38:30.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:38:30.013+0000 7f05543ff700 1 -- 172.21.15.111:0/2972007011 wait complete. 2023-10-11T16:38:30.040 INFO:tasks.cephfs.filesystem:scrub status for tag:e4b353b6-bcd8-4549-a24b-3247f2c5740c - {'path': '/', 'tag': 'e4b353b6-bcd8-4549-a24b-3247f2c5740c', 'options': 'recursive,force'} 2023-10-11T16:38:31.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:31 smithi130 ceph-mon[131859]: pgmap v179: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 696 op/s 2023-10-11T16:38:31.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:31 smithi111 ceph-mon[98527]: pgmap v179: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 696 op/s 2023-10-11T16:38:31.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:31 smithi150 ceph-mon[132217]: pgmap v179: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.2 MiB/s wr, 696 op/s 2023-10-11T16:38:33.828 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 -- 172.21.15.111:0/3940100951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 msgr2=0x7feb7410d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:33.829 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 --2- 172.21.15.111:0/3940100951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 0x7feb7410d170 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b0d0 tx=0x7feb6c00b3e0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.829 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 -- 172.21.15.111:0/3940100951 shutdown_connections 2023-10-11T16:38:33.829 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 --2- 172.21.15.111:0/3940100951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 0x7feb7410d170 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 --2- 172.21.15.111:0/3940100951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb74100560 0x7feb74100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 --2- 172.21.15.111:0/3940100951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb74100f00 0x7feb74104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 -- 172.21.15.111:0/3940100951 >> 172.21.15.111:0/3940100951 conn(0x7feb740781b0 msgr2=0x7feb740785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 -- 172.21.15.111:0/3940100951 shutdown_connections 2023-10-11T16:38:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.823+0000 7feb793ff700 1 -- 172.21.15.111:0/3940100951 wait complete. 2023-10-11T16:38:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.824+0000 7feb793ff700 1 Processor -- start 2023-10-11T16:38:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.824+0000 7feb793ff700 1 -- start start 2023-10-11T16:38:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.824+0000 7feb793ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb74100560 0x7feb7419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.824+0000 7feb793ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb74100f00 0x7feb7419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.824+0000 7feb793ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 0x7feb741985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.824+0000 7feb793ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74198c30 con 0x7feb741055a0 2023-10-11T16:38:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.824+0000 7feb793ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74198da0 con 0x7feb74100560 2023-10-11T16:38:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.824+0000 7feb793ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74198f60 con 0x7feb74100f00 2023-10-11T16:38:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb6bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb74100f00 0x7feb7419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 0x7feb741985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb6bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb74100f00 0x7feb7419ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:33538/0 (socket says 172.21.15.111:33538) 2023-10-11T16:38:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb6bfff700 1 -- 172.21.15.111:0/2255075129 learned_addr learned my addr 172.21.15.111:0/2255075129 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb72ffd700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb74100560 0x7feb7419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb737fe700 1 -- 172.21.15.111:0/2255075129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb74100f00 msgr2=0x7feb7419ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb737fe700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb74100f00 0x7feb7419ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb737fe700 1 -- 172.21.15.111:0/2255075129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb74100560 msgr2=0x7feb7419e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb737fe700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb74100560 0x7feb7419e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb737fe700 1 -- 172.21.15.111:0/2255075129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c009d00 con 0x7feb741055a0 2023-10-11T16:38:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb72ffd700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb74100560 0x7feb7419e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:38:33.836 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.825+0000 7feb737fe700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 0x7feb741985c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009fd0 tx=0x7feb6c007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.826+0000 7feb70ff9700 1 -- 172.21.15.111:0/2255075129 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c00e040 con 0x7feb741055a0 2023-10-11T16:38:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.826+0000 7feb70ff9700 1 -- 172.21.15.111:0/2255075129 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6c00bd30 con 0x7feb741055a0 2023-10-11T16:38:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.826+0000 7feb70ff9700 1 -- 172.21.15.111:0/2255075129 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c01d470 con 0x7feb741055a0 2023-10-11T16:38:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.826+0000 7feb793ff700 1 -- 172.21.15.111:0/2255075129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb741103c0 con 0x7feb741055a0 2023-10-11T16:38:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.826+0000 7feb793ff700 1 -- 172.21.15.111:0/2255075129 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb74110230 con 0x7feb741055a0 2023-10-11T16:38:33.839 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:33 smithi130 ceph-mon[131859]: pgmap v180: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 490 op/s 2023-10-11T16:38:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.829+0000 7feb70ff9700 1 -- 172.21.15.111:0/2255075129 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7feb6c004a50 con 0x7feb741055a0 2023-10-11T16:38:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb8ccd7b80 1 Processor -- start 2023-10-11T16:38:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb8ccd7b80 1 -- start start 2023-10-11T16:38:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb70ff9700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7feb600a1340 0x7feb600a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb70ff9700 1 -- 172.21.15.111:0/2255075129 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7feb6c0ce350 con 0x7feb741055a0 2023-10-11T16:38:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254ae76b0 0x563254c911e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c8b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91b90 0x563254c8bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254b91000 con 0x563254ae76b0 2023-10-11T16:38:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254be3840 con 0x563254c91720 2023-10-11T16:38:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.830+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254bdccd0 con 0x563254c91b90 2023-10-11T16:38:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb6bfff700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7feb600a1340 0x7feb600a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254ae76b0 0x563254c911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254ae76b0 0x563254c911e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34970/0 (socket says 172.21.15.111:34970) 2023-10-11T16:38:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 -- 172.21.15.111:0/648312678 learned_addr learned my addr 172.21.15.111:0/648312678 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb6bfff700 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c8b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 -- 172.21.15.111:0/648312678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91b90 msgr2=0x563254c8bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb737fe700 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91b90 0x563254c8bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91b90 0x563254c8bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 -- 172.21.15.111:0/648312678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 msgr2=0x563254c8b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c8b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 -- 172.21.15.111:0/648312678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563254a5e5d0 con 0x563254ae76b0 2023-10-11T16:38:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb6bfff700 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c8b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:38:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb6bfff700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7feb600a1340 0x7feb600a37f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feb74106870 tx=0x7feb5c0127d0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.831+0000 7feb72ffd700 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254ae76b0 0x563254c911e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feb64009de0 tx=0x7feb640116c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa8b3f4c8e1094 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.832+0000 7feb697fa700 1 -- 172.21.15.111:0/648312678 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb64011ed0 con 0x563254ae76b0 2023-10-11T16:38:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.832+0000 7feb697fa700 1 -- 172.21.15.111:0/648312678 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb64012910 con 0x563254ae76b0 2023-10-11T16:38:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.832+0000 7feb697fa700 1 -- 172.21.15.111:0/648312678 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6400f8b0 con 0x563254ae76b0 2023-10-11T16:38:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.832+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/648312678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254ae76b0 msgr2=0x563254c911e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.832+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254ae76b0 0x563254c911e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feb64009de0 tx=0x7feb640116c0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.832+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/648312678 shutdown_connections 2023-10-11T16:38:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.832+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91b90 0x563254c8bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.832+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254ae76b0 0x563254c911e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.833+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/648312678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c8b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.833+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/648312678 >> 172.21.15.111:0/648312678 conn(0x563254c80f50 msgr2=0x563254c86270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.833+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/648312678 shutdown_connections 2023-10-11T16:38:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.833+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/648312678 wait complete. 2023-10-11T16:38:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.833+0000 7feb8ccd7b80 1 -- start start 2023-10-11T16:38:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.833+0000 7feb8ccd7b80 1 Processor -- start 2023-10-11T16:38:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c97a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 0x563254c94c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c95260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254b7e880 con 0x563254c91b90 2023-10-11T16:38:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254b7e520 con 0x563254ae76b0 2023-10-11T16:38:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254a608c0 con 0x563254c91720 2023-10-11T16:38:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb8ccd7b80 10 client.0 _ll_register_callbacks cb 0x563254b9c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c95260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c95260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34972/0 (socket says 172.21.15.111:34972) 2023-10-11T16:38:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb737fe700 1 -- 172.21.15.111:0/306804410 learned_addr learned my addr 172.21.15.111:0/306804410 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb72ffd700 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c97a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb6bfff700 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 0x563254c94c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb737fe700 1 -- 172.21.15.111:0/306804410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 msgr2=0x563254c94c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:33.856 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb737fe700 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 0x563254c94c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb737fe700 1 -- 172.21.15.111:0/306804410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 msgr2=0x563254c97a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb737fe700 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c97a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.834+0000 7feb737fe700 1 -- 172.21.15.111:0/306804410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb64009a10 con 0x563254c91b90 2023-10-11T16:38:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb737fe700 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c95260 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0a2f30 tx=0x7feb6c0974d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ae2b451a7d27cf server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb5a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:38:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb5a7fc700 1 -- 172.21.15.111:0/306804410 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c02d2f0 con 0x563254c91b90 2023-10-11T16:38:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/306804410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563254c7e710 con 0x563254c91b90 2023-10-11T16:38:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb5a7fc700 1 -- 172.21.15.111:0/306804410 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7feb6c09b0a0 con 0x563254c91b90 2023-10-11T16:38:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb5a7fc700 1 -- 172.21.15.111:0/306804410 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb6c09f0d0 con 0x563254c91b90 2023-10-11T16:38:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 10 client.16431 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/306804410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563254ba3130 con 0x563254c91b90 2023-10-11T16:38:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb5a7fc700 1 -- 172.21.15.111:0/306804410 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563254c7e710 con 0x563254c91b90 2023-10-11T16:38:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb5a7fc700 1 -- 172.21.15.111:0/306804410 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7feb6c09b300 con 0x563254c91b90 2023-10-11T16:38:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 10 client.16431 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 10 client.16431 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 20 client.16431 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563254ca4cf0 0x563254ca7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 4 client.16431 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:38:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.835+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/306804410 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563254b9f040 con 0x563254ca4cf0 2023-10-11T16:38:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.836+0000 7feb72ffd700 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563254ca4cf0 0x563254ca7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.836+0000 7feb5a7fc700 10 client.16431 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.836+0000 7feb72ffd700 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563254ca4cf0 0x563254ca7100 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ebf6f3799419872 server_cookie=e12cf06f160a0632 in_seq=0 out_seq=0 2023-10-11T16:38:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb5a7fc700 1 -- 172.21.15.111:0/306804410 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563254b9f040 con 0x563254ca4cf0 2023-10-11T16:38:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb5a7fc700 10 client.16431 handle_command_reply: tid=0 2023-10-11T16:38:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 client.16431 shutdown 2023-10-11T16:38:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/306804410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 msgr2=0x563254c95260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c95260 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0a2f30 tx=0x7feb6c0974d0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/306804410 shutdown_connections 2023-10-11T16:38:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563254ca4cf0 0x563254ca7100 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c95260 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c97a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb5a7fc700 0 client.16431 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/306804410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 0x563254c94c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/306804410 >> 172.21.15.111:0/306804410 conn(0x563254c80f50 msgr2=0x563254c857e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.837+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/306804410 shutdown_connections 2023-10-11T16:38:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.838+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/306804410 wait complete. 2023-10-11T16:38:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.838+0000 7feb8ccd7b80 20 client.16431 trim_cache size 0 max 16384 2023-10-11T16:38:33.873 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.852+0000 7feb8ccd7b80 1 Processor -- start 2023-10-11T16:38:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb8ccd7b80 1 -- start start 2023-10-11T16:38:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254ae76b0 0x563254c9c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c92670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c85f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254c931f0 con 0x563254c91b90 2023-10-11T16:38:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254c8f970 con 0x563254c91720 2023-10-11T16:38:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254c911e0 con 0x563254ae76b0 2023-10-11T16:38:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c85f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb72ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254ae76b0 0x563254c9c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c85f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34982/0 (socket says 172.21.15.111:34982) 2023-10-11T16:38:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb72ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254ae76b0 0x563254c9c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:33558/0 (socket says 172.21.15.111:33558) 2023-10-11T16:38:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb737fe700 1 -- 172.21.15.111:0/1053195923 learned_addr learned my addr 172.21.15.111:0/1053195923 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb6bfff700 1 --2- 172.21.15.111:0/1053195923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c92670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb737fe700 1 -- 172.21.15.111:0/1053195923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254ae76b0 msgr2=0x563254c9c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb737fe700 1 --2- 172.21.15.111:0/1053195923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254ae76b0 0x563254c9c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb737fe700 1 -- 172.21.15.111:0/1053195923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 msgr2=0x563254c92670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb737fe700 1 --2- 172.21.15.111:0/1053195923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c92670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.853+0000 7feb737fe700 1 -- 172.21.15.111:0/1053195923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563254c826f0 con 0x563254c91b90 2023-10-11T16:38:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb737fe700 1 --2- 172.21.15.111:0/1053195923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c85f60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b0d0 tx=0x7feb6c00b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e6be38f02f20 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb69ffb700 1 -- 172.21.15.111:0/1053195923 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c09ae70 con 0x563254c91b90 2023-10-11T16:38:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb69ffb700 1 -- 172.21.15.111:0/1053195923 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6c0b3250 con 0x563254c91b90 2023-10-11T16:38:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb69ffb700 1 -- 172.21.15.111:0/1053195923 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0ce030 con 0x563254c91b90 2023-10-11T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/1053195923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 msgr2=0x563254c85f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/1053195923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c85f60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b0d0 tx=0x7feb6c00b920 comp rx=0 tx=0).stop 2023-10-11T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/1053195923 shutdown_connections 2023-10-11T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/1053195923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c85f60 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/1053195923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254ae76b0 0x563254c9c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/1053195923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254c91720 0x563254c92670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.854+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/1053195923 >> 172.21.15.111:0/1053195923 conn(0x563254c80f50 msgr2=0x563254c91f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.855+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/1053195923 shutdown_connections 2023-10-11T16:38:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.855+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/1053195923 wait complete. 2023-10-11T16:38:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.855+0000 7feb8ccd7b80 1 -- start start 2023-10-11T16:38:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.855+0000 7feb8ccd7b80 1 Processor -- start 2023-10-11T16:38:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c9c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 0x563254c8cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb8ccd7b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c8d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254c8d870 con 0x563254c91b90 2023-10-11T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254c8d9e0 con 0x563254ae76b0 2023-10-11T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb8ccd7b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563254c8dba0 con 0x563254c91720 2023-10-11T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb8ccd7b80 10 client.0 _ll_register_callbacks cb 0x563254b9c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c8d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c8d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34998/0 (socket says 172.21.15.111:34998) 2023-10-11T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb737fe700 1 -- 172.21.15.111:0/4101782290 learned_addr learned my addr 172.21.15.111:0/4101782290 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb72ffd700 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c9c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.856+0000 7feb6bfff700 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 0x563254c8cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb72ffd700 1 -- 172.21.15.111:0/4101782290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 msgr2=0x563254c8cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb72ffd700 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 0x563254c8cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb72ffd700 1 -- 172.21.15.111:0/4101782290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 msgr2=0x563254c8d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb72ffd700 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c8d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb72ffd700 1 -- 172.21.15.111:0/4101782290 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563254ba3130 con 0x563254ae76b0 2023-10-11T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb737fe700 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c8d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb72ffd700 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c9c240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7feb64011ac0 tx=0x7feb640098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dd76f50de3d00ad server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb5a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb5a7fc700 1 -- 172.21.15.111:0/4101782290 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb640125d0 con 0x563254ae76b0 2023-10-11T16:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/4101782290 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb64012910 con 0x563254ae76b0 2023-10-11T16:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.857+0000 7feb5a7fc700 1 -- 172.21.15.111:0/4101782290 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7feb64012730 con 0x563254ae76b0 2023-10-11T16:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb5a7fc700 1 -- 172.21.15.111:0/4101782290 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb64029b50 con 0x563254ae76b0 2023-10-11T16:38:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb8ccd7b80 10 client.26552 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/4101782290 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563254a0ad80 con 0x563254ae76b0 2023-10-11T16:38:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb5a7fc700 1 -- 172.21.15.111:0/4101782290 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb64012910 con 0x563254ae76b0 2023-10-11T16:38:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb5a7fc700 1 -- 172.21.15.111:0/4101782290 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7feb640113e0 con 0x563254ae76b0 2023-10-11T16:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb8ccd7b80 10 client.26552 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb8ccd7b80 10 client.26552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb8ccd7b80 20 client.26552 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563254cf93a0 0x563254cfb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb8ccd7b80 4 client.26552 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/4101782290 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563254bab600 con 0x563254cf93a0 2023-10-11T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.858+0000 7feb6bfff700 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563254cf93a0 0x563254cfb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.859+0000 7feb5a7fc700 10 client.26552 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:33.859+0000 7feb6bfff700 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563254cf93a0 0x563254cfb7b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5d2b0619f2c4c80 server_cookie=4e126b687e4adc63 in_seq=0 out_seq=0 2023-10-11T16:38:33.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:33 smithi111 ceph-mon[98527]: pgmap v180: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 490 op/s 2023-10-11T16:38:33.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:33 smithi150 ceph-mon[132217]: pgmap v180: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 490 op/s 2023-10-11T16:38:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb5a7fc700 1 -- 172.21.15.111:0/4101782290 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563254bab600 con 0x563254cf93a0 2023-10-11T16:38:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb5a7fc700 10 client.26552 handle_command_reply: tid=0 2023-10-11T16:38:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 client.26552 shutdown 2023-10-11T16:38:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 msgr2=0x563254c9c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c9c240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7feb64011ac0 tx=0x7feb640098c0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/4101782290 shutdown_connections 2023-10-11T16:38:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563254cf93a0 0x563254cfb7b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563254c91b90 0x563254c8d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563254ae76b0 0x563254c9c240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb5a7fc700 0 client.26552 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 --2- 172.21.15.111:0/4101782290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563254c91720 0x563254c8cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/4101782290 >> 172.21.15.111:0/4101782290 conn(0x563254c80f50 msgr2=0x563254c927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.017+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/4101782290 shutdown_connections 2023-10-11T16:38:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb8ccd7b80 1 -- 172.21.15.111:0/4101782290 wait complete. 2023-10-11T16:38:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb8ccd7b80 20 client.26552 trim_cache size 0 max 16384 2023-10-11T16:38:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:38:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:38:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:38:34.024 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:38:34.024 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 -- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7feb600a1340 msgr2=0x7feb600a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7feb600a1340 0x7feb600a37f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feb74106870 tx=0x7feb5c0127d0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 -- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 msgr2=0x7feb741985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 0x7feb741985c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009fd0 tx=0x7feb6c007860 comp rx=0 tx=0).stop 2023-10-11T16:38:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 -- 172.21.15.111:0/2255075129 shutdown_connections 2023-10-11T16:38:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7feb600a1340 0x7feb600a37f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb741055a0 0x7feb741985c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feb74100560 0x7feb7419e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 --2- 172.21.15.111:0/2255075129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb74100f00 0x7feb7419ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.018+0000 7feb793ff700 1 -- 172.21.15.111:0/2255075129 >> 172.21.15.111:0/2255075129 conn(0x7feb740781b0 msgr2=0x7feb740fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.019+0000 7feb793ff700 1 -- 172.21.15.111:0/2255075129 shutdown_connections 2023-10-11T16:38:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:34.019+0000 7feb793ff700 1 -- 172.21.15.111:0/2255075129 wait complete. 2023-10-11T16:38:35.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:35 smithi130 ceph-mon[131859]: pgmap v181: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 492 op/s 2023-10-11T16:38:35.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:35 smithi111 ceph-mon[98527]: pgmap v181: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 492 op/s 2023-10-11T16:38:35.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:35 smithi150 ceph-mon[132217]: pgmap v181: 129 pgs: 129 active+clean; 670 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 492 op/s 2023-10-11T16:38:36.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:36 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:38:36.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:36 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:38:36.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:36 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:38:36.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:36 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:38:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:36 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:38:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:36 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:38:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:36 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:38:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:36 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:38:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:36 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:38:37.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:37 smithi130 ceph-mon[131859]: pgmap v182: 129 pgs: 129 active+clean; 662 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 451 op/s 2023-10-11T16:38:37.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:37 smithi111 ceph-mon[98527]: pgmap v182: 129 pgs: 129 active+clean; 662 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 451 op/s 2023-10-11T16:38:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:37 smithi150 ceph-mon[132217]: pgmap v182: 129 pgs: 129 active+clean; 662 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 MiB/s wr, 451 op/s 2023-10-11T16:38:39.370 INFO:tasks.workunit.client.0.smithi111.stdout:ok 2023-10-11T16:38:39.377 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-11T16:38:39.377 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-11T16:38:39.515 INFO:tasks.workunit:Running workunit fs/misc/filelock_interrupt.py... 2023-10-11T16:38:39.515 DEBUG:teuthology.orchestra.run.smithi111:workunit test fs/misc/filelock_interrupt.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_interrupt.py 2023-10-11T16:38:39.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:39 smithi130 ceph-mon[131859]: pgmap v183: 129 pgs: 129 active+clean; 662 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 268 KiB/s wr, 262 op/s 2023-10-11T16:38:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:39 smithi111 ceph-mon[98527]: pgmap v183: 129 pgs: 129 active+clean; 662 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 268 KiB/s wr, 262 op/s 2023-10-11T16:38:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:39 smithi150 ceph-mon[132217]: pgmap v183: 129 pgs: 129 active+clean; 662 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 268 KiB/s wr, 262 op/s 2023-10-11T16:38:41.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:41 smithi130 ceph-mon[131859]: pgmap v184: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 316 KiB/s wr, 320 op/s 2023-10-11T16:38:41.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:41 smithi111 ceph-mon[98527]: pgmap v184: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 316 KiB/s wr, 320 op/s 2023-10-11T16:38:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:41 smithi150 ceph-mon[132217]: pgmap v184: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 316 KiB/s wr, 320 op/s 2023-10-11T16:38:43.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:43 smithi130 ceph-mon[131859]: pgmap v185: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 92 KiB/s wr, 94 op/s 2023-10-11T16:38:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:43 smithi111 ceph-mon[98527]: pgmap v185: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 92 KiB/s wr, 94 op/s 2023-10-11T16:38:43.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:43 smithi150 ceph-mon[132217]: pgmap v185: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 92 KiB/s wr, 94 op/s 2023-10-11T16:38:45.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:45 smithi130 ceph-mon[131859]: pgmap v186: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 97 KiB/s wr, 96 op/s 2023-10-11T16:38:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:45 smithi111 ceph-mon[98527]: pgmap v186: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 97 KiB/s wr, 96 op/s 2023-10-11T16:38:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:45 smithi150 ceph-mon[132217]: pgmap v186: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 97 KiB/s wr, 96 op/s 2023-10-11T16:38:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2966241569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100440 msgr2=0x7ff1cc100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2966241569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100440 0x7ff1cc100810 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80099c0 tx=0x7ff1c8009cd0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2966241569 shutdown_connections 2023-10-11T16:38:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2966241569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1cc105480 0x7ff1cc10d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2966241569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100440 0x7ff1cc100810 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2966241569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc100de0 0x7ff1cc104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2966241569 >> 172.21.15.111:0/2966241569 conn(0x7ff1cc0fbcb0 msgr2=0x7ff1cc0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2966241569 shutdown_connections 2023-10-11T16:38:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.171+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2966241569 wait complete. 2023-10-11T16:38:47.177 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.172+0000 7ff1d41ff700 1 Processor -- start 2023-10-11T16:38:47.177 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.172+0000 7ff1d41ff700 1 -- start start 2023-10-11T16:38:47.177 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.172+0000 7ff1d41ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1cc100440 0x7ff1cc072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.177 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.172+0000 7ff1d41ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100de0 0x7ff1cc073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.177 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.172+0000 7ff1d41ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc105480 0x7ff1cc073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.172+0000 7ff1d41ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1a8d00 con 0x7ff1cc100de0 2023-10-11T16:38:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.172+0000 7ff1d41ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1a8e40 con 0x7ff1cc100440 2023-10-11T16:38:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.172+0000 7ff1d41ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc1a8f80 con 0x7ff1cc105480 2023-10-11T16:38:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100de0 0x7ff1cc073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100de0 0x7ff1cc073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33428/0 (socket says 172.21.15.111:33428) 2023-10-11T16:38:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 -- 172.21.15.111:0/2623709123 learned_addr learned my addr 172.21.15.111:0/2623709123 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1cc100440 0x7ff1cc072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d279c700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc105480 0x7ff1cc073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.179 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 -- 172.21.15.111:0/2623709123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc105480 msgr2=0x7ff1cc073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc105480 0x7ff1cc073870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 -- 172.21.15.111:0/2623709123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1cc100440 msgr2=0x7ff1cc072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1cc100440 0x7ff1cc072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 -- 172.21.15.111:0/2623709123 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c80095f0 con 0x7ff1cc100de0 2023-10-11T16:38:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1cc100440 0x7ff1cc072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:38:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.173+0000 7ff1d179a700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100de0 0x7ff1cc073330 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00e940 tx=0x7ff1bc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.174+0000 7ff1c2ffd700 1 -- 172.21.15.111:0/2623709123 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc009730 con 0x7ff1cc100de0 2023-10-11T16:38:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.174+0000 7ff1c2ffd700 1 -- 172.21.15.111:0/2623709123 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1bc010430 con 0x7ff1cc100de0 2023-10-11T16:38:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.174+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2623709123 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc110240 con 0x7ff1cc100de0 2023-10-11T16:38:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.174+0000 7ff1c2ffd700 1 -- 172.21.15.111:0/2623709123 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc00cea0 con 0x7ff1cc100de0 2023-10-11T16:38:47.182 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.174+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2623709123 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc1100b0 con 0x7ff1cc100de0 2023-10-11T16:38:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.177+0000 7ff1c2ffd700 1 -- 172.21.15.111:0/2623709123 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7ff1bc010810 con 0x7ff1cc100de0 2023-10-11T16:38:47.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.179+0000 7ff1e7c08b80 1 Processor -- start 2023-10-11T16:38:47.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.179+0000 7ff1c2ffd700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff1b80a1340 0x7ff1b80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.179+0000 7ff1e7c08b80 1 -- start start 2023-10-11T16:38:47.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.179+0000 7ff1c2ffd700 1 -- 172.21.15.111:0/2623709123 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff1bc014070 con 0x7ff1cc100de0 2023-10-11T16:38:47.187 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d279c700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff1b80a1340 0x7ff1b80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa03e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa038b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593a9f3e000 con 0x5593aa03e720 2023-10-11T16:38:47.188 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593a9f90840 con 0x5593aa03eb90 2023-10-11T16:38:47.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593a9f89cd0 con 0x5593a9e946b0 2023-10-11T16:38:47.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d1f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa03e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d179a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d279c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa038b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.189 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d1f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa03e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:50636/0 (socket says 172.21.15.111:50636) 2023-10-11T16:38:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d1f9b700 1 -- 172.21.15.111:0/1185008420 learned_addr learned my addr 172.21.15.111:0/1185008420 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d1f9b700 1 -- 172.21.15.111:0/1185008420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 msgr2=0x5593aa038b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/1185008420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa038b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d1f9b700 1 -- 172.21.15.111:0/1185008420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 msgr2=0x5593aa038560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/1185008420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d1f9b700 1 -- 172.21.15.111:0/1185008420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593a9e0b5d0 con 0x5593a9e946b0 2023-10-11T16:38:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.180+0000 7ff1d179a700 1 --2- 172.21.15.111:0/1185008420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:38:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.181+0000 7ff1d279c700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff1b80a1340 0x7ff1b80a37f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400f870 tx=0x7ff1c400fc30 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.181+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/1185008420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa03e1e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80058e0 tx=0x7ff1c8005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b93f759505bce9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.181+0000 7ff1b7fff700 1 -- 172.21.15.111:0/1185008420 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c800bd40 con 0x5593a9e946b0 2023-10-11T16:38:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.181+0000 7ff1b7fff700 1 -- 172.21.15.111:0/1185008420 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1c801e040 con 0x5593a9e946b0 2023-10-11T16:38:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.181+0000 7ff1b7fff700 1 -- 172.21.15.111:0/1185008420 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8005580 con 0x5593a9e946b0 2023-10-11T16:38:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1185008420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 msgr2=0x5593aa03e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1185008420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa03e1e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80058e0 tx=0x7ff1c8005fb0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1185008420 shutdown_connections 2023-10-11T16:38:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1185008420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa038b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.193 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1185008420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa03e1e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.193 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1185008420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.193 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1185008420 >> 172.21.15.111:0/1185008420 conn(0x5593aa02df50 msgr2=0x5593aa033270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:47.193 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1185008420 shutdown_connections 2023-10-11T16:38:47.193 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1185008420 wait complete. 2023-10-11T16:38:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.182+0000 7ff1e7c08b80 1 -- start start 2023-10-11T16:38:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1e7c08b80 1 Processor -- start 2023-10-11T16:38:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa047a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03e720 0x5593aa047fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.194 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03eb90 0x5593aa041cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593a9f2b880 con 0x5593aa03eb90 2023-10-11T16:38:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593a9f2b520 con 0x5593aa03e720 2023-10-11T16:38:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593a9e0d8c0 con 0x5593a9e946b0 2023-10-11T16:38:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1e7c08b80 10 client.0 _ll_register_callbacks cb 0x5593a9f49720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:47.195 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1d179a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03e720 0x5593aa047fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1d1f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa047a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1d179a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03e720 0x5593aa047fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:43106/0 (socket says 172.21.15.111:43106) 2023-10-11T16:38:47.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1d179a700 1 -- 172.21.15.111:0/3669383842 learned_addr learned my addr 172.21.15.111:0/3669383842 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:47.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.183+0000 7ff1d279c700 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03eb90 0x5593aa041cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.196 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1d179a700 1 -- 172.21.15.111:0/3669383842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 msgr2=0x5593aa047a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:47.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1d179a700 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa047a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1d179a700 1 -- 172.21.15.111:0/3669383842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03eb90 msgr2=0x5593aa041cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:47.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1d179a700 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03eb90 0x5593aa041cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1d179a700 1 -- 172.21.15.111:0/3669383842 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc10d890 con 0x5593aa03e720 2023-10-11T16:38:47.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1d179a700 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03e720 0x5593aa047fa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc096920 tx=0x7ff1bc099ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7943538a01378885 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:47.197 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1b57fa700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:38:47.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1b57fa700 1 -- 172.21.15.111:0/3669383842 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0995a0 con 0x5593aa03e720 2023-10-11T16:38:47.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/3669383842 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593aa02b710 con 0x5593aa03e720 2023-10-11T16:38:47.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1b57fa700 1 -- 172.21.15.111:0/3669383842 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff1bc0a1e70 con 0x5593aa03e720 2023-10-11T16:38:47.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1b57fa700 1 -- 172.21.15.111:0/3669383842 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1bc0b82b0 con 0x5593aa03e720 2023-10-11T16:38:47.198 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.184+0000 7ff1e7c08b80 10 client.26564 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:47.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/3669383842 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5593a9f50130 con 0x5593aa03e720 2023-10-11T16:38:47.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1b57fa700 1 -- 172.21.15.111:0/3669383842 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1bc0ca080 con 0x5593aa03e720 2023-10-11T16:38:47.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1b57fa700 1 -- 172.21.15.111:0/3669383842 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff1bc0adbb0 con 0x5593aa03e720 2023-10-11T16:38:47.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1e7c08b80 10 client.26564 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:47.199 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1e7c08b80 10 client.26564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:47.200 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1e7c08b80 20 client.26564 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:47.200 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593aa051fa0 0x5593aa0543b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.201 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1e7c08b80 4 client.26564 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:38:47.202 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/3669383842 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5593a9f4c040 con 0x5593aa051fa0 2023-10-11T16:38:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.185+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593aa051fa0 0x5593aa0543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.186+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593aa051fa0 0x5593aa0543b0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bad9a4e8e24fb334 server_cookie=c1e7f3933eb3f8e1 in_seq=0 out_seq=0 2023-10-11T16:38:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.186+0000 7ff1b57fa700 10 client.26564 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.186+0000 7ff1b57fa700 1 -- 172.21.15.111:0/3669383842 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5593a9f4c040 con 0x5593aa051fa0 2023-10-11T16:38:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1b57fa700 10 client.26564 handle_command_reply: tid=0 2023-10-11T16:38:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 client.26564 shutdown 2023-10-11T16:38:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/3669383842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03e720 msgr2=0x5593aa047fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03e720 0x5593aa047fa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc096920 tx=0x7ff1bc099ff0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/3669383842 shutdown_connections 2023-10-11T16:38:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593aa051fa0 0x5593aa0543b0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03eb90 0x5593aa041cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa047a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/3669383842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03e720 0x5593aa047fa0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1b57fa700 0 client.26564 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/3669383842 >> 172.21.15.111:0/3669383842 conn(0x5593aa02df50 msgr2=0x5593aa0327e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/3669383842 shutdown_connections 2023-10-11T16:38:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/3669383842 wait complete. 2023-10-11T16:38:47.207 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.187+0000 7ff1e7c08b80 20 client.26564 trim_cache size 0 max 16384 2023-10-11T16:38:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.202+0000 7ff1e7c08b80 1 Processor -- start 2023-10-11T16:38:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.202+0000 7ff1e7c08b80 1 -- start start 2023-10-11T16:38:47.211 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa041e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa03d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593aa03c7f0 con 0x5593aa03e720 2023-10-11T16:38:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593aa035030 con 0x5593aa03eb90 2023-10-11T16:38:47.212 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593aa04e880 con 0x5593a9e946b0 2023-10-11T16:38:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d179a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d179a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33448/0 (socket says 172.21.15.111:33448) 2023-10-11T16:38:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d179a700 1 -- 172.21.15.111:0/4085419371 learned_addr learned my addr 172.21.15.111:0/4085419371 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa041e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.213 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d279c700 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa03d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d179a700 1 -- 172.21.15.111:0/4085419371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 msgr2=0x5593aa041e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d179a700 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa041e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d179a700 1 -- 172.21.15.111:0/4085419371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 msgr2=0x5593aa03d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d179a700 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa03d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.214 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d179a700 1 -- 172.21.15.111:0/4085419371 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593a9f4fe80 con 0x5593aa03e720 2023-10-11T16:38:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.203+0000 7ff1d279c700 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa03d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:38:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.204+0000 7ff1d179a700 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0a1dd0 tx=0x7ff1bc099c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f97237538d624fe server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.204+0000 7ff1c0ff9700 1 -- 172.21.15.111:0/4085419371 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0b82b0 con 0x5593aa03e720 2023-10-11T16:38:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.204+0000 7ff1c0ff9700 1 -- 172.21.15.111:0/4085419371 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5593aa02b710 con 0x5593aa03e720 2023-10-11T16:38:47.215 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.204+0000 7ff1c0ff9700 1 -- 172.21.15.111:0/4085419371 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0a20d0 con 0x5593aa03e720 2023-10-11T16:38:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.204+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/4085419371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 msgr2=0x5593aa038560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.204+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0a1dd0 tx=0x7ff1bc099c80 comp rx=0 tx=0).stop 2023-10-11T16:38:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/4085419371 shutdown_connections 2023-10-11T16:38:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa03d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa041e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.216 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/4085419371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa038560 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/4085419371 >> 172.21.15.111:0/4085419371 conn(0x5593aa02df50 msgr2=0x5593aa03cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/4085419371 shutdown_connections 2023-10-11T16:38:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/4085419371 wait complete. 2023-10-11T16:38:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 -- start start 2023-10-11T16:38:47.217 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.205+0000 7ff1e7c08b80 1 Processor -- start 2023-10-11T16:38:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa045ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa046020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1e7c08b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593aa0a53a0 con 0x5593aa03e720 2023-10-11T16:38:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593aa0a54e0 con 0x5593aa03eb90 2023-10-11T16:38:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1e7c08b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593aa0a5650 con 0x5593a9e946b0 2023-10-11T16:38:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1e7c08b80 10 client.0 _ll_register_callbacks cb 0x5593a9f49720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:38:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d1f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa045ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d179a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa046020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d1f9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa045ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:50672/0 (socket says 172.21.15.111:50672) 2023-10-11T16:38:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d1f9b700 1 -- 172.21.15.111:0/1797546328 learned_addr learned my addr 172.21.15.111:0/1797546328 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:38:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d279c700 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d179a700 1 -- 172.21.15.111:0/1797546328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 msgr2=0x5593aa045ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d179a700 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa045ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d179a700 1 -- 172.21.15.111:0/1797546328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 msgr2=0x5593aa0a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:38:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d179a700 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa0a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d179a700 1 -- 172.21.15.111:0/1797546328 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5593a9f50130 con 0x5593aa03e720 2023-10-11T16:38:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1d179a700 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa046020 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc09e0a0 tx=0x7ff1bc0a23c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd83650c44f1dd server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:38:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.206+0000 7ff1b57fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:38:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1b57fa700 1 -- 172.21.15.111:0/1797546328 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0a2070 con 0x5593aa03e720 2023-10-11T16:38:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1797546328 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1c801e040 con 0x5593aa03e720 2023-10-11T16:38:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1b57fa700 1 -- 172.21.15.111:0/1797546328 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff1bc0ca3f0 con 0x5593aa03e720 2023-10-11T16:38:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1b57fa700 1 -- 172.21.15.111:0/1797546328 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1bc0993a0 con 0x5593aa03e720 2023-10-11T16:38:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 10 client.16455 fetch_fsmap learned FSMap version 33 2023-10-11T16:38:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1797546328 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5593a9da0c00 con 0x5593aa03e720 2023-10-11T16:38:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1b57fa700 1 -- 172.21.15.111:0/1797546328 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1bc082030 con 0x5593aa03e720 2023-10-11T16:38:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1b57fa700 1 -- 172.21.15.111:0/1797546328 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff1bc094ac0 con 0x5593aa03e720 2023-10-11T16:38:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 10 client.16455 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:38:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 10 client.16455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-11T16:38:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 20 client.16455 populate_metadata read hostname 'smithi111' 2023-10-11T16:38:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593aa0a6940 0x5593aa0a8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:38:47.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 4 client.16455 mds_command: new command op to 24569 tid=0[{"prefix": "flush journal"}] 2023-10-11T16:38:47.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.207+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1797546328 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5593a9f58600 con 0x5593aa0a6940 2023-10-11T16:38:47.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.208+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593aa0a6940 0x5593aa0a8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:38:47.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.208+0000 7ff1b57fa700 10 client.16455 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:47.224 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.208+0000 7ff1d1f9b700 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593aa0a6940 0x5593aa0a8d50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3fbf55f55684673 server_cookie=6973d1febff1a37a in_seq=0 out_seq=0 2023-10-11T16:38:47.312 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.310+0000 7ff1b57fa700 1 -- 172.21.15.111:0/1797546328 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5593a9f58600 con 0x5593aa0a6940 2023-10-11T16:38:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.310+0000 7ff1b57fa700 10 client.16455 handle_command_reply: tid=0 2023-10-11T16:38:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.310+0000 7ff1e7c08b80 1 client.16455 shutdown 2023-10-11T16:38:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.310+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1797546328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 msgr2=0x5593aa046020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.313 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.310+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa046020 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc09e0a0 tx=0x7ff1bc0a23c0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.310+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1797546328 shutdown_connections 2023-10-11T16:38:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.310+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5593aa0a6940 0x5593aa0a8d50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5593aa03eb90 0x5593aa0a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5593a9e946b0 0x5593aa045ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.314 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1b57fa700 0 client.16455 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:38:47.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1e7c08b80 1 --2- 172.21.15.111:0/1797546328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5593aa03e720 0x5593aa046020 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1797546328 >> 172.21.15.111:0/1797546328 conn(0x5593aa02df50 msgr2=0x5593aa03ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:47.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1797546328 shutdown_connections 2023-10-11T16:38:47.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1e7c08b80 1 -- 172.21.15.111:0/1797546328 wait complete. 2023-10-11T16:38:47.315 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1e7c08b80 20 client.16455 trim_cache size 0 max 16384 2023-10-11T16:38:47.316 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:{ 2023-10-11T16:38:47.316 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "message": "", 2023-10-11T16:38:47.316 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout: "return_code": 0 2023-10-11T16:38:47.316 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stdout:} 2023-10-11T16:38:47.317 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff1b80a1340 msgr2=0x7ff1b80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.317 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff1b80a1340 0x7ff1b80a37f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400f870 tx=0x7ff1c400fc30 comp rx=0 tx=0).stop 2023-10-11T16:38:47.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100de0 msgr2=0x7ff1cc073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:38:47.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.311+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100de0 0x7ff1cc073330 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00e940 tx=0x7ff1bc00ec50 comp rx=0 tx=0).stop 2023-10-11T16:38:47.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.312+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2623709123 shutdown_connections 2023-10-11T16:38:47.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.312+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff1b80a1340 0x7ff1b80a37f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.318 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.312+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc105480 0x7ff1cc073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.312+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1cc100440 0x7ff1cc072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.312+0000 7ff1d41ff700 1 --2- 172.21.15.111:0/2623709123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1cc100de0 0x7ff1cc073330 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:38:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.312+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2623709123 >> 172.21.15.111:0/2623709123 conn(0x7ff1cc0fbcb0 msgr2=0x7ff1cc0fc760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:38:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.312+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2623709123 shutdown_connections 2023-10-11T16:38:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi111.front.sepia.ceph.com.smithi111.stderr:2023-10-11T16:38:47.312+0000 7ff1d41ff700 1 -- 172.21.15.111:0/2623709123 wait complete. 2023-10-11T16:38:47.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:47 smithi130 ceph-mon[131859]: pgmap v187: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 97 KiB/s wr, 95 op/s 2023-10-11T16:38:47.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:47 smithi111 ceph-mon[98527]: pgmap v187: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 97 KiB/s wr, 95 op/s 2023-10-11T16:38:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:47 smithi150 ceph-mon[132217]: pgmap v187: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 97 KiB/s wr, 95 op/s 2023-10-11T16:38:49.625 INFO:tasks.workunit.client.0.smithi111.stdout:ok 2023-10-11T16:38:49.630 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-11T16:38:49.630 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-11T16:38:49.722 INFO:tasks.workunit:Running workunit fs/misc/general_vxattrs.sh... 2023-10-11T16:38:49.723 DEBUG:teuthology.orchestra.run.smithi111:workunit test fs/misc/general_vxattrs.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh 2023-10-11T16:38:49.800 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir -p dir 2023-10-11T16:38:49.804 INFO:tasks.workunit.client.0.smithi111.stderr:+ setfattr -n ceph.dir.pin dir 2023-10-11T16:38:49.807 INFO:tasks.workunit.client.0.smithi111.stderr:+ exit 0 2023-10-11T16:38:49.808 INFO:tasks.workunit.client.0.smithi111.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-10-11T16:38:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:38:49.811 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2023-10-11T16:38:49.811 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-11T16:38:49.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:49 smithi130 ceph-mon[131859]: pgmap v188: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 62 op/s 2023-10-11T16:38:49.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:49 smithi111 ceph-mon[98527]: pgmap v188: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 62 op/s 2023-10-11T16:38:49.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:49 smithi150 ceph-mon[132217]: pgmap v188: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 62 op/s 2023-10-11T16:38:51.342 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-10-11T16:38:51.480 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4dc6a431bc2c451d9a5061e5357d01a8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_3f7a9d091654926539d989dd4c83dfee517abc9e/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-10-11T16:38:51.483 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-10-11T16:38:51.493 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:38:51.507 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2023-10-11T16:38:51.535 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2023-10-11T16:38:51.535 INFO:teuthology.task.exec:Executing custom commands... 2023-10-11T16:38:51.536 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:38:51.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2023-10-11T16:38:51.607 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:51 smithi111 ceph-mon[98527]: pgmap v189: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 60 KiB/s wr, 149 op/s 2023-10-11T16:38:51.665 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2023-10-11T16:38:51.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:51 smithi130 ceph-mon[131859]: pgmap v189: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 60 KiB/s wr, 149 op/s 2023-10-11T16:38:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:51 smithi150 ceph-mon[132217]: pgmap v189: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 60 KiB/s wr, 149 op/s 2023-10-11T16:38:53.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:53 smithi130 ceph-mon[131859]: pgmap v190: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 90 op/s 2023-10-11T16:38:53.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:53 smithi111 ceph-mon[98527]: pgmap v190: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 90 op/s 2023-10-11T16:38:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:53 smithi150 ceph-mon[132217]: pgmap v190: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 90 op/s 2023-10-11T16:38:55.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:55 smithi130 ceph-mon[131859]: pgmap v191: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 92 op/s 2023-10-11T16:38:55.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:55 smithi111 ceph-mon[98527]: pgmap v191: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 92 op/s 2023-10-11T16:38:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:55 smithi150 ceph-mon[132217]: pgmap v191: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 92 op/s 2023-10-11T16:38:57.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:57 smithi130 ceph-mon[131859]: pgmap v192: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 92 op/s 2023-10-11T16:38:57.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:57 smithi111 ceph-mon[98527]: pgmap v192: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 92 op/s 2023-10-11T16:38:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:57 smithi150 ceph-mon[132217]: pgmap v192: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 92 op/s 2023-10-11T16:38:59.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:38:59 smithi130 ceph-mon[131859]: pgmap v193: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 11 KiB/s wr, 90 op/s 2023-10-11T16:38:59.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:38:59 smithi111 ceph-mon[98527]: pgmap v193: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 11 KiB/s wr, 90 op/s 2023-10-11T16:38:59.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:38:59 smithi150 ceph-mon[132217]: pgmap v193: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 11 KiB/s wr, 90 op/s 2023-10-11T16:39:00.041 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:39:00.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 -- 172.21.15.111:0/527568044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12f8101ca0 msgr2=0x7f12f810a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 --2- 172.21.15.111:0/527568044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12f8101ca0 0x7f12f810a370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009990 tx=0x7f12e0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 -- 172.21.15.111:0/527568044 shutdown_connections 2023-10-11T16:39:00.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 --2- 172.21.15.111:0/527568044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f810a8b0 0x7f12f810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 --2- 172.21.15.111:0/527568044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12f8101390 0x7f12f8101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 --2- 172.21.15.111:0/527568044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12f8101ca0 0x7f12f810a370 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 -- 172.21.15.111:0/527568044 >> 172.21.15.111:0/527568044 conn(0x7f12f80770b0 msgr2=0x7f12f80774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:00.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 -- 172.21.15.111:0/527568044 shutdown_connections 2023-10-11T16:39:00.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.266+0000 7f12fcdff700 1 -- 172.21.15.111:0/527568044 wait complete. 2023-10-11T16:39:00.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.267+0000 7f12fcdff700 1 Processor -- start 2023-10-11T16:39:00.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.267+0000 7f12fcdff700 1 -- start start 2023-10-11T16:39:00.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.267+0000 7f12fcdff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12f8101390 0x7f12f80ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12fcdff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8101ca0 0x7f12f80ff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12fcdff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12f810a8b0 0x7f12f81a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12fcdff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80fff30 con 0x7f12f810a8b0 2023-10-11T16:39:00.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12fcdff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f81a4b40 con 0x7f12f8101390 2023-10-11T16:39:00.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12fcdff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f81a4c80 con 0x7f12f8101ca0 2023-10-11T16:39:00.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12f810a8b0 0x7f12f81a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12f810a8b0 0x7f12f81a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33256/0 (socket says 172.21.15.111:33256) 2023-10-11T16:39:00.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f6d9d700 1 -- 172.21.15.111:0/1287647302 learned_addr learned my addr 172.21.15.111:0/1287647302 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:00.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12effff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8101ca0 0x7f12f80ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f659c700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12f8101390 0x7f12f80ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f6d9d700 1 -- 172.21.15.111:0/1287647302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8101ca0 msgr2=0x7f12f80ff700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8101ca0 0x7f12f80ff700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f6d9d700 1 -- 172.21.15.111:0/1287647302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12f8101390 msgr2=0x7f12f80ff1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12f8101390 0x7f12f80ff1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.268+0000 7f12f6d9d700 1 -- 172.21.15.111:0/1287647302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e00095f0 con 0x7f12f810a8b0 2023-10-11T16:39:00.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.269+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12f810a8b0 0x7f12f81a4600 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f12f0007ec0 tx=0x7f12f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:00.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.269+0000 7f12ef7fe700 1 -- 172.21.15.111:0/1287647302 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0017070 con 0x7f12f810a8b0 2023-10-11T16:39:00.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.269+0000 7f12ef7fe700 1 -- 172.21.15.111:0/1287647302 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12f000f040 con 0x7f12f810a8b0 2023-10-11T16:39:00.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.269+0000 7f12fcdff700 1 -- 172.21.15.111:0/1287647302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f8110390 con 0x7f12f810a8b0 2023-10-11T16:39:00.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.269+0000 7f12ef7fe700 1 -- 172.21.15.111:0/1287647302 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0004020 con 0x7f12f810a8b0 2023-10-11T16:39:00.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.269+0000 7f12fcdff700 1 -- 172.21.15.111:0/1287647302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f8110200 con 0x7f12f810a8b0 2023-10-11T16:39:00.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.272+0000 7f12ef7fe700 1 -- 172.21.15.111:0/1287647302 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f12f000db20 con 0x7f12f810a8b0 2023-10-11T16:39:00.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.274+0000 7f13106bcb80 1 Processor -- start 2023-10-11T16:39:00.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.274+0000 7f13106bcb80 1 -- start start 2023-10-11T16:39:00.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.274+0000 7f12ef7fe700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f12e40a12c0 0x7f12e40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12ef7fe700 1 -- 172.21.15.111:0/1287647302 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f12f00cc530 con 0x7f12f810a8b0 2023-10-11T16:39:00.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d120f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 0x55d44d125370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d125d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12effff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f12e40a12c0 0x7f12e40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f13106bcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d026060 con 0x55d44d121470 2023-10-11T16:39:00.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f13106bcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d0788a0 con 0x55d44d1258b0 2023-10-11T16:39:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f13106bcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d071d30 con 0x55d44d120b60 2023-10-11T16:39:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d120f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 0x55d44d125370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d120f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40662/0 (socket says 172.21.15.111:40662) 2023-10-11T16:39:00.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f659c700 1 -- 172.21.15.111:0/2474881545 learned_addr learned my addr 172.21.15.111:0/2474881545 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:00.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/2474881545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d125d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f659c700 1 -- 172.21.15.111:0/2474881545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 msgr2=0x55d44d125d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f659c700 1 --2- 172.21.15.111:0/2474881545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d125d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f659c700 1 -- 172.21.15.111:0/2474881545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 msgr2=0x55d44d125370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f659c700 1 --2- 172.21.15.111:0/2474881545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 0x55d44d125370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.275+0000 7f12f659c700 1 -- 172.21.15.111:0/2474881545 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d44cef3630 con 0x55d44d120b60 2023-10-11T16:39:00.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.276+0000 7f12effff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f12e40a12c0 0x7f12e40a3770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f12e00095c0 tx=0x7f12e0005c20 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:00.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.276+0000 7f12f659c700 1 --2- 172.21.15.111:0/2474881545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d120f30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009d80 tx=0x7f12e8011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46806f9b1a3de2b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:00.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.276+0000 7f12ecff9700 1 -- 172.21.15.111:0/2474881545 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8004230 con 0x55d44d120b60 2023-10-11T16:39:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.276+0000 7f12ecff9700 1 -- 172.21.15.111:0/2474881545 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12e8004390 con 0x55d44d120b60 2023-10-11T16:39:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.276+0000 7f12ecff9700 1 -- 172.21.15.111:0/2474881545 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e80197a0 con 0x55d44d120b60 2023-10-11T16:39:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 -- 172.21.15.111:0/2474881545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 msgr2=0x55d44d120f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2474881545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d120f30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009d80 tx=0x7f12e8011720 comp rx=0 tx=0).stop 2023-10-11T16:39:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 -- 172.21.15.111:0/2474881545 shutdown_connections 2023-10-11T16:39:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2474881545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d125d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2474881545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d120f30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2474881545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 0x55d44d125370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 -- 172.21.15.111:0/2474881545 >> 172.21.15.111:0/2474881545 conn(0x55d44d116360 msgr2=0x55d44d11bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 -- 172.21.15.111:0/2474881545 shutdown_connections 2023-10-11T16:39:00.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 -- 172.21.15.111:0/2474881545 wait complete. 2023-10-11T16:39:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.277+0000 7f13106bcb80 1 -- start start 2023-10-11T16:39:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f13106bcb80 1 Processor -- start 2023-10-11T16:39:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d120b60 0x55d44d12d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 0x55d44d12a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d1258b0 0x55d44d12a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f13106bcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d0138e0 con 0x55d44d1258b0 2023-10-11T16:39:00.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f13106bcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d013580 con 0x55d44d120b60 2023-10-11T16:39:00.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f13106bcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44cef5920 con 0x55d44d121470 2023-10-11T16:39:00.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f13106bcb80 10 client.0 _ll_register_callbacks cb 0x55d44d031780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:00.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f12f6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d1258b0 0x55d44d12a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f12f6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d1258b0 0x55d44d12a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33266/0 (socket says 172.21.15.111:33266) 2023-10-11T16:39:00.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f12f6d9d700 1 -- 172.21.15.111:0/1343878341 learned_addr learned my addr 172.21.15.111:0/1343878341 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:00.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f12f659c700 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d120b60 0x55d44d12d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.278+0000 7f12effff700 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 0x55d44d12a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12f6d9d700 1 -- 172.21.15.111:0/1343878341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 msgr2=0x55d44d12a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 0x55d44d12a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12f6d9d700 1 -- 172.21.15.111:0/1343878341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d120b60 msgr2=0x55d44d12d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:00.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d120b60 0x55d44d12d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12f6d9d700 1 -- 172.21.15.111:0/1343878341 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e80099e0 con 0x55d44d1258b0 2023-10-11T16:39:00.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d1258b0 0x55d44d12a840 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f12f000d700 tx=0x7f12f00192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554e0777966502a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:00.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12ddffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:00.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12ddffb700 1 -- 172.21.15.111:0/1343878341 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0099360 con 0x55d44d1258b0 2023-10-11T16:39:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12ddffb700 1 -- 172.21.15.111:0/1343878341 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f12f009d310 con 0x55d44d1258b0 2023-10-11T16:39:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f13106bcb80 1 -- 172.21.15.111:0/1343878341 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d44d113b20 con 0x55d44d1258b0 2023-10-11T16:39:00.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f12ddffb700 1 -- 172.21.15.111:0/1343878341 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12f00a16a0 con 0x55d44d1258b0 2023-10-11T16:39:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f13106bcb80 10 client.16467 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.279+0000 7f13106bcb80 1 -- 172.21.15.111:0/1343878341 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d44d038190 con 0x55d44d1258b0 2023-10-11T16:39:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f12ddffb700 1 -- 172.21.15.111:0/1343878341 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d44d113b20 con 0x55d44d1258b0 2023-10-11T16:39:00.301 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f12ddffb700 1 -- 172.21.15.111:0/1343878341 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f12f00cc1c0 con 0x55d44d1258b0 2023-10-11T16:39:00.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f13106bcb80 10 client.16467 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:00.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f13106bcb80 10 client.16467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:00.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f13106bcb80 20 client.16467 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:00.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f13106bcb80 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d44d13a8a0 0x55d44d13ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f13106bcb80 4 client.16467 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:00.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f13106bcb80 1 -- 172.21.15.111:0/1343878341 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d44d0340a0 con 0x55d44d13a8a0 2023-10-11T16:39:00.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.280+0000 7f12f659c700 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d44d13a8a0 0x55d44d13ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.281+0000 7f12ddffb700 10 client.16467 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:00.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.281+0000 7f12f659c700 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d44d13a8a0 0x55d44d13ccb0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c80c396014fa06e server_cookie=c2aade50b52accf0 in_seq=0 out_seq=0 2023-10-11T16:39:00.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.281+0000 7f12ddffb700 1 -- 172.21.15.111:0/1343878341 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d44d0340a0 con 0x55d44d13a8a0 2023-10-11T16:39:00.304 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f12ddffb700 10 client.16467 handle_command_reply: tid=0 2023-10-11T16:39:00.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 client.16467 shutdown 2023-10-11T16:39:00.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 -- 172.21.15.111:0/1343878341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d1258b0 msgr2=0x55d44d12a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d1258b0 0x55d44d12a840 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f12f000d700 tx=0x7f12f00192c0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 -- 172.21.15.111:0/1343878341 shutdown_connections 2023-10-11T16:39:00.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d44d13a8a0 0x55d44d13ccb0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d1258b0 0x55d44d12a840 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f12ddffb700 0 client.16467 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:00.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d120b60 0x55d44d12d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 --2- 172.21.15.111:0/1343878341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 0x55d44d12a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 -- 172.21.15.111:0/1343878341 >> 172.21.15.111:0/1343878341 conn(0x55d44d116360 msgr2=0x55d44d126d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:00.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 -- 172.21.15.111:0/1343878341 shutdown_connections 2023-10-11T16:39:00.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 1 -- 172.21.15.111:0/1343878341 wait complete. 2023-10-11T16:39:00.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.282+0000 7f13106bcb80 20 client.16467 trim_cache size 0 max 16384 2023-10-11T16:39:00.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.299+0000 7f13106bcb80 1 Processor -- start 2023-10-11T16:39:00.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.299+0000 7f13106bcb80 1 -- start start 2023-10-11T16:39:00.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d11bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 0x55d44d1319d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d124820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f13106bcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d137f00 con 0x55d44d121470 2023-10-11T16:39:00.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f13106bcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d1139a0 con 0x55d44d1258b0 2023-10-11T16:39:00.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f13106bcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d136890 con 0x55d44d120b60 2023-10-11T16:39:00.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 0x55d44d1319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d11bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f6d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d124820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d11bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40682/0 (socket says 172.21.15.111:40682) 2023-10-11T16:39:00.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f659c700 1 -- 172.21.15.111:0/689039790 learned_addr learned my addr 172.21.15.111:0/689039790 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:00.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f6d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d124820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:58730/0 (socket says 172.21.15.111:58730) 2023-10-11T16:39:00.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f659c700 1 -- 172.21.15.111:0/689039790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 msgr2=0x55d44d124820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:00.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f659c700 1 --2- 172.21.15.111:0/689039790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d124820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f659c700 1 -- 172.21.15.111:0/689039790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 msgr2=0x55d44d1319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:00.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f659c700 1 --2- 172.21.15.111:0/689039790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 0x55d44d1319d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.300+0000 7f12f659c700 1 -- 172.21.15.111:0/689039790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d44d117b00 con 0x55d44d120b60 2023-10-11T16:39:00.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.301+0000 7f12f659c700 1 --2- 172.21.15.111:0/689039790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d11bbe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f12e80046c0 tx=0x7f12e8009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42aaf21a3f4e947f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:00.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.301+0000 7f12ed7fa700 1 -- 172.21.15.111:0/689039790 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8029070 con 0x55d44d120b60 2023-10-11T16:39:00.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.301+0000 7f12ed7fa700 1 -- 172.21.15.111:0/689039790 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12e8003f40 con 0x55d44d120b60 2023-10-11T16:39:00.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.301+0000 7f12ed7fa700 1 -- 172.21.15.111:0/689039790 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e80038e0 con 0x55d44d120b60 2023-10-11T16:39:00.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.301+0000 7f13106bcb80 1 -- 172.21.15.111:0/689039790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 msgr2=0x55d44d11bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.301+0000 7f13106bcb80 1 --2- 172.21.15.111:0/689039790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d11bbe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f12e80046c0 tx=0x7f12e8009f50 comp rx=0 tx=0).stop 2023-10-11T16:39:00.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 -- 172.21.15.111:0/689039790 shutdown_connections 2023-10-11T16:39:00.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 --2- 172.21.15.111:0/689039790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d124820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 --2- 172.21.15.111:0/689039790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d120b60 0x55d44d11bbe0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 --2- 172.21.15.111:0/689039790 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d121470 0x55d44d1319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 -- 172.21.15.111:0/689039790 >> 172.21.15.111:0/689039790 conn(0x55d44d116360 msgr2=0x55d44d136180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:00.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 -- 172.21.15.111:0/689039790 shutdown_connections 2023-10-11T16:39:00.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 -- 172.21.15.111:0/689039790 wait complete. 2023-10-11T16:39:00.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 -- start start 2023-10-11T16:39:00.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.302+0000 7f13106bcb80 1 Processor -- start 2023-10-11T16:39:00.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d120b60 0x55d44d1314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 0x55d44d131a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f13106bcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d12f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f13106bcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d1300a0 con 0x55d44d120b60 2023-10-11T16:39:00.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f13106bcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d130210 con 0x55d44d1258b0 2023-10-11T16:39:00.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f13106bcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44d1303d0 con 0x55d44d121470 2023-10-11T16:39:00.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f13106bcb80 10 client.0 _ll_register_callbacks cb 0x7f12f0098e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:00.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d120b60 0x55d44d1314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d120b60 0x55d44d1314f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33288/0 (socket says 172.21.15.111:33288) 2023-10-11T16:39:00.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f659c700 1 -- 172.21.15.111:0/2135707077 learned_addr learned my addr 172.21.15.111:0/2135707077 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:00.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d12f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12effff700 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 0x55d44d131a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f659c700 1 -- 172.21.15.111:0/2135707077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 msgr2=0x55d44d131a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:00.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f659c700 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 0x55d44d131a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f659c700 1 -- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 msgr2=0x55d44d12f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f659c700 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d12f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f659c700 1 -- 172.21.15.111:0/2135707077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d44cef3630 con 0x55d44d120b60 2023-10-11T16:39:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.303+0000 7f12f6d9d700 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d12f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f12f659c700 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d120b60 0x55d44d1314f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f12e8005fe0 tx=0x7f12e8004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b1cd272d619eb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f12ddffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f12ddffb700 1 -- 172.21.15.111:0/2135707077 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8029040 con 0x55d44d120b60 2023-10-11T16:39:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f12ddffb700 1 -- 172.21.15.111:0/2135707077 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f12e8003880 con 0x55d44d120b60 2023-10-11T16:39:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f13106bcb80 1 -- 172.21.15.111:0/2135707077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12e8004390 con 0x55d44d120b60 2023-10-11T16:39:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f12ddffb700 1 -- 172.21.15.111:0/2135707077 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12e8003a40 con 0x55d44d120b60 2023-10-11T16:39:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f13106bcb80 10 client.16473 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:00.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f13106bcb80 1 -- 172.21.15.111:0/2135707077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d44ce9fde0 con 0x55d44d120b60 2023-10-11T16:39:00.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f12ddffb700 1 -- 172.21.15.111:0/2135707077 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12e8004390 con 0x55d44d120b60 2023-10-11T16:39:00.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f12ddffb700 1 -- 172.21.15.111:0/2135707077 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f12e8011440 con 0x55d44d120b60 2023-10-11T16:39:00.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f13106bcb80 10 client.16473 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f13106bcb80 10 client.16473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.304+0000 7f13106bcb80 20 client.16473 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:00.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.305+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d44d18f670 0x55d44d191a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.305+0000 7f13106bcb80 4 client.16473 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:39:00.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.305+0000 7f13106bcb80 1 -- 172.21.15.111:0/2135707077 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d44d040660 con 0x55d44d18f670 2023-10-11T16:39:00.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.305+0000 7f12effff700 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d44d18f670 0x55d44d191a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.305+0000 7f12ddffb700 10 client.16473 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.305+0000 7f12effff700 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d44d18f670 0x55d44d191a80 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ccf247bf38db7b3 server_cookie=226de66330a9e83b in_seq=0 out_seq=0 2023-10-11T16:39:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f12ddffb700 1 -- 172.21.15.111:0/2135707077 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d44d040660 con 0x55d44d18f670 2023-10-11T16:39:00.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f12ddffb700 10 client.16473 handle_command_reply: tid=0 2023-10-11T16:39:00.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 client.16473 shutdown 2023-10-11T16:39:00.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 -- 172.21.15.111:0/2135707077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d120b60 msgr2=0x55d44d1314f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d120b60 0x55d44d1314f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f12e8005fe0 tx=0x7f12e8004590 comp rx=0 tx=0).stop 2023-10-11T16:39:00.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 -- 172.21.15.111:0/2135707077 shutdown_connections 2023-10-11T16:39:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d44d18f670 0x55d44d191a80 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d44d1258b0 0x55d44d12f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d44d120b60 0x55d44d1314f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f12ddffb700 0 client.16473 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:00.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 --2- 172.21.15.111:0/2135707077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d44d121470 0x55d44d131a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 -- 172.21.15.111:0/2135707077 >> 172.21.15.111:0/2135707077 conn(0x55d44d116360 msgr2=0x55d44d1289e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:00.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 -- 172.21.15.111:0/2135707077 shutdown_connections 2023-10-11T16:39:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 1 -- 172.21.15.111:0/2135707077 wait complete. 2023-10-11T16:39:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.306+0000 7f13106bcb80 20 client.16473 trim_cache size 0 max 16384 2023-10-11T16:39:00.338 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:39:00.338 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:39:00.338 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:39:00.338 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:39:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 -- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f12e40a12c0 msgr2=0x7f12e40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f12e40a12c0 0x7f12e40a3770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f12e00095c0 tx=0x7f12e0005c20 comp rx=0 tx=0).stop 2023-10-11T16:39:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 -- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12f810a8b0 msgr2=0x7f12f81a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12f810a8b0 0x7f12f81a4600 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f12f0007ec0 tx=0x7f12f000d3b0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 -- 172.21.15.111:0/1287647302 shutdown_connections 2023-10-11T16:39:00.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f12e40a12c0 0x7f12e40a3770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12f810a8b0 0x7f12f81a4600 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f12f8101390 0x7f12f80ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 --2- 172.21.15.111:0/1287647302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12f8101ca0 0x7f12f80ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 -- 172.21.15.111:0/1287647302 >> 172.21.15.111:0/1287647302 conn(0x7f12f80770b0 msgr2=0x7f12f80fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:00.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 -- 172.21.15.111:0/1287647302 shutdown_connections 2023-10-11T16:39:00.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.307+0000 7f12fcdff700 1 -- 172.21.15.111:0/1287647302 wait complete. 2023-10-11T16:39:00.343 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:39:00.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:39:00.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 -- 172.21.15.111:0/746869128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 msgr2=0x7ff6cc10ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/746869128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 0x7ff6cc10ef70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009f90 tx=0x7ff6c400b330 comp rx=0 tx=0).stop 2023-10-11T16:39:00.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 -- 172.21.15.111:0/746869128 shutdown_connections 2023-10-11T16:39:00.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/746869128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 0x7ff6cc10ef70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/746869128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc072e30 0x7ff6cc073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/746869128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6cc073740 0x7ff6cc10c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 -- 172.21.15.111:0/746869128 >> 172.21.15.111:0/746869128 conn(0x7ff6cc077080 msgr2=0x7ff6cc077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:00.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 -- 172.21.15.111:0/746869128 shutdown_connections 2023-10-11T16:39:00.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.544+0000 7ff6d27ff700 1 -- 172.21.15.111:0/746869128 wait complete. 2023-10-11T16:39:00.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.546+0000 7ff6d27ff700 1 Processor -- start 2023-10-11T16:39:00.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.546+0000 7ff6d27ff700 1 -- start start 2023-10-11T16:39:00.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6d27ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6cc072e30 0x7ff6cc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6d27ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc073740 0x7ff6cc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6d27ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 0x7ff6cc19bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6d27ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc19c5c0 con 0x7ff6cc10cb90 2023-10-11T16:39:00.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6d27ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc19c730 con 0x7ff6cc073740 2023-10-11T16:39:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6d27ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc19c8f0 con 0x7ff6cc072e30 2023-10-11T16:39:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6cbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6cc072e30 0x7ff6cc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6cbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6cc072e30 0x7ff6cc1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40712/0 (socket says 172.21.15.111:40712) 2023-10-11T16:39:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6cbfff700 1 -- 172.21.15.111:0/3267372753 learned_addr learned my addr 172.21.15.111:0/3267372753 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.547+0000 7ff6cb7fe700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc073740 0x7ff6cc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6d0d9c700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 0x7ff6cc19bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6cbfff700 1 -- 172.21.15.111:0/3267372753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc073740 msgr2=0x7ff6cc198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6cbfff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc073740 0x7ff6cc198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6cbfff700 1 -- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 msgr2=0x7ff6cc19bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6cbfff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 0x7ff6cc19bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6cbfff700 1 -- 172.21.15.111:0/3267372753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c00095f0 con 0x7ff6cc072e30 2023-10-11T16:39:00.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6cb7fe700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc073740 0x7ff6cc198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:00.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6d0d9c700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 0x7ff6cc19bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:39:00.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6cbfff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6cc072e30 0x7ff6cc1986d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc00e9a0 tx=0x7ff6bc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:00.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6c97fa700 1 -- 172.21.15.111:0/3267372753 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc009790 con 0x7ff6cc072e30 2023-10-11T16:39:00.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c4009c40 con 0x7ff6cc072e30 2023-10-11T16:39:00.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6c97fa700 1 -- 172.21.15.111:0/3267372753 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6bc00cbd0 con 0x7ff6cc072e30 2023-10-11T16:39:00.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6c97fa700 1 -- 172.21.15.111:0/3267372753 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc018680 con 0x7ff6cc072e30 2023-10-11T16:39:00.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.548+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6cc112570 con 0x7ff6cc072e30 2023-10-11T16:39:00.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.549+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6cc1123e0 con 0x7ff6cc072e30 2023-10-11T16:39:00.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.552+0000 7ff6c97fa700 1 -- 172.21.15.111:0/3267372753 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7ff6bc00cd40 con 0x7ff6cc072e30 2023-10-11T16:39:00.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.553+0000 7ff6c97fa700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff6b40a1340 0x7ff6b40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:00.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.553+0000 7ff6c97fa700 1 -- 172.21.15.111:0/3267372753 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7ff6bc014070 con 0x7ff6cc072e30 2023-10-11T16:39:00.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.554+0000 7ff6cb7fe700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff6b40a1340 0x7ff6b40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:00.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.554+0000 7ff6cb7fe700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff6b40a1340 0x7ff6b40a37f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc199ce0 tx=0x7ff6c0009310 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:00.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.555+0000 7ff6c97fa700 1 -- 172.21.15.111:0/3267372753 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff6bc0965c0 con 0x7ff6cc072e30 2023-10-11T16:39:00.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.811+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6cc112700 con 0x7ff6cc072e30 2023-10-11T16:39:00.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.812+0000 7ff6c97fa700 1 -- 172.21.15.111:0/3267372753 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10829 (secure 0 0 0) 0x7ff6bc026090 con 0x7ff6cc072e30 2023-10-11T16:39:00.814 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:39:00.814 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:36:18.517722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:39:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff6b40a1340 msgr2=0x7ff6b40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff6b40a1340 0x7ff6b40a37f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc199ce0 tx=0x7ff6c0009310 comp rx=0 tx=0).stop 2023-10-11T16:39:00.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6cc072e30 msgr2=0x7ff6cc1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6cc072e30 0x7ff6cc1986d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc00e9a0 tx=0x7ff6bc00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 shutdown_connections 2023-10-11T16:39:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff6b40a1340 0x7ff6b40a37f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6cc10cb90 0x7ff6cc19bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6cc072e30 0x7ff6cc1986d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 --2- 172.21.15.111:0/3267372753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff6cc073740 0x7ff6cc198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 >> 172.21.15.111:0/3267372753 conn(0x7ff6cc077080 msgr2=0x7ff6cc0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:00.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 shutdown_connections 2023-10-11T16:39:00.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:00.815+0000 7ff6d27ff700 1 -- 172.21.15.111:0/3267372753 wait complete. 2023-10-11T16:39:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-10-11T16:39:00.835 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:39:01.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 -- 172.21.15.111:0/1793653258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d8101660 msgr2=0x7f11d8104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 --2- 172.21.15.111:0/1793653258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d8101660 0x7f11d8104ef0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009990 tx=0x7f11c0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 -- 172.21.15.111:0/1793653258 shutdown_connections 2023-10-11T16:39:01.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 --2- 172.21.15.111:0/1793653258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11d8105620 0x7f11d810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 --2- 172.21.15.111:0/1793653258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d8100cc0 0x7f11d8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 --2- 172.21.15.111:0/1793653258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d8101660 0x7f11d8104ef0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 -- 172.21.15.111:0/1793653258 >> 172.21.15.111:0/1793653258 conn(0x7f11d8077560 msgr2=0x7f11d8077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 -- 172.21.15.111:0/1793653258 shutdown_connections 2023-10-11T16:39:01.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.036+0000 7f11de7ff700 1 -- 172.21.15.111:0/1793653258 wait complete. 2023-10-11T16:39:01.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.037+0000 7f11de7ff700 1 Processor -- start 2023-10-11T16:39:01.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.037+0000 7f11de7ff700 1 -- start start 2023-10-11T16:39:01.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11de7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d8100cc0 0x7f11d8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11de7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11d8101660 0x7f11d8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11de7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d8105620 0x7f11d819bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11de7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d819c5f0 con 0x7f11d8101660 2023-10-11T16:39:01.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11de7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d819c760 con 0x7f11d8100cc0 2023-10-11T16:39:01.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11de7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d819c920 con 0x7f11d8105620 2023-10-11T16:39:01.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11cffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11d8101660 0x7f11d8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11cffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11d8101660 0x7f11d8198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33322/0 (socket says 172.21.15.111:33322) 2023-10-11T16:39:01.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11cffff700 1 -- 172.21.15.111:0/2923171578 learned_addr learned my addr 172.21.15.111:0/2923171578 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11d7fff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d8100cc0 0x7f11d8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11dcd9c700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d8105620 0x7f11d819bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11cffff700 1 -- 172.21.15.111:0/2923171578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d8105620 msgr2=0x7f11d819bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11cffff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d8105620 0x7f11d819bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11cffff700 1 -- 172.21.15.111:0/2923171578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d8100cc0 msgr2=0x7f11d8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11cffff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d8100cc0 0x7f11d8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.038+0000 7f11cffff700 1 -- 172.21.15.111:0/2923171578 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c00095f0 con 0x7f11d8101660 2023-10-11T16:39:01.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.039+0000 7f11cffff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11d8101660 0x7f11d8198c40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f11c0006010 tx=0x7f11c0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.039+0000 7f11d5ffb700 1 -- 172.21.15.111:0/2923171578 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c001d070 con 0x7f11d8101660 2023-10-11T16:39:01.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.039+0000 7f11d5ffb700 1 -- 172.21.15.111:0/2923171578 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11c0022470 con 0x7f11d8101660 2023-10-11T16:39:01.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.039+0000 7f11de7ff700 1 -- 172.21.15.111:0/2923171578 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d8110320 con 0x7f11d8101660 2023-10-11T16:39:01.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.039+0000 7f11d5ffb700 1 -- 172.21.15.111:0/2923171578 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0017650 con 0x7f11d8101660 2023-10-11T16:39:01.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.039+0000 7f11de7ff700 1 -- 172.21.15.111:0/2923171578 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d8110190 con 0x7f11d8101660 2023-10-11T16:39:01.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.042+0000 7f11d5ffb700 1 -- 172.21.15.111:0/2923171578 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f11c0022890 con 0x7f11d8101660 2023-10-11T16:39:01.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.043+0000 7f11f215db80 1 Processor -- start 2023-10-11T16:39:01.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.043+0000 7f11f215db80 1 -- start start 2023-10-11T16:39:01.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.043+0000 7f11d5ffb700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f11c40a1340 0x7f11c40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d5ffb700 1 -- 172.21.15.111:0/2923171578 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f11c00ce2e0 con 0x7f11d8101660 2023-10-11T16:39:01.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11f215db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d098f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11f215db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d09d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11f215db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d09dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11f215db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16cf9e060 con 0x55f16d09d8b0 2023-10-11T16:39:01.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11f215db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16cff08a0 con 0x55f16d098b60 2023-10-11T16:39:01.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11f215db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16cfe9d30 con 0x55f16d099470 2023-10-11T16:39:01.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11dcd9c700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f11c40a1340 0x7f11c40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d098f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d098f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:58796/0 (socket says 172.21.15.111:58796) 2023-10-11T16:39:01.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11dcd9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d09dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d7fff700 1 -- 172.21.15.111:0/2226958615 learned_addr learned my addr 172.21.15.111:0/2226958615 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11cffff700 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d09d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d7fff700 1 -- 172.21.15.111:0/2226958615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 msgr2=0x55f16d09d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d7fff700 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d09d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d7fff700 1 -- 172.21.15.111:0/2226958615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 msgr2=0x55f16d09dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d7fff700 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d09dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11d7fff700 1 -- 172.21.15.111:0/2226958615 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f16ce6b630 con 0x55f16d098b60 2023-10-11T16:39:01.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.044+0000 7f11cffff700 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d09d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:01.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.045+0000 7f11dcd9c700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f11c40a1340 0x7f11c40a37f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009ec0 tx=0x7f11d00092f0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.045+0000 7f11d7fff700 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d098f30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009d80 tx=0x7f11c8011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde0f42fbffe99f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.045+0000 7f11ce7fc700 1 -- 172.21.15.111:0/2226958615 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8004230 con 0x55f16d098b60 2023-10-11T16:39:01.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.045+0000 7f11ce7fc700 1 -- 172.21.15.111:0/2226958615 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11c8004390 con 0x55f16d098b60 2023-10-11T16:39:01.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.045+0000 7f11ce7fc700 1 -- 172.21.15.111:0/2226958615 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80197a0 con 0x55f16d098b60 2023-10-11T16:39:01.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 -- 172.21.15.111:0/2226958615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 msgr2=0x55f16d098f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d098f30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009d80 tx=0x7f11c8011720 comp rx=0 tx=0).stop 2023-10-11T16:39:01.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 -- 172.21.15.111:0/2226958615 shutdown_connections 2023-10-11T16:39:01.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d09dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d098f30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 --2- 172.21.15.111:0/2226958615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d09d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 -- 172.21.15.111:0/2226958615 >> 172.21.15.111:0/2226958615 conn(0x55f16d08e360 msgr2=0x55f16d093b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 -- 172.21.15.111:0/2226958615 shutdown_connections 2023-10-11T16:39:01.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.046+0000 7f11f215db80 1 -- 172.21.15.111:0/2226958615 wait complete. 2023-10-11T16:39:01.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.047+0000 7f11f215db80 1 -- start start 2023-10-11T16:39:01.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.047+0000 7f11f215db80 1 Processor -- start 2023-10-11T16:39:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11f215db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d0a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11f215db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0a2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11f215db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d0a2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11f215db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16cf8b8e0 con 0x55f16d09d8b0 2023-10-11T16:39:01.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11f215db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16cf8b580 con 0x55f16d098b60 2023-10-11T16:39:01.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11f215db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16ce6d920 con 0x55f16d099470 2023-10-11T16:39:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11f215db80 10 client.0 _ll_register_callbacks cb 0x55f16cfa9780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11dcd9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d0a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11dcd9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d0a2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33338/0 (socket says 172.21.15.111:33338) 2023-10-11T16:39:01.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11dcd9c700 1 -- 172.21.15.111:0/3974407674 learned_addr learned my addr 172.21.15.111:0/3974407674 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11cffff700 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11d7fff700 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d0a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11dcd9c700 1 -- 172.21.15.111:0/3974407674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 msgr2=0x55f16d0a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11dcd9c700 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11dcd9c700 1 -- 172.21.15.111:0/3974407674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 msgr2=0x55f16d0a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11dcd9c700 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d0a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.048+0000 7f11dcd9c700 1 -- 172.21.15.111:0/3974407674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c80099e0 con 0x55f16d09d8b0 2023-10-11T16:39:01.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.049+0000 7f11dcd9c700 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d0a2870 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f11d001aeb0 tx=0x7f11d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe4f6077a18c92f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.049+0000 7f11b77fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:01.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.049+0000 7f11b77fe700 1 -- 172.21.15.111:0/3974407674 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d000e070 con 0x55f16d09d8b0 2023-10-11T16:39:01.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.049+0000 7f11b77fe700 1 -- 172.21.15.111:0/3974407674 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f11d0021040 con 0x55f16d09d8b0 2023-10-11T16:39:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.049+0000 7f11f215db80 1 -- 172.21.15.111:0/3974407674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f16d08bb20 con 0x55f16d09d8b0 2023-10-11T16:39:01.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.049+0000 7f11b77fe700 1 -- 172.21.15.111:0/3974407674 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11d001aee0 con 0x55f16d09d8b0 2023-10-11T16:39:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.049+0000 7f11f215db80 10 client.16488 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.049+0000 7f11f215db80 1 -- 172.21.15.111:0/3974407674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f16cfb0190 con 0x55f16d09d8b0 2023-10-11T16:39:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11b77fe700 1 -- 172.21.15.111:0/3974407674 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f16d08bb20 con 0x55f16d09d8b0 2023-10-11T16:39:01.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11b77fe700 1 -- 172.21.15.111:0/3974407674 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f11d0025070 con 0x55f16d09d8b0 2023-10-11T16:39:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11f215db80 10 client.16488 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11f215db80 10 client.16488 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:39:01.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11f215db80 20 client.16488 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11f215db80 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f16d0b2880 0x55f16d0b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11f215db80 4 client.16488 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:01.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11f215db80 1 -- 172.21.15.111:0/3974407674 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f16cfac0a0 con 0x55f16d0b2880 2023-10-11T16:39:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.050+0000 7f11d7fff700 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f16d0b2880 0x55f16d0b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.051+0000 7f11b77fe700 10 client.16488 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:39:01.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.051+0000 7f11d7fff700 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f16d0b2880 0x55f16d0b4c90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1d87029c1f5d36ae server_cookie=e5e4ed7999b58f92 in_seq=0 out_seq=0 2023-10-11T16:39:01.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11b77fe700 1 -- 172.21.15.111:0/3974407674 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f16cfac0a0 con 0x55f16d0b2880 2023-10-11T16:39:01.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11b77fe700 10 client.16488 handle_command_reply: tid=0 2023-10-11T16:39:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 client.16488 shutdown 2023-10-11T16:39:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 -- 172.21.15.111:0/3974407674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 msgr2=0x55f16d0a2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d0a2870 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f11d001aeb0 tx=0x7f11d000b040 comp rx=0 tx=0).stop 2023-10-11T16:39:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 -- 172.21.15.111:0/3974407674 shutdown_connections 2023-10-11T16:39:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f16d0b2880 0x55f16d0b4c90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d09d8b0 0x55f16d0a2870 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d098b60 0x55f16d0a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11b77fe700 0 client.16488 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:39:01.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 --2- 172.21.15.111:0/3974407674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 -- 172.21.15.111:0/3974407674 >> 172.21.15.111:0/3974407674 conn(0x55f16d08e360 msgr2=0x55f16d092550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 -- 172.21.15.111:0/3974407674 shutdown_connections 2023-10-11T16:39:01.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 1 -- 172.21.15.111:0/3974407674 wait complete. 2023-10-11T16:39:01.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.052+0000 7f11f215db80 20 client.16488 trim_cache size 0 max 16384 2023-10-11T16:39:01.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11f215db80 1 Processor -- start 2023-10-11T16:39:01.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11f215db80 1 -- start start 2023-10-11T16:39:01.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11f215db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d098b60 0x55f16d0a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11f215db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d099470 0x55f16d093670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11f215db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 0x55f16d093bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11f215db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16d0a3840 con 0x55f16d099470 2023-10-11T16:39:01.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11f215db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16d08dee0 con 0x55f16d09d8b0 2023-10-11T16:39:01.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11f215db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16d08e170 con 0x55f16d098b60 2023-10-11T16:39:01.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11cffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d099470 0x55f16d093670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11cffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d099470 0x55f16d093670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33354/0 (socket says 172.21.15.111:33354) 2023-10-11T16:39:01.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.066+0000 7f11cffff700 1 -- 172.21.15.111:0/1531376497 learned_addr learned my addr 172.21.15.111:0/1531376497 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11d7fff700 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d098b60 0x55f16d0a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11dcd9c700 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 0x55f16d093bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11cffff700 1 -- 172.21.15.111:0/1531376497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d098b60 msgr2=0x55f16d0a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11cffff700 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d098b60 0x55f16d0a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11cffff700 1 -- 172.21.15.111:0/1531376497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 msgr2=0x55f16d093bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11cffff700 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 0x55f16d093bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11cffff700 1 -- 172.21.15.111:0/1531376497 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f16d08fb00 con 0x55f16d099470 2023-10-11T16:39:01.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11cffff700 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d099470 0x55f16d093670 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f11c00cb680 tx=0x7f11c001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829cd2ac01257cf server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11ceffd700 1 -- 172.21.15.111:0/1531376497 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c00b2070 con 0x55f16d099470 2023-10-11T16:39:01.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11ceffd700 1 -- 172.21.15.111:0/1531376497 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11c009a990 con 0x55f16d099470 2023-10-11T16:39:01.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.067+0000 7f11ceffd700 1 -- 172.21.15.111:0/1531376497 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c009b290 con 0x55f16d099470 2023-10-11T16:39:01.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 -- 172.21.15.111:0/1531376497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d099470 msgr2=0x55f16d093670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d099470 0x55f16d093670 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f11c00cb680 tx=0x7f11c001a040 comp rx=0 tx=0).stop 2023-10-11T16:39:01.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 -- 172.21.15.111:0/1531376497 shutdown_connections 2023-10-11T16:39:01.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 0x55f16d093bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d098b60 0x55f16d0a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 --2- 172.21.15.111:0/1531376497 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d099470 0x55f16d093670 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 -- 172.21.15.111:0/1531376497 >> 172.21.15.111:0/1531376497 conn(0x55f16d08e360 msgr2=0x55f16d0aebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 -- 172.21.15.111:0/1531376497 shutdown_connections 2023-10-11T16:39:01.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 -- 172.21.15.111:0/1531376497 wait complete. 2023-10-11T16:39:01.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 -- start start 2023-10-11T16:39:01.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.068+0000 7f11f215db80 1 Processor -- start 2023-10-11T16:39:01.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11f215db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d098b60 0x55f16d0ff170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11f215db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11f215db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 0x55f16d0a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11f215db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16d0a9550 con 0x55f16d098b60 2023-10-11T16:39:01.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11f215db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16d0a96c0 con 0x55f16d09d8b0 2023-10-11T16:39:01.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11f215db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16d0a9880 con 0x55f16d099470 2023-10-11T16:39:01.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11f215db80 10 client.0 _ll_register_callbacks cb 0x55f16cfa9780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d098b60 0x55f16d0ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d098b60 0x55f16d0ff170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33366/0 (socket says 172.21.15.111:33366) 2023-10-11T16:39:01.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11d7fff700 1 -- 172.21.15.111:0/323776910 learned_addr learned my addr 172.21.15.111:0/323776910 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11cffff700 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.069+0000 7f11dcd9c700 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 0x55f16d0a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11d7fff700 1 -- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 msgr2=0x55f16d0ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11d7fff700 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0ff6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11d7fff700 1 -- 172.21.15.111:0/323776910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 msgr2=0x55f16d0a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11d7fff700 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 0x55f16d0a8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11d7fff700 1 -- 172.21.15.111:0/323776910 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d8110320 con 0x55f16d098b60 2023-10-11T16:39:01.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11cffff700 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:01.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11d7fff700 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d098b60 0x55f16d0ff170 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f11c8000c00 tx=0x7f11c80096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8da05e440658a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11b77fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:01.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11b77fe700 1 -- 172.21.15.111:0/323776910 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c802a7f0 con 0x55f16d098b60 2023-10-11T16:39:01.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11b77fe700 1 -- 172.21.15.111:0/323776910 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f11c8029070 con 0x55f16d098b60 2023-10-11T16:39:01.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11f215db80 1 -- 172.21.15.111:0/323776910 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11c8004390 con 0x55f16d098b60 2023-10-11T16:39:01.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11b77fe700 1 -- 172.21.15.111:0/323776910 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11c802a950 con 0x55f16d098b60 2023-10-11T16:39:01.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11f215db80 10 client.16494 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:01.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.070+0000 7f11f215db80 1 -- 172.21.15.111:0/323776910 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f16ce17de0 con 0x55f16d098b60 2023-10-11T16:39:01.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11b77fe700 1 -- 172.21.15.111:0/323776910 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11c8004390 con 0x55f16d098b60 2023-10-11T16:39:01.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11b77fe700 1 -- 172.21.15.111:0/323776910 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f11c8011440 con 0x55f16d098b60 2023-10-11T16:39:01.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11f215db80 10 client.16494 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:01.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11f215db80 10 client.16494 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:39:01.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11f215db80 20 client.16494 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:01.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11f215db80 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f16d106fd0 0x55f16d1093e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11f215db80 4 client.16494 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:39:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11f215db80 1 -- 172.21.15.111:0/323776910 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f16cfb8660 con 0x55f16d106fd0 2023-10-11T16:39:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.071+0000 7f11cffff700 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f16d106fd0 0x55f16d1093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11cffff700 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f16d106fd0 0x55f16d1093e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=50b1c4ec4b294b3c server_cookie=821d3a801164ba02 in_seq=0 out_seq=0 2023-10-11T16:39:01.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11b77fe700 10 client.16494 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:39:01.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11b77fe700 1 -- 172.21.15.111:0/323776910 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f16cfb8660 con 0x55f16d106fd0 2023-10-11T16:39:01.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11b77fe700 10 client.16494 handle_command_reply: tid=0 2023-10-11T16:39:01.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 client.16494 shutdown 2023-10-11T16:39:01.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 -- 172.21.15.111:0/323776910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d098b60 msgr2=0x55f16d0ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d098b60 0x55f16d0ff170 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f11c8000c00 tx=0x7f11c80096d0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 -- 172.21.15.111:0/323776910 shutdown_connections 2023-10-11T16:39:01.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f16d106fd0 0x55f16d1093e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f16d09d8b0 0x55f16d0a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f16d098b60 0x55f16d0ff170 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 --2- 172.21.15.111:0/323776910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f16d099470 0x55f16d0ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11b77fe700 0 client.16494 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:39:01.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 -- 172.21.15.111:0/323776910 >> 172.21.15.111:0/323776910 conn(0x55f16d08e360 msgr2=0x55f16d0935d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 -- 172.21.15.111:0/323776910 shutdown_connections 2023-10-11T16:39:01.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 1 -- 172.21.15.111:0/323776910 wait complete. 2023-10-11T16:39:01.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.072+0000 7f11f215db80 20 client.16494 trim_cache size 0 max 16384 2023-10-11T16:39:01.123 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:39:01.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 -- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f11c40a1340 msgr2=0x7f11c40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f11c40a1340 0x7f11c40a37f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f11d0009ec0 tx=0x7f11d00092f0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 -- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11d8101660 msgr2=0x7f11d8198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11d8101660 0x7f11d8198c40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f11c0006010 tx=0x7f11c0005720 comp rx=0 tx=0).stop 2023-10-11T16:39:01.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 -- 172.21.15.111:0/2923171578 shutdown_connections 2023-10-11T16:39:01.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f11c40a1340 0x7f11c40a37f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d8105620 0x7f11d819bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11d8100cc0 0x7f11d8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 --2- 172.21.15.111:0/2923171578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11d8101660 0x7f11d8198c40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 -- 172.21.15.111:0/2923171578 >> 172.21.15.111:0/2923171578 conn(0x7f11d8077560 msgr2=0x7f11d80feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 -- 172.21.15.111:0/2923171578 shutdown_connections 2023-10-11T16:39:01.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.073+0000 7f11de7ff700 1 -- 172.21.15.111:0/2923171578 wait complete. 2023-10-11T16:39:01.128 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:39:01.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 -- 172.21.15.111:0/20266884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe0101390 msgr2=0x7fefe0101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 --2- 172.21.15.111:0/20266884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe0101390 0x7fefe0101760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009990 tx=0x7fefd0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 -- 172.21.15.111:0/20266884 shutdown_connections 2023-10-11T16:39:01.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 --2- 172.21.15.111:0/20266884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe010a8b0 0x7fefe010ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 --2- 172.21.15.111:0/20266884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe0101390 0x7fefe0101760 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 --2- 172.21.15.111:0/20266884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe0101ca0 0x7fefe010a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 -- 172.21.15.111:0/20266884 >> 172.21.15.111:0/20266884 conn(0x7fefe00770b0 msgr2=0x7fefe00774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 -- 172.21.15.111:0/20266884 shutdown_connections 2023-10-11T16:39:01.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.328+0000 7fefe65ff700 1 -- 172.21.15.111:0/20266884 wait complete. 2023-10-11T16:39:01.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.329+0000 7fefe65ff700 1 Processor -- start 2023-10-11T16:39:01.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.329+0000 7fefe65ff700 1 -- start start 2023-10-11T16:39:01.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.329+0000 7fefe65ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe0101390 0x7fefe0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.329+0000 7fefe65ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe0101ca0 0x7fefe0198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.329+0000 7fefe65ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe010a8b0 0x7fefe019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.329+0000 7fefe65ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe019d5e0 con 0x7fefe0101ca0 2023-10-11T16:39:01.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.329+0000 7fefe65ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe019d750 con 0x7fefe010a8b0 2023-10-11T16:39:01.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.329+0000 7fefe65ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe019d910 con 0x7fefe0101390 2023-10-11T16:39:01.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe0101ca0 0x7fefe0198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe0101ca0 0x7fefe0198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33374/0 (socket says 172.21.15.111:33374) 2023-10-11T16:39:01.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 -- 172.21.15.111:0/3466600110 learned_addr learned my addr 172.21.15.111:0/3466600110 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdffff700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe0101390 0x7fefe0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe010a8b0 0x7fefe019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 -- 172.21.15.111:0/3466600110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe0101390 msgr2=0x7fefe0198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe0101390 0x7fefe0198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 -- 172.21.15.111:0/3466600110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe010a8b0 msgr2=0x7fefe019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe010a8b0 0x7fefe019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 -- 172.21.15.111:0/3466600110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd00095f0 con 0x7fefe0101ca0 2023-10-11T16:39:01.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe0101ca0 0x7fefe0198b40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fefd400e9a0 tx=0x7fefd400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.330+0000 7fefdd7fa700 1 -- 172.21.15.111:0/3466600110 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd4009790 con 0x7fefe0101ca0 2023-10-11T16:39:01.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.331+0000 7fefe65ff700 1 -- 172.21.15.111:0/3466600110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe0110330 con 0x7fefe0101ca0 2023-10-11T16:39:01.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.331+0000 7fefdd7fa700 1 -- 172.21.15.111:0/3466600110 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefd4010430 con 0x7fefe0101ca0 2023-10-11T16:39:01.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.331+0000 7fefdd7fa700 1 -- 172.21.15.111:0/3466600110 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd4018680 con 0x7fefe0101ca0 2023-10-11T16:39:01.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.331+0000 7fefe65ff700 1 -- 172.21.15.111:0/3466600110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe01101a0 con 0x7fefe0101ca0 2023-10-11T16:39:01.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.334+0000 7fefdd7fa700 1 -- 172.21.15.111:0/3466600110 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fefd40105a0 con 0x7fefe0101ca0 2023-10-11T16:39:01.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.335+0000 7feff9ea4b80 1 Processor -- start 2023-10-11T16:39:01.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.335+0000 7fefdd7fa700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fefc80a12c0 0x7fefc80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.335+0000 7fefdd7fa700 1 -- 172.21.15.111:0/3466600110 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fefd4014070 con 0x7fefe0101ca0 2023-10-11T16:39:01.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.335+0000 7feff9ea4b80 1 -- start start 2023-10-11T16:39:01.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.335+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fefc80a12c0 0x7fefc80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.335+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e55f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e56470 0x55a1a6e5a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6d5b060 con 0x55a1a6e5a8b0 2023-10-11T16:39:01.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6dad8a0 con 0x55a1a6e56470 2023-10-11T16:39:01.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6da6d30 con 0x55a1a6e55b60 2023-10-11T16:39:01.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefdffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e55f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefdffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e55f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40766/0 (socket says 172.21.15.111:40766) 2023-10-11T16:39:01.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefdffff700 1 -- 172.21.15.111:0/3031319182 learned_addr learned my addr 172.21.15.111:0/3031319182 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e56470 0x55a1a6e5a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fefc80a12c0 0x7fefc80a3770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe006a4a0 tx=0x7fefd8009920 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefe4b9c700 1 -- 172.21.15.111:0/3031319182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 msgr2=0x55a1a6e55f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e55f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefe4b9c700 1 -- 172.21.15.111:0/3031319182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e56470 msgr2=0x55a1a6e5a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e56470 0x55a1a6e5a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefe4b9c700 1 -- 172.21.15.111:0/3031319182 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1a6c28630 con 0x55a1a6e5a8b0 2023-10-11T16:39:01.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.336+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e56470 0x55a1a6e5a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:01.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.337+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5ad00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009fd0 tx=0x7fefd801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f08e18483faae1f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.337+0000 7fefc67fc700 1 -- 172.21.15.111:0/3031319182 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd800e040 con 0x55a1a6e5a8b0 2023-10-11T16:39:01.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.337+0000 7fefc67fc700 1 -- 172.21.15.111:0/3031319182 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefd800f040 con 0x55a1a6e5a8b0 2023-10-11T16:39:01.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.337+0000 7fefc67fc700 1 -- 172.21.15.111:0/3031319182 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8021430 con 0x55a1a6e5a8b0 2023-10-11T16:39:01.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 -- 172.21.15.111:0/3031319182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 msgr2=0x55a1a6e5ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5ad00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009fd0 tx=0x7fefd801c000 comp rx=0 tx=0).stop 2023-10-11T16:39:01.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 -- 172.21.15.111:0/3031319182 shutdown_connections 2023-10-11T16:39:01.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5ad00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e55f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/3031319182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e56470 0x55a1a6e5a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 -- 172.21.15.111:0/3031319182 >> 172.21.15.111:0/3031319182 conn(0x55a1a6e4b360 msgr2=0x55a1a6e50b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 -- 172.21.15.111:0/3031319182 shutdown_connections 2023-10-11T16:39:01.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 -- 172.21.15.111:0/3031319182 wait complete. 2023-10-11T16:39:01.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 -- start start 2023-10-11T16:39:01.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.338+0000 7feff9ea4b80 1 Processor -- start 2023-10-11T16:39:01.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e62080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e5f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6d488e0 con 0x55a1a6e56470 2023-10-11T16:39:01.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6d48580 con 0x55a1a6e5a8b0 2023-10-11T16:39:01.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6c2a920 con 0x55a1a6e55b60 2023-10-11T16:39:01.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7feff9ea4b80 10 client.0 _ll_register_callbacks cb 0x55a1a6d66780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:01.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7fefdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e5f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7fefdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e5f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33392/0 (socket says 172.21.15.111:33392) 2023-10-11T16:39:01.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7fefdf7fe700 1 -- 172.21.15.111:0/1840425136 learned_addr learned my addr 172.21.15.111:0/1840425136 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7fefdffff700 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e62080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.339+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefdf7fe700 1 -- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 msgr2=0x55a1a6e62080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e62080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefdf7fe700 1 -- 172.21.15.111:0/1840425136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 msgr2=0x55a1a6e5f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefdf7fe700 1 -- 172.21.15.111:0/1840425136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd8019460 con 0x55a1a6e56470 2023-10-11T16:39:01.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefdffff700 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e62080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:01.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e5f290 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fefd4096a20 tx=0x7fefd4099fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73790bb8d1fecd2b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefbf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:01.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefbf7fe700 1 -- 172.21.15.111:0/1840425136 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd40996a0 con 0x55a1a6e56470 2023-10-11T16:39:01.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefbf7fe700 1 -- 172.21.15.111:0/1840425136 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fefd40a2400 con 0x55a1a6e56470 2023-10-11T16:39:01.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7feff9ea4b80 1 -- 172.21.15.111:0/1840425136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a1a6e48b20 con 0x55a1a6e56470 2023-10-11T16:39:01.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7fefbf7fe700 1 -- 172.21.15.111:0/1840425136 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefd40a1f70 con 0x55a1a6e56470 2023-10-11T16:39:01.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7feff9ea4b80 10 client.16506 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:01.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.340+0000 7feff9ea4b80 1 -- 172.21.15.111:0/1840425136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a1a6d6d190 con 0x55a1a6e56470 2023-10-11T16:39:01.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7fefbf7fe700 1 -- 172.21.15.111:0/1840425136 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a1a6e48b20 con 0x55a1a6e56470 2023-10-11T16:39:01.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7fefbf7fe700 1 -- 172.21.15.111:0/1840425136 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fefd4098f10 con 0x55a1a6e56470 2023-10-11T16:39:01.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7feff9ea4b80 10 client.16506 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:01.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7feff9ea4b80 10 client.16506 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:39:01.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7feff9ea4b80 20 client.16506 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:01.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a1a6e6f8c0 0x55a1a6e71cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7feff9ea4b80 4 client.16506 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:01.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7feff9ea4b80 1 -- 172.21.15.111:0/1840425136 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a1a6d690a0 con 0x55a1a6e6f8c0 2023-10-11T16:39:01.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.341+0000 7fefdffff700 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a1a6e6f8c0 0x55a1a6e71cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.342+0000 7fefdffff700 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a1a6e6f8c0 0x55a1a6e71cd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d96de9e4187d552 server_cookie=475333ec31a015fa in_seq=0 out_seq=0 2023-10-11T16:39:01.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.342+0000 7fefbf7fe700 10 client.16506 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:39:01.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7fefbf7fe700 1 -- 172.21.15.111:0/1840425136 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a1a6d690a0 con 0x55a1a6e6f8c0 2023-10-11T16:39:01.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7fefbf7fe700 10 client.16506 handle_command_reply: tid=0 2023-10-11T16:39:01.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 client.16506 shutdown 2023-10-11T16:39:01.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 -- 172.21.15.111:0/1840425136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 msgr2=0x55a1a6e5f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e5f290 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fefd4096a20 tx=0x7fefd4099fb0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 -- 172.21.15.111:0/1840425136 shutdown_connections 2023-10-11T16:39:01.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a1a6e6f8c0 0x55a1a6e71cd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e5f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e62080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/1840425136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e5f290 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 -- 172.21.15.111:0/1840425136 >> 172.21.15.111:0/1840425136 conn(0x55a1a6e4b360 msgr2=0x55a1a6e5bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7fefbf7fe700 0 client.16506 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:39:01.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 -- 172.21.15.111:0/1840425136 shutdown_connections 2023-10-11T16:39:01.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.343+0000 7feff9ea4b80 1 -- 172.21.15.111:0/1840425136 wait complete. 2023-10-11T16:39:01.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.344+0000 7feff9ea4b80 20 client.16506 trim_cache size 0 max 16384 2023-10-11T16:39:01.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.357+0000 7feff9ea4b80 1 Processor -- start 2023-10-11T16:39:01.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.357+0000 7feff9ea4b80 1 -- start start 2023-10-11T16:39:01.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e55b60 0x55a1a6e718d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e56470 0x55a1a6e4fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e61bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6e56bc0 con 0x55a1a6e5a8b0 2023-10-11T16:39:01.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6e593c0 con 0x55a1a6e55b60 2023-10-11T16:39:01.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6e56900 con 0x55a1a6e56470 2023-10-11T16:39:01.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e55b60 0x55a1a6e718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e55b60 0x55a1a6e718d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:58886/0 (socket says 172.21.15.111:58886) 2023-10-11T16:39:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdffff700 1 -- 172.21.15.111:0/2806899913 learned_addr learned my addr 172.21.15.111:0/2806899913 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e61bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e56470 0x55a1a6e4fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdffff700 1 -- 172.21.15.111:0/2806899913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e56470 msgr2=0x55a1a6e4fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdffff700 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e56470 0x55a1a6e4fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdffff700 1 -- 172.21.15.111:0/2806899913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 msgr2=0x55a1a6e61bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdffff700 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e61bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.358+0000 7fefdffff700 1 -- 172.21.15.111:0/2806899913 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1a6d6cee0 con 0x55a1a6e55b60 2023-10-11T16:39:01.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.359+0000 7fefdffff700 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e55b60 0x55a1a6e718d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009990 tx=0x7fefd0005960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a5e88f0ff724c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.359+0000 7fefc6ffd700 1 -- 172.21.15.111:0/2806899913 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd001d040 con 0x55a1a6e55b60 2023-10-11T16:39:01.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.359+0000 7fefc6ffd700 1 -- 172.21.15.111:0/2806899913 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefd001e040 con 0x55a1a6e55b60 2023-10-11T16:39:01.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.359+0000 7fefc6ffd700 1 -- 172.21.15.111:0/2806899913 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00053b0 con 0x55a1a6e55b60 2023-10-11T16:39:01.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 -- 172.21.15.111:0/2806899913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e55b60 msgr2=0x55a1a6e718d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e55b60 0x55a1a6e718d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009990 tx=0x7fefd0005960 comp rx=0 tx=0).stop 2023-10-11T16:39:01.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 -- 172.21.15.111:0/2806899913 shutdown_connections 2023-10-11T16:39:01.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e61bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e55b60 0x55a1a6e718d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/2806899913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e56470 0x55a1a6e4fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 -- 172.21.15.111:0/2806899913 >> 172.21.15.111:0/2806899913 conn(0x55a1a6e4b360 msgr2=0x55a1a6e5ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 -- 172.21.15.111:0/2806899913 shutdown_connections 2023-10-11T16:39:01.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 -- 172.21.15.111:0/2806899913 wait complete. 2023-10-11T16:39:01.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 -- start start 2023-10-11T16:39:01.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.360+0000 7feff9ea4b80 1 Processor -- start 2023-10-11T16:39:01.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e649c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7feff9ea4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e64f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6e65440 con 0x55a1a6e56470 2023-10-11T16:39:01.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6e655b0 con 0x55a1a6e5a8b0 2023-10-11T16:39:01.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7feff9ea4b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1a6e65770 con 0x55a1a6e55b60 2023-10-11T16:39:01.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7feff9ea4b80 10 client.0 _ll_register_callbacks cb 0x7fefd40cd0c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:01.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7fefdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e649c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7fefdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e649c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33414/0 (socket says 172.21.15.111:33414) 2023-10-11T16:39:01.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7fefdf7fe700 1 -- 172.21.15.111:0/186258042 learned_addr learned my addr 172.21.15.111:0/186258042 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.361+0000 7fefdffff700 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e64f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefdf7fe700 1 -- 172.21.15.111:0/186258042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 msgr2=0x55a1a6e669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e669e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefdf7fe700 1 -- 172.21.15.111:0/186258042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 msgr2=0x55a1a6e64f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e64f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefdf7fe700 1 -- 172.21.15.111:0/186258042 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe010d990 con 0x55a1a6e56470 2023-10-11T16:39:01.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefe4b9c700 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e64f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:39:01.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e649c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fefd40964b0 tx=0x7fefd4005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ff5bd6f5d19f5d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefbf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:01.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefbf7fe700 1 -- 172.21.15.111:0/186258042 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd40962f0 con 0x55a1a6e56470 2023-10-11T16:39:01.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefbf7fe700 1 -- 172.21.15.111:0/186258042 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fefd40a2400 con 0x55a1a6e56470 2023-10-11T16:39:01.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7feff9ea4b80 1 -- 172.21.15.111:0/186258042 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fefd800f040 con 0x55a1a6e56470 2023-10-11T16:39:01.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7fefbf7fe700 1 -- 172.21.15.111:0/186258042 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefd40994a0 con 0x55a1a6e56470 2023-10-11T16:39:01.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7feff9ea4b80 10 client.16512 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:01.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.362+0000 7feff9ea4b80 1 -- 172.21.15.111:0/186258042 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a1a6bbdc60 con 0x55a1a6e56470 2023-10-11T16:39:01.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7fefbf7fe700 1 -- 172.21.15.111:0/186258042 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefd800f040 con 0x55a1a6e56470 2023-10-11T16:39:01.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7fefbf7fe700 1 -- 172.21.15.111:0/186258042 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fefd4095da0 con 0x55a1a6e56470 2023-10-11T16:39:01.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7feff9ea4b80 10 client.16512 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:01.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7feff9ea4b80 10 client.16512 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:39:01.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7feff9ea4b80 20 client.16512 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:01.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a1a6ec3890 0x55a1a6ec5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7feff9ea4b80 4 client.16512 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:39:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7feff9ea4b80 1 -- 172.21.15.111:0/186258042 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a1a6d75660 con 0x55a1a6ec3890 2023-10-11T16:39:01.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.363+0000 7fefdffff700 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a1a6ec3890 0x55a1a6ec5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7fefbf7fe700 10 client.16512 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:39:01.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7fefdffff700 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a1a6ec3890 0x55a1a6ec5ca0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=11a7b1ae6a65fbb4 server_cookie=7171acf2765dd235 in_seq=0 out_seq=0 2023-10-11T16:39:01.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7fefbf7fe700 1 -- 172.21.15.111:0/186258042 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a1a6d75660 con 0x55a1a6ec3890 2023-10-11T16:39:01.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7fefbf7fe700 10 client.16512 handle_command_reply: tid=0 2023-10-11T16:39:01.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 client.16512 shutdown 2023-10-11T16:39:01.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 -- 172.21.15.111:0/186258042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 msgr2=0x55a1a6e649c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e649c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fefd40964b0 tx=0x7fefd4005bb0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 -- 172.21.15.111:0/186258042 shutdown_connections 2023-10-11T16:39:01.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a1a6ec3890 0x55a1a6ec5ca0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a1a6e5a8b0 0x55a1a6e64f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a1a6e55b60 0x55a1a6e669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 --2- 172.21.15.111:0/186258042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a1a6e56470 0x55a1a6e649c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7fefbf7fe700 0 client.16512 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:39:01.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 -- 172.21.15.111:0/186258042 >> 172.21.15.111:0/186258042 conn(0x55a1a6e4b360 msgr2=0x55a1a6e718d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 -- 172.21.15.111:0/186258042 shutdown_connections 2023-10-11T16:39:01.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 1 -- 172.21.15.111:0/186258042 wait complete. 2023-10-11T16:39:01.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.364+0000 7feff9ea4b80 20 client.16512 trim_cache size 0 max 16384 2023-10-11T16:39:01.409 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:39:01.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 -- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fefc80a12c0 msgr2=0x7fefc80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fefc80a12c0 0x7fefc80a3770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe006a4a0 tx=0x7fefd8009920 comp rx=0 tx=0).stop 2023-10-11T16:39:01.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 -- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe0101ca0 msgr2=0x7fefe0198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe0101ca0 0x7fefe0198b40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fefd400e9a0 tx=0x7fefd400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 -- 172.21.15.111:0/3466600110 shutdown_connections 2023-10-11T16:39:01.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fefc80a12c0 0x7fefc80a3770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefe010a8b0 0x7fefe019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe0101390 0x7fefe0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 --2- 172.21.15.111:0/3466600110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe0101ca0 0x7fefe0198b40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 -- 172.21.15.111:0/3466600110 >> 172.21.15.111:0/3466600110 conn(0x7fefe00770b0 msgr2=0x7fefe00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 -- 172.21.15.111:0/3466600110 shutdown_connections 2023-10-11T16:39:01.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.365+0000 7fefe65ff700 1 -- 172.21.15.111:0/3466600110 wait complete. 2023-10-11T16:39:01.414 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:39:01.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 -- 172.21.15.111:0/485668942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 msgr2=0x7f201810d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 --2- 172.21.15.111:0/485668942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 0x7f201810d1a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f201000b0d0 tx=0x7f201000b3e0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 -- 172.21.15.111:0/485668942 shutdown_connections 2023-10-11T16:39:01.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 --2- 172.21.15.111:0/485668942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 0x7f201810d1a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 --2- 172.21.15.111:0/485668942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2018100cc0 0x7f2018101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 --2- 172.21.15.111:0/485668942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2018101660 0x7f2018104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 -- 172.21.15.111:0/485668942 >> 172.21.15.111:0/485668942 conn(0x7f2018077560 msgr2=0x7f2018077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 -- 172.21.15.111:0/485668942 shutdown_connections 2023-10-11T16:39:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.614+0000 7f201e1ff700 1 -- 172.21.15.111:0/485668942 wait complete. 2023-10-11T16:39:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.615+0000 7f201e1ff700 1 Processor -- start 2023-10-11T16:39:01.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.615+0000 7f201e1ff700 1 -- start start 2023-10-11T16:39:01.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.615+0000 7f201e1ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2018100cc0 0x7f201819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.615+0000 7f201e1ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2018101660 0x7f201819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.615+0000 7f201e1ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 0x7f20181986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.615+0000 7f201e1ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2018198d20 con 0x7f2018105620 2023-10-11T16:39:01.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.615+0000 7f201e1ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2018198e90 con 0x7f2018101660 2023-10-11T16:39:01.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.615+0000 7f201e1ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2018199050 con 0x7f2018100cc0 2023-10-11T16:39:01.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 0x7f20181986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 0x7f20181986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33426/0 (socket says 172.21.15.111:33426) 2023-10-11T16:39:01.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 -- 172.21.15.111:0/1908726977 learned_addr learned my addr 172.21.15.111:0/1908726977 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2016ffd700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2018101660 0x7f201819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f20177fe700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2018100cc0 0x7f201819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 -- 172.21.15.111:0/1908726977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2018100cc0 msgr2=0x7f201819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2018100cc0 0x7f201819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 -- 172.21.15.111:0/1908726977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2018101660 msgr2=0x7f201819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2018101660 0x7f201819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 -- 172.21.15.111:0/1908726977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010009d00 con 0x7f2018105620 2023-10-11T16:39:01.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2017fff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 0x7f20181986b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2010007820 tx=0x7f2010007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2014ff9700 1 -- 172.21.15.111:0/1908726977 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201000e040 con 0x7f2018105620 2023-10-11T16:39:01.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.616+0000 7f2014ff9700 1 -- 172.21.15.111:0/1908726977 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f201000bcb0 con 0x7f2018105620 2023-10-11T16:39:01.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.617+0000 7f201e1ff700 1 -- 172.21.15.111:0/1908726977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20181103e0 con 0x7f2018105620 2023-10-11T16:39:01.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.617+0000 7f2014ff9700 1 -- 172.21.15.111:0/1908726977 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20100058a0 con 0x7f2018105620 2023-10-11T16:39:01.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.617+0000 7f201e1ff700 1 -- 172.21.15.111:0/1908726977 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2018110250 con 0x7f2018105620 2023-10-11T16:39:01.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.619+0000 7f2014ff9700 1 -- 172.21.15.111:0/1908726977 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2010004a50 con 0x7f2018105620 2023-10-11T16:39:01.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2031b66b80 1 Processor -- start 2023-10-11T16:39:01.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2014ff9700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f20000a12c0 0x7f20000a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2031b66b80 1 -- start start 2023-10-11T16:39:01.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2014ff9700 1 -- 172.21.15.111:0/1908726977 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f20100ce1e0 con 0x7f2018105620 2023-10-11T16:39:01.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2016ffd700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f20000a12c0 0x7f20000a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b970f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b975370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 0x55877b975d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2031b66b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b876060 con 0x55877b970b60 2023-10-11T16:39:01.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2031b66b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b8c88a0 con 0x55877b9758b0 2023-10-11T16:39:01.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.621+0000 7f2031b66b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b8c1d30 con 0x55877b971470 2023-10-11T16:39:01.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b970f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b970f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33432/0 (socket says 172.21.15.111:33432) 2023-10-11T16:39:01.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 -- 172.21.15.111:0/1392546117 learned_addr learned my addr 172.21.15.111:0/1392546117 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f2016ffd700 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b975370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f2017fff700 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 0x55877b975d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f2016ffd700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f20000a12c0 0x7f20000a3770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2018106ab0 tx=0x7f200c009f80 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 -- 172.21.15.111:0/1392546117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 msgr2=0x55877b975370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b975370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 -- 172.21.15.111:0/1392546117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 msgr2=0x55877b975d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 0x55877b975d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 -- 172.21.15.111:0/1392546117 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55877b743630 con 0x55877b970b60 2023-10-11T16:39:01.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f2016ffd700 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b975370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:01.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.622+0000 7f20177fe700 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b970f30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2008009d80 tx=0x7f2008011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5afe98a915b8416f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f1ffdffb700 1 -- 172.21.15.111:0/1392546117 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008004230 con 0x55877b970b60 2023-10-11T16:39:01.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f1ffdffb700 1 -- 172.21.15.111:0/1392546117 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2008009e10 con 0x55877b970b60 2023-10-11T16:39:01.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f1ffdffb700 1 -- 172.21.15.111:0/1392546117 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080197e0 con 0x55877b970b60 2023-10-11T16:39:01.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f2031b66b80 1 -- 172.21.15.111:0/1392546117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 msgr2=0x55877b970f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f2031b66b80 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b970f30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2008009d80 tx=0x7f2008011720 comp rx=0 tx=0).stop 2023-10-11T16:39:01.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f2031b66b80 1 -- 172.21.15.111:0/1392546117 shutdown_connections 2023-10-11T16:39:01.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f2031b66b80 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 0x55877b975d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f2031b66b80 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b970f30 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f2031b66b80 1 --2- 172.21.15.111:0/1392546117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b975370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f2031b66b80 1 -- 172.21.15.111:0/1392546117 >> 172.21.15.111:0/1392546117 conn(0x55877b966360 msgr2=0x55877b96bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.623+0000 7f2031b66b80 1 -- 172.21.15.111:0/1392546117 shutdown_connections 2023-10-11T16:39:01.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.624+0000 7f2031b66b80 1 -- 172.21.15.111:0/1392546117 wait complete. 2023-10-11T16:39:01.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.624+0000 7f2031b66b80 1 -- start start 2023-10-11T16:39:01.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.624+0000 7f2031b66b80 1 Processor -- start 2023-10-11T16:39:01.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 0x55877b97d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b971470 0x55877b97a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b9758b0 0x55877b97a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2031b66b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b8638e0 con 0x55877b971470 2023-10-11T16:39:01.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2031b66b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b863580 con 0x55877b970b60 2023-10-11T16:39:01.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2031b66b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b745920 con 0x55877b9758b0 2023-10-11T16:39:01.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2031b66b80 10 client.0 _ll_register_callbacks cb 0x55877b881780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:01.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2016ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b971470 0x55877b97a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2016ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b971470 0x55877b97a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33438/0 (socket says 172.21.15.111:33438) 2023-10-11T16:39:01.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2016ffd700 1 -- 172.21.15.111:0/157083293 learned_addr learned my addr 172.21.15.111:0/157083293 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f20177fe700 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 0x55877b97d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2017fff700 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b9758b0 0x55877b97a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2016ffd700 1 -- 172.21.15.111:0/157083293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b9758b0 msgr2=0x55877b97a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2016ffd700 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b9758b0 0x55877b97a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2016ffd700 1 -- 172.21.15.111:0/157083293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 msgr2=0x55877b97d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2016ffd700 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 0x55877b97d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.625+0000 7f2016ffd700 1 -- 172.21.15.111:0/157083293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20080099e0 con 0x55877b971470 2023-10-11T16:39:01.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f2016ffd700 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b971470 0x55877b97a280 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f200c01e6d0 tx=0x7f200c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5edacfb26389c8fe server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f1ff6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:01.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/157083293 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c01eea0 con 0x55877b971470 2023-10-11T16:39:01.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f2031b66b80 1 -- 172.21.15.111:0/157083293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55877b963b20 con 0x55877b971470 2023-10-11T16:39:01.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/157083293 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f200c01e220 con 0x55877b971470 2023-10-11T16:39:01.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/157083293 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f200c004830 con 0x55877b971470 2023-10-11T16:39:01.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f2031b66b80 10 client.16524 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:01.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f2031b66b80 1 -- 172.21.15.111:0/157083293 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55877b888190 con 0x55877b971470 2023-10-11T16:39:01.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/157083293 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55877b963b20 con 0x55877b971470 2023-10-11T16:39:01.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.626+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/157083293 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f200c00f690 con 0x55877b971470 2023-10-11T16:39:01.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.627+0000 7f2031b66b80 10 client.16524 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:01.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.627+0000 7f2031b66b80 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:01.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.627+0000 7f2031b66b80 20 client.16524 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:01.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.627+0000 7f2031b66b80 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55877b98a8f0 0x55877b98cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.627+0000 7f2031b66b80 4 client.16524 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:01.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.627+0000 7f2031b66b80 1 -- 172.21.15.111:0/157083293 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55877b8840a0 con 0x55877b98a8f0 2023-10-11T16:39:01.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.627+0000 7f20177fe700 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55877b98a8f0 0x55877b98cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.628+0000 7f20177fe700 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55877b98a8f0 0x55877b98cd00 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2828e2d307e4367c server_cookie=d1734f260cd8ba51 in_seq=0 out_seq=0 2023-10-11T16:39:01.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.628+0000 7f1ff6ffd700 10 client.16524 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:01.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.628+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/157083293 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55877b8840a0 con 0x55877b98a8f0 2023-10-11T16:39:01.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.628+0000 7f1ff6ffd700 10 client.16524 handle_command_reply: tid=0 2023-10-11T16:39:01.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 client.16524 shutdown 2023-10-11T16:39:01.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 -- 172.21.15.111:0/157083293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b971470 msgr2=0x55877b97a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b971470 0x55877b97a280 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f200c01e6d0 tx=0x7f200c0047c0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 -- 172.21.15.111:0/157083293 shutdown_connections 2023-10-11T16:39:01.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55877b98a8f0 0x55877b98cd00 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b9758b0 0x55877b97a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f1ff6ffd700 0 client.16524 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:01.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 0x55877b97d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 --2- 172.21.15.111:0/157083293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b971470 0x55877b97a280 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 -- 172.21.15.111:0/157083293 >> 172.21.15.111:0/157083293 conn(0x55877b966360 msgr2=0x55877b976d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 -- 172.21.15.111:0/157083293 shutdown_connections 2023-10-11T16:39:01.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 1 -- 172.21.15.111:0/157083293 wait complete. 2023-10-11T16:39:01.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.629+0000 7f2031b66b80 20 client.16524 trim_cache size 0 max 16384 2023-10-11T16:39:01.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:01 smithi111 ceph-mon[98527]: pgmap v194: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 94 op/s 2023-10-11T16:39:01.661 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:01 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3267372753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:39:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2031b66b80 1 Processor -- start 2023-10-11T16:39:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2031b66b80 1 -- start start 2023-10-11T16:39:01.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b986150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b9738b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 0x55877b97cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2031b66b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b9787e0 con 0x55877b970b60 2023-10-11T16:39:01.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2031b66b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b97aeb0 con 0x55877b9758b0 2023-10-11T16:39:01.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2031b66b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b971900 con 0x55877b971470 2023-10-11T16:39:01.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f20177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b986150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f20177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b986150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33442/0 (socket says 172.21.15.111:33442) 2023-10-11T16:39:01.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f20177fe700 1 -- 172.21.15.111:0/898236380 learned_addr learned my addr 172.21.15.111:0/898236380 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2016ffd700 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b9738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.643+0000 7f2017fff700 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 0x55877b97cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f20177fe700 1 -- 172.21.15.111:0/898236380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 msgr2=0x55877b9738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f20177fe700 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b9738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f20177fe700 1 -- 172.21.15.111:0/898236380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 msgr2=0x55877b97cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f20177fe700 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 0x55877b97cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f20177fe700 1 -- 172.21.15.111:0/898236380 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55877b887ee0 con 0x55877b970b60 2023-10-11T16:39:01.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f20177fe700 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b986150 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2008014040 tx=0x7f2008011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6490fe2b43fc712 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f1ffe7fc700 1 -- 172.21.15.111:0/898236380 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200802a2b0 con 0x55877b970b60 2023-10-11T16:39:01.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f1ffe7fc700 1 -- 172.21.15.111:0/898236380 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f200802acb0 con 0x55877b970b60 2023-10-11T16:39:01.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.644+0000 7f1ffe7fc700 1 -- 172.21.15.111:0/898236380 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008005510 con 0x55877b970b60 2023-10-11T16:39:01.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 -- 172.21.15.111:0/898236380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 msgr2=0x55877b986150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b986150 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f2008014040 tx=0x7f2008011720 comp rx=0 tx=0).stop 2023-10-11T16:39:01.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 -- 172.21.15.111:0/898236380 shutdown_connections 2023-10-11T16:39:01.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b9758b0 0x55877b97cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b970b60 0x55877b986150 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 --2- 172.21.15.111:0/898236380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b9738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 -- 172.21.15.111:0/898236380 >> 172.21.15.111:0/898236380 conn(0x55877b966360 msgr2=0x55877b977c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 -- 172.21.15.111:0/898236380 shutdown_connections 2023-10-11T16:39:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 -- 172.21.15.111:0/898236380 wait complete. 2023-10-11T16:39:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.645+0000 7f2031b66b80 1 -- start start 2023-10-11T16:39:01.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2031b66b80 1 Processor -- start 2023-10-11T16:39:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 0x55877b9d7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b9d7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2031b66b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b9758b0 0x55877b97f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2031b66b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b97fed0 con 0x55877b9758b0 2023-10-11T16:39:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2031b66b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b980040 con 0x55877b970b60 2023-10-11T16:39:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2031b66b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877b980200 con 0x55877b971470 2023-10-11T16:39:01.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2031b66b80 10 client.0 _ll_register_callbacks cb 0x7f200c01e490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:01.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2017fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b9758b0 0x55877b97f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f20177fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 0x55877b9d7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2017fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b9758b0 0x55877b97f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33444/0 (socket says 172.21.15.111:33444) 2023-10-11T16:39:01.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2017fff700 1 -- 172.21.15.111:0/2255683189 learned_addr learned my addr 172.21.15.111:0/2255683189 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:01.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.646+0000 7f2016ffd700 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b9d7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2017fff700 1 -- 172.21.15.111:0/2255683189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 msgr2=0x55877b9d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2017fff700 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b9d7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2017fff700 1 -- 172.21.15.111:0/2255683189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 msgr2=0x55877b9d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:01.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2017fff700 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 0x55877b9d7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2017fff700 1 -- 172.21.15.111:0/2255683189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55877b743630 con 0x55877b9758b0 2023-10-11T16:39:01.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2017fff700 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b9758b0 0x55877b97f990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f20100078c0 tx=0x7f201009adf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6897a6c5d62ef91e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:01.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f1ff6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:01.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/2255683189 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20100a3060 con 0x55877b9758b0 2023-10-11T16:39:01.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/2255683189 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f201000be20 con 0x55877b9758b0 2023-10-11T16:39:01.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2031b66b80 1 -- 172.21.15.111:0/2255683189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2008009e10 con 0x55877b9758b0 2023-10-11T16:39:01.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/2255683189 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f201009af30 con 0x55877b9758b0 2023-10-11T16:39:01.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2031b66b80 10 client.16530 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:01.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f2031b66b80 1 -- 172.21.15.111:0/2255683189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55877b6d8c60 con 0x55877b9758b0 2023-10-11T16:39:01.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.647+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/2255683189 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2008009e10 con 0x55877b9758b0 2023-10-11T16:39:01.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/2255683189 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f201009b200 con 0x55877b9758b0 2023-10-11T16:39:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f2031b66b80 10 client.16530 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f2031b66b80 10 client.16530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f2031b66b80 20 client.16530 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:01.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f2031b66b80 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55877b9df5c0 0x55877b9e19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:01.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f2031b66b80 4 client.16530 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:39:01.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f2031b66b80 1 -- 172.21.15.111:0/2255683189 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55877b890660 con 0x55877b9df5c0 2023-10-11T16:39:01.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f20177fe700 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55877b9df5c0 0x55877b9e19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:01.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f1ff6ffd700 10 client.16530 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:01.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.648+0000 7f20177fe700 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55877b9df5c0 0x55877b9e19d0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=154e08fc1727e459 server_cookie=19ef0884d074e1b3 in_seq=0 out_seq=0 2023-10-11T16:39:01.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f1ff6ffd700 1 -- 172.21.15.111:0/2255683189 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55877b890660 con 0x55877b9df5c0 2023-10-11T16:39:01.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f1ff6ffd700 10 client.16530 handle_command_reply: tid=0 2023-10-11T16:39:01.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 client.16530 shutdown 2023-10-11T16:39:01.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 -- 172.21.15.111:0/2255683189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b9758b0 msgr2=0x55877b97f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b9758b0 0x55877b97f990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f20100078c0 tx=0x7f201009adf0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 -- 172.21.15.111:0/2255683189 shutdown_connections 2023-10-11T16:39:01.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55877b9df5c0 0x55877b9e19d0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55877b9758b0 0x55877b97f990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55877b970b60 0x55877b9d7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 --2- 172.21.15.111:0/2255683189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55877b971470 0x55877b9d7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f1ff6ffd700 0 client.16530 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:01.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 -- 172.21.15.111:0/2255683189 >> 172.21.15.111:0/2255683189 conn(0x55877b966360 msgr2=0x55877b986150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 -- 172.21.15.111:0/2255683189 shutdown_connections 2023-10-11T16:39:01.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 1 -- 172.21.15.111:0/2255683189 wait complete. 2023-10-11T16:39:01.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.649+0000 7f2031b66b80 20 client.16530 trim_cache size 0 max 16384 2023-10-11T16:39:01.691 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:39:01.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 -- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f20000a12c0 msgr2=0x7f20000a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f20000a12c0 0x7f20000a3770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2018106ab0 tx=0x7f200c009f80 comp rx=0 tx=0).stop 2023-10-11T16:39:01.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 -- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 msgr2=0x7f20181986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:01.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 0x7f20181986b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2010007820 tx=0x7f2010007850 comp rx=0 tx=0).stop 2023-10-11T16:39:01.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 -- 172.21.15.111:0/1908726977 shutdown_connections 2023-10-11T16:39:01.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f20000a12c0 0x7f20000a3770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2018105620 0x7f20181986b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2018100cc0 0x7f201819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 --2- 172.21.15.111:0/1908726977 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2018101660 0x7f201819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:01.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 -- 172.21.15.111:0/1908726977 >> 172.21.15.111:0/1908726977 conn(0x7f2018077560 msgr2=0x7f20180fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:01.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 -- 172.21.15.111:0/1908726977 shutdown_connections 2023-10-11T16:39:01.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:01.650+0000 7f201e1ff700 1 -- 172.21.15.111:0/1908726977 wait complete. 2023-10-11T16:39:01.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:01 smithi130 ceph-mon[131859]: pgmap v194: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 94 op/s 2023-10-11T16:39:01.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:01 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3267372753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:39:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:01 smithi150 ceph-mon[132217]: pgmap v194: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 94 op/s 2023-10-11T16:39:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:01 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3267372753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:39:02.696 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:39:02.697 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:39:02.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 -- 172.21.15.111:0/2973371373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a4100a50 msgr2=0x7fa5a4100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:02.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 --2- 172.21.15.111:0/2973371373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a4100a50 0x7fa5a4100ec0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa594009990 tx=0x7fa594009ca0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 -- 172.21.15.111:0/2973371373 shutdown_connections 2023-10-11T16:39:02.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 --2- 172.21.15.111:0/2973371373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a4101400 0x7fa5a410d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 --2- 172.21.15.111:0/2973371373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5a41063f0 0x7fa5a41067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 --2- 172.21.15.111:0/2973371373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a4100a50 0x7fa5a4100ec0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 -- 172.21.15.111:0/2973371373 >> 172.21.15.111:0/2973371373 conn(0x7fa5a4077590 msgr2=0x7fa5a4077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:02.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 -- 172.21.15.111:0/2973371373 shutdown_connections 2023-10-11T16:39:02.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.906+0000 7fa5abbff700 1 -- 172.21.15.111:0/2973371373 wait complete. 2023-10-11T16:39:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.908+0000 7fa5abbff700 1 Processor -- start 2023-10-11T16:39:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.908+0000 7fa5abbff700 1 -- start start 2023-10-11T16:39:02.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5abbff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a4100a50 0x7fa5a4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5abbff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5a4101400 0x7fa5a4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5abbff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a41063f0 0x7fa5a419cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5abbff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a419d5f0 con 0x7fa5a41063f0 2023-10-11T16:39:02.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5abbff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a419d760 con 0x7fa5a4101400 2023-10-11T16:39:02.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5abbff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a419d920 con 0x7fa5a4100a50 2023-10-11T16:39:02.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a41063f0 0x7fa5a419cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a41063f0 0x7fa5a419cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33458/0 (socket says 172.21.15.111:33458) 2023-10-11T16:39:02.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 -- 172.21.15.111:0/3843872633 learned_addr learned my addr 172.21.15.111:0/3843872633 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:02.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5a999b700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a4100a50 0x7fa5a4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5a919a700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5a4101400 0x7fa5a4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 -- 172.21.15.111:0/3843872633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a4100a50 msgr2=0x7fa5a4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:02.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a4100a50 0x7fa5a4198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 -- 172.21.15.111:0/3843872633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5a4101400 msgr2=0x7fa5a4198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:02.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5a4101400 0x7fa5a4198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 -- 172.21.15.111:0/3843872633 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5940095f0 con 0x7fa5a41063f0 2023-10-11T16:39:02.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5a919a700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5a4101400 0x7fa5a4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:02.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.909+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a41063f0 0x7fa5a419cfb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00ef30 tx=0x7fa59c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:02.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.910+0000 7fa59affd700 1 -- 172.21.15.111:0/3843872633 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c018070 con 0x7fa5a41063f0 2023-10-11T16:39:02.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.910+0000 7fa59affd700 1 -- 172.21.15.111:0/3843872633 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa59c00f040 con 0x7fa5a41063f0 2023-10-11T16:39:02.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.910+0000 7fa5abbff700 1 -- 172.21.15.111:0/3843872633 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a4110410 con 0x7fa5a41063f0 2023-10-11T16:39:02.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.910+0000 7fa59affd700 1 -- 172.21.15.111:0/3843872633 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c00cd00 con 0x7fa5a41063f0 2023-10-11T16:39:02.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.910+0000 7fa5abbff700 1 -- 172.21.15.111:0/3843872633 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a4110280 con 0x7fa5a41063f0 2023-10-11T16:39:02.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.913+0000 7fa59affd700 1 -- 172.21.15.111:0/3843872633 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fa59c003a00 con 0x7fa5a41063f0 2023-10-11T16:39:02.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.914+0000 7fa5bf43fb80 1 Processor -- start 2023-10-11T16:39:02.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.914+0000 7fa59affd700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa5900a12c0 0x7fa5900a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.914+0000 7fa5bf43fb80 1 -- start start 2023-10-11T16:39:02.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.914+0000 7fa59affd700 1 -- 172.21.15.111:0/3843872633 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa59c0ccc50 con 0x7fa5a41063f0 2023-10-11T16:39:02.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a919a700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa5900a12c0 0x7fa5900a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb675fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb67a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb57b0e0 con 0x55d1eb6764f0 2023-10-11T16:39:02.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb5cd920 con 0x55d1eb67a930 2023-10-11T16:39:02.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb5c6db0 con 0x55d1eb675be0 2023-10-11T16:39:02.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a999b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb675fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a919a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb67a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a999b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb675fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:40896/0 (socket says 172.21.15.111:40896) 2023-10-11T16:39:02.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a999b700 1 -- 172.21.15.111:0/3642383279 learned_addr learned my addr 172.21.15.111:0/3642383279 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:02.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a999b700 1 -- 172.21.15.111:0/3642383279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 msgr2=0x55d1eb67ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:02.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a999b700 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a999b700 1 -- 172.21.15.111:0/3642383279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 msgr2=0x55d1eb67a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:02.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.915+0000 7fa5a999b700 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb67a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.916+0000 7fa5a999b700 1 -- 172.21.15.111:0/3642383279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1eb4486b0 con 0x55d1eb675be0 2023-10-11T16:39:02.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.916+0000 7fa5a919a700 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb67a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:02.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.916+0000 7fa5a919a700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa5900a12c0 0x7fa5900a3770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4199c20 tx=0x7fa59400bc00 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:02.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.916+0000 7fa5a999b700 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb675fb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009d00 tx=0x7fa5a0011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd0b039868097665 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:02.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.916+0000 7fa58ffff700 1 -- 172.21.15.111:0/3642383279 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0004230 con 0x55d1eb675be0 2023-10-11T16:39:02.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.916+0000 7fa58ffff700 1 -- 172.21.15.111:0/3642383279 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5a0004390 con 0x55d1eb675be0 2023-10-11T16:39:02.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.916+0000 7fa58ffff700 1 -- 172.21.15.111:0/3642383279 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0019750 con 0x55d1eb675be0 2023-10-11T16:39:02.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/3642383279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 msgr2=0x55d1eb675fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:02.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb675fb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009d00 tx=0x7fa5a0011720 comp rx=0 tx=0).stop 2023-10-11T16:39:02.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/3642383279 shutdown_connections 2023-10-11T16:39:02.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb675fb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/3642383279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb67a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/3642383279 >> 172.21.15.111:0/3642383279 conn(0x55d1eb66b3e0 msgr2=0x55d1eb670bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:02.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/3642383279 shutdown_connections 2023-10-11T16:39:02.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/3642383279 wait complete. 2023-10-11T16:39:02.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.917+0000 7fa5bf43fb80 1 -- start start 2023-10-11T16:39:02.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5bf43fb80 1 Processor -- start 2023-10-11T16:39:02.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb675be0 0x55d1eb6820d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 0x55d1eb67f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb568960 con 0x55d1eb675be0 2023-10-11T16:39:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb568600 con 0x55d1eb67a930 2023-10-11T16:39:02.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb44a9a0 con 0x55d1eb6764f0 2023-10-11T16:39:02.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5bf43fb80 10 client.0 _ll_register_callbacks cb 0x55d1eb586800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5a999b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb675be0 0x55d1eb6820d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5a999b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb675be0 0x55d1eb6820d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33482/0 (socket says 172.21.15.111:33482) 2023-10-11T16:39:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.918+0000 7fa5a999b700 1 -- 172.21.15.111:0/41516751 learned_addr learned my addr 172.21.15.111:0/41516751 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5a919a700 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 0x55d1eb67f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5a999b700 1 -- 172.21.15.111:0/41516751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 msgr2=0x55d1eb67f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:02.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5a999b700 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 0x55d1eb67f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5a999b700 1 -- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 msgr2=0x55d1eb67f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5a999b700 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5a999b700 1 -- 172.21.15.111:0/41516751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a0009960 con 0x55d1eb675be0 2023-10-11T16:39:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa5a999b700 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb675be0 0x55d1eb6820d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a00038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c423ebe3941af3f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa58d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:02.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.919+0000 7fa58d7fa700 1 -- 172.21.15.111:0/41516751 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0005450 con 0x55d1eb675be0 2023-10-11T16:39:02.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa58d7fa700 1 -- 172.21.15.111:0/41516751 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa5a00055b0 con 0x55d1eb675be0 2023-10-11T16:39:02.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/41516751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1eb668ba0 con 0x55d1eb675be0 2023-10-11T16:39:02.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa58d7fa700 1 -- 172.21.15.111:0/41516751 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5a0004500 con 0x55d1eb675be0 2023-10-11T16:39:02.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 10 client.16542 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:02.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/41516751 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d1eb58d210 con 0x55d1eb675be0 2023-10-11T16:39:02.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa58d7fa700 1 -- 172.21.15.111:0/41516751 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d1eb668ba0 con 0x55d1eb675be0 2023-10-11T16:39:02.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa58d7fa700 1 -- 172.21.15.111:0/41516751 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa5a0028020 con 0x55d1eb675be0 2023-10-11T16:39:02.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 10 client.16542 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:02.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:02.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 20 client.16542 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1eb68f8d0 0x55d1eb691ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 4 client.16542 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:02.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.920+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/41516751 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1eb589120 con 0x55d1eb68f8d0 2023-10-11T16:39:02.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.921+0000 7fa5a919a700 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1eb68f8d0 0x55d1eb691ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.921+0000 7fa5a919a700 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1eb68f8d0 0x55d1eb691ce0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c198fe08c03b1f07 server_cookie=505ee84e4c50ab3b in_seq=0 out_seq=0 2023-10-11T16:39:02.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.921+0000 7fa58d7fa700 10 client.16542 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:02.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.922+0000 7fa58d7fa700 1 -- 172.21.15.111:0/41516751 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d1eb589120 con 0x55d1eb68f8d0 2023-10-11T16:39:02.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.922+0000 7fa58d7fa700 10 client.16542 handle_command_reply: tid=0 2023-10-11T16:39:02.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.922+0000 7fa5bf43fb80 1 client.16542 shutdown 2023-10-11T16:39:02.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.922+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/41516751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb675be0 msgr2=0x55d1eb6820d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:02.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.922+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb675be0 0x55d1eb6820d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a00038b0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/41516751 shutdown_connections 2023-10-11T16:39:02.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1eb68f8d0 0x55d1eb691ce0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb67f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb675be0 0x55d1eb6820d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/41516751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 0x55d1eb67f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/41516751 >> 172.21.15.111:0/41516751 conn(0x55d1eb66b3e0 msgr2=0x55d1eb66fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:02.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa58d7fa700 0 client.16542 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:02.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/41516751 shutdown_connections 2023-10-11T16:39:02.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/41516751 wait complete. 2023-10-11T16:39:02.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.923+0000 7fa5bf43fb80 20 client.16542 trim_cache size 0 max 16384 2023-10-11T16:39:02.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.940+0000 7fa5bf43fb80 1 Processor -- start 2023-10-11T16:39:02.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.940+0000 7fa5bf43fb80 1 -- start start 2023-10-11T16:39:02.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb686a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb6750f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb681c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb680930 con 0x55d1eb6764f0 2023-10-11T16:39:02.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb668a30 con 0x55d1eb67a930 2023-10-11T16:39:02.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb672240 con 0x55d1eb675be0 2023-10-11T16:39:02.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5a919a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb6750f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5a919a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb6750f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33498/0 (socket says 172.21.15.111:33498) 2023-10-11T16:39:02.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5a919a700 1 -- 172.21.15.111:0/939248053 learned_addr learned my addr 172.21.15.111:0/939248053 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:02.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5a999b700 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb686a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.941+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb681c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa5a919a700 1 -- 172.21.15.111:0/939248053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 msgr2=0x55d1eb686a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:02.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa5a919a700 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb686a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa5a919a700 1 -- 172.21.15.111:0/939248053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 msgr2=0x55d1eb681c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:02.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa5a919a700 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb681c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa5a919a700 1 -- 172.21.15.111:0/939248053 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1eb58cf60 con 0x55d1eb6764f0 2023-10-11T16:39:02.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa5a919a700 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb6750f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa594005ff0 tx=0x7fa594027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c941fdbef4d40e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:02.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa598bff700 1 -- 172.21.15.111:0/939248053 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59401d070 con 0x55d1eb6764f0 2023-10-11T16:39:02.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa598bff700 1 -- 172.21.15.111:0/939248053 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa594004500 con 0x55d1eb6764f0 2023-10-11T16:39:02.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.942+0000 7fa598bff700 1 -- 172.21.15.111:0/939248053 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594017410 con 0x55d1eb6764f0 2023-10-11T16:39:02.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/939248053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 msgr2=0x55d1eb6750f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:02.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb6750f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa594005ff0 tx=0x7fa594027000 comp rx=0 tx=0).stop 2023-10-11T16:39:02.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/939248053 shutdown_connections 2023-10-11T16:39:02.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb67a930 0x55d1eb681c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb675be0 0x55d1eb686a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/939248053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb6764f0 0x55d1eb6750f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/939248053 >> 172.21.15.111:0/939248053 conn(0x55d1eb66b3e0 msgr2=0x55d1eb6749e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:02.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/939248053 shutdown_connections 2023-10-11T16:39:02.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/939248053 wait complete. 2023-10-11T16:39:02.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 -- start start 2023-10-11T16:39:02.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.943+0000 7fa5bf43fb80 1 Processor -- start 2023-10-11T16:39:02.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb675be0 0x55d1eb685030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 0x55d1eb685570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5bf43fb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb67a930 0x55d1eb685ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:02.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb6e3760 con 0x55d1eb67a930 2023-10-11T16:39:02.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb6e38a0 con 0x55d1eb675be0 2023-10-11T16:39:02.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5bf43fb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1eb6e39e0 con 0x55d1eb6764f0 2023-10-11T16:39:02.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5bf43fb80 10 client.0 _ll_register_callbacks cb 0x7fa59c009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:02.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5aa19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb67a930 0x55d1eb685ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5aa19c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb67a930 0x55d1eb685ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33500/0 (socket says 172.21.15.111:33500) 2023-10-11T16:39:02.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5aa19c700 1 -- 172.21.15.111:0/2045515381 learned_addr learned my addr 172.21.15.111:0/2045515381 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:02.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5a919a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 0x55d1eb685570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5a999b700 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb675be0 0x55d1eb685030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:02.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5aa19c700 1 -- 172.21.15.111:0/2045515381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 msgr2=0x55d1eb685570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:02.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 0x55d1eb685570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5aa19c700 1 -- 172.21.15.111:0/2045515381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb675be0 msgr2=0x55d1eb685030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:02.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb675be0 0x55d1eb685030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:02.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.944+0000 7fa5aa19c700 1 -- 172.21.15.111:0/2045515381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59402e040 con 0x55d1eb67a930 2023-10-11T16:39:02.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa5aa19c700 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb67a930 0x55d1eb685ab0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00ef00 tx=0x7fa59c0955c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa1e9cecd77e00 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:02.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa58d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:02.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa58d7fa700 1 -- 172.21.15.111:0/2045515381 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c00ce60 con 0x55d1eb67a930 2023-10-11T16:39:02.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa58d7fa700 1 -- 172.21.15.111:0/2045515381 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7fa59c0b1070 con 0x55d1eb67a930 2023-10-11T16:39:02.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/2045515381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5a0004390 con 0x55d1eb67a930 2023-10-11T16:39:02.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa58d7fa700 1 -- 172.21.15.111:0/2045515381 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa59c0999a0 con 0x55d1eb67a930 2023-10-11T16:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa5bf43fb80 10 client.16548 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/2045515381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d1eb3ddce0 con 0x55d1eb67a930 2023-10-11T16:39:02.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa58d7fa700 1 -- 172.21.15.111:0/2045515381 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5a0004390 con 0x55d1eb67a930 2023-10-11T16:39:03.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.945+0000 7fa58d7fa700 1 -- 172.21.15.111:0/2045515381 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa59c01b9d0 con 0x55d1eb67a930 2023-10-11T16:39:03.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa5bf43fb80 10 client.16548 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:03.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa5bf43fb80 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:03.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa5bf43fb80 20 client.16548 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:03.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1eb6e5150 0x55d1eb6e7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa5bf43fb80 4 client.16548 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:39:03.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/2045515381 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d1eb5956e0 con 0x55d1eb6e5150 2023-10-11T16:39:03.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa5a999b700 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1eb6e5150 0x55d1eb6e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa58d7fa700 10 client.16548 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:03.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.946+0000 7fa5a999b700 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1eb6e5150 0x55d1eb6e7560 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ba4b47705f5cfe0 server_cookie=e2cf04a9ff3e11ed in_seq=0 out_seq=0 2023-10-11T16:39:03.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa58d7fa700 1 -- 172.21.15.111:0/2045515381 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d1eb5956e0 con 0x55d1eb6e5150 2023-10-11T16:39:03.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa58d7fa700 10 client.16548 handle_command_reply: tid=0 2023-10-11T16:39:03.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 client.16548 shutdown 2023-10-11T16:39:03.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/2045515381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb67a930 msgr2=0x55d1eb685ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb67a930 0x55d1eb685ab0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00ef00 tx=0x7fa59c0955c0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/2045515381 shutdown_connections 2023-10-11T16:39:03.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1eb6e5150 0x55d1eb6e7560 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1eb67a930 0x55d1eb685ab0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1eb675be0 0x55d1eb685030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 --2- 172.21.15.111:0/2045515381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1eb6764f0 0x55d1eb685570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa58d7fa700 0 client.16548 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:03.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.947+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/2045515381 >> 172.21.15.111:0/2045515381 conn(0x55d1eb66b3e0 msgr2=0x55d1eb675270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:03.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/2045515381 shutdown_connections 2023-10-11T16:39:03.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5bf43fb80 1 -- 172.21.15.111:0/2045515381 wait complete. 2023-10-11T16:39:03.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5bf43fb80 20 client.16548 trim_cache size 0 max 16384 2023-10-11T16:39:03.007 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:39:03.007 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:39:03.007 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "d340f37f-1db9-4d09-999a-a996a4a9a7ab", 2023-10-11T16:39:03.008 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:39:03.008 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:39:03.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 -- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa5900a12c0 msgr2=0x7fa5900a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa5900a12c0 0x7fa5900a3770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4199c20 tx=0x7fa59400bc00 comp rx=0 tx=0).stop 2023-10-11T16:39:03.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 -- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a41063f0 msgr2=0x7fa5a419cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a41063f0 0x7fa5a419cfb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa59c00ef30 tx=0x7fa59c00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 -- 172.21.15.111:0/3843872633 shutdown_connections 2023-10-11T16:39:03.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa5900a12c0 0x7fa5900a3770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5a41063f0 0x7fa5a419cfb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a4100a50 0x7fa5a4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 --2- 172.21.15.111:0/3843872633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5a4101400 0x7fa5a4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.948+0000 7fa5abbff700 1 -- 172.21.15.111:0/3843872633 >> 172.21.15.111:0/3843872633 conn(0x7fa5a4077590 msgr2=0x7fa5a40ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:03.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.949+0000 7fa5abbff700 1 -- 172.21.15.111:0/3843872633 shutdown_connections 2023-10-11T16:39:03.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:02.949+0000 7fa5abbff700 1 -- 172.21.15.111:0/3843872633 wait complete. 2023-10-11T16:39:03.013 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:39:03.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 -- 172.21.15.111:0/1644772645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4101660 msgr2=0x7f13c4104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 --2- 172.21.15.111:0/1644772645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4101660 0x7f13c4104ef0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f13b40099c0 tx=0x7f13b4009cd0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 -- 172.21.15.111:0/1644772645 shutdown_connections 2023-10-11T16:39:03.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 --2- 172.21.15.111:0/1644772645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13c4105620 0x7f13c410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 --2- 172.21.15.111:0/1644772645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c4100cc0 0x7f13c4101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 --2- 172.21.15.111:0/1644772645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4101660 0x7f13c4104ef0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 -- 172.21.15.111:0/1644772645 >> 172.21.15.111:0/1644772645 conn(0x7f13c4077560 msgr2=0x7f13c4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:03.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 -- 172.21.15.111:0/1644772645 shutdown_connections 2023-10-11T16:39:03.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.212+0000 7f13cbfff700 1 -- 172.21.15.111:0/1644772645 wait complete. 2023-10-11T16:39:03.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.213+0000 7f13cbfff700 1 Processor -- start 2023-10-11T16:39:03.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.213+0000 7f13cbfff700 1 -- start start 2023-10-11T16:39:03.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13cbfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c4100cc0 0x7f13c4072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13cbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13c4101660 0x7f13c4073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4105620 0x7f13c41a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13cbfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c4073b60 con 0x7f13c4105620 2023-10-11T16:39:03.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13cbfff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c41a4c00 con 0x7f13c4100cc0 2023-10-11T16:39:03.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13cbfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c41a4d40 con 0x7f13c4101660 2023-10-11T16:39:03.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c9d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c4100cc0 0x7f13c4072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c9d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c4100cc0 0x7f13c4072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:59028/0 (socket says 172.21.15.111:59028) 2023-10-11T16:39:03.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4105620 0x7f13c41a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c9d9b700 1 -- 172.21.15.111:0/61680412 learned_addr learned my addr 172.21.15.111:0/61680412 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:03.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c959a700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13c4101660 0x7f13c4073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c9d9b700 1 -- 172.21.15.111:0/61680412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13c4101660 msgr2=0x7f13c4073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13c4101660 0x7f13c4073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c9d9b700 1 -- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4105620 msgr2=0x7f13c41a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4105620 0x7f13c41a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13c9d9b700 1 -- 172.21.15.111:0/61680412 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b40095f0 con 0x7f13c4100cc0 2023-10-11T16:39:03.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.214+0000 7f13ca59c700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4105620 0x7f13c41a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:03.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.215+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c4100cc0 0x7f13c4072df0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f13c000e9a0 tx=0x7f13c000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.215+0000 7f13baffd700 1 -- 172.21.15.111:0/61680412 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0009790 con 0x7f13c4100cc0 2023-10-11T16:39:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.215+0000 7f13cbfff700 1 -- 172.21.15.111:0/61680412 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c4110370 con 0x7f13c4100cc0 2023-10-11T16:39:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.215+0000 7f13baffd700 1 -- 172.21.15.111:0/61680412 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13c0010430 con 0x7f13c4100cc0 2023-10-11T16:39:03.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.215+0000 7f13baffd700 1 -- 172.21.15.111:0/61680412 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0018680 con 0x7f13c4100cc0 2023-10-11T16:39:03.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.215+0000 7f13cbfff700 1 -- 172.21.15.111:0/61680412 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c41101e0 con 0x7f13c4100cc0 2023-10-11T16:39:03.230 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.218+0000 7f13baffd700 1 -- 172.21.15.111:0/61680412 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f13c00105a0 con 0x7f13c4100cc0 2023-10-11T16:39:03.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.219+0000 7f13df940b80 1 Processor -- start 2023-10-11T16:39:03.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.219+0000 7f13df940b80 1 -- start start 2023-10-11T16:39:03.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.219+0000 7f13df940b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a00f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.219+0000 7f13df940b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a01470 0x55d1f0a05370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13df940b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a058b0 0x55d1f0a05d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13df940b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f0906060 con 0x55d1f0a00b60 2023-10-11T16:39:03.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13df940b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f09588a0 con 0x55d1f0a01470 2023-10-11T16:39:03.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13df940b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f0951d30 con 0x55d1f0a058b0 2023-10-11T16:39:03.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a00f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a00f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33530/0 (socket says 172.21.15.111:33530) 2023-10-11T16:39:03.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13c959a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a058b0 0x55d1f0a05d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13c9d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a01470 0x55d1f0a05370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13ca59c700 1 -- 172.21.15.111:0/3846884374 learned_addr learned my addr 172.21.15.111:0/3846884374 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:03.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13baffd700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f13b00a12c0 0x7f13b00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13c959a700 1 -- 172.21.15.111:0/3846884374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a01470 msgr2=0x55d1f0a05370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:03.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13c959a700 1 --2- 172.21.15.111:0/3846884374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a01470 0x55d1f0a05370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13c959a700 1 -- 172.21.15.111:0/3846884374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 msgr2=0x55d1f0a00f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:03.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13c959a700 1 --2- 172.21.15.111:0/3846884374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a00f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13c959a700 1 -- 172.21.15.111:0/3846884374 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f07d3630 con 0x55d1f0a058b0 2023-10-11T16:39:03.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13baffd700 1 -- 172.21.15.111:0/61680412 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f13c0014070 con 0x7f13c4100cc0 2023-10-11T16:39:03.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.220+0000 7f13ca59c700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f13b00a12c0 0x7f13b00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.221+0000 7f13c959a700 1 --2- 172.21.15.111:0/3846884374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a058b0 0x55d1f0a05d00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f13c406a350 tx=0x7f13b40047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0b3b6edea9221c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:03.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.221+0000 7f13affff700 1 -- 172.21.15.111:0/3846884374 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4026070 con 0x55d1f0a058b0 2023-10-11T16:39:03.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.221+0000 7f13affff700 1 -- 172.21.15.111:0/3846884374 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13b401e040 con 0x55d1f0a058b0 2023-10-11T16:39:03.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.221+0000 7f13affff700 1 -- 172.21.15.111:0/3846884374 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b400fad0 con 0x55d1f0a058b0 2023-10-11T16:39:03.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 -- 172.21.15.111:0/3846884374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a058b0 msgr2=0x55d1f0a05d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13ca59c700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f13b00a12c0 0x7f13b00a3770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009790 tx=0x7f13bc00b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:03.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 --2- 172.21.15.111:0/3846884374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a058b0 0x55d1f0a05d00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f13c406a350 tx=0x7f13b40047e0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 -- 172.21.15.111:0/3846884374 shutdown_connections 2023-10-11T16:39:03.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 --2- 172.21.15.111:0/3846884374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a058b0 0x55d1f0a05d00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 --2- 172.21.15.111:0/3846884374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a00f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 --2- 172.21.15.111:0/3846884374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a01470 0x55d1f0a05370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 -- 172.21.15.111:0/3846884374 >> 172.21.15.111:0/3846884374 conn(0x55d1f09f6360 msgr2=0x55d1f09fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:03.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 -- 172.21.15.111:0/3846884374 shutdown_connections 2023-10-11T16:39:03.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 -- 172.21.15.111:0/3846884374 wait complete. 2023-10-11T16:39:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.222+0000 7f13df940b80 1 -- start start 2023-10-11T16:39:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13df940b80 1 Processor -- start 2023-10-11T16:39:03.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13df940b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13df940b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a0a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13df940b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13df940b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f08f38e0 con 0x55d1f0a00b60 2023-10-11T16:39:03.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13df940b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f08f3580 con 0x55d1f0a058b0 2023-10-11T16:39:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13df940b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f07d5920 con 0x55d1f0a01470 2023-10-11T16:39:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13df940b80 10 client.0 _ll_register_callbacks cb 0x55d1f0911780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:03.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13c9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13c9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33544/0 (socket says 172.21.15.111:33544) 2023-10-11T16:39:03.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13c959a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a0a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13c9d9b700 1 -- 172.21.15.111:0/823385075 learned_addr learned my addr 172.21.15.111:0/823385075 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:03.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.223+0000 7f13ca59c700 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13c9d9b700 1 -- 172.21.15.111:0/823385075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 msgr2=0x55d1f0a0a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a0a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13c9d9b700 1 -- 172.21.15.111:0/823385075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a058b0 msgr2=0x55d1f0a0a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:03.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13c9d9b700 1 -- 172.21.15.111:0/823385075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c410d9b0 con 0x55d1f0a00b60 2023-10-11T16:39:03.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13c959a700 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a0a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0cff0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f13c00964a0 tx=0x7f13c0096070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844a9a2eea78b02e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:03.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13ad7fa700 1 -- 172.21.15.111:0/823385075 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00cf2e0 con 0x55d1f0a00b60 2023-10-11T16:39:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13ad7fa700 1 -- 172.21.15.111:0/823385075 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f13c009dfd0 con 0x55d1f0a00b60 2023-10-11T16:39:03.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13df940b80 1 -- 172.21.15.111:0/823385075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1f09f3b20 con 0x55d1f0a00b60 2023-10-11T16:39:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13ad7fa700 1 -- 172.21.15.111:0/823385075 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13c00a23f0 con 0x55d1f0a00b60 2023-10-11T16:39:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13df940b80 10 client.16560 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.224+0000 7f13df940b80 1 -- 172.21.15.111:0/823385075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d1f0918190 con 0x55d1f0a00b60 2023-10-11T16:39:03.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13ad7fa700 1 -- 172.21.15.111:0/823385075 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d1f09f3b20 con 0x55d1f0a00b60 2023-10-11T16:39:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13ad7fa700 1 -- 172.21.15.111:0/823385075 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f13c00a1f60 con 0x55d1f0a00b60 2023-10-11T16:39:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13df940b80 10 client.16560 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13df940b80 10 client.16560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:03.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13df940b80 20 client.16560 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:03.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13df940b80 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1f0a1a780 0x55d1f0a1cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13df940b80 4 client.16560 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:03.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13df940b80 1 -- 172.21.15.111:0/823385075 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1f09140a0 con 0x55d1f0a1a780 2023-10-11T16:39:03.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.225+0000 7f13c959a700 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1f0a1a780 0x55d1f0a1cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.226+0000 7f13c959a700 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1f0a1a780 0x55d1f0a1cb90 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65a7e2134b38c302 server_cookie=ed761c8094bdc45b in_seq=0 out_seq=0 2023-10-11T16:39:03.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.226+0000 7f13ad7fa700 10 client.16560 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:03.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.226+0000 7f13ad7fa700 1 -- 172.21.15.111:0/823385075 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d1f09140a0 con 0x55d1f0a1a780 2023-10-11T16:39:03.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.226+0000 7f13ad7fa700 10 client.16560 handle_command_reply: tid=0 2023-10-11T16:39:03.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 client.16560 shutdown 2023-10-11T16:39:03.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 -- 172.21.15.111:0/823385075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 msgr2=0x55d1f0a0cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0cff0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f13c00964a0 tx=0x7f13c0096070 comp rx=0 tx=0).stop 2023-10-11T16:39:03.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 -- 172.21.15.111:0/823385075 shutdown_connections 2023-10-11T16:39:03.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1f0a1a780 0x55d1f0a1cb90 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0cff0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13ad7fa700 0 client.16560 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:03.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 --2- 172.21.15.111:0/823385075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a0a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 -- 172.21.15.111:0/823385075 >> 172.21.15.111:0/823385075 conn(0x55d1f09f6360 msgr2=0x55d1f0a06d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:03.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 -- 172.21.15.111:0/823385075 shutdown_connections 2023-10-11T16:39:03.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 1 -- 172.21.15.111:0/823385075 wait complete. 2023-10-11T16:39:03.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.227+0000 7f13df940b80 20 client.16560 trim_cache size 0 max 16384 2023-10-11T16:39:03.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.244+0000 7f13df940b80 1 Processor -- start 2023-10-11T16:39:03.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.244+0000 7f13df940b80 1 -- start start 2023-10-11T16:39:03.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13df940b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a11940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13df940b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a160f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13df940b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13df940b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f0a0b240 con 0x55d1f0a058b0 2023-10-11T16:39:03.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13df940b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f0a04410 con 0x55d1f0a00b60 2023-10-11T16:39:03.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13df940b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f09f3960 con 0x55d1f0a01470 2023-10-11T16:39:03.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33550/0 (socket says 172.21.15.111:33550) 2023-10-11T16:39:03.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13ca59c700 1 -- 172.21.15.111:0/2807796409 learned_addr learned my addr 172.21.15.111:0/2807796409 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:03.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a11940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13c959a700 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13ca59c700 1 -- 172.21.15.111:0/2807796409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 msgr2=0x55d1f0a160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:03.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13ca59c700 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13ca59c700 1 -- 172.21.15.111:0/2807796409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 msgr2=0x55d1f0a11940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:03.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13ca59c700 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a11940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.245+0000 7f13ca59c700 1 -- 172.21.15.111:0/2807796409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f0917ee0 con 0x55d1f0a058b0 2023-10-11T16:39:03.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.246+0000 7f13ca59c700 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0cb90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f13bc017a20 tx=0x7f13bc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f4b4c8cf41337b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:03.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.246+0000 7f13b8ff9700 1 -- 172.21.15.111:0/2807796409 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc016a60 con 0x55d1f0a058b0 2023-10-11T16:39:03.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.246+0000 7f13b8ff9700 1 -- 172.21.15.111:0/2807796409 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13bc00f040 con 0x55d1f0a058b0 2023-10-11T16:39:03.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.246+0000 7f13b8ff9700 1 -- 172.21.15.111:0/2807796409 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc023430 con 0x55d1f0a058b0 2023-10-11T16:39:03.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.246+0000 7f13df940b80 1 -- 172.21.15.111:0/2807796409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 msgr2=0x55d1f0a0cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.246+0000 7f13df940b80 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0cb90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f13bc017a20 tx=0x7f13bc016000 comp rx=0 tx=0).stop 2023-10-11T16:39:03.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 -- 172.21.15.111:0/2807796409 shutdown_connections 2023-10-11T16:39:03.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a0cb90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a11940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 --2- 172.21.15.111:0/2807796409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 -- 172.21.15.111:0/2807796409 >> 172.21.15.111:0/2807796409 conn(0x55d1f09f6360 msgr2=0x55d1f0a03970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:03.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 -- 172.21.15.111:0/2807796409 shutdown_connections 2023-10-11T16:39:03.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 -- 172.21.15.111:0/2807796409 wait complete. 2023-10-11T16:39:03.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 -- start start 2023-10-11T16:39:03.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.247+0000 7f13df940b80 1 Processor -- start 2023-10-11T16:39:03.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13df940b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13df940b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a10060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13df940b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a6d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13df940b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f0a6dba0 con 0x55d1f0a058b0 2023-10-11T16:39:03.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13df940b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f0a6dce0 con 0x55d1f0a00b60 2023-10-11T16:39:03.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13df940b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f0a6dea0 con 0x55d1f0a01470 2023-10-11T16:39:03.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13df940b80 10 client.0 _ll_register_callbacks cb 0x55d1f0911780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:03.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c9d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.274 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c9d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:59076/0 (socket says 172.21.15.111:59076) 2023-10-11T16:39:03.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c9d9b700 1 -- 172.21.15.111:0/4276209116 learned_addr learned my addr 172.21.15.111:0/4276209116 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:03.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a6d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.275 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c959a700 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a10060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c9d9b700 1 -- 172.21.15.111:0/4276209116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 msgr2=0x55d1f0a10060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:03.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a10060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c9d9b700 1 -- 172.21.15.111:0/4276209116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 msgr2=0x55d1f0a6d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:03.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a6d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.248+0000 7f13c9d9b700 1 -- 172.21.15.111:0/4276209116 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc0176d0 con 0x55d1f0a00b60 2023-10-11T16:39:03.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13c9d9b700 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0fb20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f13c00ca760 tx=0x7f13c009a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dbbd067f3f038af server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:03.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:39:03.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13ad7fa700 1 -- 172.21.15.111:0/4276209116 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00962e0 con 0x55d1f0a00b60 2023-10-11T16:39:03.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 1 -- 172.21.15.111:0/4276209116 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f13b401e040 con 0x55d1f0a00b60 2023-10-11T16:39:03.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13ad7fa700 1 -- 172.21.15.111:0/4276209116 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9468+0+0 (secure 0 0 0) 0x7f13c009dfd0 con 0x55d1f0a00b60 2023-10-11T16:39:03.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13ad7fa700 1 -- 172.21.15.111:0/4276209116 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13c00a23f0 con 0x55d1f0a00b60 2023-10-11T16:39:03.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 10 client.26681 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:03.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 1 -- 172.21.15.111:0/4276209116 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d1f0768c60 con 0x55d1f0a00b60 2023-10-11T16:39:03.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13ad7fa700 1 -- 172.21.15.111:0/4276209116 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13b401e040 con 0x55d1f0a00b60 2023-10-11T16:39:03.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13ad7fa700 1 -- 172.21.15.111:0/4276209116 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f13c000cc80 con 0x55d1f0a00b60 2023-10-11T16:39:03.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 10 client.26681 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:03.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 10 client.26681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:03.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 20 client.26681 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:03.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1f0a6f370 0x55d1f0a71780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:03.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 4 client.26681 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:39:03.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.249+0000 7f13df940b80 1 -- 172.21.15.111:0/4276209116 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d1f0920660 con 0x55d1f0a6f370 2023-10-11T16:39:03.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.250+0000 7f13c959a700 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1f0a6f370 0x55d1f0a71780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:03.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.250+0000 7f13ad7fa700 10 client.26681 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:03.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.250+0000 7f13c959a700 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1f0a6f370 0x55d1f0a71780 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e721add30bc8401 server_cookie=bb7f8a7c76be8b01 in_seq=0 out_seq=0 2023-10-11T16:39:03.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.250+0000 7f13ad7fa700 1 -- 172.21.15.111:0/4276209116 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d1f0920660 con 0x55d1f0a6f370 2023-10-11T16:39:03.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.250+0000 7f13ad7fa700 10 client.26681 handle_command_reply: tid=0 2023-10-11T16:39:03.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.250+0000 7f13df940b80 1 client.26681 shutdown 2023-10-11T16:39:03.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 -- 172.21.15.111:0/4276209116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 msgr2=0x55d1f0a0fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0fb20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f13c00ca760 tx=0x7f13c009a430 comp rx=0 tx=0).stop 2023-10-11T16:39:03.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 -- 172.21.15.111:0/4276209116 shutdown_connections 2023-10-11T16:39:03.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d1f0a6f370 0x55d1f0a71780 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d1f0a058b0 0x55d1f0a6d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d1f0a00b60 0x55d1f0a0fb20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13ad7fa700 0 client.26681 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:03.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 --2- 172.21.15.111:0/4276209116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d1f0a01470 0x55d1f0a10060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 -- 172.21.15.111:0/4276209116 >> 172.21.15.111:0/4276209116 conn(0x55d1f09f6360 msgr2=0x55d1f0a160f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:03.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 -- 172.21.15.111:0/4276209116 shutdown_connections 2023-10-11T16:39:03.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 1 -- 172.21.15.111:0/4276209116 wait complete. 2023-10-11T16:39:03.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13df940b80 20 client.26681 trim_cache size 0 max 16384 2023-10-11T16:39:03.287 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:39:03.288 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:39:03.288 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:39:03.288 INFO:teuthology.orchestra.run.smithi111.stdout: "d340f37f-1db9-4d09-999a-a996a4a9a7ab": { 2023-10-11T16:39:03.288 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:39:03.289 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "d340f37f-1db9-4d09-999a-a996a4a9a7ab", 2023-10-11T16:39:03.289 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:39:03.289 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:39:03.290 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:39:03.290 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13cbfff700 1 -- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f13b00a12c0 msgr2=0x7f13b00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13cbfff700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f13b00a12c0 0x7f13b00a3770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009790 tx=0x7f13bc00b040 comp rx=0 tx=0).stop 2023-10-11T16:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13cbfff700 1 -- 172.21.15.111:0/61680412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c4100cc0 msgr2=0x7f13c4072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.251+0000 7f13cbfff700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c4100cc0 0x7f13c4072df0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f13c000e9a0 tx=0x7f13c000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.252+0000 7f13cbfff700 1 -- 172.21.15.111:0/61680412 shutdown_connections 2023-10-11T16:39:03.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.252+0000 7f13cbfff700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f13b00a12c0 0x7f13b00a3770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.252+0000 7f13cbfff700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13c4105620 0x7f13c41a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.252+0000 7f13cbfff700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13c4100cc0 0x7f13c4072df0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.252+0000 7f13cbfff700 1 --2- 172.21.15.111:0/61680412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13c4101660 0x7f13c4073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:03.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.252+0000 7f13cbfff700 1 -- 172.21.15.111:0/61680412 >> 172.21.15.111:0/61680412 conn(0x7f13c4077560 msgr2=0x7f13c40ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:03.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.252+0000 7f13cbfff700 1 -- 172.21.15.111:0/61680412 shutdown_connections 2023-10-11T16:39:03.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:03.252+0000 7f13cbfff700 1 -- 172.21.15.111:0/61680412 wait complete. 2023-10-11T16:39:03.295 INFO:tasks.cephfs.filesystem:scrub status for tag:d340f37f-1db9-4d09-999a-a996a4a9a7ab - {'path': '/', 'tag': 'd340f37f-1db9-4d09-999a-a996a4a9a7ab', 'options': 'recursive,force'} 2023-10-11T16:39:03.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:03 smithi130 ceph-mon[131859]: pgmap v195: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2023-10-11T16:39:03.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:03 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:39:03.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:03 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:03.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:03 smithi111 ceph-mon[98527]: pgmap v195: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2023-10-11T16:39:03.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:03 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:39:03.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:03 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:03 smithi150 ceph-mon[132217]: pgmap v195: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2023-10-11T16:39:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:03 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:39:03.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:03 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:04.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:04 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:39:04.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:04 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:04.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:04 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:39:04.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:04 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:04.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:04 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:39:04.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:04 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:04.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:04 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:39:04.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:04 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:04.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:04 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:39:04.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:04 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:04.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:04 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:39:04.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:04 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:05.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:05 smithi130 ceph-mon[131859]: pgmap v196: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2023-10-11T16:39:05.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:05 smithi111 ceph-mon[98527]: pgmap v196: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2023-10-11T16:39:05.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:05 smithi150 ceph-mon[132217]: pgmap v196: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 9 op/s 2023-10-11T16:39:07.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:07 smithi130 ceph-mon[131859]: pgmap v197: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 49 KiB/s wr, 25 op/s 2023-10-11T16:39:07.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:07 smithi130 ceph-mon[131859]: osdmap e93: 12 total, 12 up, 12 in 2023-10-11T16:39:07.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:07 smithi111 ceph-mon[98527]: pgmap v197: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 49 KiB/s wr, 25 op/s 2023-10-11T16:39:07.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:07 smithi111 ceph-mon[98527]: osdmap e93: 12 total, 12 up, 12 in 2023-10-11T16:39:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:07 smithi150 ceph-mon[132217]: pgmap v197: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 49 KiB/s wr, 25 op/s 2023-10-11T16:39:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:07 smithi150 ceph-mon[132217]: osdmap e93: 12 total, 12 up, 12 in 2023-10-11T16:39:09.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:09 smithi130 ceph-mon[131859]: pgmap v199: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2023-10-11T16:39:09.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:09 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:39:09.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:09 smithi111 ceph-mon[98527]: pgmap v199: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2023-10-11T16:39:09.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:09 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:39:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:09 smithi150 ceph-mon[132217]: pgmap v199: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 59 KiB/s wr, 28 op/s 2023-10-11T16:39:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:09 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:39:10.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:10 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:39:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:10 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:39:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:10 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:39:10.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:10 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:39:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:10 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:39:10.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:10 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:39:10.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:10 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:39:10.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:10 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:39:10.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:10 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:39:11.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:11 smithi130 ceph-mon[131859]: pgmap v200: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 27 op/s 2023-10-11T16:39:11.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:11 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:39:11.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:11 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:39:11.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:11 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:39:11.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:11 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:39:11.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:11 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:39:11.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:11 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:39:11.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:11 smithi111 ceph-mon[98527]: pgmap v200: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 27 op/s 2023-10-11T16:39:11.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:11 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:39:11.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:11 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:39:11.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:11 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:39:11.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:11 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:39:11.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:11 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:39:11.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:11 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:39:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:11 smithi150 ceph-mon[132217]: pgmap v200: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 27 op/s 2023-10-11T16:39:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:11 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:39:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:11 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:39:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:11 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:39:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:11 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:39:11.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:11 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:39:11.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:11 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:39:13.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:13 smithi130 ceph-mon[131859]: pgmap v201: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 27 op/s 2023-10-11T16:39:13.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:13 smithi111 ceph-mon[98527]: pgmap v201: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 27 op/s 2023-10-11T16:39:13.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:13 smithi150 ceph-mon[132217]: pgmap v201: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 55 KiB/s wr, 27 op/s 2023-10-11T16:39:15.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:15 smithi130 ceph-mon[131859]: pgmap v202: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 26 op/s 2023-10-11T16:39:15.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:15 smithi130 ceph-mon[131859]: osdmap e94: 12 total, 12 up, 12 in 2023-10-11T16:39:15.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:15 smithi111 ceph-mon[98527]: pgmap v202: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 26 op/s 2023-10-11T16:39:15.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:15 smithi111 ceph-mon[98527]: osdmap e94: 12 total, 12 up, 12 in 2023-10-11T16:39:15.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:15 smithi150 ceph-mon[132217]: pgmap v202: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 55 KiB/s wr, 26 op/s 2023-10-11T16:39:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:15 smithi150 ceph-mon[132217]: osdmap e94: 12 total, 12 up, 12 in 2023-10-11T16:39:17.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:17 smithi130 ceph-mon[131859]: pgmap v204: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 331 B/s wr, 7 op/s 2023-10-11T16:39:17.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:17 smithi111 ceph-mon[98527]: pgmap v204: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 331 B/s wr, 7 op/s 2023-10-11T16:39:17.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:17 smithi150 ceph-mon[132217]: pgmap v204: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 331 B/s wr, 7 op/s 2023-10-11T16:39:19.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:19 smithi130 ceph-mon[131859]: pgmap v205: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-10-11T16:39:19.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:19 smithi111 ceph-mon[98527]: pgmap v205: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-10-11T16:39:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:19 smithi150 ceph-mon[132217]: pgmap v205: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-10-11T16:39:21.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:21 smithi130 ceph-mon[131859]: pgmap v206: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:21.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:21 smithi111 ceph-mon[98527]: pgmap v206: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:21.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:21 smithi150 ceph-mon[132217]: pgmap v206: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:23.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:23 smithi111 ceph-mon[98527]: pgmap v207: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:23.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:23 smithi150 ceph-mon[132217]: pgmap v207: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:24.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:23 smithi130 ceph-mon[131859]: pgmap v207: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:25.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:25 smithi111 ceph-mon[98527]: pgmap v208: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:25 smithi150 ceph-mon[132217]: pgmap v208: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:26.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:25 smithi130 ceph-mon[131859]: pgmap v208: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:39:27.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:27 smithi111 ceph-mon[98527]: pgmap v209: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-11T16:39:27.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:27 smithi150 ceph-mon[132217]: pgmap v209: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-11T16:39:28.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:27 smithi130 ceph-mon[131859]: pgmap v209: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-11T16:39:29.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:29 smithi111 ceph-mon[98527]: pgmap v210: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:29.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:29 smithi150 ceph-mon[132217]: pgmap v210: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:30.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:29 smithi130 ceph-mon[131859]: pgmap v210: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:31.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:31 smithi111 ceph-mon[98527]: pgmap v211: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:39:31.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:31 smithi150 ceph-mon[132217]: pgmap v211: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:39:32.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:31 smithi130 ceph-mon[131859]: pgmap v211: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:39:33.296 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:39:33.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 -- 172.21.15.111:0/3854078280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34101660 msgr2=0x7ffb34104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 --2- 172.21.15.111:0/3854078280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34101660 0x7ffb34104ef0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffb200099c0 tx=0x7ffb20009cd0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 -- 172.21.15.111:0/3854078280 shutdown_connections 2023-10-11T16:39:33.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 --2- 172.21.15.111:0/3854078280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb34105620 0x7ffb3410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 --2- 172.21.15.111:0/3854078280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb34100cc0 0x7ffb34101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 --2- 172.21.15.111:0/3854078280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34101660 0x7ffb34104ef0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 -- 172.21.15.111:0/3854078280 >> 172.21.15.111:0/3854078280 conn(0x7ffb34077560 msgr2=0x7ffb34077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:33.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 -- 172.21.15.111:0/3854078280 shutdown_connections 2023-10-11T16:39:33.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.524+0000 7ffb3abff700 1 -- 172.21.15.111:0/3854078280 wait complete. 2023-10-11T16:39:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.525+0000 7ffb3abff700 1 Processor -- start 2023-10-11T16:39:33.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.525+0000 7ffb3abff700 1 -- start start 2023-10-11T16:39:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3abff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34100cc0 0x7ffb34072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3abff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb34101660 0x7ffb34073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3abff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb34105620 0x7ffb341a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3abff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb34073b60 con 0x7ffb34100cc0 2023-10-11T16:39:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3abff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb341a4c00 con 0x7ffb34101660 2023-10-11T16:39:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3abff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb341a4d40 con 0x7ffb34105620 2023-10-11T16:39:33.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3899b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34100cc0 0x7ffb34072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb33fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb34101660 0x7ffb34073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3899b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34100cc0 0x7ffb34072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47176/0 (socket says 172.21.15.111:47176) 2023-10-11T16:39:33.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3899b700 1 -- 172.21.15.111:0/2555934292 learned_addr learned my addr 172.21.15.111:0/2555934292 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3919c700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb34105620 0x7ffb341a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3899b700 1 -- 172.21.15.111:0/2555934292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb34105620 msgr2=0x7ffb341a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb34105620 0x7ffb341a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3899b700 1 -- 172.21.15.111:0/2555934292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb34101660 msgr2=0x7ffb34073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb34101660 0x7ffb34073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.526+0000 7ffb3899b700 1 -- 172.21.15.111:0/2555934292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb200095f0 con 0x7ffb34100cc0 2023-10-11T16:39:33.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.527+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34100cc0 0x7ffb34072df0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800d980 tx=0x7ffb2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.527+0000 7ffb31ffb700 1 -- 172.21.15.111:0/2555934292 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28004220 con 0x7ffb34100cc0 2023-10-11T16:39:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.527+0000 7ffb31ffb700 1 -- 172.21.15.111:0/2555934292 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb28007640 con 0x7ffb34100cc0 2023-10-11T16:39:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.527+0000 7ffb31ffb700 1 -- 172.21.15.111:0/2555934292 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28005690 con 0x7ffb34100cc0 2023-10-11T16:39:33.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.527+0000 7ffb3abff700 1 -- 172.21.15.111:0/2555934292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb34110370 con 0x7ffb34100cc0 2023-10-11T16:39:33.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.527+0000 7ffb3abff700 1 -- 172.21.15.111:0/2555934292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb341101e0 con 0x7ffb34100cc0 2023-10-11T16:39:33.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.530+0000 7ffb31ffb700 1 -- 172.21.15.111:0/2555934292 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7ffb28004750 con 0x7ffb34100cc0 2023-10-11T16:39:33.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.531+0000 7ffb4e56db80 1 Processor -- start 2023-10-11T16:39:33.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.531+0000 7ffb31ffb700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ffb240a1340 0x7ffb240a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb4e56db80 1 -- start start 2023-10-11T16:39:33.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb31ffb700 1 -- 172.21.15.111:0/2555934292 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7ffb280ccee0 con 0x7ffb34100cc0 2023-10-11T16:39:33.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb3919c700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ffb240a1340 0x7ffb240a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f1470 0x55bebf9f5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf8f6060 con 0x55bebf9f0b60 2023-10-11T16:39:33.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf9488a0 con 0x55bebf9f58b0 2023-10-11T16:39:33.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf941d30 con 0x55bebf9f1470 2023-10-11T16:39:33.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb3899b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb3899b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47178/0 (socket says 172.21.15.111:47178) 2023-10-11T16:39:33.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb3899b700 1 -- 172.21.15.111:0/2037202416 learned_addr learned my addr 172.21.15.111:0/2037202416 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:33.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb33fff700 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f1470 0x55bebf9f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.532+0000 7ffb3919c700 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb3899b700 1 -- 172.21.15.111:0/2037202416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f1470 msgr2=0x55bebf9f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:33.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f1470 0x55bebf9f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb3899b700 1 -- 172.21.15.111:0/2037202416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 msgr2=0x55bebf9f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:33.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb3899b700 1 -- 172.21.15.111:0/2037202416 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bebf7c3630 con 0x55bebf9f0b60 2023-10-11T16:39:33.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb3919c700 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:33.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb3919c700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ffb240a1340 0x7ffb240a37f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c009550 tx=0x7ffb2c009c20 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:33.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f0f30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffb280965b0 tx=0x7ffb28022420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94b93989c626e6d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:33.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb1effd700 1 -- 172.21.15.111:0/2037202416 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb280a1f90 con 0x55bebf9f0b60 2023-10-11T16:39:33.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb1effd700 1 -- 172.21.15.111:0/2037202416 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb28004380 con 0x55bebf9f0b60 2023-10-11T16:39:33.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.533+0000 7ffb1effd700 1 -- 172.21.15.111:0/2037202416 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb280a1b00 con 0x55bebf9f0b60 2023-10-11T16:39:33.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2037202416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 msgr2=0x55bebf9f0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f0f30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ffb280965b0 tx=0x7ffb28022420 comp rx=0 tx=0).stop 2023-10-11T16:39:33.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2037202416 shutdown_connections 2023-10-11T16:39:33.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f0f30 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2037202416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f1470 0x55bebf9f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2037202416 >> 172.21.15.111:0/2037202416 conn(0x55bebf9e6360 msgr2=0x55bebf9ebb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:33.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2037202416 shutdown_connections 2023-10-11T16:39:33.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.534+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2037202416 wait complete. 2023-10-11T16:39:33.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 1 -- start start 2023-10-11T16:39:33.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 1 Processor -- start 2023-10-11T16:39:33.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9fd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f1470 0x55bebf9fa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9fab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf8e38e0 con 0x55bebf9f1470 2023-10-11T16:39:33.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf8e3580 con 0x55bebf9f58b0 2023-10-11T16:39:33.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf7c5920 con 0x55bebf9f0b60 2023-10-11T16:39:33.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.535+0000 7ffb4e56db80 10 client.0 _ll_register_callbacks cb 0x55bebf901780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:33.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f1470 0x55bebf9fa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f1470 0x55bebf9fa3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47184/0 (socket says 172.21.15.111:47184) 2023-10-11T16:39:33.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 -- 172.21.15.111:0/1398793869 learned_addr learned my addr 172.21.15.111:0/1398793869 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:33.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb3899b700 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9fd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb3919c700 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9fab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 -- 172.21.15.111:0/1398793869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 msgr2=0x55bebf9fd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:33.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9fd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 -- 172.21.15.111:0/1398793869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 msgr2=0x55bebf9fab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:33.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9fab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 -- 172.21.15.111:0/1398793869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb341101e0 con 0x55bebf9f1470 2023-10-11T16:39:33.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb3899b700 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9fd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:33.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb33fff700 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f1470 0x55bebf9fa3e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ffb20000c00 tx=0x7ffb20005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a493cc8d36c5b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:33.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.536+0000 7ffb13fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:33.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb13fff700 1 -- 172.21.15.111:0/1398793869 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2001d070 con 0x55bebf9f1470 2023-10-11T16:39:33.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1398793869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bebf9e3b20 con 0x55bebf9f1470 2023-10-11T16:39:33.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb13fff700 1 -- 172.21.15.111:0/1398793869 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7ffb2000bc60 con 0x55bebf9f1470 2023-10-11T16:39:33.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb13fff700 1 -- 172.21.15.111:0/1398793869 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb2000be20 con 0x55bebf9f1470 2023-10-11T16:39:33.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb4e56db80 10 client.16578 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:33.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1398793869 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55bebf908190 con 0x55bebf9f1470 2023-10-11T16:39:33.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb13fff700 1 -- 172.21.15.111:0/1398793869 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bebf9e3b20 con 0x55bebf9f1470 2023-10-11T16:39:33.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb13fff700 1 -- 172.21.15.111:0/1398793869 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ffb20004730 con 0x55bebf9f1470 2023-10-11T16:39:33.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb4e56db80 10 client.16578 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:33.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.537+0000 7ffb4e56db80 10 client.16578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:33.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.538+0000 7ffb4e56db80 20 client.16578 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:33.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.538+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bebfa0a970 0x55bebfa0cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.538+0000 7ffb4e56db80 4 client.16578 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:33.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.538+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1398793869 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bebf9040a0 con 0x55bebfa0a970 2023-10-11T16:39:33.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.538+0000 7ffb3899b700 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bebfa0a970 0x55bebfa0cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.538+0000 7ffb3899b700 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bebfa0a970 0x55bebfa0cd80 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d77b306fd368bde5 server_cookie=5fb2f2543bbcb2b0 in_seq=0 out_seq=0 2023-10-11T16:39:33.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.538+0000 7ffb13fff700 10 client.16578 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:33.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.539+0000 7ffb13fff700 1 -- 172.21.15.111:0/1398793869 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bebf9040a0 con 0x55bebfa0a970 2023-10-11T16:39:33.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.539+0000 7ffb13fff700 10 client.16578 handle_command_reply: tid=0 2023-10-11T16:39:33.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.539+0000 7ffb4e56db80 1 client.16578 shutdown 2023-10-11T16:39:33.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1398793869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f1470 msgr2=0x55bebf9fa3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f1470 0x55bebf9fa3e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ffb20000c00 tx=0x7ffb20005a90 comp rx=0 tx=0).stop 2023-10-11T16:39:33.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1398793869 shutdown_connections 2023-10-11T16:39:33.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bebfa0a970 0x55bebfa0cd80 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f58b0 0x55bebf9fab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9fd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb13fff700 0 client.16578 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:33.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1398793869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f1470 0x55bebf9fa3e0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1398793869 >> 172.21.15.111:0/1398793869 conn(0x55bebf9e6360 msgr2=0x55bebf9f6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:33.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1398793869 shutdown_connections 2023-10-11T16:39:33.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1398793869 wait complete. 2023-10-11T16:39:33.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.540+0000 7ffb4e56db80 20 client.16578 trim_cache size 0 max 16384 2023-10-11T16:39:33.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.557+0000 7ffb4e56db80 1 Processor -- start 2023-10-11T16:39:33.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.557+0000 7ffb4e56db80 1 -- start start 2023-10-11T16:39:33.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f58b0 0x55bebf9fa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf9f1bc0 con 0x55bebf9f0b60 2023-10-11T16:39:33.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf9e5ee0 con 0x55bebf9f1470 2023-10-11T16:39:33.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebf9fa750 con 0x55bebf9f58b0 2023-10-11T16:39:33.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb3899b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.595 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb3899b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47192/0 (socket says 172.21.15.111:47192) 2023-10-11T16:39:33.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb3899b700 1 -- 172.21.15.111:0/1580767585 learned_addr learned my addr 172.21.15.111:0/1580767585 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:33.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb33fff700 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb3919c700 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f58b0 0x55bebf9fa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb3899b700 1 -- 172.21.15.111:0/1580767585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f58b0 msgr2=0x55bebf9fa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.558+0000 7ffb3899b700 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f58b0 0x55bebf9fa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.559+0000 7ffb3899b700 1 -- 172.21.15.111:0/1580767585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 msgr2=0x55bebf9f9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.559+0000 7ffb3899b700 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9f9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.559+0000 7ffb3899b700 1 -- 172.21.15.111:0/1580767585 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bebf907ee0 con 0x55bebf9f0b60 2023-10-11T16:39:33.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.559+0000 7ffb3899b700 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f31a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffb28099840 tx=0x7ffb280050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6dfa15c8c3ec33 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:33.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.559+0000 7ffb1f7fe700 1 -- 172.21.15.111:0/1580767585 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb280b2230 con 0x55bebf9f0b60 2023-10-11T16:39:33.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.559+0000 7ffb1f7fe700 1 -- 172.21.15.111:0/1580767585 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb28099da0 con 0x55bebf9f0b60 2023-10-11T16:39:33.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.559+0000 7ffb1f7fe700 1 -- 172.21.15.111:0/1580767585 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb280a1b00 con 0x55bebf9f0b60 2023-10-11T16:39:33.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1580767585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 msgr2=0x55bebf9f31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f31a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffb28099840 tx=0x7ffb280050b0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1580767585 shutdown_connections 2023-10-11T16:39:33.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f58b0 0x55bebf9fa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f0b60 0x55bebf9f31a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/1580767585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9f9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1580767585 >> 172.21.15.111:0/1580767585 conn(0x55bebf9e6360 msgr2=0x55bebf9f3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:33.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1580767585 shutdown_connections 2023-10-11T16:39:33.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 -- 172.21.15.111:0/1580767585 wait complete. 2023-10-11T16:39:33.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 -- start start 2023-10-11T16:39:33.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.560+0000 7ffb4e56db80 1 Processor -- start 2023-10-11T16:39:33.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb4e56db80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f58b0 0x55bebfa00370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebfa5d9d0 con 0x55bebf9f58b0 2023-10-11T16:39:33.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebfa5db40 con 0x55bebf9f1470 2023-10-11T16:39:33.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb4e56db80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bebfa5dd00 con 0x55bebf9f0b60 2023-10-11T16:39:33.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb4e56db80 10 client.0 _ll_register_callbacks cb 0x7ffb20005350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:33.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb3899b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb33fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb3899b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9ff8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41626/0 (socket says 172.21.15.111:41626) 2023-10-11T16:39:33.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb33fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9ffe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:33714/0 (socket says 172.21.15.111:33714) 2023-10-11T16:39:33.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb3899b700 1 -- 172.21.15.111:0/2389233630 learned_addr learned my addr 172.21.15.111:0/2389233630 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:33.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb3919c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f58b0 0x55bebfa00370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb33fff700 1 -- 172.21.15.111:0/2389233630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 msgr2=0x55bebf9ff8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:33.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb33fff700 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9ff8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb33fff700 1 -- 172.21.15.111:0/2389233630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f58b0 msgr2=0x55bebfa00370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:33.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb33fff700 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f58b0 0x55bebfa00370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb33fff700 1 -- 172.21.15.111:0/2389233630 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bebf7c3630 con 0x55bebf9f1470 2023-10-11T16:39:33.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.561+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9ff8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:33.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.562+0000 7ffb33fff700 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9ffe10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffb200098e0 tx=0x7ffb2000bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf830921698f99d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:33.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.562+0000 7ffb13fff700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:39:33.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.562+0000 7ffb13fff700 1 -- 172.21.15.111:0/2389233630 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2001d040 con 0x55bebf9f1470 2023-10-11T16:39:33.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.562+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2389233630 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb28004380 con 0x55bebf9f1470 2023-10-11T16:39:33.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.562+0000 7ffb13fff700 1 -- 172.21.15.111:0/2389233630 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7ffb20005bc0 con 0x55bebf9f1470 2023-10-11T16:39:33.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.562+0000 7ffb13fff700 1 -- 172.21.15.111:0/2389233630 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb2000b630 con 0x55bebf9f1470 2023-10-11T16:39:33.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.562+0000 7ffb4e56db80 10 client.26699 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:33.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.562+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2389233630 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55bebf758c60 con 0x55bebf9f1470 2023-10-11T16:39:33.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb13fff700 1 -- 172.21.15.111:0/2389233630 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb28004380 con 0x55bebf9f1470 2023-10-11T16:39:33.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb13fff700 1 -- 172.21.15.111:0/2389233630 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ffb20003d80 con 0x55bebf9f1470 2023-10-11T16:39:33.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb4e56db80 10 client.26699 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:33.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb4e56db80 10 client.26699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:33.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb4e56db80 20 client.26699 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:33.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bebfa5efe0 0x55bebfa613f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb4e56db80 4 client.26699 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:39:33.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2389233630 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bebf910660 con 0x55bebfa5efe0 2023-10-11T16:39:33.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.563+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bebfa5efe0 0x55bebfa613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb13fff700 10 client.26699 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:33.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb3899b700 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bebfa5efe0 0x55bebfa613f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a03f9e6ae07de86 server_cookie=cda6921eaeef3bc0 in_seq=0 out_seq=0 2023-10-11T16:39:33.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb13fff700 1 -- 172.21.15.111:0/2389233630 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55bebf910660 con 0x55bebfa5efe0 2023-10-11T16:39:33.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb13fff700 10 client.26699 handle_command_reply: tid=0 2023-10-11T16:39:33.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 client.26699 shutdown 2023-10-11T16:39:33.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2389233630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 msgr2=0x55bebf9ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9ffe10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffb200098e0 tx=0x7ffb2000bd10 comp rx=0 tx=0).stop 2023-10-11T16:39:33.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2389233630 shutdown_connections 2023-10-11T16:39:33.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55bebfa5efe0 0x55bebfa613f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55bebf9f58b0 0x55bebfa00370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb13fff700 0 client.26699 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:33.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55bebf9f0b60 0x55bebf9ff8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 --2- 172.21.15.111:0/2389233630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bebf9f1470 0x55bebf9ffe10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2389233630 >> 172.21.15.111:0/2389233630 conn(0x55bebf9e6360 msgr2=0x55bebf9ef960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:33.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2389233630 shutdown_connections 2023-10-11T16:39:33.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 1 -- 172.21.15.111:0/2389233630 wait complete. 2023-10-11T16:39:33.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.564+0000 7ffb4e56db80 20 client.26699 trim_cache size 0 max 16384 2023-10-11T16:39:33.625 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:39:33.625 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:39:33.626 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:39:33.626 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:39:33.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 -- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ffb240a1340 msgr2=0x7ffb240a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ffb240a1340 0x7ffb240a37f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c009550 tx=0x7ffb2c009c20 comp rx=0 tx=0).stop 2023-10-11T16:39:33.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 -- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34100cc0 msgr2=0x7ffb34072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34100cc0 0x7ffb34072df0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800d980 tx=0x7ffb2800dc90 comp rx=0 tx=0).stop 2023-10-11T16:39:33.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 -- 172.21.15.111:0/2555934292 shutdown_connections 2023-10-11T16:39:33.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ffb240a1340 0x7ffb240a37f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb34105620 0x7ffb341a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb34100cc0 0x7ffb34072df0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 --2- 172.21.15.111:0/2555934292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffb34101660 0x7ffb34073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 -- 172.21.15.111:0/2555934292 >> 172.21.15.111:0/2555934292 conn(0x7ffb34077560 msgr2=0x7ffb340ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:33.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 -- 172.21.15.111:0/2555934292 shutdown_connections 2023-10-11T16:39:33.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.565+0000 7ffb3abff700 1 -- 172.21.15.111:0/2555934292 wait complete. 2023-10-11T16:39:33.634 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:39:33.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:39:33.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:33 smithi111 ceph-mon[98527]: pgmap v212: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:33.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 -- 172.21.15.111:0/3395530280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b00ff080 msgr2=0x7f37b00ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3395530280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b00ff080 0x7f37b00ff450 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009990 tx=0x7f37a0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 -- 172.21.15.111:0/3395530280 shutdown_connections 2023-10-11T16:39:33.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3395530280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b010cc30 0x7f37b010f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3395530280 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b00ff080 0x7f37b00ff450 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3395530280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37b00ff990 0x7f37b010c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 -- 172.21.15.111:0/3395530280 >> 172.21.15.111:0/3395530280 conn(0x7f37b0076310 msgr2=0x7f37b0076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:33.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 -- 172.21.15.111:0/3395530280 shutdown_connections 2023-10-11T16:39:33.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.835+0000 7f37b61ff700 1 -- 172.21.15.111:0/3395530280 wait complete. 2023-10-11T16:39:33.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.836+0000 7f37b61ff700 1 Processor -- start 2023-10-11T16:39:33.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.836+0000 7f37b61ff700 1 -- start start 2023-10-11T16:39:33.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.836+0000 7f37b61ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b00ff080 0x7f37b01986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.836+0000 7f37b61ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37b00ff990 0x7f37b0198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.836+0000 7f37b61ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b010cc30 0x7f37b019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.836+0000 7f37b61ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b019d5e0 con 0x7f37b010cc30 2023-10-11T16:39:33.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.836+0000 7f37b61ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b019d750 con 0x7f37b00ff990 2023-10-11T16:39:33.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.836+0000 7f37b61ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b019d910 con 0x7f37b00ff080 2023-10-11T16:39:33.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b010cc30 0x7f37b019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b010cc30 0x7f37b019cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47212/0 (socket says 172.21.15.111:47212) 2023-10-11T16:39:33.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 -- 172.21.15.111:0/3573234292 learned_addr learned my addr 172.21.15.111:0/3573234292 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:33.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37af7fe700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b00ff080 0x7f37b01986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37aeffd700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37b00ff990 0x7f37b0198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 -- 172.21.15.111:0/3573234292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b00ff080 msgr2=0x7f37b01986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b00ff080 0x7f37b01986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 -- 172.21.15.111:0/3573234292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37b00ff990 msgr2=0x7f37b0198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:33.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37b00ff990 0x7f37b0198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:33.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 -- 172.21.15.111:0/3573234292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a00095f0 con 0x7f37b010cc30 2023-10-11T16:39:33.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37af7fe700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b00ff080 0x7f37b01986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:33.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37affff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b010cc30 0x7f37b019cfa0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f37a800ef30 tx=0x7f37a800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:33.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37acff9700 1 -- 172.21.15.111:0/3573234292 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8018070 con 0x7f37b010cc30 2023-10-11T16:39:33.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37acff9700 1 -- 172.21.15.111:0/3573234292 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37a800f040 con 0x7f37b010cc30 2023-10-11T16:39:33.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b0112520 con 0x7f37b010cc30 2023-10-11T16:39:33.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.837+0000 7f37acff9700 1 -- 172.21.15.111:0/3573234292 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8003e10 con 0x7f37b010cc30 2023-10-11T16:39:33.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.838+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b0112390 con 0x7f37b010cc30 2023-10-11T16:39:33.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.839+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b01126b0 con 0x7f37b010cc30 2023-10-11T16:39:33.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.840+0000 7f37acff9700 1 -- 172.21.15.111:0/3573234292 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f37a8004ad0 con 0x7f37b010cc30 2023-10-11T16:39:33.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.842+0000 7f37acff9700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f37980a1340 0x7f37980a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:33.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.842+0000 7f37acff9700 1 -- 172.21.15.111:0/3573234292 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f37a80cccd0 con 0x7f37b010cc30 2023-10-11T16:39:33.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.842+0000 7f37af7fe700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f37980a1340 0x7f37980a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:33.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.843+0000 7f37af7fe700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f37980a1340 0x7f37980a37f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f37a000b2e0 tx=0x7f37a0005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:33.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:33.846+0000 7f37acff9700 1 -- 172.21.15.111:0/3573234292 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f37a8095d00 con 0x7f37b010cc30 2023-10-11T16:39:33.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:33 smithi150 ceph-mon[132217]: pgmap v212: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:34.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:33 smithi130 ceph-mon[131859]: pgmap v212: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:34.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.087+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37b0067a60 con 0x7f37b010cc30 2023-10-11T16:39:34.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.088+0000 7f37acff9700 1 -- 172.21.15.111:0/3573234292 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10829 (secure 0 0 0) 0x7f37a8099320 con 0x7f37b010cc30 2023-10-11T16:39:34.090 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:39:34.090 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:36:18.517722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:39:34.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f37980a1340 msgr2=0x7f37980a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f37980a1340 0x7f37980a37f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f37a000b2e0 tx=0x7f37a0005fb0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b010cc30 msgr2=0x7f37b019cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b010cc30 0x7f37b019cfa0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f37a800ef30 tx=0x7f37a800c450 comp rx=0 tx=0).stop 2023-10-11T16:39:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 shutdown_connections 2023-10-11T16:39:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f37980a1340 0x7f37980a37f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37b010cc30 0x7f37b019cfa0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37b00ff080 0x7f37b01986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 --2- 172.21.15.111:0/3573234292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37b00ff990 0x7f37b0198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 >> 172.21.15.111:0/3573234292 conn(0x7f37b0076310 msgr2=0x7f37b00fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 shutdown_connections 2023-10-11T16:39:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.090+0000 7f37b61ff700 1 -- 172.21.15.111:0/3573234292 wait complete. 2023-10-11T16:39:34.095 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-10-11T16:39:34.111 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:39:34.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 -- 172.21.15.111:0/1636611747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 msgr2=0x7f8f20100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 --2- 172.21.15.111:0/1636611747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 0x7f8f20100ec0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8f140098f0 tx=0x7f8f14009c00 comp rx=0 tx=0).stop 2023-10-11T16:39:34.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 -- 172.21.15.111:0/1636611747 shutdown_connections 2023-10-11T16:39:34.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 --2- 172.21.15.111:0/1636611747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f20101400 0x7f8f2010d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 --2- 172.21.15.111:0/1636611747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f201063f0 0x7f8f201067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 --2- 172.21.15.111:0/1636611747 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 0x7f8f20100ec0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 -- 172.21.15.111:0/1636611747 >> 172.21.15.111:0/1636611747 conn(0x7f8f20077590 msgr2=0x7f8f20077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 -- 172.21.15.111:0/1636611747 shutdown_connections 2023-10-11T16:39:34.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.311+0000 7f8f25dff700 1 -- 172.21.15.111:0/1636611747 wait complete. 2023-10-11T16:39:34.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.312+0000 7f8f25dff700 1 Processor -- start 2023-10-11T16:39:34.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f25dff700 1 -- start start 2023-10-11T16:39:34.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f25dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 0x7f8f20196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f25dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f20101400 0x7f8f20196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f25dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f201063f0 0x7f8f2019aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f25dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2019b4e0 con 0x7f8f20100a50 2023-10-11T16:39:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f25dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2019b650 con 0x7f8f201063f0 2023-10-11T16:39:34.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f25dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2019b810 con 0x7f8f20101400 2023-10-11T16:39:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f1f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 0x7f8f20196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f1f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 0x7f8f20196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47244/0 (socket says 172.21.15.111:47244) 2023-10-11T16:39:34.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/3435250800 learned_addr learned my addr 172.21.15.111:0/3435250800 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f201063f0 0x7f8f2019aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.313+0000 7f8f1effd700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f20101400 0x7f8f20196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/3435250800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f20101400 msgr2=0x7f8f20196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f20101400 0x7f8f20196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/3435250800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f201063f0 msgr2=0x7f8f2019aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f201063f0 0x7f8f2019aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/3435250800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f100095f0 con 0x7f8f20100a50 2023-10-11T16:39:34.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1effd700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f20101400 0x7f8f20196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:39:34.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f201063f0 0x7f8f2019aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:34.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 0x7f8f20196500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000ec10 tx=0x7f8f1000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1cff9700 1 -- 172.21.15.111:0/3435250800 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10009a10 con 0x7f8f20100a50 2023-10-11T16:39:34.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f25dff700 1 -- 172.21.15.111:0/3435250800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f14009520 con 0x7f8f20100a50 2023-10-11T16:39:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1cff9700 1 -- 172.21.15.111:0/3435250800 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f100044e0 con 0x7f8f20100a50 2023-10-11T16:39:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f1cff9700 1 -- 172.21.15.111:0/3435250800 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10010450 con 0x7f8f20100a50 2023-10-11T16:39:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.314+0000 7f8f25dff700 1 -- 172.21.15.111:0/3435250800 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f20110410 con 0x7f8f20100a50 2023-10-11T16:39:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.317+0000 7f8f1cff9700 1 -- 172.21.15.111:0/3435250800 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f8f10003680 con 0x7f8f20100a50 2023-10-11T16:39:34.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f397ceb80 1 Processor -- start 2023-10-11T16:39:34.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f397ceb80 1 -- start start 2023-10-11T16:39:34.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f1cff9700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8f080a12c0 0x7f8f080a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868af4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af4880 0x557868af8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868af9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f1cff9700 1 -- 172.21.15.111:0/3435250800 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f8f10014070 con 0x7f8f20100a50 2023-10-11T16:39:34.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578689f5060 con 0x557868af8cc0 2023-10-11T16:39:34.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557868a478a0 con 0x557868af4880 2023-10-11T16:39:34.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557868a40d30 con 0x557868af3f70 2023-10-11T16:39:34.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8f080a12c0 0x7f8f080a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.319+0000 7f8f1ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868af9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1effd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af4880 0x557868af8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868af9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47248/0 (socket says 172.21.15.111:47248) 2023-10-11T16:39:34.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1ffff700 1 -- 172.21.15.111:0/1142999285 learned_addr learned my addr 172.21.15.111:0/1142999285 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868af4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1effd700 1 -- 172.21.15.111:0/1142999285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 msgr2=0x557868af4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1effd700 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868af4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1effd700 1 -- 172.21.15.111:0/1142999285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 msgr2=0x557868af9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1effd700 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868af9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1effd700 1 -- 172.21.15.111:0/1142999285 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578688c2630 con 0x557868af4880 2023-10-11T16:39:34.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868af4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:34.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868af9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8f080a12c0 0x7f8f080a3770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8f18012ef0 tx=0x7f8f180124f0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f1effd700 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af4880 0x557868af8780 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009230 tx=0x7f8f14009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69a7821b4201216c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.320+0000 7f8f05ffb700 1 -- 172.21.15.111:0/1142999285 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1401c070 con 0x557868af4880 2023-10-11T16:39:34.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f05ffb700 1 -- 172.21.15.111:0/1142999285 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f1400f9a0 con 0x557868af4880 2023-10-11T16:39:34.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f05ffb700 1 -- 172.21.15.111:0/1142999285 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f14005370 con 0x557868af4880 2023-10-11T16:39:34.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1142999285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af4880 msgr2=0x557868af8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af4880 0x557868af8780 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009230 tx=0x7f8f14009460 comp rx=0 tx=0).stop 2023-10-11T16:39:34.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1142999285 shutdown_connections 2023-10-11T16:39:34.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868af9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868af4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1142999285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af4880 0x557868af8780 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1142999285 >> 172.21.15.111:0/1142999285 conn(0x557868ae5360 msgr2=0x557868aeab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.321+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1142999285 shutdown_connections 2023-10-11T16:39:34.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.322+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1142999285 wait complete. 2023-10-11T16:39:34.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.322+0000 7f8f397ceb80 1 -- start start 2023-10-11T16:39:34.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.322+0000 7f8f397ceb80 1 Processor -- start 2023-10-11T16:39:34.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af3f70 0x557868b00490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af4880 0x557868afd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868afdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578689e28e0 con 0x557868af8cc0 2023-10-11T16:39:34.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578689e2580 con 0x557868af3f70 2023-10-11T16:39:34.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578688c4920 con 0x557868af4880 2023-10-11T16:39:34.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f397ceb80 10 client.0 _ll_register_callbacks cb 0x557868a00780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:34.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868afdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868afdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47252/0 (socket says 172.21.15.111:47252) 2023-10-11T16:39:34.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1ffff700 1 -- 172.21.15.111:0/1073542037 learned_addr learned my addr 172.21.15.111:0/1073542037 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1effd700 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af4880 0x557868afd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af3f70 0x557868b00490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1ffff700 1 -- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af4880 msgr2=0x557868afd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af4880 0x557868afd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1ffff700 1 -- 172.21.15.111:0/1073542037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af3f70 msgr2=0x557868b00490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af3f70 0x557868b00490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1ffff700 1 -- 172.21.15.111:0/1073542037 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f2010da50 con 0x557868af8cc0 2023-10-11T16:39:34.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.323+0000 7f8f1effd700 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af4880 0x557868afd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868afdde0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8f180122f0 tx=0x7f8f1800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cb335f480816b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8efeffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:34.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8efeffd700 1 -- 172.21.15.111:0/1073542037 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f18029070 con 0x557868af8cc0 2023-10-11T16:39:34.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8efeffd700 1 -- 172.21.15.111:0/1073542037 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f8f18020040 con 0x557868af8cc0 2023-10-11T16:39:34.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1073542037 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557868ae2b20 con 0x557868af8cc0 2023-10-11T16:39:34.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8efeffd700 1 -- 172.21.15.111:0/1073542037 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f1800e040 con 0x557868af8cc0 2023-10-11T16:39:34.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8f397ceb80 10 client.16602 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:34.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1073542037 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557868a07190 con 0x557868af8cc0 2023-10-11T16:39:34.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.324+0000 7f8efeffd700 1 -- 172.21.15.111:0/1073542037 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557868ae2b20 con 0x557868af8cc0 2023-10-11T16:39:34.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.325+0000 7f8efeffd700 1 -- 172.21.15.111:0/1073542037 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8f1802c020 con 0x557868af8cc0 2023-10-11T16:39:34.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.325+0000 7f8f397ceb80 10 client.16602 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:34.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.325+0000 7f8f397ceb80 10 client.16602 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:39:34.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.325+0000 7f8f397ceb80 20 client.16602 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:34.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.325+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x557868b0dd10 0x557868b10120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.325+0000 7f8f397ceb80 4 client.16602 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:34.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.325+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1073542037 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557868a030a0 con 0x557868b0dd10 2023-10-11T16:39:34.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.325+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x557868b0dd10 0x557868b10120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.326+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x557868b0dd10 0x557868b10120 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad2dc0b678d9d715 server_cookie=746814d4e8c0c438 in_seq=0 out_seq=0 2023-10-11T16:39:34.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.326+0000 7f8efeffd700 10 client.16602 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:39:34.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.326+0000 7f8efeffd700 1 -- 172.21.15.111:0/1073542037 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557868a030a0 con 0x557868b0dd10 2023-10-11T16:39:34.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.326+0000 7f8efeffd700 10 client.16602 handle_command_reply: tid=0 2023-10-11T16:39:34.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.326+0000 7f8f397ceb80 1 client.16602 shutdown 2023-10-11T16:39:34.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1073542037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 msgr2=0x557868afdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868afdde0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8f180122f0 tx=0x7f8f1800b040 comp rx=0 tx=0).stop 2023-10-11T16:39:34.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1073542037 shutdown_connections 2023-10-11T16:39:34.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x557868b0dd10 0x557868b10120 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af8cc0 0x557868afdde0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af3f70 0x557868b00490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1073542037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af4880 0x557868afd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8efeffd700 0 client.16602 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:39:34.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1073542037 >> 172.21.15.111:0/1073542037 conn(0x557868ae5360 msgr2=0x557868afa170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1073542037 shutdown_connections 2023-10-11T16:39:34.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1073542037 wait complete. 2023-10-11T16:39:34.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.327+0000 7f8f397ceb80 20 client.16602 trim_cache size 0 max 16384 2023-10-11T16:39:34.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.340+0000 7f8f397ceb80 1 Processor -- start 2023-10-11T16:39:34.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.340+0000 7f8f397ceb80 1 -- start start 2023-10-11T16:39:34.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b09fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868afffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868aea560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557868af4fd0 con 0x557868af4880 2023-10-11T16:39:34.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557868ae29f0 con 0x557868af8cc0 2023-10-11T16:39:34.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557868af7870 con 0x557868af3f70 2023-10-11T16:39:34.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868afffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b09fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868aea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868afffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47266/0 (socket says 172.21.15.111:47266) 2023-10-11T16:39:34.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1effd700 1 -- 172.21.15.111:0/2102967763 learned_addr learned my addr 172.21.15.111:0/2102967763 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/2102967763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 msgr2=0x557868aea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868aea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/2102967763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 msgr2=0x557868afffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868afffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1f7fe700 1 -- 172.21.15.111:0/2102967763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557868afd210 con 0x557868af3f70 2023-10-11T16:39:34.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1effd700 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868afffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.341+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868aea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b09fd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8f100caa40 tx=0x7f8f10085040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d227aafc0bf9c8e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f067fc700 1 -- 172.21.15.111:0/2102967763 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1009a3c0 con 0x557868af3f70 2023-10-11T16:39:34.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f067fc700 1 -- 172.21.15.111:0/2102967763 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f1009e3f0 con 0x557868af3f70 2023-10-11T16:39:34.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f067fc700 1 -- 172.21.15.111:0/2102967763 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10096c20 con 0x557868af3f70 2023-10-11T16:39:34.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f397ceb80 1 -- 172.21.15.111:0/2102967763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 msgr2=0x557868b09fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b09fd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8f100caa40 tx=0x7f8f10085040 comp rx=0 tx=0).stop 2023-10-11T16:39:34.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f397ceb80 1 -- 172.21.15.111:0/2102967763 shutdown_connections 2023-10-11T16:39:34.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868aea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b09fd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/2102967763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868afffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.342+0000 7f8f397ceb80 1 -- 172.21.15.111:0/2102967763 >> 172.21.15.111:0/2102967763 conn(0x557868ae5360 msgr2=0x557868b14a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.343+0000 7f8f397ceb80 1 -- 172.21.15.111:0/2102967763 shutdown_connections 2023-10-11T16:39:34.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.343+0000 7f8f397ceb80 1 -- 172.21.15.111:0/2102967763 wait complete. 2023-10-11T16:39:34.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.343+0000 7f8f397ceb80 1 -- start start 2023-10-11T16:39:34.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.343+0000 7f8f397ceb80 1 Processor -- start 2023-10-11T16:39:34.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b02d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868b03280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f397ceb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868b037c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557868b60cb0 con 0x557868af4880 2023-10-11T16:39:34.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557868b60e20 con 0x557868af8cc0 2023-10-11T16:39:34.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f397ceb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557868b60fe0 con 0x557868af3f70 2023-10-11T16:39:34.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f397ceb80 10 client.0 _ll_register_callbacks cb 0x7f8f18009350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:34.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f1effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868b03280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f1effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868b03280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47274/0 (socket says 172.21.15.111:47274) 2023-10-11T16:39:34.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f1ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868b037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f1effd700 1 -- 172.21.15.111:0/1751755536 learned_addr learned my addr 172.21.15.111:0/1751755536 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.344+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b02d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f1effd700 1 -- 172.21.15.111:0/1751755536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 msgr2=0x557868b02d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f1effd700 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b02d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f1effd700 1 -- 172.21.15.111:0/1751755536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 msgr2=0x557868b037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f1effd700 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868b037c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f1effd700 1 -- 172.21.15.111:0/1751755536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557868a07190 con 0x557868af4880 2023-10-11T16:39:34.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f1ffff700 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868b037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f1effd700 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868b03280 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8f140098c0 tx=0x7f8f1400bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4808ee87686805 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8efeffd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:34.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8efeffd700 1 -- 172.21.15.111:0/1751755536 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1402d070 con 0x557868af4880 2023-10-11T16:39:34.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1751755536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f1400f9a0 con 0x557868af4880 2023-10-11T16:39:34.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8efeffd700 1 -- 172.21.15.111:0/1751755536 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f8f14005250 con 0x557868af4880 2023-10-11T16:39:34.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8efeffd700 1 -- 172.21.15.111:0/1751755536 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f1401c040 con 0x557868af4880 2023-10-11T16:39:34.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f397ceb80 10 client.16608 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:34.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.345+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1751755536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f8f20110410 con 0x557868af4880 2023-10-11T16:39:34.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8efeffd700 1 -- 172.21.15.111:0/1751755536 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f1400f9a0 con 0x557868af4880 2023-10-11T16:39:34.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8efeffd700 1 -- 172.21.15.111:0/1751755536 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8f14030020 con 0x557868af4880 2023-10-11T16:39:34.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8f397ceb80 10 client.16608 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:34.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8f397ceb80 10 client.16608 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:39:34.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8f397ceb80 20 client.16608 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:34.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x557868b62360 0x557868b64770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8f397ceb80 4 client.16608 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:39:34.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1751755536 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557868a0f660 con 0x557868b62360 2023-10-11T16:39:34.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.346+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x557868b62360 0x557868b64770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8efeffd700 10 client.16608 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:39:34.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f1f7fe700 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x557868b62360 0x557868b64770 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a6614067dc20de server_cookie=a83c0e47dcad70ec in_seq=0 out_seq=0 2023-10-11T16:39:34.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8efeffd700 1 -- 172.21.15.111:0/1751755536 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557868a0f660 con 0x557868b62360 2023-10-11T16:39:34.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8efeffd700 10 client.16608 handle_command_reply: tid=0 2023-10-11T16:39:34.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 client.16608 shutdown 2023-10-11T16:39:34.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1751755536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 msgr2=0x557868b03280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868b03280 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8f140098c0 tx=0x7f8f1400bb50 comp rx=0 tx=0).stop 2023-10-11T16:39:34.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1751755536 shutdown_connections 2023-10-11T16:39:34.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x557868b62360 0x557868b64770 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557868af8cc0 0x557868b037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557868af3f70 0x557868b02d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 --2- 172.21.15.111:0/1751755536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557868af4880 0x557868b03280 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8efeffd700 0 client.16608 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:39:34.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1751755536 >> 172.21.15.111:0/1751755536 conn(0x557868ae5360 msgr2=0x557868af6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1751755536 shutdown_connections 2023-10-11T16:39:34.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.347+0000 7f8f397ceb80 1 -- 172.21.15.111:0/1751755536 wait complete. 2023-10-11T16:39:34.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f397ceb80 20 client.16608 trim_cache size 0 max 16384 2023-10-11T16:39:34.395 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:39:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 -- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8f080a12c0 msgr2=0x7f8f080a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8f080a12c0 0x7f8f080a3770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8f18012ef0 tx=0x7f8f180124f0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 -- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 msgr2=0x7f8f20196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 0x7f8f20196500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000ec10 tx=0x7f8f1000c2d0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 -- 172.21.15.111:0/3435250800 shutdown_connections 2023-10-11T16:39:34.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8f080a12c0 0x7f8f080a3770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f201063f0 0x7f8f2019aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f20100a50 0x7f8f20196500 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 --2- 172.21.15.111:0/3435250800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f20101400 0x7f8f20196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 -- 172.21.15.111:0/3435250800 >> 172.21.15.111:0/3435250800 conn(0x7f8f20077590 msgr2=0x7f8f200ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 -- 172.21.15.111:0/3435250800 shutdown_connections 2023-10-11T16:39:34.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.348+0000 7f8f25dff700 1 -- 172.21.15.111:0/3435250800 wait complete. 2023-10-11T16:39:34.400 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:39:34.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 -- 172.21.15.111:0/2904089780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 msgr2=0x7f4b701067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 --2- 172.21.15.111:0/2904089780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 0x7f4b701067c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009990 tx=0x7f4b64009ca0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 -- 172.21.15.111:0/2904089780 shutdown_connections 2023-10-11T16:39:34.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 --2- 172.21.15.111:0/2904089780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b70101400 0x7f4b7010d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 --2- 172.21.15.111:0/2904089780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 0x7f4b701067c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 --2- 172.21.15.111:0/2904089780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b70100a50 0x7f4b70100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 -- 172.21.15.111:0/2904089780 >> 172.21.15.111:0/2904089780 conn(0x7f4b70077590 msgr2=0x7f4b70077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 -- 172.21.15.111:0/2904089780 shutdown_connections 2023-10-11T16:39:34.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.601+0000 7f4b76fff700 1 -- 172.21.15.111:0/2904089780 wait complete. 2023-10-11T16:39:34.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.602+0000 7f4b76fff700 1 Processor -- start 2023-10-11T16:39:34.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.602+0000 7f4b76fff700 1 -- start start 2023-10-11T16:39:34.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.602+0000 7f4b76fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b70100a50 0x7f4b70198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.602+0000 7f4b76fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b70101400 0x7f4b70198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.602+0000 7f4b76fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 0x7f4b7019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.602+0000 7f4b76fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7019d620 con 0x7f4b701063f0 2023-10-11T16:39:34.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.602+0000 7f4b76fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7019d790 con 0x7f4b70101400 2023-10-11T16:39:34.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.602+0000 7f4b76fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7019d950 con 0x7f4b70100a50 2023-10-11T16:39:34.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b70100a50 0x7f4b70198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b70100a50 0x7f4b70198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41716/0 (socket says 172.21.15.111:41716) 2023-10-11T16:39:34.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b6ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b70101400 0x7f4b70198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 -- 172.21.15.111:0/690907022 learned_addr learned my addr 172.21.15.111:0/690907022 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b7559c700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 0x7f4b7019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 -- 172.21.15.111:0/690907022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b70101400 msgr2=0x7f4b70198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b70101400 0x7f4b70198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 -- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 msgr2=0x7f4b7019cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 0x7f4b7019cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 -- 172.21.15.111:0/690907022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b640095f0 con 0x7f4b70100a50 2023-10-11T16:39:34.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b70101400 0x7f4b70198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b7559c700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 0x7f4b7019cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.603+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b70100a50 0x7f4b70198640 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4b64005910 tx=0x7f4b64004710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.607+0000 7f4b6dffb700 1 -- 172.21.15.111:0/690907022 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6401d070 con 0x7f4b70100a50 2023-10-11T16:39:34.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.607+0000 7f4b6dffb700 1 -- 172.21.15.111:0/690907022 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b64022470 con 0x7f4b70100a50 2023-10-11T16:39:34.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.607+0000 7f4b6dffb700 1 -- 172.21.15.111:0/690907022 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64017650 con 0x7f4b70100a50 2023-10-11T16:39:34.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.607+0000 7f4b76fff700 1 -- 172.21.15.111:0/690907022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b701103d0 con 0x7f4b70100a50 2023-10-11T16:39:34.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.607+0000 7f4b76fff700 1 -- 172.21.15.111:0/690907022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b70110240 con 0x7f4b70100a50 2023-10-11T16:39:34.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.610+0000 7f4b6dffb700 1 -- 172.21.15.111:0/690907022 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f4b640225e0 con 0x7f4b70100a50 2023-10-11T16:39:34.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.611+0000 7f4b6dffb700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4b600a1340 0x7f4b600a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b8a9e3b80 1 Processor -- start 2023-10-11T16:39:34.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b6dffb700 1 -- 172.21.15.111:0/690907022 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4b640ce2b0 con 0x7f4b70100a50 2023-10-11T16:39:34.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b8a9e3b80 1 -- start start 2023-10-11T16:39:34.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4b600a1340 0x7f4b600a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07abb60 0x555ea07abf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07b08b0 0x555ea07b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea06b1060 con 0x555ea07ac470 2023-10-11T16:39:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea07038a0 con 0x555ea07abb60 2023-10-11T16:39:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.612+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea06fcd30 con 0x555ea07b08b0 2023-10-11T16:39:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47304/0 (socket says 172.21.15.111:47304) 2023-10-11T16:39:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 -- 172.21.15.111:0/808591305 learned_addr learned my addr 172.21.15.111:0/808591305 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07abb60 0x555ea07abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4b600a1340 0x7f4b600a37f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4b70199c50 tx=0x7f4b5c009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b7559c700 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07b08b0 0x555ea07b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 -- 172.21.15.111:0/808591305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07b08b0 msgr2=0x555ea07b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07b08b0 0x555ea07b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 -- 172.21.15.111:0/808591305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07abb60 msgr2=0x555ea07abf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07abb60 0x555ea07abf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 -- 172.21.15.111:0/808591305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ea057e630 con 0x555ea07ac470 2023-10-11T16:39:34.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07abb60 0x555ea07abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b0370 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c004380 tx=0x7f4b5c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd93bbf185e44bab server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.613+0000 7f4b5affd700 1 -- 172.21.15.111:0/808591305 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c003c90 con 0x555ea07ac470 2023-10-11T16:39:34.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b5affd700 1 -- 172.21.15.111:0/808591305 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b5c003df0 con 0x555ea07ac470 2023-10-11T16:39:34.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b5affd700 1 -- 172.21.15.111:0/808591305 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c00e770 con 0x555ea07ac470 2023-10-11T16:39:34.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/808591305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 msgr2=0x555ea07b0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b0370 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c004380 tx=0x7f4b5c014000 comp rx=0 tx=0).stop 2023-10-11T16:39:34.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/808591305 shutdown_connections 2023-10-11T16:39:34.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07b08b0 0x555ea07b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07abb60 0x555ea07abf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/808591305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b0370 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/808591305 >> 172.21.15.111:0/808591305 conn(0x555ea07a1360 msgr2=0x555ea07a6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/808591305 shutdown_connections 2023-10-11T16:39:34.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.614+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/808591305 wait complete. 2023-10-11T16:39:34.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.615+0000 7f4b8a9e3b80 1 -- start start 2023-10-11T16:39:34.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.615+0000 7f4b8a9e3b80 1 Processor -- start 2023-10-11T16:39:34.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea07b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea069e8e0 con 0x555ea07abb60 2023-10-11T16:39:34.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea069e580 con 0x555ea07b08b0 2023-10-11T16:39:34.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea0580920 con 0x555ea07ac470 2023-10-11T16:39:34.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b8a9e3b80 10 client.0 _ll_register_callbacks cb 0x555ea06bc780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:34.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b74d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b74d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07b80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47310/0 (socket says 172.21.15.111:47310) 2023-10-11T16:39:34.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.616+0000 7f4b74d9b700 1 -- 172.21.15.111:0/1681527074 learned_addr learned my addr 172.21.15.111:0/1681527074 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b7559c700 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea07b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b74d9b700 1 -- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 msgr2=0x555ea07b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b74d9b700 1 -- 172.21.15.111:0/1681527074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 msgr2=0x555ea07b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea07b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b74d9b700 1 -- 172.21.15.111:0/1681527074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c017070 con 0x555ea07abb60 2023-10-11T16:39:34.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:34.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07b80a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4b640cb650 tx=0x7f4b6409a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a37a6b32e647758 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b4ffff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:34.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1681527074 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640b2070 con 0x555ea07abb60 2023-10-11T16:39:34.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1681527074 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4b6409f030 con 0x555ea07abb60 2023-10-11T16:39:34.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1681527074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ea079eb20 con 0x555ea07abb60 2023-10-11T16:39:34.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1681527074 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b640a33c0 con 0x555ea07abb60 2023-10-11T16:39:34.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b8a9e3b80 10 client.16620 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:34.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.617+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1681527074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555ea06c3190 con 0x555ea07abb60 2023-10-11T16:39:34.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1681527074 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b640c9080 con 0x555ea07abb60 2023-10-11T16:39:34.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1681527074 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f4b640a2f30 con 0x555ea07abb60 2023-10-11T16:39:34.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b8a9e3b80 10 client.16620 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:34.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b8a9e3b80 10 client.16620 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:39:34.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b8a9e3b80 20 client.16620 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:34.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x555ea07c5400 0x555ea07c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b8a9e3b80 4 client.16620 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:34.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1681527074 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ea06bf0a0 con 0x555ea07c5400 2023-10-11T16:39:34.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.618+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x555ea07c5400 0x555ea07c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.619+0000 7f4b4ffff700 10 client.16620 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:39:34.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.619+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x555ea07c5400 0x555ea07c77e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ad0f3db8d300412f server_cookie=226b431ad6589bc3 in_seq=0 out_seq=0 2023-10-11T16:39:34.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.619+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1681527074 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555ea06bf0a0 con 0x555ea07c5400 2023-10-11T16:39:34.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b4ffff700 10 client.16620 handle_command_reply: tid=0 2023-10-11T16:39:34.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 client.16620 shutdown 2023-10-11T16:39:34.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1681527074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 msgr2=0x555ea07b80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07b80a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4b640cb650 tx=0x7f4b6409a960 comp rx=0 tx=0).stop 2023-10-11T16:39:34.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1681527074 shutdown_connections 2023-10-11T16:39:34.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x555ea07c5400 0x555ea07c77e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea07b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07b80a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b4ffff700 0 client.16620 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:39:34.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1681527074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1681527074 >> 172.21.15.111:0/1681527074 conn(0x555ea07a1360 msgr2=0x555ea07a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1681527074 shutdown_connections 2023-10-11T16:39:34.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1681527074 wait complete. 2023-10-11T16:39:34.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.620+0000 7f4b8a9e3b80 20 client.16620 trim_cache size 0 max 16384 2023-10-11T16:39:34.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.633+0000 7f4b8a9e3b80 1 Processor -- start 2023-10-11T16:39:34.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b8a9e3b80 1 -- start start 2023-10-11T16:39:34.654 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:34 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3573234292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:39:34.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea07b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea06bf0a0 con 0x555ea07abb60 2023-10-11T16:39:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea079e750 con 0x555ea07b08b0 2023-10-11T16:39:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea07acbc0 con 0x555ea07ac470 2023-10-11T16:39:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b74d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b7559c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea07b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b74d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07bc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47320/0 (socket says 172.21.15.111:47320) 2023-10-11T16:39:34.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b74d9b700 1 -- 172.21.15.111:0/2138484727 learned_addr learned my addr 172.21.15.111:0/2138484727 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/2138484727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.634+0000 7f4b74d9b700 1 -- 172.21.15.111:0/2138484727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 msgr2=0x555ea07b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.635+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/2138484727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.635+0000 7f4b74d9b700 1 -- 172.21.15.111:0/2138484727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 msgr2=0x555ea07b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.635+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/2138484727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea07b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.635+0000 7f4b74d9b700 1 -- 172.21.15.111:0/2138484727 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ea07b0f50 con 0x555ea07abb60 2023-10-11T16:39:34.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.635+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/2138484727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07bc9a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4b640a2e90 tx=0x7f4b64000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701dc9d526e611d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.635+0000 7f4b5b7fe700 1 -- 172.21.15.111:0/2138484727 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640b2020 con 0x555ea07abb60 2023-10-11T16:39:34.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.635+0000 7f4b5b7fe700 1 -- 172.21.15.111:0/2138484727 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555ea079eb20 con 0x555ea07abb60 2023-10-11T16:39:34.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.635+0000 7f4b5b7fe700 1 -- 172.21.15.111:0/2138484727 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6409a4f0 con 0x555ea07abb60 2023-10-11T16:39:34.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/2138484727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 msgr2=0x555ea07bc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/2138484727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07bc9a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4b640a2e90 tx=0x7f4b64000bc0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/2138484727 shutdown_connections 2023-10-11T16:39:34.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/2138484727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea07b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/2138484727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07abb60 0x555ea07bc9a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/2138484727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07ac470 0x555ea07b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/2138484727 >> 172.21.15.111:0/2138484727 conn(0x555ea07a1360 msgr2=0x555ea07c1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/2138484727 shutdown_connections 2023-10-11T16:39:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/2138484727 wait complete. 2023-10-11T16:39:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 -- start start 2023-10-11T16:39:34.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.636+0000 7f4b8a9e3b80 1 Processor -- start 2023-10-11T16:39:34.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07abb60 0x555ea07b8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b8a9e3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea0818280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea07b9630 con 0x555ea07ac470 2023-10-11T16:39:34.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea07b97a0 con 0x555ea07b08b0 2023-10-11T16:39:34.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b8a9e3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea08187c0 con 0x555ea07abb60 2023-10-11T16:39:34.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b8a9e3b80 10 client.0 _ll_register_callbacks cb 0x7f4b64097340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:34.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b74d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07abb60 0x555ea07b8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b6ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b74d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07abb60 0x555ea07b8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:41754/0 (socket says 172.21.15.111:41754) 2023-10-11T16:39:34.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b6ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47328/0 (socket says 172.21.15.111:47328) 2023-10-11T16:39:34.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b74d9b700 1 -- 172.21.15.111:0/1284065201 learned_addr learned my addr 172.21.15.111:0/1284065201 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b7559c700 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea0818280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b6ffff700 1 -- 172.21.15.111:0/1284065201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07abb60 msgr2=0x555ea07b8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07abb60 0x555ea07b8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b6ffff700 1 -- 172.21.15.111:0/1284065201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 msgr2=0x555ea0818280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea0818280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.637+0000 7f4b6ffff700 1 -- 172.21.15.111:0/1284065201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ea06c3190 con 0x555ea07ac470 2023-10-11T16:39:34.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b6ffff700 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b8ec0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009750 tx=0x7f4b5c01edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e548e3c113a1561d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b4ffff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:34.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1284065201 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c02b050 con 0x555ea07ac470 2023-10-11T16:39:34.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1284065201 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4b5c032960 con 0x555ea07ac470 2023-10-11T16:39:34.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1284065201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b5c003df0 con 0x555ea07ac470 2023-10-11T16:39:34.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1284065201 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b5c003a10 con 0x555ea07ac470 2023-10-11T16:39:34.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b8a9e3b80 10 client.16626 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:34.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1284065201 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555ea052ade0 con 0x555ea07ac470 2023-10-11T16:39:34.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.638+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1284065201 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b5c003df0 con 0x555ea07ac470 2023-10-11T16:39:34.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1284065201 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f4b5c024020 con 0x555ea07ac470 2023-10-11T16:39:34.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b8a9e3b80 10 client.16626 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:34.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b8a9e3b80 10 client.16626 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:39:34.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b8a9e3b80 20 client.16626 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:34.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x555ea0819f40 0x555ea081c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b8a9e3b80 4 client.16626 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:39:34.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1284065201 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555ea06cb660 con 0x555ea0819f40 2023-10-11T16:39:34.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x555ea0819f40 0x555ea081c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b74d9b700 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x555ea0819f40 0x555ea081c350 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ac0b887da245300 server_cookie=20acb277d13f61db in_seq=0 out_seq=0 2023-10-11T16:39:34.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.639+0000 7f4b4ffff700 10 client.16626 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:39:34.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b4ffff700 1 -- 172.21.15.111:0/1284065201 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555ea06cb660 con 0x555ea0819f40 2023-10-11T16:39:34.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b4ffff700 10 client.16626 handle_command_reply: tid=0 2023-10-11T16:39:34.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 client.16626 shutdown 2023-10-11T16:39:34.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1284065201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 msgr2=0x555ea07b8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b8ec0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009750 tx=0x7f4b5c01edc0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1284065201 shutdown_connections 2023-10-11T16:39:34.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x555ea0819f40 0x555ea081c350 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ea07b08b0 0x555ea0818280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b4ffff700 0 client.16626 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:39:34.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555ea07abb60 0x555ea07b8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 --2- 172.21.15.111:0/1284065201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555ea07ac470 0x555ea07b8ec0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1284065201 >> 172.21.15.111:0/1284065201 conn(0x555ea07a1360 msgr2=0x555ea07aa960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1284065201 shutdown_connections 2023-10-11T16:39:34.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 1 -- 172.21.15.111:0/1284065201 wait complete. 2023-10-11T16:39:34.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.640+0000 7f4b8a9e3b80 20 client.16626 trim_cache size 0 max 16384 2023-10-11T16:39:34.683 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:39:34.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 -- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4b600a1340 msgr2=0x7f4b600a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4b600a1340 0x7f4b600a37f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4b70199c50 tx=0x7f4b5c009260 comp rx=0 tx=0).stop 2023-10-11T16:39:34.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 -- 172.21.15.111:0/690907022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b70100a50 msgr2=0x7f4b70198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b70100a50 0x7f4b70198640 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4b64005910 tx=0x7f4b64004710 comp rx=0 tx=0).stop 2023-10-11T16:39:34.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 -- 172.21.15.111:0/690907022 shutdown_connections 2023-10-11T16:39:34.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4b600a1340 0x7f4b600a37f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b701063f0 0x7f4b7019cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b70100a50 0x7f4b70198640 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 --2- 172.21.15.111:0/690907022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b70101400 0x7f4b70198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 -- 172.21.15.111:0/690907022 >> 172.21.15.111:0/690907022 conn(0x7f4b70077590 msgr2=0x7f4b7010b750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 -- 172.21.15.111:0/690907022 shutdown_connections 2023-10-11T16:39:34.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.641+0000 7f4b76fff700 1 -- 172.21.15.111:0/690907022 wait complete. 2023-10-11T16:39:34.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:39:34.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 -- 172.21.15.111:0/1712137495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 msgr2=0x7fb64410d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/1712137495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 0x7fb64410d1a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb63c00b0d0 tx=0x7fb63c00b3e0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 -- 172.21.15.111:0/1712137495 shutdown_connections 2023-10-11T16:39:34.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/1712137495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 0x7fb64410d1a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/1712137495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb644100cc0 0x7fb644101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/1712137495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644101660 0x7fb644104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 -- 172.21.15.111:0/1712137495 >> 172.21.15.111:0/1712137495 conn(0x7fb644077560 msgr2=0x7fb644077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 -- 172.21.15.111:0/1712137495 shutdown_connections 2023-10-11T16:39:34.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.888+0000 7fb64a9ff700 1 -- 172.21.15.111:0/1712137495 wait complete. 2023-10-11T16:39:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.889+0000 7fb64a9ff700 1 Processor -- start 2023-10-11T16:39:34.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.889+0000 7fb64a9ff700 1 -- start start 2023-10-11T16:39:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb64a9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb644100cc0 0x7fb644072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb64a9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644101660 0x7fb644073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb64a9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 0x7fb6441a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb64a9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb644073b60 con 0x7fb644105620 2023-10-11T16:39:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb64a9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6441a4d50 con 0x7fb644100cc0 2023-10-11T16:39:34.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb64a9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6441a4e90 con 0x7fb644101660 2023-10-11T16:39:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644101660 0x7fb644073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb648f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 0x7fb6441a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644101660 0x7fb644073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56080/0 (socket says 172.21.15.111:56080) 2023-10-11T16:39:34.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb63bfff700 1 -- 172.21.15.111:0/38740958 learned_addr learned my addr 172.21.15.111:0/38740958 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb643fff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb644100cc0 0x7fb644072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb648f9c700 1 -- 172.21.15.111:0/38740958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644101660 msgr2=0x7fb644073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb648f9c700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644101660 0x7fb644073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb648f9c700 1 -- 172.21.15.111:0/38740958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb644100cc0 msgr2=0x7fb644072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb648f9c700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb644100cc0 0x7fb644072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb648f9c700 1 -- 172.21.15.111:0/38740958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c009d00 con 0x7fb644105620 2023-10-11T16:39:34.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.890+0000 7fb648f9c700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 0x7fb6441a4810 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009fd0 tx=0x7fb63c0077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.891+0000 7fb641ffb700 1 -- 172.21.15.111:0/38740958 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c00e040 con 0x7fb644105620 2023-10-11T16:39:34.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.891+0000 7fb641ffb700 1 -- 172.21.15.111:0/38740958 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb63c00bcb0 con 0x7fb644105620 2023-10-11T16:39:34.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.891+0000 7fb641ffb700 1 -- 172.21.15.111:0/38740958 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c005840 con 0x7fb644105620 2023-10-11T16:39:34.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.891+0000 7fb64a9ff700 1 -- 172.21.15.111:0/38740958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6441103e0 con 0x7fb644105620 2023-10-11T16:39:34.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.891+0000 7fb64a9ff700 1 -- 172.21.15.111:0/38740958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb644110250 con 0x7fb644105620 2023-10-11T16:39:34.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.894+0000 7fb641ffb700 1 -- 172.21.15.111:0/38740958 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fb63c004a50 con 0x7fb644105620 2023-10-11T16:39:34.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.895+0000 7fb641ffb700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6300a1340 0x7fb6300a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.895+0000 7fb641ffb700 1 -- 172.21.15.111:0/38740958 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fb63c0ce1b0 con 0x7fb644105620 2023-10-11T16:39:34.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.895+0000 7fb65e2b6b80 1 Processor -- start 2023-10-11T16:39:34.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.895+0000 7fb643fff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6300a1340 0x7fb6300a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb65e2b6b80 1 -- start start 2023-10-11T16:39:34.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a70f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a75370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a758b0 0x557811a75d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811976060 con 0x557811a758b0 2023-10-11T16:39:34.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578119c88a0 con 0x557811a70b60 2023-10-11T16:39:34.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578119c1d30 con 0x557811a71470 2023-10-11T16:39:34.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb643fff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6300a1340 0x7fb6300a37f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb634009c60 tx=0x7fb634009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb648f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a758b0 0x557811a75d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a75370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a75370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56082/0 (socket says 172.21.15.111:56082) 2023-10-11T16:39:34.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb648f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a758b0 0x557811a75d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34176/0 (socket says 172.21.15.111:34176) 2023-10-11T16:39:34.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb63bfff700 1 -- 172.21.15.111:0/1068406741 learned_addr learned my addr 172.21.15.111:0/1068406741 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.896+0000 7fb643fff700 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a70f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63bfff700 1 -- 172.21.15.111:0/1068406741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 msgr2=0x557811a70f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63bfff700 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a70f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63bfff700 1 -- 172.21.15.111:0/1068406741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a758b0 msgr2=0x557811a75d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63bfff700 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a758b0 0x557811a75d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63bfff700 1 -- 172.21.15.111:0/1068406741 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557811843630 con 0x557811a71470 2023-10-11T16:39:34.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb643fff700 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a70f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:34.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63bfff700 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a75370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009d10 tx=0x7fb62c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0f40006e28f0e42 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63a7fc700 1 -- 172.21.15.111:0/1068406741 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c004220 con 0x557811a71470 2023-10-11T16:39:34.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63a7fc700 1 -- 172.21.15.111:0/1068406741 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb62c004380 con 0x557811a71470 2023-10-11T16:39:34.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.897+0000 7fb63a7fc700 1 -- 172.21.15.111:0/1068406741 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0197d0 con 0x557811a71470 2023-10-11T16:39:34.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1068406741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 msgr2=0x557811a75370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a75370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009d10 tx=0x7fb62c011720 comp rx=0 tx=0).stop 2023-10-11T16:39:34.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1068406741 shutdown_connections 2023-10-11T16:39:34.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a758b0 0x557811a75d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a70f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1068406741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a75370 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1068406741 >> 172.21.15.111:0/1068406741 conn(0x557811a66360 msgr2=0x557811a6bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1068406741 shutdown_connections 2023-10-11T16:39:34.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.898+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1068406741 wait complete. 2023-10-11T16:39:34.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 1 -- start start 2023-10-11T16:39:34.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 1 Processor -- start 2023-10-11T16:39:34.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a7d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a71470 0x557811a7a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a758b0 0x557811a7a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578119638e0 con 0x557811a71470 2023-10-11T16:39:34.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811963580 con 0x557811a70b60 2023-10-11T16:39:34.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811845920 con 0x557811a758b0 2023-10-11T16:39:34.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.899+0000 7fb65e2b6b80 10 client.0 _ll_register_callbacks cb 0x557811981780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:34.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a71470 0x557811a7a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb643fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a7d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a71470 0x557811a7a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34192/0 (socket says 172.21.15.111:34192) 2023-10-11T16:39:34.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb63bfff700 1 -- 172.21.15.111:0/504698360 learned_addr learned my addr 172.21.15.111:0/504698360 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb648f9c700 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a758b0 0x557811a7a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.923 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb643fff700 1 -- 172.21.15.111:0/504698360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a758b0 msgr2=0x557811a7a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb643fff700 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a758b0 0x557811a7a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb643fff700 1 -- 172.21.15.111:0/504698360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a71470 msgr2=0x557811a7a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb643fff700 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a71470 0x557811a7a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb643fff700 1 -- 172.21.15.111:0/504698360 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c009940 con 0x557811a70b60 2023-10-11T16:39:34.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb63bfff700 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a71470 0x557811a7a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb643fff700 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a7d0c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb634011ab0 tx=0x7fb6340047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c151d8e6c3a9fcea server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb6237fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:39:34.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.900+0000 7fb6237fe700 1 -- 172.21.15.111:0/504698360 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63400ec50 con 0x557811a70b60 2023-10-11T16:39:34.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/504698360 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557811a63b20 con 0x557811a70b60 2023-10-11T16:39:34.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb6237fe700 1 -- 172.21.15.111:0/504698360 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fb634009e50 con 0x557811a70b60 2023-10-11T16:39:34.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb6237fe700 1 -- 172.21.15.111:0/504698360 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb63400edb0 con 0x557811a70b60 2023-10-11T16:39:34.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 10 client.26753 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:34.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/504698360 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557811988190 con 0x557811a70b60 2023-10-11T16:39:34.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb6237fe700 1 -- 172.21.15.111:0/504698360 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557811a63b20 con 0x557811a70b60 2023-10-11T16:39:34.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb6237fe700 1 -- 172.21.15.111:0/504698360 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb634003c80 con 0x557811a70b60 2023-10-11T16:39:34.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 10 client.26753 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:34.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 10 client.26753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:34.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 20 client.26753 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:34.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557811a8a950 0x557811a8cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 4 client.26753 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:34.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.901+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/504698360 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578119840a0 con 0x557811a8a950 2023-10-11T16:39:34.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.902+0000 7fb63bfff700 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557811a8a950 0x557811a8cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.902+0000 7fb63bfff700 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557811a8a950 0x557811a8cd60 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67574bd48b0b5b2c server_cookie=5f2641899499f817 in_seq=0 out_seq=0 2023-10-11T16:39:34.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.902+0000 7fb6237fe700 10 client.26753 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:34.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb6237fe700 1 -- 172.21.15.111:0/504698360 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5578119840a0 con 0x557811a8a950 2023-10-11T16:39:34.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb6237fe700 10 client.26753 handle_command_reply: tid=0 2023-10-11T16:39:34.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 client.26753 shutdown 2023-10-11T16:39:34.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/504698360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 msgr2=0x557811a7d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a7d0c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb634011ab0 tx=0x7fb6340047c0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/504698360 shutdown_connections 2023-10-11T16:39:34.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557811a8a950 0x557811a8cd60 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a758b0 0x557811a7a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a70b60 0x557811a7d0c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/504698360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a71470 0x557811a7a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb6237fe700 0 client.26753 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:34.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/504698360 >> 172.21.15.111:0/504698360 conn(0x557811a66360 msgr2=0x557811a76d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/504698360 shutdown_connections 2023-10-11T16:39:34.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.903+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/504698360 wait complete. 2023-10-11T16:39:34.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.904+0000 7fb65e2b6b80 20 client.26753 trim_cache size 0 max 16384 2023-10-11T16:39:34.940 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.917+0000 7fb65e2b6b80 1 Processor -- start 2023-10-11T16:39:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.917+0000 7fb65e2b6b80 1 -- start start 2023-10-11T16:39:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.917+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a6a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.917+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a86150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.917+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811a7cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.917+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811a6aa90 con 0x557811a70b60 2023-10-11T16:39:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.917+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811a67830 con 0x557811a758b0 2023-10-11T16:39:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.917+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811a679f0 con 0x557811a71470 2023-10-11T16:39:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a6a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a6a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34202/0 (socket says 172.21.15.111:34202) 2023-10-11T16:39:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 -- 172.21.15.111:0/3556190136 learned_addr learned my addr 172.21.15.111:0/3556190136 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb63bfff700 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a86150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb648f9c700 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811a7cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 -- 172.21.15.111:0/3556190136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 msgr2=0x557811a86150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a86150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 -- 172.21.15.111:0/3556190136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 msgr2=0x557811a7cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811a7cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 -- 172.21.15.111:0/3556190136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557811987ee0 con 0x557811a70b60 2023-10-11T16:39:34.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb643fff700 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a6a550 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb63400f430 tx=0x7fb634009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8344abaa19b2df server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.918+0000 7fb63affd700 1 -- 172.21.15.111:0/3556190136 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634009ed0 con 0x557811a70b60 2023-10-11T16:39:34.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb63affd700 1 -- 172.21.15.111:0/3556190136 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6340265e0 con 0x557811a70b60 2023-10-11T16:39:34.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb63affd700 1 -- 172.21.15.111:0/3556190136 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634004a70 con 0x557811a70b60 2023-10-11T16:39:34.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/3556190136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 msgr2=0x557811a6a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a6a550 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb63400f430 tx=0x7fb634009990 comp rx=0 tx=0).stop 2023-10-11T16:39:34.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/3556190136 shutdown_connections 2023-10-11T16:39:34.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811a7cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a6a550 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/3556190136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a86150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/3556190136 >> 172.21.15.111:0/3556190136 conn(0x557811a66360 msgr2=0x557811a916c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/3556190136 shutdown_connections 2023-10-11T16:39:34.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.919+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/3556190136 wait complete. 2023-10-11T16:39:34.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 1 -- start start 2023-10-11T16:39:34.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 1 Processor -- start 2023-10-11T16:39:34.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a7f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a7fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811add810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811a80680 con 0x557811a70b60 2023-10-11T16:39:34.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811a807f0 con 0x557811a758b0 2023-10-11T16:39:34.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557811addd80 con 0x557811a71470 2023-10-11T16:39:34.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb65e2b6b80 10 client.0 _ll_register_callbacks cb 0x7fb63401e7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:34.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.920+0000 7fb643fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a7f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a7fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb648f9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811add810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a7fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56092/0 (socket says 172.21.15.111:56092) 2023-10-11T16:39:34.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb63bfff700 1 -- 172.21.15.111:0/1023614712 learned_addr learned my addr 172.21.15.111:0/1023614712 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:34.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb648f9c700 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811add810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40768/0 (socket says 172.21.15.111:40768) 2023-10-11T16:39:34.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb648f9c700 1 -- 172.21.15.111:0/1023614712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 msgr2=0x557811a7fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb648f9c700 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a7fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb648f9c700 1 -- 172.21.15.111:0/1023614712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 msgr2=0x557811a7f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:34.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb648f9c700 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a7f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb648f9c700 1 -- 172.21.15.111:0/1023614712 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557811988190 con 0x557811a758b0 2023-10-11T16:39:34.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb643fff700 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:34.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb648f9c700 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811add810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0cb490 tx=0x7fb63c0059a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf20a8cd95eebd server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb6237fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:39:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb6237fe700 1 -- 172.21.15.111:0/1023614712 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0a0040 con 0x557811a758b0 2023-10-11T16:39:34.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1023614712 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb62c004380 con 0x557811a758b0 2023-10-11T16:39:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.921+0000 7fb6237fe700 1 -- 172.21.15.111:0/1023614712 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fb63c0b2030 con 0x557811a758b0 2023-10-11T16:39:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb6237fe700 1 -- 172.21.15.111:0/1023614712 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb63c09a800 con 0x557811a758b0 2023-10-11T16:39:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb65e2b6b80 10 client.26759 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1023614712 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5578117d8c60 con 0x557811a758b0 2023-10-11T16:39:34.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb6237fe700 1 -- 172.21.15.111:0/1023614712 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb62c004380 con 0x557811a758b0 2023-10-11T16:39:34.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb6237fe700 1 -- 172.21.15.111:0/1023614712 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb63c0b90d0 con 0x557811a758b0 2023-10-11T16:39:34.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb65e2b6b80 10 client.26759 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:34.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb65e2b6b80 10 client.26759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:34.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb65e2b6b80 20 client.26759 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:34.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557811adf2b0 0x557811ae16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:34.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb65e2b6b80 4 client.26759 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:39:34.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1023614712 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557811990660 con 0x557811adf2b0 2023-10-11T16:39:34.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.922+0000 7fb643fff700 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557811adf2b0 0x557811ae16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:34.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb6237fe700 10 client.26759 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:34.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb643fff700 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557811adf2b0 0x557811ae16c0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c800e702469d809 server_cookie=7e70da004c2f0f96 in_seq=0 out_seq=0 2023-10-11T16:39:34.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb6237fe700 1 -- 172.21.15.111:0/1023614712 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557811990660 con 0x557811adf2b0 2023-10-11T16:39:34.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb6237fe700 10 client.26759 handle_command_reply: tid=0 2023-10-11T16:39:34.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 client.26759 shutdown 2023-10-11T16:39:34.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 msgr2=0x557811add810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811add810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0cb490 tx=0x7fb63c0059a0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1023614712 shutdown_connections 2023-10-11T16:39:34.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557811adf2b0 0x557811ae16c0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557811a758b0 0x557811add810 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557811a70b60 0x557811a7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 --2- 172.21.15.111:0/1023614712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557811a71470 0x557811a7fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb6237fe700 0 client.26759 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:34.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1023614712 >> 172.21.15.111:0/1023614712 conn(0x557811a66360 msgr2=0x557811a738f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1023614712 shutdown_connections 2023-10-11T16:39:34.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.923+0000 7fb65e2b6b80 1 -- 172.21.15.111:0/1023614712 wait complete. 2023-10-11T16:39:34.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb65e2b6b80 20 client.26759 trim_cache size 0 max 16384 2023-10-11T16:39:34.965 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:39:34.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 -- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6300a1340 msgr2=0x7fb6300a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6300a1340 0x7fb6300a37f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb634009c60 tx=0x7fb634009260 comp rx=0 tx=0).stop 2023-10-11T16:39:34.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 -- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 msgr2=0x7fb6441a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:34.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 0x7fb6441a4810 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009fd0 tx=0x7fb63c0077f0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 -- 172.21.15.111:0/38740958 shutdown_connections 2023-10-11T16:39:34.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb6300a1340 0x7fb6300a37f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb644105620 0x7fb6441a4810 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb644100cc0 0x7fb644072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 --2- 172.21.15.111:0/38740958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb644101660 0x7fb644073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 -- 172.21.15.111:0/38740958 >> 172.21.15.111:0/38740958 conn(0x7fb644077560 msgr2=0x7fb644103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 -- 172.21.15.111:0/38740958 shutdown_connections 2023-10-11T16:39:34.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:34.924+0000 7fb64a9ff700 1 -- 172.21.15.111:0/38740958 wait complete. 2023-10-11T16:39:34.970 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:34 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3573234292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:39:35.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:34 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3573234292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:39:35.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:35 smithi111 ceph-mon[98527]: pgmap v213: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:39:35.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:35 smithi150 ceph-mon[132217]: pgmap v213: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:39:35.971 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:39:35.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:39:36.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:35 smithi130 ceph-mon[131859]: pgmap v213: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:39:36.185 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.183+0000 7fd6f6dff700 1 -- 172.21.15.111:0/510495770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f0100a50 msgr2=0x7fd6f0100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.183+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/510495770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f0100a50 0x7fd6f0100ec0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009990 tx=0x7fd6dc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.183+0000 7fd6f6dff700 1 -- 172.21.15.111:0/510495770 shutdown_connections 2023-10-11T16:39:36.186 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.183+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/510495770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6f0101400 0x7fd6f010d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.183+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/510495770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6f01063f0 0x7fd6f01067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.183+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/510495770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f0100a50 0x7fd6f0100ec0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.187 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.184+0000 7fd6f6dff700 1 -- 172.21.15.111:0/510495770 >> 172.21.15.111:0/510495770 conn(0x7fd6f0077590 msgr2=0x7fd6f0077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.184+0000 7fd6f6dff700 1 -- 172.21.15.111:0/510495770 shutdown_connections 2023-10-11T16:39:36.188 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.184+0000 7fd6f6dff700 1 -- 172.21.15.111:0/510495770 wait complete. 2023-10-11T16:39:36.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f6dff700 1 Processor -- start 2023-10-11T16:39:36.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f6dff700 1 -- start start 2023-10-11T16:39:36.190 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f6dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6f0100a50 0x7fd6f0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f6dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f0101400 0x7fd6f0198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f6dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6f01063f0 0x7fd6f019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f6dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f019d5e0 con 0x7fd6f0101400 2023-10-11T16:39:36.191 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f6dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f019d750 con 0x7fd6f0100a50 2023-10-11T16:39:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f6dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f019d910 con 0x7fd6f01063f0 2023-10-11T16:39:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f4b9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6f0100a50 0x7fd6f0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f0101400 0x7fd6f0198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.192 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f4b9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6f0100a50 0x7fd6f0198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40786/0 (socket says 172.21.15.111:40786) 2023-10-11T16:39:36.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f4b9b700 1 -- 172.21.15.111:0/4181542353 learned_addr learned my addr 172.21.15.111:0/4181542353 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.185+0000 7fd6f539c700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6f01063f0 0x7fd6f019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.193 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6f4b9b700 1 -- 172.21.15.111:0/4181542353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6f01063f0 msgr2=0x7fd6f019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6f01063f0 0x7fd6f019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6f4b9b700 1 -- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f0101400 msgr2=0x7fd6f0198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f0101400 0x7fd6f0198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.194 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6f4b9b700 1 -- 172.21.15.111:0/4181542353 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc0095f0 con 0x7fd6f0100a50 2023-10-11T16:39:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6f0100a50 0x7fd6f0198600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400e9a0 tx=0x7fd6e400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.195 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6edffb700 1 -- 172.21.15.111:0/4181542353 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4009790 con 0x7fd6f0100a50 2023-10-11T16:39:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6f6dff700 1 -- 172.21.15.111:0/4181542353 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f0110410 con 0x7fd6f0100a50 2023-10-11T16:39:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6edffb700 1 -- 172.21.15.111:0/4181542353 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6e4010430 con 0x7fd6f0100a50 2023-10-11T16:39:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6edffb700 1 -- 172.21.15.111:0/4181542353 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4018680 con 0x7fd6f0100a50 2023-10-11T16:39:36.196 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.186+0000 7fd6f6dff700 1 -- 172.21.15.111:0/4181542353 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f0110280 con 0x7fd6f0100a50 2023-10-11T16:39:36.202 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.190+0000 7fd6edffb700 1 -- 172.21.15.111:0/4181542353 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fd6e40105a0 con 0x7fd6f0100a50 2023-10-11T16:39:36.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.190+0000 7fd70a681b80 1 Processor -- start 2023-10-11T16:39:36.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd70a681b80 1 -- start start 2023-10-11T16:39:36.203 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e65fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd70a681b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6d6b0e0 con 0x55eed6e6a930 2023-10-11T16:39:36.204 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd70a681b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6dbd920 con 0x55eed6e65be0 2023-10-11T16:39:36.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd70a681b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6db6db0 con 0x55eed6e664f0 2023-10-11T16:39:36.205 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd6edffb700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd6e00a12c0 0x7fd6e00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd6effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd6f4b9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd6effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34250/0 (socket says 172.21.15.111:34250) 2023-10-11T16:39:36.206 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd6effff700 1 -- 172.21.15.111:0/3024040889 learned_addr learned my addr 172.21.15.111:0/3024040889 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd6f539c700 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e65fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd6edffb700 1 -- 172.21.15.111:0/4181542353 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fd6e4014070 con 0x7fd6f0100a50 2023-10-11T16:39:36.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.191+0000 7fd6f539c700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd6e00a12c0 0x7fd6e00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.207 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6effff700 1 -- 172.21.15.111:0/3024040889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 msgr2=0x55eed6e6a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6effff700 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6effff700 1 -- 172.21.15.111:0/3024040889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 msgr2=0x55eed6e65fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.208 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6effff700 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e65fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6effff700 1 -- 172.21.15.111:0/3024040889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eed6c386b0 con 0x55eed6e6a930 2023-10-11T16:39:36.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6f539c700 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e65fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:36.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6effff700 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6ad80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc015000 tx=0x7fd6dc005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1300ec27f71d10b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.209 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6f539c700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd6e00a12c0 0x7fd6e00a3770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009ec0 tx=0x7fd6e800b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6daffd700 1 -- 172.21.15.111:0/3024040889 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc01d070 con 0x55eed6e6a930 2023-10-11T16:39:36.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6daffd700 1 -- 172.21.15.111:0/3024040889 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6dc01e040 con 0x55eed6e6a930 2023-10-11T16:39:36.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.192+0000 7fd6daffd700 1 -- 172.21.15.111:0/3024040889 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc0047f0 con 0x55eed6e6a930 2023-10-11T16:39:36.210 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 -- 172.21.15.111:0/3024040889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 msgr2=0x55eed6e6ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6ad80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc015000 tx=0x7fd6dc005ea0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 -- 172.21.15.111:0/3024040889 shutdown_connections 2023-10-11T16:39:36.211 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6ad80 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e65fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 --2- 172.21.15.111:0/3024040889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 -- 172.21.15.111:0/3024040889 >> 172.21.15.111:0/3024040889 conn(0x55eed6e5b3e0 msgr2=0x55eed6e60bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.212 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 -- 172.21.15.111:0/3024040889 shutdown_connections 2023-10-11T16:39:36.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.193+0000 7fd70a681b80 1 -- 172.21.15.111:0/3024040889 wait complete. 2023-10-11T16:39:36.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.194+0000 7fd70a681b80 1 -- start start 2023-10-11T16:39:36.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.194+0000 7fd70a681b80 1 Processor -- start 2023-10-11T16:39:36.213 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e72110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.214 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd70a681b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6d58960 con 0x55eed6e6a930 2023-10-11T16:39:36.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd70a681b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6d58600 con 0x55eed6e65be0 2023-10-11T16:39:36.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd70a681b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6c3a9a0 con 0x55eed6e664f0 2023-10-11T16:39:36.215 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd70a681b80 10 client.0 _ll_register_callbacks cb 0x55eed6d76800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:36.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6f4b9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e72110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6effff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.216 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6f4b9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e72110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40808/0 (socket says 172.21.15.111:40808) 2023-10-11T16:39:36.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6f4b9b700 1 -- 172.21.15.111:0/422057528 learned_addr learned my addr 172.21.15.111:0/422057528 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6f539c700 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6effff700 1 -- 172.21.15.111:0/422057528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 msgr2=0x55eed6e72110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6effff700 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e72110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6effff700 1 -- 172.21.15.111:0/422057528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 msgr2=0x55eed6e6fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.195+0000 7fd6effff700 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd6effff700 1 -- 172.21.15.111:0/422057528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f010da50 con 0x55eed6e664f0 2023-10-11T16:39:36.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd6f539c700 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd6effff700 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f320 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00bb30 tx=0x7fd6dc00fb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cef34d0452d5bb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd6cffff700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:39:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd6cffff700 1 -- 172.21.15.111:0/422057528 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc01d040 con 0x55eed6e664f0 2023-10-11T16:39:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd70a681b80 1 -- 172.21.15.111:0/422057528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eed6e58ba0 con 0x55eed6e664f0 2023-10-11T16:39:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd6cffff700 1 -- 172.21.15.111:0/422057528 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fd6dc005980 con 0x55eed6e664f0 2023-10-11T16:39:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd6cffff700 1 -- 172.21.15.111:0/422057528 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6dc00fe60 con 0x55eed6e664f0 2023-10-11T16:39:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd70a681b80 10 client.26785 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:36.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd70a681b80 1 -- 172.21.15.111:0/422057528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55eed6d7d210 con 0x55eed6e664f0 2023-10-11T16:39:36.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.196+0000 7fd6cffff700 1 -- 172.21.15.111:0/422057528 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eed6e58ba0 con 0x55eed6e664f0 2023-10-11T16:39:36.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.197+0000 7fd6cffff700 1 -- 172.21.15.111:0/422057528 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fd6dc004a50 con 0x55eed6e664f0 2023-10-11T16:39:36.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.197+0000 7fd70a681b80 10 client.26785 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.197+0000 7fd70a681b80 10 client.26785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.197+0000 7fd70a681b80 20 client.26785 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.197+0000 7fd70a681b80 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eed6e7f450 0x55eed6e81860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.197+0000 7fd70a681b80 4 client.26785 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:36.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.197+0000 7fd70a681b80 1 -- 172.21.15.111:0/422057528 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eed6d79120 con 0x55eed6e7f450 2023-10-11T16:39:36.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.197+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eed6e7f450 0x55eed6e81860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.198+0000 7fd6cffff700 10 client.26785 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:36.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.198+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eed6e7f450 0x55eed6e81860 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c64ec0ea1f7edd8b server_cookie=57d1e420ca6a3538 in_seq=0 out_seq=0 2023-10-11T16:39:36.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.198+0000 7fd6cffff700 1 -- 172.21.15.111:0/422057528 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55eed6d79120 con 0x55eed6e7f450 2023-10-11T16:39:36.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.198+0000 7fd6cffff700 10 client.26785 handle_command_reply: tid=0 2023-10-11T16:39:36.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.198+0000 7fd70a681b80 1 client.26785 shutdown 2023-10-11T16:39:36.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.198+0000 7fd70a681b80 1 -- 172.21.15.111:0/422057528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 msgr2=0x55eed6e6f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.198+0000 7fd70a681b80 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f320 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00bb30 tx=0x7fd6dc00fb40 comp rx=0 tx=0).stop 2023-10-11T16:39:36.225 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 1 -- 172.21.15.111:0/422057528 shutdown_connections 2023-10-11T16:39:36.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eed6e7f450 0x55eed6e81860 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e6a930 0x55eed6e6fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e65be0 0x55eed6e72110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.226 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 1 --2- 172.21.15.111:0/422057528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f320 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd6cffff700 0 client.26785 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:36.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 1 -- 172.21.15.111:0/422057528 >> 172.21.15.111:0/422057528 conn(0x55eed6e5b3e0 msgr2=0x55eed6e6bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.227 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 1 -- 172.21.15.111:0/422057528 shutdown_connections 2023-10-11T16:39:36.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 1 -- 172.21.15.111:0/422057528 wait complete. 2023-10-11T16:39:36.228 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.199+0000 7fd70a681b80 20 client.26785 trim_cache size 0 max 16384 2023-10-11T16:39:36.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.217+0000 7fd70a681b80 1 Processor -- start 2023-10-11T16:39:36.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.217+0000 7fd70a681b80 1 -- start start 2023-10-11T16:39:36.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e65be0 0x55eed6e60d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e769c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.234 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd70a681b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6e62240 con 0x55eed6e65be0 2023-10-11T16:39:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd70a681b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6e69470 con 0x55eed6e6a930 2023-10-11T16:39:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd70a681b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6e70540 con 0x55eed6e664f0 2023-10-11T16:39:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd6f4b9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e65be0 0x55eed6e60d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd6effff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd6f4b9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e65be0 0x55eed6e60d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34276/0 (socket says 172.21.15.111:34276) 2023-10-11T16:39:36.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd6f4b9b700 1 -- 172.21.15.111:0/3974572943 learned_addr learned my addr 172.21.15.111:0/3974572943 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.218+0000 7fd6f539c700 1 --2- 172.21.15.111:0/3974572943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6effff700 1 -- 172.21.15.111:0/3974572943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 msgr2=0x55eed6e769c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6effff700 1 --2- 172.21.15.111:0/3974572943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e769c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6effff700 1 -- 172.21.15.111:0/3974572943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e65be0 msgr2=0x55eed6e60d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6effff700 1 --2- 172.21.15.111:0/3974572943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e65be0 0x55eed6e60d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6effff700 1 -- 172.21.15.111:0/3974572943 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eed6d7cf60 con 0x55eed6e664f0 2023-10-11T16:39:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6effff700 1 --2- 172.21.15.111:0/3974572943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f2d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009990 tx=0x7fd6dc009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e524d30a14a6dc12 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6db7fe700 1 -- 172.21.15.111:0/3974572943 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc0224d0 con 0x55eed6e664f0 2023-10-11T16:39:36.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6db7fe700 1 -- 172.21.15.111:0/3974572943 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6dc005980 con 0x55eed6e664f0 2023-10-11T16:39:36.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.219+0000 7fd6db7fe700 1 -- 172.21.15.111:0/3974572943 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc004920 con 0x55eed6e664f0 2023-10-11T16:39:36.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 -- 172.21.15.111:0/3974572943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 msgr2=0x55eed6e6f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 --2- 172.21.15.111:0/3974572943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f2d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009990 tx=0x7fd6dc009310 comp rx=0 tx=0).stop 2023-10-11T16:39:36.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 -- 172.21.15.111:0/3974572943 shutdown_connections 2023-10-11T16:39:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 --2- 172.21.15.111:0/3974572943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e769c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 --2- 172.21.15.111:0/3974572943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e65be0 0x55eed6e60d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 --2- 172.21.15.111:0/3974572943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e664f0 0x55eed6e6f2d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 -- 172.21.15.111:0/3974572943 >> 172.21.15.111:0/3974572943 conn(0x55eed6e5b3e0 msgr2=0x55eed6e6ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 -- 172.21.15.111:0/3974572943 shutdown_connections 2023-10-11T16:39:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 -- 172.21.15.111:0/3974572943 wait complete. 2023-10-11T16:39:36.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.220+0000 7fd70a681b80 1 -- start start 2023-10-11T16:39:36.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd70a681b80 1 Processor -- start 2023-10-11T16:39:36.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e65be0 0x55eed6e769c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e664f0 0x55eed6e74de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd70a681b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e75320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd70a681b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6e759a0 con 0x55eed6e664f0 2023-10-11T16:39:36.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd70a681b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6e75b10 con 0x55eed6e6a930 2023-10-11T16:39:36.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd70a681b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eed6ed3320 con 0x55eed6e65be0 2023-10-11T16:39:36.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd70a681b80 10 client.0 _ll_register_callbacks cb 0x7fd6e4010b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:36.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd6effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e664f0 0x55eed6e74de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd6effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e664f0 0x55eed6e74de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34290/0 (socket says 172.21.15.111:34290) 2023-10-11T16:39:36.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd6effff700 1 -- 172.21.15.111:0/1823421067 learned_addr learned my addr 172.21.15.111:0/1823421067 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.221+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e65be0 0x55eed6e769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6f539c700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e75320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6effff700 1 -- 172.21.15.111:0/1823421067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e65be0 msgr2=0x55eed6e769c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6effff700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e65be0 0x55eed6e769c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6effff700 1 -- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 msgr2=0x55eed6e75320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6effff700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e75320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6effff700 1 -- 172.21.15.111:0/1823421067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eed6d7d210 con 0x55eed6e664f0 2023-10-11T16:39:36.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e65be0 0x55eed6e769c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:36.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6f539c700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e75320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:36.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6effff700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e664f0 0x55eed6e74de0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0098e0 tx=0x7fd6dc005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7096aa2ba4e0ba7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6cffff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6cffff700 1 -- 172.21.15.111:0/1823421067 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc0224d0 con 0x55eed6e664f0 2023-10-11T16:39:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd70a681b80 1 -- 172.21.15.111:0/1823421067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6dc01e040 con 0x55eed6e664f0 2023-10-11T16:39:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6cffff700 1 -- 172.21.15.111:0/1823421067 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fd6dc000e40 con 0x55eed6e664f0 2023-10-11T16:39:36.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.222+0000 7fd6cffff700 1 -- 172.21.15.111:0/1823421067 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6dc0045a0 con 0x55eed6e664f0 2023-10-11T16:39:36.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd70a681b80 10 client.16662 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:36.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd70a681b80 1 -- 172.21.15.111:0/1823421067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55eed6bcdce0 con 0x55eed6e664f0 2023-10-11T16:39:36.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd6cffff700 1 -- 172.21.15.111:0/1823421067 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6dc01e040 con 0x55eed6e664f0 2023-10-11T16:39:36.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd6cffff700 1 -- 172.21.15.111:0/1823421067 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fd6dc022c80 con 0x55eed6e664f0 2023-10-11T16:39:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd70a681b80 10 client.16662 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd70a681b80 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:36.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd70a681b80 20 client.16662 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd70a681b80 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eed6ed4490 0x55eed6ed68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd70a681b80 4 client.16662 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:39:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd70a681b80 1 -- 172.21.15.111:0/1823421067 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55eed6d856e0 con 0x55eed6ed4490 2023-10-11T16:39:36.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.223+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eed6ed4490 0x55eed6ed68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.224+0000 7fd6cffff700 10 client.16662 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:36.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.224+0000 7fd6f4b9b700 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eed6ed4490 0x55eed6ed68a0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52d8198f078cebe9 server_cookie=ed9255698e0572d2 in_seq=0 out_seq=0 2023-10-11T16:39:36.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.224+0000 7fd6cffff700 1 -- 172.21.15.111:0/1823421067 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55eed6d856e0 con 0x55eed6ed4490 2023-10-11T16:39:36.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.224+0000 7fd6cffff700 10 client.16662 handle_command_reply: tid=0 2023-10-11T16:39:36.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.224+0000 7fd70a681b80 1 client.16662 shutdown 2023-10-11T16:39:36.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 -- 172.21.15.111:0/1823421067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e664f0 msgr2=0x55eed6e74de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e664f0 0x55eed6e74de0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0098e0 tx=0x7fd6dc005fb0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 -- 172.21.15.111:0/1823421067 shutdown_connections 2023-10-11T16:39:36.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55eed6ed4490 0x55eed6ed68a0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eed6e6a930 0x55eed6e75320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55eed6e65be0 0x55eed6e769c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 --2- 172.21.15.111:0/1823421067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55eed6e664f0 0x55eed6e74de0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 -- 172.21.15.111:0/1823421067 >> 172.21.15.111:0/1823421067 conn(0x55eed6e5b3e0 msgr2=0x55eed6e65270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd6cffff700 0 client.16662 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:36.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 -- 172.21.15.111:0/1823421067 shutdown_connections 2023-10-11T16:39:36.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 1 -- 172.21.15.111:0/1823421067 wait complete. 2023-10-11T16:39:36.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.225+0000 7fd70a681b80 20 client.16662 trim_cache size 0 max 16384 2023-10-11T16:39:36.257 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:39:36.257 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:39:36.258 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "2fad7908-0223-4693-978e-0b32ef0c2afd", 2023-10-11T16:39:36.258 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:39:36.258 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:39:36.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 -- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd6e00a12c0 msgr2=0x7fd6e00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd6e00a12c0 0x7fd6e00a3770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009ec0 tx=0x7fd6e800b040 comp rx=0 tx=0).stop 2023-10-11T16:39:36.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 -- 172.21.15.111:0/4181542353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6f0100a50 msgr2=0x7fd6f0198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6f0100a50 0x7fd6f0198600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400e9a0 tx=0x7fd6e400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 -- 172.21.15.111:0/4181542353 shutdown_connections 2023-10-11T16:39:36.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fd6e00a12c0 0x7fd6e00a3770 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6f01063f0 0x7fd6f019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6f0100a50 0x7fd6f0198600 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 --2- 172.21.15.111:0/4181542353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6f0101400 0x7fd6f0198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 -- 172.21.15.111:0/4181542353 >> 172.21.15.111:0/4181542353 conn(0x7fd6f0077590 msgr2=0x7fd6f00ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 -- 172.21.15.111:0/4181542353 shutdown_connections 2023-10-11T16:39:36.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.226+0000 7fd6f6dff700 1 -- 172.21.15.111:0/4181542353 wait complete. 2023-10-11T16:39:36.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:39:36.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 -- 172.21.15.111:0/2920714480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc1063f0 msgr2=0x7f7ecc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/2920714480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc1063f0 0x7f7ecc1067c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009990 tx=0x7f7ec8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 -- 172.21.15.111:0/2920714480 shutdown_connections 2023-10-11T16:39:36.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/2920714480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ecc101400 0x7f7ecc10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/2920714480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc1063f0 0x7f7ecc1067c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/2920714480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ecc100a50 0x7f7ecc100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 -- 172.21.15.111:0/2920714480 >> 172.21.15.111:0/2920714480 conn(0x7f7ecc077590 msgr2=0x7f7ecc077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 -- 172.21.15.111:0/2920714480 shutdown_connections 2023-10-11T16:39:36.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.461+0000 7f7ed39ff700 1 -- 172.21.15.111:0/2920714480 wait complete. 2023-10-11T16:39:36.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.462+0000 7f7ed39ff700 1 Processor -- start 2023-10-11T16:39:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.462+0000 7f7ed39ff700 1 -- start start 2023-10-11T16:39:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed39ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc100a50 0x7f7ecc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed39ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ecc101400 0x7f7ecc198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed39ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ecc1063f0 0x7f7ecc19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed39ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc19d5a0 con 0x7f7ecc100a50 2023-10-11T16:39:36.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed39ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc19d710 con 0x7f7ecc1063f0 2023-10-11T16:39:36.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed39ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc19d8d0 con 0x7f7ecc101400 2023-10-11T16:39:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc100a50 0x7f7ecc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc100a50 0x7f7ecc1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34314/0 (socket says 172.21.15.111:34314) 2023-10-11T16:39:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed179b700 1 -- 172.21.15.111:0/261186458 learned_addr learned my addr 172.21.15.111:0/261186458 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ecc101400 0x7f7ecc198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed1f9c700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ecc1063f0 0x7f7ecc19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed179b700 1 -- 172.21.15.111:0/261186458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ecc101400 msgr2=0x7f7ecc198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed179b700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ecc101400 0x7f7ecc198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed179b700 1 -- 172.21.15.111:0/261186458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ecc1063f0 msgr2=0x7f7ecc19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.463+0000 7f7ed179b700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ecc1063f0 0x7f7ecc19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.464+0000 7f7ed179b700 1 -- 172.21.15.111:0/261186458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc009520 con 0x7f7ecc100a50 2023-10-11T16:39:36.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.464+0000 7f7ed179b700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc100a50 0x7f7ecc1985c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8000c00 tx=0x7f7ec8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.464+0000 7f7ec27fc700 1 -- 172.21.15.111:0/261186458 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec801d070 con 0x7f7ecc100a50 2023-10-11T16:39:36.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.464+0000 7f7ed39ff700 1 -- 172.21.15.111:0/261186458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec80095f0 con 0x7f7ecc100a50 2023-10-11T16:39:36.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.464+0000 7f7ec27fc700 1 -- 172.21.15.111:0/261186458 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec8022470 con 0x7f7ecc100a50 2023-10-11T16:39:36.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.464+0000 7f7ec27fc700 1 -- 172.21.15.111:0/261186458 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80175f0 con 0x7f7ecc100a50 2023-10-11T16:39:36.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.464+0000 7f7ed39ff700 1 -- 172.21.15.111:0/261186458 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc110380 con 0x7f7ecc100a50 2023-10-11T16:39:36.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.467+0000 7f7ec27fc700 1 -- 172.21.15.111:0/261186458 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f7ec80225e0 con 0x7f7ecc100a50 2023-10-11T16:39:36.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.468+0000 7f7ee7296b80 1 Processor -- start 2023-10-11T16:39:36.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ee7296b80 1 -- start start 2023-10-11T16:39:36.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ec27fc700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7eb80a12c0 0x7f7eb80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b0bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b10370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b10d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2a11060 con 0x555fb2b0bb60 2023-10-11T16:39:36.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2a638a0 con 0x555fb2b0c470 2023-10-11T16:39:36.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2a5cd30 con 0x555fb2b108b0 2023-10-11T16:39:36.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ec27fc700 1 -- 172.21.15.111:0/261186458 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f7ec80ce540 con 0x7f7ecc100a50 2023-10-11T16:39:36.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b0bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ed1f9c700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7eb80a12c0 0x7f7eb80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ed0f9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b10370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b0bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34318/0 (socket says 172.21.15.111:34318) 2023-10-11T16:39:36.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ed179b700 1 -- 172.21.15.111:0/3366700015 learned_addr learned my addr 172.21.15.111:0/3366700015 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.469+0000 7f7ed0f9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b10370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40860/0 (socket says 172.21.15.111:40860) 2023-10-11T16:39:36.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed1f9c700 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b10d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed179b700 1 -- 172.21.15.111:0/3366700015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 msgr2=0x555fb2b10d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed179b700 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b10d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed179b700 1 -- 172.21.15.111:0/3366700015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 msgr2=0x555fb2b10370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed179b700 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b10370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed179b700 1 -- 172.21.15.111:0/3366700015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fb28de630 con 0x555fb2b0bb60 2023-10-11T16:39:36.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b10370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:39:36.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed1f9c700 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b10d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:39:36.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed179b700 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b0bf30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80a30c0 tx=0x7f7ec80975f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a094f440a6f5b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7ed1f9c700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7eb80a12c0 0x7f7eb80a3770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009fd0 tx=0x7f7ec4009920 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7eb77fe700 1 -- 172.21.15.111:0/3366700015 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec809ad70 con 0x555fb2b0bb60 2023-10-11T16:39:36.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7eb77fe700 1 -- 172.21.15.111:0/3366700015 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec809ab90 con 0x555fb2b0bb60 2023-10-11T16:39:36.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.470+0000 7f7eb77fe700 1 -- 172.21.15.111:0/3366700015 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80aaa30 con 0x555fb2b0bb60 2023-10-11T16:39:36.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 -- 172.21.15.111:0/3366700015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 msgr2=0x555fb2b0bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b0bf30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80a30c0 tx=0x7f7ec80975f0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 -- 172.21.15.111:0/3366700015 shutdown_connections 2023-10-11T16:39:36.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b10d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b0bf30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/3366700015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b10370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 -- 172.21.15.111:0/3366700015 >> 172.21.15.111:0/3366700015 conn(0x555fb2b01360 msgr2=0x555fb2b06b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 -- 172.21.15.111:0/3366700015 shutdown_connections 2023-10-11T16:39:36.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.471+0000 7f7ee7296b80 1 -- 172.21.15.111:0/3366700015 wait complete. 2023-10-11T16:39:36.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.472+0000 7f7ee7296b80 1 -- start start 2023-10-11T16:39:36.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.472+0000 7f7ee7296b80 1 Processor -- start 2023-10-11T16:39:36.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0bb60 0x555fb2b17f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0c470 0x555fb2b151b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 0x555fb2b156f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb29fe8e0 con 0x555fb2b108b0 2023-10-11T16:39:36.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb29fe580 con 0x555fb2b0bb60 2023-10-11T16:39:36.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb28e0920 con 0x555fb2b0c470 2023-10-11T16:39:36.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ee7296b80 10 client.0 _ll_register_callbacks cb 0x555fb2a1c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:36.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed1f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 0x555fb2b156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0bb60 0x555fb2b17f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0bb60 0x555fb2b17f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40868/0 (socket says 172.21.15.111:40868) 2023-10-11T16:39:36.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed179b700 1 -- 172.21.15.111:0/1171832244 learned_addr learned my addr 172.21.15.111:0/1171832244 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0c470 0x555fb2b151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed179b700 1 -- 172.21.15.111:0/1171832244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0c470 msgr2=0x555fb2b151b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed179b700 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0c470 0x555fb2b151b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed179b700 1 -- 172.21.15.111:0/1171832244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 msgr2=0x555fb2b156f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed179b700 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 0x555fb2b156f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.473+0000 7f7ed179b700 1 -- 172.21.15.111:0/1171832244 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc110380 con 0x555fb2b0bb60 2023-10-11T16:39:36.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7ed179b700 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0bb60 0x555fb2b17f30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8096fb0 tx=0x7f7ec8009540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c857f5176bdb3d49 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7eb4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:39:36.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/1171832244 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80aa4b0 con 0x555fb2b0bb60 2023-10-11T16:39:36.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7ee7296b80 1 -- 172.21.15.111:0/1171832244 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555fb2afeb20 con 0x555fb2b0bb60 2023-10-11T16:39:36.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/1171832244 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f7ec80a10d0 con 0x555fb2b0bb60 2023-10-11T16:39:36.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/1171832244 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ec80a8000 con 0x555fb2b0bb60 2023-10-11T16:39:36.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7ee7296b80 10 client.26789 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:36.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7ee7296b80 1 -- 172.21.15.111:0/1171832244 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555fb2a23190 con 0x555fb2b0bb60 2023-10-11T16:39:36.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/1171832244 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555fb2afeb20 con 0x555fb2b0bb60 2023-10-11T16:39:36.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.474+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/1171832244 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7ec809a720 con 0x555fb2b0bb60 2023-10-11T16:39:36.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7ee7296b80 10 client.26789 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:36.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7ee7296b80 10 client.26789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:36.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7ee7296b80 20 client.26789 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:36.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555fb2b24fc0 0x555fb2b273a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7ee7296b80 4 client.26789 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:39:36.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7ee7296b80 1 -- 172.21.15.111:0/1171832244 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555fb2a1f0a0 con 0x555fb2b24fc0 2023-10-11T16:39:36.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555fb2b24fc0 0x555fb2b273a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7eb4ff9700 10 client.26789 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:36.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.475+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555fb2b24fc0 0x555fb2b273a0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a19025f5773f7b7 server_cookie=184ca7303fb68019 in_seq=0 out_seq=0 2023-10-11T16:39:36.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.476+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/1171832244 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555fb2a1f0a0 con 0x555fb2b24fc0 2023-10-11T16:39:36.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.476+0000 7f7eb4ff9700 10 client.26789 handle_command_reply: tid=0 2023-10-11T16:39:36.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.476+0000 7f7ee7296b80 1 client.26789 shutdown 2023-10-11T16:39:36.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 -- 172.21.15.111:0/1171832244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0bb60 msgr2=0x555fb2b17f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0bb60 0x555fb2b17f30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8096fb0 tx=0x7f7ec8009540 comp rx=0 tx=0).stop 2023-10-11T16:39:36.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 -- 172.21.15.111:0/1171832244 shutdown_connections 2023-10-11T16:39:36.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555fb2b24fc0 0x555fb2b273a0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 0x555fb2b156f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0bb60 0x555fb2b17f30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/1171832244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0c470 0x555fb2b151b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7eb4ff9700 0 client.26789 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 -- 172.21.15.111:0/1171832244 >> 172.21.15.111:0/1171832244 conn(0x555fb2b01360 msgr2=0x555fb2b11d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 -- 172.21.15.111:0/1171832244 shutdown_connections 2023-10-11T16:39:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 1 -- 172.21.15.111:0/1171832244 wait complete. 2023-10-11T16:39:36.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.477+0000 7f7ee7296b80 20 client.26789 trim_cache size 0 max 16384 2023-10-11T16:39:36.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.494+0000 7f7ee7296b80 1 Processor -- start 2023-10-11T16:39:36.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.494+0000 7f7ee7296b80 1 -- start start 2023-10-11T16:39:36.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.494+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0bb60 0x555fb2b17ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.494+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b0f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.494+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 0x555fb2b0fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.494+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2b0cbc0 con 0x555fb2b108b0 2023-10-11T16:39:36.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.494+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2b16790 con 0x555fb2b0c470 2023-10-11T16:39:36.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.494+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2b077b0 con 0x555fb2b0bb60 2023-10-11T16:39:36.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0bb60 0x555fb2b17ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed1f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 0x555fb2b0fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0bb60 0x555fb2b17ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56216/0 (socket says 172.21.15.111:56216) 2023-10-11T16:39:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 -- 172.21.15.111:0/41516940 learned_addr learned my addr 172.21.15.111:0/41516940 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/41516940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b0f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 -- 172.21.15.111:0/41516940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 msgr2=0x555fb2b0f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 --2- 172.21.15.111:0/41516940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b0f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 -- 172.21.15.111:0/41516940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 msgr2=0x555fb2b0fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 --2- 172.21.15.111:0/41516940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 0x555fb2b0fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 -- 172.21.15.111:0/41516940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fb2b05820 con 0x555fb2b0bb60 2023-10-11T16:39:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7ed179b700 1 --2- 172.21.15.111:0/41516940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0bb60 0x555fb2b17ad0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8097ca0 tx=0x7f7ec809b250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33fabb6f5464f559 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7eb7fff700 1 -- 172.21.15.111:0/41516940 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80aa4b0 con 0x555fb2b0bb60 2023-10-11T16:39:36.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7eb7fff700 1 -- 172.21.15.111:0/41516940 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec80a8030 con 0x555fb2b0bb60 2023-10-11T16:39:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.495+0000 7f7eb7fff700 1 -- 172.21.15.111:0/41516940 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80a31b0 con 0x555fb2b0bb60 2023-10-11T16:39:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 -- 172.21.15.111:0/41516940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0bb60 msgr2=0x555fb2b17ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/41516940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0bb60 0x555fb2b17ad0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8097ca0 tx=0x7f7ec809b250 comp rx=0 tx=0).stop 2023-10-11T16:39:36.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 -- 172.21.15.111:0/41516940 shutdown_connections 2023-10-11T16:39:36.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/41516940 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b108b0 0x555fb2b0fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/41516940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b0bb60 0x555fb2b17ad0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/41516940 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b0f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 -- 172.21.15.111:0/41516940 >> 172.21.15.111:0/41516940 conn(0x555fb2b01360 msgr2=0x555fb2b06670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 -- 172.21.15.111:0/41516940 shutdown_connections 2023-10-11T16:39:36.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.496+0000 7f7ee7296b80 1 -- 172.21.15.111:0/41516940 wait complete. 2023-10-11T16:39:36.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.497+0000 7f7ee7296b80 1 -- start start 2023-10-11T16:39:36.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.497+0000 7f7ee7296b80 1 Processor -- start 2023-10-11T16:39:36.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b6cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b6d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ee7296b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b77e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2b6dc60 con 0x555fb2b0bb60 2023-10-11T16:39:36.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2b78390 con 0x555fb2b0c470 2023-10-11T16:39:36.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ee7296b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb2b78550 con 0x555fb2b108b0 2023-10-11T16:39:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ee7296b80 10 client.0 _ll_register_callbacks cb 0x555fb2a1c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:39:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b6cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b6cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34328/0 (socket says 172.21.15.111:34328) 2023-10-11T16:39:36.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 -- 172.21.15.111:0/5116764 learned_addr learned my addr 172.21.15.111:0/5116764 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:39:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed1f9c700 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b77e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b6d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 -- 172.21.15.111:0/5116764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 msgr2=0x555fb2b77e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b77e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 -- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 msgr2=0x555fb2b6d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:39:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b6d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 -- 172.21.15.111:0/5116764 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555fb2a23190 con 0x555fb2b0bb60 2023-10-11T16:39:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b6d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:39:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7ed179b700 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b6cf90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80cbe20 tx=0x7f7ec80aa760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f4c17fb5c739e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:39:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.498+0000 7f7eb4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:39:36.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/5116764 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80aeea0 con 0x555fb2b0bb60 2023-10-11T16:39:36.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/5116764 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f7ec80a10d0 con 0x555fb2b0bb60 2023-10-11T16:39:36.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 1 -- 172.21.15.111:0/5116764 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ec809ab90 con 0x555fb2b0bb60 2023-10-11T16:39:36.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/5116764 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ec80aa4b0 con 0x555fb2b0bb60 2023-10-11T16:39:36.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 10 client.16680 fetch_fsmap learned FSMap version 33 2023-10-11T16:39:36.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 1 -- 172.21.15.111:0/5116764 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555fb288ade0 con 0x555fb2b0bb60 2023-10-11T16:39:36.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/5116764 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ec809ab90 con 0x555fb2b0bb60 2023-10-11T16:39:36.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/5116764 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7ec8097020 con 0x555fb2b0bb60 2023-10-11T16:39:36.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 10 client.16680 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:39:36.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 10 client.16680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:39:36.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 20 client.16680 populate_metadata read hostname 'smithi111' 2023-10-11T16:39:36.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555fb2b797b0 0x555fb2b7bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:39:36.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 4 client.16680 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:39:36.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.499+0000 7f7ee7296b80 1 -- 172.21.15.111:0/5116764 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555fb2a2b660 con 0x555fb2b797b0 2023-10-11T16:39:36.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.500+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555fb2b797b0 0x555fb2b7bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:39:36.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.500+0000 7f7ed0f9a700 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555fb2b797b0 0x555fb2b7bb90 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74741f0715734cb8 server_cookie=4fb200529f9fa7b1 in_seq=0 out_seq=0 2023-10-11T16:39:36.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.500+0000 7f7eb4ff9700 10 client.16680 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:36.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.500+0000 7f7eb4ff9700 1 -- 172.21.15.111:0/5116764 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555fb2a2b660 con 0x555fb2b797b0 2023-10-11T16:39:36.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.500+0000 7f7eb4ff9700 10 client.16680 handle_command_reply: tid=0 2023-10-11T16:39:36.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.500+0000 7f7ee7296b80 1 client.16680 shutdown 2023-10-11T16:39:36.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 -- 172.21.15.111:0/5116764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 msgr2=0x555fb2b6cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b6cf90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80cbe20 tx=0x7f7ec80aa760 comp rx=0 tx=0).stop 2023-10-11T16:39:36.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 -- 172.21.15.111:0/5116764 shutdown_connections 2023-10-11T16:39:36.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x555fb2b797b0 0x555fb2b7bb90 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x555fb2b108b0 0x555fb2b77e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x555fb2b0bb60 0x555fb2b6cf90 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7eb4ff9700 0 client.16680 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:39:36.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 --2- 172.21.15.111:0/5116764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fb2b0c470 0x555fb2b6d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 -- 172.21.15.111:0/5116764 >> 172.21.15.111:0/5116764 conn(0x555fb2b01360 msgr2=0x555fb2b0f770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 -- 172.21.15.111:0/5116764 shutdown_connections 2023-10-11T16:39:36.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 1 -- 172.21.15.111:0/5116764 wait complete. 2023-10-11T16:39:36.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.501+0000 7f7ee7296b80 20 client.16680 trim_cache size 0 max 16384 2023-10-11T16:39:36.534 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:39:36.534 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:39:36.534 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:39:36.534 INFO:teuthology.orchestra.run.smithi111.stdout: "2fad7908-0223-4693-978e-0b32ef0c2afd": { 2023-10-11T16:39:36.535 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:39:36.535 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "2fad7908-0223-4693-978e-0b32ef0c2afd", 2023-10-11T16:39:36.535 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:39:36.536 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:39:36.536 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:39:36.536 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:39:36.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 -- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7eb80a12c0 msgr2=0x7f7eb80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7eb80a12c0 0x7f7eb80a3770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009fd0 tx=0x7f7ec4009920 comp rx=0 tx=0).stop 2023-10-11T16:39:36.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 -- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc100a50 msgr2=0x7f7ecc1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:39:36.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc100a50 0x7f7ecc1985c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8000c00 tx=0x7f7ec8005740 comp rx=0 tx=0).stop 2023-10-11T16:39:36.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 -- 172.21.15.111:0/261186458 shutdown_connections 2023-10-11T16:39:36.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f7eb80a12c0 0x7f7eb80a3770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ecc1063f0 0x7f7ecc19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ecc100a50 0x7f7ecc1985c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 --2- 172.21.15.111:0/261186458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ecc101400 0x7f7ecc198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:39:36.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 -- 172.21.15.111:0/261186458 >> 172.21.15.111:0/261186458 conn(0x7f7ecc077590 msgr2=0x7f7ecc10b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:39:36.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 -- 172.21.15.111:0/261186458 shutdown_connections 2023-10-11T16:39:36.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:39:36.502+0000 7f7ed39ff700 1 -- 172.21.15.111:0/261186458 wait complete. 2023-10-11T16:39:36.541 INFO:tasks.cephfs.filesystem:scrub status for tag:2fad7908-0223-4693-978e-0b32ef0c2afd - {'path': '/', 'tag': '2fad7908-0223-4693-978e-0b32ef0c2afd', 'options': 'recursive,force'} 2023-10-11T16:39:36.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:36 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:39:36.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:36 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:36 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:39:36.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:36 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:37.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:36 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:39:37.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:36 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:37.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:37 smithi111 ceph-mon[98527]: pgmap v214: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:39:37.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:37 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:39:37.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:37 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:37.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:37 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:39:37.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:37 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:37 smithi150 ceph-mon[132217]: pgmap v214: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:39:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:37 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:39:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:37 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:37 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:39:37.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:37 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:38.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:37 smithi130 ceph-mon[131859]: pgmap v214: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:39:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:37 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:39:38.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:37 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:37 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:39:38.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:37 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:39:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:39 smithi111 ceph-mon[98527]: pgmap v215: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:39 smithi150 ceph-mon[132217]: pgmap v215: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:40.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:39 smithi130 ceph-mon[131859]: pgmap v215: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:41 smithi111 ceph-mon[98527]: pgmap v216: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:39:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:41 smithi150 ceph-mon[132217]: pgmap v216: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:39:42.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:41 smithi130 ceph-mon[131859]: pgmap v216: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:39:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:43 smithi111 ceph-mon[98527]: pgmap v217: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:39:43.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:43 smithi150 ceph-mon[132217]: pgmap v217: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:39:44.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:43 smithi130 ceph-mon[131859]: pgmap v217: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:39:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:45 smithi111 ceph-mon[98527]: pgmap v218: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-11T16:39:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:45 smithi150 ceph-mon[132217]: pgmap v218: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-11T16:39:46.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:45 smithi130 ceph-mon[131859]: pgmap v218: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-11T16:39:46.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:46 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:39:46.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:46 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:39:47.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:46 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:39:47.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:46 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:39:47.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:46 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:39:47.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:46 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:39:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:47 smithi150 ceph-mon[132217]: pgmap v219: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:39:48.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:47 smithi130 ceph-mon[131859]: pgmap v219: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:39:48.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:47 smithi111 ceph-mon[98527]: pgmap v219: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:39:49.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:49 smithi150 ceph-mon[132217]: pgmap v220: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:39:50.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:49 smithi130 ceph-mon[131859]: pgmap v220: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:39:50.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:49 smithi111 ceph-mon[98527]: pgmap v220: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:39:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:51 smithi150 ceph-mon[132217]: pgmap v221: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:39:52.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:51 smithi130 ceph-mon[131859]: pgmap v221: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:39:52.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:51 smithi111 ceph-mon[98527]: pgmap v221: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:39:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:53 smithi150 ceph-mon[132217]: pgmap v222: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:54.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:53 smithi130 ceph-mon[131859]: pgmap v222: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:54.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:53 smithi111 ceph-mon[98527]: pgmap v222: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:39:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:55 smithi150 ceph-mon[132217]: pgmap v223: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:39:56.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:55 smithi130 ceph-mon[131859]: pgmap v223: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:39:56.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:55 smithi111 ceph-mon[98527]: pgmap v223: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:39:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:57 smithi150 ceph-mon[132217]: pgmap v224: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:39:58.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:57 smithi130 ceph-mon[131859]: pgmap v224: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:39:58.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:57 smithi111 ceph-mon[98527]: pgmap v224: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:39:59.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:39:59 smithi150 ceph-mon[132217]: pgmap v225: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:00.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:39:59 smithi130 ceph-mon[131859]: pgmap v225: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:00.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:39:59 smithi111 ceph-mon[98527]: pgmap v225: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:00.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:00 smithi150 ceph-mon[132217]: overall HEALTH_OK 2023-10-11T16:40:01.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:00 smithi130 ceph-mon[131859]: overall HEALTH_OK 2023-10-11T16:40:01.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:00 smithi111 ceph-mon[98527]: overall HEALTH_OK 2023-10-11T16:40:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:01 smithi150 ceph-mon[132217]: pgmap v226: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:40:02.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:01 smithi130 ceph-mon[131859]: pgmap v226: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:40:02.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:01 smithi111 ceph-mon[98527]: pgmap v226: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:40:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:03 smithi150 ceph-mon[132217]: pgmap v227: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:04.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:03 smithi130 ceph-mon[131859]: pgmap v227: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:04.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:03 smithi111 ceph-mon[98527]: pgmap v227: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:05.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:05 smithi150 ceph-mon[132217]: pgmap v228: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-11T16:40:06.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:05 smithi130 ceph-mon[131859]: pgmap v228: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-11T16:40:06.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:05 smithi111 ceph-mon[98527]: pgmap v228: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-11T16:40:06.542 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:40:06.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 -- 172.21.15.111:0/2186258266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 msgr2=0x7f4f4c10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 --2- 172.21.15.111:0/2186258266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 0x7f4f4c10ccd0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400b0a0 tx=0x7f4f4400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 -- 172.21.15.111:0/2186258266 shutdown_connections 2023-10-11T16:40:06.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 --2- 172.21.15.111:0/2186258266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 0x7f4f4c10ccd0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 --2- 172.21.15.111:0/2186258266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c101390 0x7f4f4c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 --2- 172.21.15.111:0/2186258266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4c101ca0 0x7f4f4c10a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 -- 172.21.15.111:0/2186258266 >> 172.21.15.111:0/2186258266 conn(0x7f4f4c0770b0 msgr2=0x7f4f4c0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:06.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 -- 172.21.15.111:0/2186258266 shutdown_connections 2023-10-11T16:40:06.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.757+0000 7f4f519ff700 1 -- 172.21.15.111:0/2186258266 wait complete. 2023-10-11T16:40:06.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.758+0000 7f4f519ff700 1 Processor -- start 2023-10-11T16:40:06.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.758+0000 7f4f519ff700 1 -- start start 2023-10-11T16:40:06.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.758+0000 7f4f519ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4c101390 0x7f4f4c1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.758+0000 7f4f519ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c101ca0 0x7f4f4c196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.758+0000 7f4f519ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 0x7f4f4c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.758+0000 7f4f519ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c19b4a0 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.758+0000 7f4f519ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c19b610 con 0x7f4f4c101ca0 2023-10-11T16:40:06.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.758+0000 7f4f519ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c19b7d0 con 0x7f4f4c101390 2023-10-11T16:40:06.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 0x7f4f4c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4c101390 0x7f4f4c1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4c101390 0x7f4f4c1964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:38814/0 (socket says 172.21.15.111:38814) 2023-10-11T16:40:06.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4affd700 1 -- 172.21.15.111:0/4233278714 learned_addr learned my addr 172.21.15.111:0/4233278714 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:06.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 0x7f4f4c19ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42838/0 (socket says 172.21.15.111:42838) 2023-10-11T16:40:06.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4a7fc700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c101ca0 0x7f4f4c196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/4233278714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4c101390 msgr2=0x7f4f4c1964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4c101390 0x7f4f4c1964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/4233278714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c101ca0 msgr2=0x7f4f4c196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c101ca0 0x7f4f4c196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/4233278714 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44009d00 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.759+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 0x7f4f4c19ae60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009fd0 tx=0x7f4f4400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:06.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.760+0000 7f4f509fd700 1 -- 172.21.15.111:0/4233278714 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400e040 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.760+0000 7f4f509fd700 1 -- 172.21.15.111:0/4233278714 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f44004e60 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.760+0000 7f4f509fd700 1 -- 172.21.15.111:0/4233278714 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44005560 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.760+0000 7f4f519ff700 1 -- 172.21.15.111:0/4233278714 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c110350 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.760+0000 7f4f519ff700 1 -- 172.21.15.111:0/4233278714 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c1101c0 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.763+0000 7f4f509fd700 1 -- 172.21.15.111:0/4233278714 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f4f4401dbb0 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f653f6b80 1 Processor -- start 2023-10-11T16:40:06.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f509fd700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f340a1340 0x7f4f340a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f653f6b80 1 -- start start 2023-10-11T16:40:06.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f509fd700 1 -- 172.21.15.111:0/4233278714 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4f440ce960 con 0x7f4f4c10a8b0 2023-10-11T16:40:06.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f4a7fc700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f340a1340 0x7f4f340a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae3d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae41780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae42110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccad3e060 con 0x55cccae41cc0 2023-10-11T16:40:06.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccad908a0 con 0x55cccae3d880 2023-10-11T16:40:06.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.764+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccad89d30 con 0x55cccae3cf70 2023-10-11T16:40:06.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae42110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4a7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae41780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae42110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42840/0 (socket says 172.21.15.111:42840) 2023-10-11T16:40:06.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/2965590057 learned_addr learned my addr 172.21.15.111:0/2965590057 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:06.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4affd700 1 --2- 172.21.15.111:0/2965590057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae3d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4a7fc700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f340a1340 0x7f4f340a37f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c197ad0 tx=0x7f4f4000b410 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:06.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/2965590057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 msgr2=0x55cccae3d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:06.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/2965590057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae3d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/2965590057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 msgr2=0x55cccae41780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:06.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/2965590057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae41780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/2965590057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cccac0b630 con 0x55cccae41cc0 2023-10-11T16:40:06.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.765+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/2965590057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae42110 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4f440a34e0 tx=0x7f4f4401df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3863f48d750c83 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:06.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f31ffb700 1 -- 172.21.15.111:0/2965590057 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4409b650 con 0x55cccae41cc0 2023-10-11T16:40:06.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f31ffb700 1 -- 172.21.15.111:0/2965590057 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f4409f680 con 0x55cccae41cc0 2023-10-11T16:40:06.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f31ffb700 1 -- 172.21.15.111:0/2965590057 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f440860b0 con 0x55cccae41cc0 2023-10-11T16:40:06.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2965590057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 msgr2=0x55cccae42110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2965590057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae42110 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4f440a34e0 tx=0x7f4f4401df50 comp rx=0 tx=0).stop 2023-10-11T16:40:06.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2965590057 shutdown_connections 2023-10-11T16:40:06.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2965590057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae42110 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2965590057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae3d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2965590057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae41780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2965590057 >> 172.21.15.111:0/2965590057 conn(0x55cccae2e360 msgr2=0x55cccae33b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.766+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2965590057 shutdown_connections 2023-10-11T16:40:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.767+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2965590057 wait complete. 2023-10-11T16:40:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.767+0000 7f4f653f6b80 1 -- start start 2023-10-11T16:40:06.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.767+0000 7f4f653f6b80 1 Processor -- start 2023-10-11T16:40:06.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae3cf70 0x55cccae49410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae46690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae41cc0 0x55cccae46cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccad2b8e0 con 0x55cccae3cf70 2023-10-11T16:40:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccad2b580 con 0x55cccae3d880 2023-10-11T16:40:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccac0d920 con 0x55cccae41cc0 2023-10-11T16:40:06.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f653f6b80 10 client.0 _ll_register_callbacks cb 0x55cccad49780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae3cf70 0x55cccae49410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae3cf70 0x55cccae49410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42856/0 (socket says 172.21.15.111:42856) 2023-10-11T16:40:06.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4affd700 1 -- 172.21.15.111:0/1063972211 learned_addr learned my addr 172.21.15.111:0/1063972211 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:06.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4a7fc700 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae46690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae41cc0 0x55cccae46cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4affd700 1 -- 172.21.15.111:0/1063972211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae41cc0 msgr2=0x55cccae46cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:06.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4affd700 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae41cc0 0x55cccae46cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4affd700 1 -- 172.21.15.111:0/1063972211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 msgr2=0x55cccae46690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:06.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4affd700 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae46690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.768+0000 7f4f4affd700 1 -- 172.21.15.111:0/1063972211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c1101c0 con 0x55cccae3cf70 2023-10-11T16:40:06.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f4affd700 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae3cf70 0x55cccae49410 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0116f0 tx=0x7f4f3c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74fd93b593e45336 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:06.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f2affd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:06.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f2affd700 1 -- 172.21.15.111:0/1063972211 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c016070 con 0x55cccae3cf70 2023-10-11T16:40:06.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f2affd700 1 -- 172.21.15.111:0/1063972211 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4f3c004260 con 0x55cccae3cf70 2023-10-11T16:40:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f653f6b80 1 -- 172.21.15.111:0/1063972211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cccae2bb20 con 0x55cccae3cf70 2023-10-11T16:40:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f2affd700 1 -- 172.21.15.111:0/1063972211 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f3c004460 con 0x55cccae3cf70 2023-10-11T16:40:06.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f653f6b80 10 client.16692 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:06.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f653f6b80 1 -- 172.21.15.111:0/1063972211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cccad50190 con 0x55cccae3cf70 2023-10-11T16:40:06.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.769+0000 7f4f2affd700 1 -- 172.21.15.111:0/1063972211 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cccae2bb20 con 0x55cccae3cf70 2023-10-11T16:40:06.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.770+0000 7f4f2affd700 1 -- 172.21.15.111:0/1063972211 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f4f3c01b660 con 0x55cccae3cf70 2023-10-11T16:40:06.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.770+0000 7f4f653f6b80 10 client.16692 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.770+0000 7f4f653f6b80 10 client.16692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.770+0000 7f4f653f6b80 20 client.16692 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:06.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.770+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cccae56b70 0x55cccae58f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.770+0000 7f4f653f6b80 4 client.16692 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:06.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.770+0000 7f4f653f6b80 1 -- 172.21.15.111:0/1063972211 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cccad4c0a0 con 0x55cccae56b70 2023-10-11T16:40:06.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.770+0000 7f4f4a7fc700 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cccae56b70 0x55cccae58f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.771+0000 7f4f4a7fc700 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cccae56b70 0x55cccae58f80 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b84496a8dceed20e server_cookie=e34296f4c4827b6c in_seq=0 out_seq=0 2023-10-11T16:40:06.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.771+0000 7f4f2affd700 10 client.16692 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:06.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.771+0000 7f4f2affd700 1 -- 172.21.15.111:0/1063972211 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cccad4c0a0 con 0x55cccae56b70 2023-10-11T16:40:06.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.771+0000 7f4f2affd700 10 client.16692 handle_command_reply: tid=0 2023-10-11T16:40:06.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 client.16692 shutdown 2023-10-11T16:40:06.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 -- 172.21.15.111:0/1063972211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae3cf70 msgr2=0x55cccae49410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae3cf70 0x55cccae49410 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0116f0 tx=0x7f4f3c011ab0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 -- 172.21.15.111:0/1063972211 shutdown_connections 2023-10-11T16:40:06.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cccae56b70 0x55cccae58f80 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae41cc0 0x55cccae46cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae3cf70 0x55cccae49410 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f2affd700 0 client.16692 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:06.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/1063972211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae46690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 -- 172.21.15.111:0/1063972211 >> 172.21.15.111:0/1063972211 conn(0x55cccae2e360 msgr2=0x55cccae43170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:06.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 -- 172.21.15.111:0/1063972211 shutdown_connections 2023-10-11T16:40:06.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 1 -- 172.21.15.111:0/1063972211 wait complete. 2023-10-11T16:40:06.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.772+0000 7f4f653f6b80 20 client.16692 trim_cache size 0 max 16384 2023-10-11T16:40:06.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f653f6b80 1 Processor -- start 2023-10-11T16:40:06.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f653f6b80 1 -- start start 2023-10-11T16:40:06.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae33be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae4dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae48fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccae2ba10 con 0x55cccae41cc0 2023-10-11T16:40:06.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccae44c10 con 0x55cccae3d880 2023-10-11T16:40:06.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccae407e0 con 0x55cccae3cf70 2023-10-11T16:40:06.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f4affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae33be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f4affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae33be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:38832/0 (socket says 172.21.15.111:38832) 2023-10-11T16:40:06.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f4affd700 1 -- 172.21.15.111:0/4268249030 learned_addr learned my addr 172.21.15.111:0/4268249030 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:06.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f4b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae48fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.790+0000 7f4f4a7fc700 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae4dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f4affd700 1 -- 172.21.15.111:0/4268249030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 msgr2=0x55cccae4dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:06.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f4affd700 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae4dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f4affd700 1 -- 172.21.15.111:0/4268249030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 msgr2=0x55cccae48fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:06.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f4affd700 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae48fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f4affd700 1 -- 172.21.15.111:0/4268249030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cccad4fee0 con 0x55cccae3cf70 2023-10-11T16:40:06.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae48fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f4affd700 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae33be0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c000c00 tx=0x7f4f3c01bbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45d3d3e794ac05a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f327fc700 1 -- 172.21.15.111:0/4268249030 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c016030 con 0x55cccae3cf70 2023-10-11T16:40:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f327fc700 1 -- 172.21.15.111:0/4268249030 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f3c017040 con 0x55cccae3cf70 2023-10-11T16:40:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.791+0000 7f4f327fc700 1 -- 172.21.15.111:0/4268249030 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c0038e0 con 0x55cccae3cf70 2023-10-11T16:40:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 -- 172.21.15.111:0/4268249030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 msgr2=0x55cccae33be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae33be0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c000c00 tx=0x7f4f3c01bbe0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 -- 172.21.15.111:0/4268249030 shutdown_connections 2023-10-11T16:40:06.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae48fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3cf70 0x55cccae33be0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/4268249030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3d880 0x55cccae4dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 -- 172.21.15.111:0/4268249030 >> 172.21.15.111:0/4268249030 conn(0x55cccae2e360 msgr2=0x55cccae3fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 -- 172.21.15.111:0/4268249030 shutdown_connections 2023-10-11T16:40:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.792+0000 7f4f653f6b80 1 -- 172.21.15.111:0/4268249030 wait complete. 2023-10-11T16:40:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 1 -- start start 2023-10-11T16:40:06.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 1 Processor -- start 2023-10-11T16:40:06.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3cf70 0x55cccae4d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3d880 0x55cccae4ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae4bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccae4c480 con 0x55cccae41cc0 2023-10-11T16:40:06.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccae4c5f0 con 0x55cccae3cf70 2023-10-11T16:40:06.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cccae4c7b0 con 0x55cccae3d880 2023-10-11T16:40:06.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f653f6b80 10 client.0 _ll_register_callbacks cb 0x7f4f3c007bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:06.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f4b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae4bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f4b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae4bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42874/0 (socket says 172.21.15.111:42874) 2023-10-11T16:40:06.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.793+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/2550583964 learned_addr learned my addr 172.21.15.111:0/2550583964 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:06.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f4affd700 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3cf70 0x55cccae4d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f4a7fc700 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3d880 0x55cccae4ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/2550583964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3d880 msgr2=0x55cccae4ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:06.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3d880 0x55cccae4ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/2550583964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3cf70 msgr2=0x55cccae4d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:06.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3cf70 0x55cccae4d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f4b7fe700 1 -- 172.21.15.111:0/2550583964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cccac0b630 con 0x55cccae41cc0 2023-10-11T16:40:06.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f4b7fe700 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae4bdb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4f44097b50 tx=0x7f4f44009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1ae4b5b6d20ecd server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:06.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f2affd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:06.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f2affd700 1 -- 172.21.15.111:0/2550583964 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f440862b0 con 0x55cccae41cc0 2023-10-11T16:40:06.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f2affd700 1 -- 172.21.15.111:0/2550583964 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4f44011e70 con 0x55cccae41cc0 2023-10-11T16:40:06.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2550583964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f4409f680 con 0x55cccae41cc0 2023-10-11T16:40:06.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f2affd700 1 -- 172.21.15.111:0/2550583964 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f4409b650 con 0x55cccae41cc0 2023-10-11T16:40:06.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.794+0000 7f4f653f6b80 10 client.16698 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:06.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2550583964 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cccad50190 con 0x55cccae41cc0 2023-10-11T16:40:06.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f2affd700 1 -- 172.21.15.111:0/2550583964 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f4409f680 con 0x55cccae41cc0 2023-10-11T16:40:06.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f2affd700 1 -- 172.21.15.111:0/2550583964 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f4f4409ab40 con 0x55cccae41cc0 2023-10-11T16:40:06.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f653f6b80 10 client.16698 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:06.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f653f6b80 10 client.16698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:06.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f653f6b80 20 client.16698 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:06.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cccaeab8e0 0x55cccaeadcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:06.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f653f6b80 4 client.16698 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:40:06.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2550583964 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cccad58660 con 0x55cccaeab8e0 2023-10-11T16:40:06.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.795+0000 7f4f4affd700 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cccaeab8e0 0x55cccaeadcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:06.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f2affd700 10 client.16698 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:06.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f4affd700 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cccaeab8e0 0x55cccaeadcf0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80b5790a9167ac1a server_cookie=c8ab067105bf0dfd in_seq=0 out_seq=0 2023-10-11T16:40:06.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f2affd700 1 -- 172.21.15.111:0/2550583964 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cccad58660 con 0x55cccaeab8e0 2023-10-11T16:40:06.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f2affd700 10 client.16698 handle_command_reply: tid=0 2023-10-11T16:40:06.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f653f6b80 1 client.16698 shutdown 2023-10-11T16:40:06.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2550583964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 msgr2=0x55cccae4bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae4bdb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4f44097b50 tx=0x7f4f44009a50 comp rx=0 tx=0).stop 2023-10-11T16:40:06.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2550583964 shutdown_connections 2023-10-11T16:40:06.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.796+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55cccaeab8e0 0x55cccaeadcf0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cccae41cc0 0x55cccae4bdb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f2affd700 0 client.16698 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:06.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cccae3cf70 0x55cccae4d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f653f6b80 1 --2- 172.21.15.111:0/2550583964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cccae3d880 0x55cccae4ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2550583964 >> 172.21.15.111:0/2550583964 conn(0x55cccae2e360 msgr2=0x55cccae37960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:06.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2550583964 shutdown_connections 2023-10-11T16:40:06.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f653f6b80 1 -- 172.21.15.111:0/2550583964 wait complete. 2023-10-11T16:40:06.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f653f6b80 20 client.16698 trim_cache size 0 max 16384 2023-10-11T16:40:06.830 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:40:06.830 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:40:06.830 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:40:06.831 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:40:06.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 -- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f340a1340 msgr2=0x7f4f340a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f340a1340 0x7f4f340a37f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c197ad0 tx=0x7f4f4000b410 comp rx=0 tx=0).stop 2023-10-11T16:40:06.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 -- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 msgr2=0x7f4f4c19ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:06.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 0x7f4f4c19ae60 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009fd0 tx=0x7f4f4400ce50 comp rx=0 tx=0).stop 2023-10-11T16:40:06.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 -- 172.21.15.111:0/4233278714 shutdown_connections 2023-10-11T16:40:06.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4f340a1340 0x7f4f340a37f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f4c10a8b0 0x7f4f4c19ae60 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f4c101390 0x7f4f4c1964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 --2- 172.21.15.111:0/4233278714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c101ca0 0x7f4f4c196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:06.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.797+0000 7f4f519ff700 1 -- 172.21.15.111:0/4233278714 >> 172.21.15.111:0/4233278714 conn(0x7f4f4c0770b0 msgr2=0x7f4f4c102290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:06.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.798+0000 7f4f519ff700 1 -- 172.21.15.111:0/4233278714 shutdown_connections 2023-10-11T16:40:06.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:06.798+0000 7f4f519ff700 1 -- 172.21.15.111:0/4233278714 wait complete. 2023-10-11T16:40:06.836 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:40:06.836 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:40:07.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.034+0000 7f7740dff700 1 -- 172.21.15.111:0/4066354839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f773c100ef0 msgr2=0x7f773c104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.034+0000 7f7740dff700 1 --2- 172.21.15.111:0/4066354839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f773c100ef0 0x7f773c104e60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7728009990 tx=0x7f7728009ca0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.035+0000 7f7740dff700 1 -- 172.21.15.111:0/4066354839 shutdown_connections 2023-10-11T16:40:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.035+0000 7f7740dff700 1 --2- 172.21.15.111:0/4066354839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f773c105540 0x7f773c10d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.035+0000 7f7740dff700 1 --2- 172.21.15.111:0/4066354839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f773c100550 0x7f773c100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.035+0000 7f7740dff700 1 --2- 172.21.15.111:0/4066354839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f773c100ef0 0x7f773c104e60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.035+0000 7f7740dff700 1 -- 172.21.15.111:0/4066354839 >> 172.21.15.111:0/4066354839 conn(0x7f773c077020 msgr2=0x7f773c077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.035+0000 7f7740dff700 1 -- 172.21.15.111:0/4066354839 shutdown_connections 2023-10-11T16:40:07.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.035+0000 7f7740dff700 1 -- 172.21.15.111:0/4066354839 wait complete. 2023-10-11T16:40:07.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7740dff700 1 Processor -- start 2023-10-11T16:40:07.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7740dff700 1 -- start start 2023-10-11T16:40:07.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7740dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f773c100550 0x7f773c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7740dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f773c100ef0 0x7f773c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7740dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f773c105540 0x7f773c1a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7740dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c073b60 con 0x7f773c100ef0 2023-10-11T16:40:07.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7740dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c1a4ab0 con 0x7f773c100550 2023-10-11T16:40:07.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7740dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c1a4bf0 con 0x7f773c105540 2023-10-11T16:40:07.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.036+0000 7f7739d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f773c100ef0 0x7f773c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7739d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f773c100ef0 0x7f773c073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42892/0 (socket says 172.21.15.111:42892) 2023-10-11T16:40:07.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7739d9b700 1 -- 172.21.15.111:0/1394786873 learned_addr learned my addr 172.21.15.111:0/1394786873 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f773ad9d700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f773c105540 0x7f773c1a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f773a59c700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f773c100550 0x7f773c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7739d9b700 1 -- 172.21.15.111:0/1394786873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f773c105540 msgr2=0x7f773c1a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7739d9b700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f773c105540 0x7f773c1a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7739d9b700 1 -- 172.21.15.111:0/1394786873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f773c100550 msgr2=0x7f773c072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7739d9b700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f773c100550 0x7f773c072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7739d9b700 1 -- 172.21.15.111:0/1394786873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280095f0 con 0x7f773c100ef0 2023-10-11T16:40:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7739d9b700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f773c100ef0 0x7f773c073330 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7728000c00 tx=0x7f7728005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f77337fe700 1 -- 172.21.15.111:0/1394786873 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772801d070 con 0x7f773c100ef0 2023-10-11T16:40:07.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f77337fe700 1 -- 172.21.15.111:0/1394786873 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7728021410 con 0x7f773c100ef0 2023-10-11T16:40:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c110310 con 0x7f773c100ef0 2023-10-11T16:40:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.037+0000 7f77337fe700 1 -- 172.21.15.111:0/1394786873 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7728017650 con 0x7f773c100ef0 2023-10-11T16:40:07.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.038+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c110180 con 0x7f773c100ef0 2023-10-11T16:40:07.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.039+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c1104a0 con 0x7f773c100ef0 2023-10-11T16:40:07.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.040+0000 7f77337fe700 1 -- 172.21.15.111:0/1394786873 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f77280217f0 con 0x7f773c100ef0 2023-10-11T16:40:07.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.043+0000 7f77337fe700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77200a12c0 0x7f77200a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.043+0000 7f77337fe700 1 -- 172.21.15.111:0/1394786873 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f77280ce320 con 0x7f773c100ef0 2023-10-11T16:40:07.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.043+0000 7f773a59c700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77200a12c0 0x7f77200a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.043+0000 7f773a59c700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77200a12c0 0x7f77200a3770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f772c009c60 tx=0x7f772c009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.047+0000 7f77337fe700 1 -- 172.21.15.111:0/1394786873 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7728097460 con 0x7f773c100ef0 2023-10-11T16:40:07.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.291+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f773c067a60 con 0x7f773c100ef0 2023-10-11T16:40:07.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.292+0000 7f77337fe700 1 -- 172.21.15.111:0/1394786873 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10829 (secure 0 0 0) 0x7f772809aa80 con 0x7f773c100ef0 2023-10-11T16:40:07.293 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:40:07.294 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:36:18.517722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:40:07.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.294+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77200a12c0 msgr2=0x7f77200a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.294+0000 7f7740dff700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77200a12c0 0x7f77200a3770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f772c009c60 tx=0x7f772c009260 comp rx=0 tx=0).stop 2023-10-11T16:40:07.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.294+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f773c100ef0 msgr2=0x7f773c073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.294+0000 7f7740dff700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f773c100ef0 0x7f773c073330 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7728000c00 tx=0x7f7728005740 comp rx=0 tx=0).stop 2023-10-11T16:40:07.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.295+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 shutdown_connections 2023-10-11T16:40:07.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.295+0000 7f7740dff700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77200a12c0 0x7f77200a3770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.295+0000 7f7740dff700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f773c105540 0x7f773c1a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.295+0000 7f7740dff700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f773c100550 0x7f773c072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.295+0000 7f7740dff700 1 --2- 172.21.15.111:0/1394786873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f773c100ef0 0x7f773c073330 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.295+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 >> 172.21.15.111:0/1394786873 conn(0x7f773c077020 msgr2=0x7f773c0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.295+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 shutdown_connections 2023-10-11T16:40:07.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.295+0000 7f7740dff700 1 -- 172.21.15.111:0/1394786873 wait complete. 2023-10-11T16:40:07.299 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-10-11T16:40:07.315 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:40:07.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.513+0000 7f55c35ff700 1 -- 172.21.15.111:0/3070427516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 msgr2=0x7f55bc10d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.513+0000 7f55c35ff700 1 --2- 172.21.15.111:0/3070427516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 0x7f55bc10d1a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f55b400b0d0 tx=0x7f55b400b3e0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.514+0000 7f55c35ff700 1 -- 172.21.15.111:0/3070427516 shutdown_connections 2023-10-11T16:40:07.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.514+0000 7f55c35ff700 1 --2- 172.21.15.111:0/3070427516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 0x7f55bc10d1a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.514+0000 7f55c35ff700 1 --2- 172.21.15.111:0/3070427516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc100cc0 0x7f55bc101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.514+0000 7f55c35ff700 1 --2- 172.21.15.111:0/3070427516 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55bc101660 0x7f55bc104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.514+0000 7f55c35ff700 1 -- 172.21.15.111:0/3070427516 >> 172.21.15.111:0/3070427516 conn(0x7f55bc077560 msgr2=0x7f55bc077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.514+0000 7f55c35ff700 1 -- 172.21.15.111:0/3070427516 shutdown_connections 2023-10-11T16:40:07.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.514+0000 7f55c35ff700 1 -- 172.21.15.111:0/3070427516 wait complete. 2023-10-11T16:40:07.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.515+0000 7f55c35ff700 1 Processor -- start 2023-10-11T16:40:07.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.515+0000 7f55c35ff700 1 -- start start 2023-10-11T16:40:07.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.515+0000 7f55c35ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc100cc0 0x7f55bc196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.515+0000 7f55c35ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55bc101660 0x7f55bc196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.515+0000 7f55c35ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 0x7f55bc19afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.515+0000 7f55c35ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc19b620 con 0x7f55bc105620 2023-10-11T16:40:07.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.515+0000 7f55c35ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc19b790 con 0x7f55bc101660 2023-10-11T16:40:07.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.515+0000 7f55c35ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc19b950 con 0x7f55bc100cc0 2023-10-11T16:40:07.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 0x7f55bc19afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 0x7f55bc19afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42910/0 (socket says 172.21.15.111:42910) 2023-10-11T16:40:07.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 -- 172.21.15.111:0/508652060 learned_addr learned my addr 172.21.15.111:0/508652060 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c139b700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc100cc0 0x7f55bc196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c0b9a700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55bc101660 0x7f55bc196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 -- 172.21.15.111:0/508652060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc100cc0 msgr2=0x7f55bc196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc100cc0 0x7f55bc196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 -- 172.21.15.111:0/508652060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55bc101660 msgr2=0x7f55bc196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55bc101660 0x7f55bc196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 -- 172.21.15.111:0/508652060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4009d00 con 0x7f55bc105620 2023-10-11T16:40:07.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 0x7f55bc19afe0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f55b4007820 tx=0x7f55b4007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55b27fc700 1 -- 172.21.15.111:0/508652060 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b400e040 con 0x7f55bc105620 2023-10-11T16:40:07.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55b27fc700 1 -- 172.21.15.111:0/508652060 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55b400bcb0 con 0x7f55bc105620 2023-10-11T16:40:07.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55c35ff700 1 -- 172.21.15.111:0/508652060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc1103e0 con 0x7f55bc105620 2023-10-11T16:40:07.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.516+0000 7f55b27fc700 1 -- 172.21.15.111:0/508652060 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b40058a0 con 0x7f55bc105620 2023-10-11T16:40:07.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.517+0000 7f55c35ff700 1 -- 172.21.15.111:0/508652060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc110250 con 0x7f55bc105620 2023-10-11T16:40:07.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.519+0000 7f55b27fc700 1 -- 172.21.15.111:0/508652060 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f55b4004a50 con 0x7f55bc105620 2023-10-11T16:40:07.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.521+0000 7f55d6f4bb80 1 Processor -- start 2023-10-11T16:40:07.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.521+0000 7f55b27fc700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f55a80a12c0 0x7f55a80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.521+0000 7f55b27fc700 1 -- 172.21.15.111:0/508652060 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f55b40ce240 con 0x7f55bc105620 2023-10-11T16:40:07.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.521+0000 7f55d6f4bb80 1 -- start start 2023-10-11T16:40:07.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.521+0000 7f55c0b9a700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f55a80a12c0 0x7f55a80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.521+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 0x55d10f857340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.521+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f85b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f85c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f758060 con 0x55d10f85bcc0 2023-10-11T16:40:07.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f7aa8a0 con 0x55d10f857880 2023-10-11T16:40:07.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f7a3d30 con 0x55d10f856f70 2023-10-11T16:40:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f85c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c0b9a700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f55a80a12c0 0x7f55a80a3770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f55bc197c50 tx=0x7f55ac009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f85c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42918/0 (socket says 172.21.15.111:42918) 2023-10-11T16:40:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 -- 172.21.15.111:0/2118312729 learned_addr learned my addr 172.21.15.111:0/2118312729 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c139b700 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 0x55d10f857340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c0b9a700 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f85b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 -- 172.21.15.111:0/2118312729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 msgr2=0x55d10f857340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 0x55d10f857340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 -- 172.21.15.111:0/2118312729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 msgr2=0x55d10f85b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f85b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 -- 172.21.15.111:0/2118312729 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d10f625630 con 0x55d10f85bcc0 2023-10-11T16:40:07.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.522+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f85c110 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f55b40a2dc0 tx=0x7f55b4097340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6acf469af135b27 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55a77fe700 1 -- 172.21.15.111:0/2118312729 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b409a890 con 0x55d10f85bcc0 2023-10-11T16:40:07.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55a77fe700 1 -- 172.21.15.111:0/2118312729 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55b400be20 con 0x55d10f85bcc0 2023-10-11T16:40:07.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55a77fe700 1 -- 172.21.15.111:0/2118312729 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b40a2e60 con 0x55d10f85bcc0 2023-10-11T16:40:07.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/2118312729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 msgr2=0x55d10f85c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f85c110 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f55b40a2dc0 tx=0x7f55b4097340 comp rx=0 tx=0).stop 2023-10-11T16:40:07.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/2118312729 shutdown_connections 2023-10-11T16:40:07.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f85c110 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 0x55d10f857340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/2118312729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f85b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.523+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/2118312729 >> 172.21.15.111:0/2118312729 conn(0x55d10f848360 msgr2=0x55d10f84db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.524+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/2118312729 shutdown_connections 2023-10-11T16:40:07.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.524+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/2118312729 wait complete. 2023-10-11T16:40:07.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.524+0000 7f55d6f4bb80 1 -- start start 2023-10-11T16:40:07.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.524+0000 7f55d6f4bb80 1 Processor -- start 2023-10-11T16:40:07.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f856f70 0x55d10f8634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f8606f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f860de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f7458e0 con 0x55d10f85bcc0 2023-10-11T16:40:07.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f745580 con 0x55d10f856f70 2023-10-11T16:40:07.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f627920 con 0x55d10f857880 2023-10-11T16:40:07.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55d6f4bb80 10 client.0 _ll_register_callbacks cb 0x55d10f763780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:07.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c1b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f860de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c139b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f856f70 0x55d10f8634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c1b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f860de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42928/0 (socket says 172.21.15.111:42928) 2023-10-11T16:40:07.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c1b9c700 1 -- 172.21.15.111:0/3304609279 learned_addr learned my addr 172.21.15.111:0/3304609279 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c0b9a700 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f8606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c1b9c700 1 -- 172.21.15.111:0/3304609279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 msgr2=0x55d10f8606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f8606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c1b9c700 1 -- 172.21.15.111:0/3304609279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f856f70 msgr2=0x55d10f8634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f856f70 0x55d10f8634e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.525+0000 7f55c1b9c700 1 -- 172.21.15.111:0/3304609279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc110250 con 0x55d10f85bcc0 2023-10-11T16:40:07.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.526+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f860de0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f55b40979a0 tx=0x7f55b40ce170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44c50c8eb699f5b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.526+0000 7f55a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.526+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3304609279 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b40b92c0 con 0x55d10f85bcc0 2023-10-11T16:40:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.526+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3304609279 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f55b409af30 con 0x55d10f85bcc0 2023-10-11T16:40:07.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.526+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3304609279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d10f845b20 con 0x55d10f85bcc0 2023-10-11T16:40:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.526+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3304609279 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55b40a32f0 con 0x55d10f85bcc0 2023-10-11T16:40:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.526+0000 7f55d6f4bb80 10 client.16716 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.526+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3304609279 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d10f76a190 con 0x55d10f85bcc0 2023-10-11T16:40:07.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.527+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3304609279 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d10f845b20 con 0x55d10f85bcc0 2023-10-11T16:40:07.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.527+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3304609279 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f55b40a2e60 con 0x55d10f85bcc0 2023-10-11T16:40:07.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.527+0000 7f55d6f4bb80 10 client.16716 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:07.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.527+0000 7f55d6f4bb80 10 client.16716 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:40:07.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.527+0000 7f55d6f4bb80 20 client.16716 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:07.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.527+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55d10f870640 0x55d10f872a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.527+0000 7f55d6f4bb80 4 client.16716 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:07.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.527+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3304609279 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d10f7660a0 con 0x55d10f870640 2023-10-11T16:40:07.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.528+0000 7f55c139b700 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55d10f870640 0x55d10f872a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.528+0000 7f55c139b700 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55d10f870640 0x55d10f872a20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b193ea5d85b25ade server_cookie=c0cbfa979c354c8b in_seq=0 out_seq=0 2023-10-11T16:40:07.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.528+0000 7f55a4ff9700 10 client.16716 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:40:07.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3304609279 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d10f7660a0 con 0x55d10f870640 2023-10-11T16:40:07.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55a4ff9700 10 client.16716 handle_command_reply: tid=0 2023-10-11T16:40:07.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 client.16716 shutdown 2023-10-11T16:40:07.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3304609279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 msgr2=0x55d10f860de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f860de0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f55b40979a0 tx=0x7f55b40ce170 comp rx=0 tx=0).stop 2023-10-11T16:40:07.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3304609279 shutdown_connections 2023-10-11T16:40:07.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55d10f870640 0x55d10f872a20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f860de0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f856f70 0x55d10f8634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3304609279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f8606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55a4ff9700 0 client.16716 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:40:07.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3304609279 >> 172.21.15.111:0/3304609279 conn(0x55d10f848360 msgr2=0x55d10f85d170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3304609279 shutdown_connections 2023-10-11T16:40:07.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3304609279 wait complete. 2023-10-11T16:40:07.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.529+0000 7f55d6f4bb80 20 client.16716 trim_cache size 0 max 16384 2023-10-11T16:40:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55d6f4bb80 1 Processor -- start 2023-10-11T16:40:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55d6f4bb80 1 -- start start 2023-10-11T16:40:07.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f856f70 0x55d10f85f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f860cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f85bcc0 0x55d10f867bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f8613e0 con 0x55d10f856f70 2023-10-11T16:40:07.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f86ca80 con 0x55d10f85bcc0 2023-10-11T16:40:07.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f85a820 con 0x55d10f857880 2023-10-11T16:40:07.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55c139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f856f70 0x55d10f85f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55c139b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f856f70 0x55d10f85f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42944/0 (socket says 172.21.15.111:42944) 2023-10-11T16:40:07.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.543+0000 7f55c139b700 1 -- 172.21.15.111:0/3550674413 learned_addr learned my addr 172.21.15.111:0/3550674413 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c0b9a700 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f860cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f85bcc0 0x55d10f867bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c139b700 1 -- 172.21.15.111:0/3550674413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 msgr2=0x55d10f860cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c139b700 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f860cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c139b700 1 -- 172.21.15.111:0/3550674413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f85bcc0 msgr2=0x55d10f867bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c139b700 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f85bcc0 0x55d10f867bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c139b700 1 -- 172.21.15.111:0/3550674413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d10f849b00 con 0x55d10f856f70 2023-10-11T16:40:07.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c0b9a700 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f860cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:07.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55c139b700 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f856f70 0x55d10f85f980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f55b8000c00 tx=0x7f55b8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5100e58adff4d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55a7fff700 1 -- 172.21.15.111:0/3550674413 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8017070 con 0x55d10f856f70 2023-10-11T16:40:07.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55a7fff700 1 -- 172.21.15.111:0/3550674413 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55b8018040 con 0x55d10f856f70 2023-10-11T16:40:07.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.544+0000 7f55a7fff700 1 -- 172.21.15.111:0/3550674413 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b801daf0 con 0x55d10f856f70 2023-10-11T16:40:07.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3550674413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f856f70 msgr2=0x55d10f85f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f856f70 0x55d10f85f980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f55b8000c00 tx=0x7f55b8009260 comp rx=0 tx=0).stop 2023-10-11T16:40:07.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3550674413 shutdown_connections 2023-10-11T16:40:07.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f85bcc0 0x55d10f867bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f856f70 0x55d10f85f980 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3550674413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f857880 0x55d10f860cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3550674413 >> 172.21.15.111:0/3550674413 conn(0x55d10f848360 msgr2=0x55d10f86c370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3550674413 shutdown_connections 2023-10-11T16:40:07.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3550674413 wait complete. 2023-10-11T16:40:07.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.545+0000 7f55d6f4bb80 1 -- start start 2023-10-11T16:40:07.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55d6f4bb80 1 Processor -- start 2023-10-11T16:40:07.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 0x55d10f865da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f8662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55d6f4bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f8c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f866a20 con 0x55d10f85bcc0 2023-10-11T16:40:07.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f8c39e0 con 0x55d10f857880 2023-10-11T16:40:07.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55d6f4bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d10f8c3b70 con 0x55d10f856f70 2023-10-11T16:40:07.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55d6f4bb80 10 client.0 _ll_register_callbacks cb 0x7f55ac0099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:07.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55c1b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f8c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55c0b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f8662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55c1b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f8c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42950/0 (socket says 172.21.15.111:42950) 2023-10-11T16:40:07.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.546+0000 7f55c1b9c700 1 -- 172.21.15.111:0/3388105378 learned_addr learned my addr 172.21.15.111:0/3388105378 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55c139b700 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 0x55d10f865da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55c1b9c700 1 -- 172.21.15.111:0/3388105378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 msgr2=0x55d10f865da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 0x55d10f865da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55c1b9c700 1 -- 172.21.15.111:0/3388105378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 msgr2=0x55d10f8662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f8662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55c1b9c700 1 -- 172.21.15.111:0/3388105378 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b8010430 con 0x55d10f85bcc0 2023-10-11T16:40:07.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55c0b9a700 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f8662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:07.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55c1b9c700 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f8c34a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f55b40077f0 tx=0x7f55b40095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e68fb7163051a56 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:07.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3388105378 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b409b830 con 0x55d10f85bcc0 2023-10-11T16:40:07.584 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3388105378 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f55b409ef60 con 0x55d10f85bcc0 2023-10-11T16:40:07.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3388105378 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f55b400be20 con 0x55d10f85bcc0 2023-10-11T16:40:07.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3388105378 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55b40b92c0 con 0x55d10f85bcc0 2023-10-11T16:40:07.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55d6f4bb80 10 client.16722 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:07.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.547+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3388105378 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d10f5d1de0 con 0x55d10f85bcc0 2023-10-11T16:40:07.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3388105378 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55b400be20 con 0x55d10f85bcc0 2023-10-11T16:40:07.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3388105378 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f55b4024c40 con 0x55d10f85bcc0 2023-10-11T16:40:07.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55d6f4bb80 10 client.16722 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:07.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55d6f4bb80 10 client.16722 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:40:07.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55d6f4bb80 20 client.16722 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:07.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55d10f8c4da0 0x55d10f8c7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55d6f4bb80 4 client.16722 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:40:07.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3388105378 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d10f772660 con 0x55d10f8c4da0 2023-10-11T16:40:07.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.548+0000 7f55c139b700 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55d10f8c4da0 0x55d10f8c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55a4ff9700 10 client.16722 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:40:07.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55c139b700 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55d10f8c4da0 0x55d10f8c7180 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15eb7fde588229f4 server_cookie=60d5b8955b3d45a4 in_seq=0 out_seq=0 2023-10-11T16:40:07.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55a4ff9700 1 -- 172.21.15.111:0/3388105378 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d10f772660 con 0x55d10f8c4da0 2023-10-11T16:40:07.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55a4ff9700 10 client.16722 handle_command_reply: tid=0 2023-10-11T16:40:07.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55d6f4bb80 1 client.16722 shutdown 2023-10-11T16:40:07.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3388105378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 msgr2=0x55d10f8c34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f8c34a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f55b40077f0 tx=0x7f55b40095e0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3388105378 shutdown_connections 2023-10-11T16:40:07.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55d10f8c4da0 0x55d10f8c7180 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d10f85bcc0 0x55d10f8c34a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d10f856f70 0x55d10f865da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55d6f4bb80 1 --2- 172.21.15.111:0/3388105378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d10f857880 0x55d10f8662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.549+0000 7f55a4ff9700 0 client.16722 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:40:07.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3388105378 >> 172.21.15.111:0/3388105378 conn(0x55d10f848360 msgr2=0x55d10f86cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3388105378 shutdown_connections 2023-10-11T16:40:07.593 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55d6f4bb80 1 -- 172.21.15.111:0/3388105378 wait complete. 2023-10-11T16:40:07.594 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55d6f4bb80 20 client.16722 trim_cache size 0 max 16384 2023-10-11T16:40:07.594 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:40:07.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 -- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f55a80a12c0 msgr2=0x7f55a80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f55a80a12c0 0x7f55a80a3770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f55bc197c50 tx=0x7f55ac009260 comp rx=0 tx=0).stop 2023-10-11T16:40:07.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 -- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 msgr2=0x7f55bc19afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 0x7f55bc19afe0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f55b4007820 tx=0x7f55b4007850 comp rx=0 tx=0).stop 2023-10-11T16:40:07.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 -- 172.21.15.111:0/508652060 shutdown_connections 2023-10-11T16:40:07.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f55a80a12c0 0x7f55a80a3770 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55bc105620 0x7f55bc19afe0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc100cc0 0x7f55bc196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 --2- 172.21.15.111:0/508652060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f55bc101660 0x7f55bc196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.550+0000 7f55c35ff700 1 -- 172.21.15.111:0/508652060 >> 172.21.15.111:0/508652060 conn(0x7f55bc077560 msgr2=0x7f55bc103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.551+0000 7f55c35ff700 1 -- 172.21.15.111:0/508652060 shutdown_connections 2023-10-11T16:40:07.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.551+0000 7f55c35ff700 1 -- 172.21.15.111:0/508652060 wait complete. 2023-10-11T16:40:07.600 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:40:07.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 -- 172.21.15.111:0/3877214866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768101400 msgr2=0x7fc76810d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/3877214866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768101400 0x7fc76810d0a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc76000b0a0 tx=0x7fc76000b3b0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 -- 172.21.15.111:0/3877214866 shutdown_connections 2023-10-11T16:40:07.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/3877214866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768101400 0x7fc76810d0a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/3877214866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7681063f0 0x7fc7681067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/3877214866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc768100a50 0x7fc768100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 -- 172.21.15.111:0/3877214866 >> 172.21.15.111:0/3877214866 conn(0x7fc768077590 msgr2=0x7fc768077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 -- 172.21.15.111:0/3877214866 shutdown_connections 2023-10-11T16:40:07.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.803+0000 7fc76f9ff700 1 -- 172.21.15.111:0/3877214866 wait complete. 2023-10-11T16:40:07.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.804+0000 7fc76f9ff700 1 Processor -- start 2023-10-11T16:40:07.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.804+0000 7fc76f9ff700 1 -- start start 2023-10-11T16:40:07.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76f9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768100a50 0x7fc76819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76f9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc768101400 0x7fc76819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76f9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7681063f0 0x7fc7681985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76f9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc768198c60 con 0x7fc768100a50 2023-10-11T16:40:07.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76f9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc768198dd0 con 0x7fc7681063f0 2023-10-11T16:40:07.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76f9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc768198f90 con 0x7fc768101400 2023-10-11T16:40:07.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76cf9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc768101400 0x7fc76819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76d79b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768100a50 0x7fc76819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76df9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7681063f0 0x7fc7681985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76d79b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768100a50 0x7fc76819e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42960/0 (socket says 172.21.15.111:42960) 2023-10-11T16:40:07.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76cf9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc768101400 0x7fc76819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:38944/0 (socket says 172.21.15.111:38944) 2023-10-11T16:40:07.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76d79b700 1 -- 172.21.15.111:0/2096027178 learned_addr learned my addr 172.21.15.111:0/2096027178 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76d79b700 1 -- 172.21.15.111:0/2096027178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc768101400 msgr2=0x7fc76819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76d79b700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc768101400 0x7fc76819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76d79b700 1 -- 172.21.15.111:0/2096027178 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7681063f0 msgr2=0x7fc7681985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76d79b700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7681063f0 0x7fc7681985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.805+0000 7fc76d79b700 1 -- 172.21.15.111:0/2096027178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760009d00 con 0x7fc768100a50 2023-10-11T16:40:07.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.806+0000 7fc76d79b700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768100a50 0x7fc76819e580 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc76400e9a0 tx=0x7fc76400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.806+0000 7fc75e7fc700 1 -- 172.21.15.111:0/2096027178 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764009790 con 0x7fc768100a50 2023-10-11T16:40:07.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.806+0000 7fc76f9ff700 1 -- 172.21.15.111:0/2096027178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc768110410 con 0x7fc768100a50 2023-10-11T16:40:07.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.806+0000 7fc75e7fc700 1 -- 172.21.15.111:0/2096027178 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc764010430 con 0x7fc768100a50 2023-10-11T16:40:07.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.806+0000 7fc75e7fc700 1 -- 172.21.15.111:0/2096027178 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764018680 con 0x7fc768100a50 2023-10-11T16:40:07.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.806+0000 7fc76f9ff700 1 -- 172.21.15.111:0/2096027178 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc768110280 con 0x7fc768100a50 2023-10-11T16:40:07.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.809+0000 7fc75e7fc700 1 -- 172.21.15.111:0/2096027178 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fc7640105a0 con 0x7fc768100a50 2023-10-11T16:40:07.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.811+0000 7fc7833f2b80 1 Processor -- start 2023-10-11T16:40:07.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.811+0000 7fc7833f2b80 1 -- start start 2023-10-11T16:40:07.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.811+0000 7fc75e7fc700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc7540a12c0 0x7fc7540a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.811+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948112b60 0x556948112f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.811+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948117370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.811+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x556948117d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.811+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x556948018060 con 0x5569481178b0 2023-10-11T16:40:07.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55694806a8a0 con 0x556948113470 2023-10-11T16:40:07.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556948063d30 con 0x556948112b60 2023-10-11T16:40:07.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc75e7fc700 1 -- 172.21.15.111:0/2096027178 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fc764014070 con 0x7fc768100a50 2023-10-11T16:40:07.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76df9c700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc7540a12c0 0x7fc7540a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76d79b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948112b60 0x556948112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76d79b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948112b60 0x556948112f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:38948/0 (socket says 172.21.15.111:38948) 2023-10-11T16:40:07.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76d79b700 1 -- 172.21.15.111:0/1600683306 learned_addr learned my addr 172.21.15.111:0/1600683306 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948117370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76df9c700 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x556948117d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76cf9a700 1 -- 172.21.15.111:0/1600683306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948112b60 msgr2=0x556948112f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948112b60 0x556948112f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76cf9a700 1 -- 172.21.15.111:0/1600683306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 msgr2=0x556948117d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x556948117d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76cf9a700 1 -- 172.21.15.111:0/1600683306 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556947ee5630 con 0x556948113470 2023-10-11T16:40:07.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.812+0000 7fc76d79b700 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948112b60 0x556948112f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:07.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.813+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948117370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc758009c80 tx=0x7fc758009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d02ac0da0d1f9fe server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.813+0000 7fc76df9c700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc7540a12c0 0x7fc7540a3770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc76000b780 tx=0x7fc760009500 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.813+0000 7fc7537fe700 1 -- 172.21.15.111:0/1600683306 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758011cb0 con 0x556948113470 2023-10-11T16:40:07.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.813+0000 7fc7537fe700 1 -- 172.21.15.111:0/1600683306 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc758011e10 con 0x556948113470 2023-10-11T16:40:07.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.813+0000 7fc7537fe700 1 -- 172.21.15.111:0/1600683306 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758005280 con 0x556948113470 2023-10-11T16:40:07.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1600683306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 msgr2=0x556948117370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948117370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc758009c80 tx=0x7fc758009f90 comp rx=0 tx=0).stop 2023-10-11T16:40:07.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1600683306 shutdown_connections 2023-10-11T16:40:07.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x556948117d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948112b60 0x556948112f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/1600683306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948117370 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1600683306 >> 172.21.15.111:0/1600683306 conn(0x556948108360 msgr2=0x55694810db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1600683306 shutdown_connections 2023-10-11T16:40:07.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1600683306 wait complete. 2023-10-11T16:40:07.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.814+0000 7fc7833f2b80 1 -- start start 2023-10-11T16:40:07.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc7833f2b80 1 Processor -- start 2023-10-11T16:40:07.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948112b60 0x55694811efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948113470 0x55694811c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x55694811c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569480058e0 con 0x5569481178b0 2023-10-11T16:40:07.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556948005580 con 0x556948112b60 2023-10-11T16:40:07.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556947ee7920 con 0x556948113470 2023-10-11T16:40:07.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc7833f2b80 10 client.0 _ll_register_callbacks cb 0x556948023780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:07.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc76df9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x55694811c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc76df9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x55694811c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42978/0 (socket says 172.21.15.111:42978) 2023-10-11T16:40:07.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc76df9c700 1 -- 172.21.15.111:0/105137153 learned_addr learned my addr 172.21.15.111:0/105137153 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc76d79b700 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948112b60 0x55694811efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.815+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948113470 0x55694811c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc76df9c700 1 -- 172.21.15.111:0/105137153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948113470 msgr2=0x55694811c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc76df9c700 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948113470 0x55694811c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc76df9c700 1 -- 172.21.15.111:0/105137153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948112b60 msgr2=0x55694811efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc76df9c700 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948112b60 0x55694811efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc76df9c700 1 -- 172.21.15.111:0/105137153 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7580098e0 con 0x5569481178b0 2023-10-11T16:40:07.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc76d79b700 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948112b60 0x55694811efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:07.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc76df9c700 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x55694811c800 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc76000b0a0 tx=0x7fc760006230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2006d6d741eac34c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc750ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:07.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc750ff9700 1 -- 172.21.15.111:0/105137153 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760026070 con 0x5569481178b0 2023-10-11T16:40:07.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc750ff9700 1 -- 172.21.15.111:0/105137153 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fc7600218c0 con 0x5569481178b0 2023-10-11T16:40:07.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc7833f2b80 1 -- 172.21.15.111:0/105137153 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556948105b20 con 0x5569481178b0 2023-10-11T16:40:07.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc750ff9700 1 -- 172.21.15.111:0/105137153 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc76000e040 con 0x5569481178b0 2023-10-11T16:40:07.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc7833f2b80 10 client.16734 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.816+0000 7fc7833f2b80 1 -- 172.21.15.111:0/105137153 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55694802a190 con 0x5569481178b0 2023-10-11T16:40:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc750ff9700 1 -- 172.21.15.111:0/105137153 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556948105b20 con 0x5569481178b0 2023-10-11T16:40:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc750ff9700 1 -- 172.21.15.111:0/105137153 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc76002ca00 con 0x5569481178b0 2023-10-11T16:40:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc7833f2b80 10 client.16734 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:07.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc7833f2b80 10 client.16734 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:40:07.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc7833f2b80 20 client.16734 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:07.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55694812c8d0 0x55694812ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc7833f2b80 4 client.16734 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:07.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc7833f2b80 1 -- 172.21.15.111:0/105137153 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569480260a0 con 0x55694812c8d0 2023-10-11T16:40:07.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.817+0000 7fc76d79b700 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55694812c8d0 0x55694812ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.818+0000 7fc750ff9700 10 client.16734 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:40:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.818+0000 7fc76d79b700 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55694812c8d0 0x55694812ece0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d28aa4d64e70d1f5 server_cookie=9720270123432fd9 in_seq=0 out_seq=0 2023-10-11T16:40:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc750ff9700 1 -- 172.21.15.111:0/105137153 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5569480260a0 con 0x55694812c8d0 2023-10-11T16:40:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc750ff9700 10 client.16734 handle_command_reply: tid=0 2023-10-11T16:40:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 client.16734 shutdown 2023-10-11T16:40:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 -- 172.21.15.111:0/105137153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 msgr2=0x55694811c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x55694811c800 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc76000b0a0 tx=0x7fc760006230 comp rx=0 tx=0).stop 2023-10-11T16:40:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 -- 172.21.15.111:0/105137153 shutdown_connections 2023-10-11T16:40:07.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55694812c8d0 0x55694812ece0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5569481178b0 0x55694811c800 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948112b60 0x55694811efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/105137153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556948113470 0x55694811c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc750ff9700 0 client.16734 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:40:07.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 -- 172.21.15.111:0/105137153 >> 172.21.15.111:0/105137153 conn(0x556948108360 msgr2=0x55694810c550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 -- 172.21.15.111:0/105137153 shutdown_connections 2023-10-11T16:40:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 1 -- 172.21.15.111:0/105137153 wait complete. 2023-10-11T16:40:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.819+0000 7fc7833f2b80 20 client.16734 trim_cache size 0 max 16384 2023-10-11T16:40:07.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc7833f2b80 1 Processor -- start 2023-10-11T16:40:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc7833f2b80 1 -- start start 2023-10-11T16:40:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x556948128b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x55694811eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 0x55694810d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x556948105750 con 0x556948112b60 2023-10-11T16:40:07.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556948113bf0 con 0x556948113470 2023-10-11T16:40:07.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x556948107f60 con 0x5569481178b0 2023-10-11T16:40:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc76d79b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x556948128b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc76d79b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x556948128b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42984/0 (socket says 172.21.15.111:42984) 2023-10-11T16:40:07.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x55694811eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc76d79b700 1 -- 172.21.15.111:0/1838295103 learned_addr learned my addr 172.21.15.111:0/1838295103 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.833+0000 7fc76df9c700 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 0x55694810d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc76cf9a700 1 -- 172.21.15.111:0/1838295103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 msgr2=0x55694810d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 0x55694810d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc76cf9a700 1 -- 172.21.15.111:0/1838295103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 msgr2=0x556948128b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x556948128b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc76cf9a700 1 -- 172.21.15.111:0/1838295103 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556948109b00 con 0x556948113470 2023-10-11T16:40:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc76d79b700 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x556948128b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x55694811eb40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc758011720 tx=0x7fc758009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3d85b3ff33f8a4a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc753fff700 1 -- 172.21.15.111:0/1838295103 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758011c30 con 0x556948113470 2023-10-11T16:40:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc753fff700 1 -- 172.21.15.111:0/1838295103 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc758003b40 con 0x556948113470 2023-10-11T16:40:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.834+0000 7fc753fff700 1 -- 172.21.15.111:0/1838295103 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758023c00 con 0x556948113470 2023-10-11T16:40:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1838295103 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 msgr2=0x55694811eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x55694811eb40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc758011720 tx=0x7fc758009770 comp rx=0 tx=0).stop 2023-10-11T16:40:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1838295103 shutdown_connections 2023-10-11T16:40:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 0x55694810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x556948128b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/1838295103 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x55694811eb40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1838295103 >> 172.21.15.111:0/1838295103 conn(0x556948108360 msgr2=0x556948119c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1838295103 shutdown_connections 2023-10-11T16:40:07.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 -- 172.21.15.111:0/1838295103 wait complete. 2023-10-11T16:40:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 -- start start 2023-10-11T16:40:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.835+0000 7fc7833f2b80 1 Processor -- start 2023-10-11T16:40:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x5569481239c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948120ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc7833f2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 0x556948121220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x556948121900 con 0x556948112b60 2023-10-11T16:40:07.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556948121a70 con 0x556948113470 2023-10-11T16:40:07.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc7833f2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55694817f790 con 0x5569481178b0 2023-10-11T16:40:07.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc7833f2b80 10 client.0 _ll_register_callbacks cb 0x7fc760003d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:07.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc76d79b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x5569481239c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc76cf9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948120ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc76d79b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x5569481239c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42986/0 (socket says 172.21.15.111:42986) 2023-10-11T16:40:07.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc76d79b700 1 -- 172.21.15.111:0/2224392710 learned_addr learned my addr 172.21.15.111:0/2224392710 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:07.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc76cf9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948120ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:38566/0 (socket says 172.21.15.111:38566) 2023-10-11T16:40:07.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.836+0000 7fc76df9c700 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 0x556948121220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc76d79b700 1 -- 172.21.15.111:0/2224392710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 msgr2=0x556948121220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc76d79b700 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 0x556948121220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc76d79b700 1 -- 172.21.15.111:0/2224392710 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 msgr2=0x556948120ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:07.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc76d79b700 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948120ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc76d79b700 1 -- 172.21.15.111:0/2224392710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc768110410 con 0x556948112b60 2023-10-11T16:40:07.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948120ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:07.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc76d79b700 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x5569481239c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc764000c00 tx=0x7fc7640cd1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ed19a874aafd60 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:07.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc750ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:07.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc750ff9700 1 -- 172.21.15.111:0/2224392710 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764099f20 con 0x556948112b60 2023-10-11T16:40:07.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc750ff9700 1 -- 172.21.15.111:0/2224392710 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fc76409df50 con 0x556948112b60 2023-10-11T16:40:07.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc7833f2b80 1 -- 172.21.15.111:0/2224392710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc758011e10 con 0x556948112b60 2023-10-11T16:40:07.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc750ff9700 1 -- 172.21.15.111:0/2224392710 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7640a2370 con 0x556948112b60 2023-10-11T16:40:07.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc7833f2b80 10 client.16740 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:07.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc7833f2b80 1 -- 172.21.15.111:0/2224392710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7fc75801d040 con 0x556948112b60 2023-10-11T16:40:07.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.837+0000 7fc750ff9700 1 -- 172.21.15.111:0/2224392710 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc758011e10 con 0x556948112b60 2023-10-11T16:40:07.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc750ff9700 1 -- 172.21.15.111:0/2224392710 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc7640a1ee0 con 0x556948112b60 2023-10-11T16:40:07.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc7833f2b80 10 client.16740 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:07.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc7833f2b80 10 client.16740 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:40:07.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc7833f2b80 20 client.16740 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:07.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x556948180d10 0x556948183120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:07.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc7833f2b80 4 client.16740 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:40:07.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc7833f2b80 1 -- 172.21.15.111:0/2224392710 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556948032660 con 0x556948180d10 2023-10-11T16:40:07.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x556948180d10 0x556948183120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:07.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc750ff9700 10 client.16740 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:40:07.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.838+0000 7fc76cf9a700 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x556948180d10 0x556948183120 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee696a5edbeb2326 server_cookie=3ae0921e6fb09802 in_seq=0 out_seq=0 2023-10-11T16:40:07.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc750ff9700 1 -- 172.21.15.111:0/2224392710 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556948032660 con 0x556948180d10 2023-10-11T16:40:07.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc750ff9700 10 client.16740 handle_command_reply: tid=0 2023-10-11T16:40:07.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 client.16740 shutdown 2023-10-11T16:40:07.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 -- 172.21.15.111:0/2224392710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 msgr2=0x5569481239c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x5569481239c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc764000c00 tx=0x7fc7640cd1f0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 -- 172.21.15.111:0/2224392710 shutdown_connections 2023-10-11T16:40:07.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x556948180d10 0x556948183120 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5569481178b0 0x556948121220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x556948112b60 0x5569481239c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 --2- 172.21.15.111:0/2224392710 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556948113470 0x556948120ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 -- 172.21.15.111:0/2224392710 >> 172.21.15.111:0/2224392710 conn(0x556948108360 msgr2=0x556948119c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc750ff9700 0 client.16740 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:40:07.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 -- 172.21.15.111:0/2224392710 shutdown_connections 2023-10-11T16:40:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 1 -- 172.21.15.111:0/2224392710 wait complete. 2023-10-11T16:40:07.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.839+0000 7fc7833f2b80 20 client.16740 trim_cache size 0 max 16384 2023-10-11T16:40:07.880 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:40:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 -- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc7540a12c0 msgr2=0x7fc7540a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc7540a12c0 0x7fc7540a3770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc76000b780 tx=0x7fc760009500 comp rx=0 tx=0).stop 2023-10-11T16:40:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 -- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768100a50 msgr2=0x7fc76819e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:07.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768100a50 0x7fc76819e580 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc76400e9a0 tx=0x7fc76400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 -- 172.21.15.111:0/2096027178 shutdown_connections 2023-10-11T16:40:07.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc7540a12c0 0x7fc7540a3770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc7681063f0 0x7fc7681985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc768100a50 0x7fc76819e580 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 --2- 172.21.15.111:0/2096027178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc768101400 0x7fc76819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 -- 172.21.15.111:0/2096027178 >> 172.21.15.111:0/2096027178 conn(0x7fc768077590 msgr2=0x7fc76810b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:07.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 -- 172.21.15.111:0/2096027178 shutdown_connections 2023-10-11T16:40:07.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:07.840+0000 7fc76f9ff700 1 -- 172.21.15.111:0/2096027178 wait complete. 2023-10-11T16:40:07.884 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:07 smithi111 ceph-mon[98527]: pgmap v229: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-10-11T16:40:07.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:07 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1394786873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:40:07.886 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:40:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:07 smithi150 ceph-mon[132217]: pgmap v229: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-10-11T16:40:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:07 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1394786873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:40:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:07 smithi130 ceph-mon[131859]: pgmap v229: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-10-11T16:40:08.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:07 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1394786873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:40:08.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 -- 172.21.15.111:0/2233741400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8105620 msgr2=0x7f94a810d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2233741400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8105620 0x7f94a810d1a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f94a00099c0 tx=0x7f94a0009cd0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 -- 172.21.15.111:0/2233741400 shutdown_connections 2023-10-11T16:40:08.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2233741400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8105620 0x7f94a810d1a0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2233741400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94a8100cc0 0x7f94a8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2233741400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a8101660 0x7f94a8104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 -- 172.21.15.111:0/2233741400 >> 172.21.15.111:0/2233741400 conn(0x7f94a8077560 msgr2=0x7f94a8077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:08.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 -- 172.21.15.111:0/2233741400 shutdown_connections 2023-10-11T16:40:08.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.086+0000 7f94af1ff700 1 -- 172.21.15.111:0/2233741400 wait complete. 2023-10-11T16:40:08.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.087+0000 7f94af1ff700 1 Processor -- start 2023-10-11T16:40:08.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94af1ff700 1 -- start start 2023-10-11T16:40:08.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94af1ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94a8100cc0 0x7f94a81964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94af1ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8101660 0x7f94a8196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94af1ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a8105620 0x7f94a819ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94af1ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a819b4b0 con 0x7f94a8101660 2023-10-11T16:40:08.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94af1ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a819b620 con 0x7f94a8105620 2023-10-11T16:40:08.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94af1ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a819b7e0 con 0x7f94a8100cc0 2023-10-11T16:40:08.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94a7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8101660 0x7f94a8196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94a7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8101660 0x7f94a8196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43002/0 (socket says 172.21.15.111:43002) 2023-10-11T16:40:08.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94a7fff700 1 -- 172.21.15.111:0/2634499264 learned_addr learned my addr 172.21.15.111:0/2634499264 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:08.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94acf9b700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94a8100cc0 0x7f94a81964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.088+0000 7f94ad79c700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a8105620 0x7f94a819ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a7fff700 1 -- 172.21.15.111:0/2634499264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94a8100cc0 msgr2=0x7f94a81964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a7fff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94a8100cc0 0x7f94a81964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a7fff700 1 -- 172.21.15.111:0/2634499264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a8105620 msgr2=0x7f94a819ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a7fff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a8105620 0x7f94a819ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a7fff700 1 -- 172.21.15.111:0/2634499264 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a00095f0 con 0x7f94a8101660 2023-10-11T16:40:08.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a7fff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8101660 0x7f94a8196a10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f949400e910 tx=0x7f949400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:08.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a5ffb700 1 -- 172.21.15.111:0/2634499264 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494004a30 con 0x7f94a8101660 2023-10-11T16:40:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a5ffb700 1 -- 172.21.15.111:0/2634499264 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94940054e0 con 0x7f94a8101660 2023-10-11T16:40:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94af1ff700 1 -- 172.21.15.111:0/2634499264 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a81103e0 con 0x7f94a8101660 2023-10-11T16:40:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94a5ffb700 1 -- 172.21.15.111:0/2634499264 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949400cc40 con 0x7f94a8101660 2023-10-11T16:40:08.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.089+0000 7f94af1ff700 1 -- 172.21.15.111:0/2634499264 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a8110250 con 0x7f94a8101660 2023-10-11T16:40:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.092+0000 7f94a5ffb700 1 -- 172.21.15.111:0/2634499264 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f9494005000 con 0x7f94a8101660 2023-10-11T16:40:08.103 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.093+0000 7f94c2a96b80 1 Processor -- start 2023-10-11T16:40:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94c2a96b80 1 -- start start 2023-10-11T16:40:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94a5ffb700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f94980a12c0 0x7f94980a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4ce7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4ceb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4cebcc0 0x557be4cec110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4be8060 con 0x557be4ce6f70 2023-10-11T16:40:08.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4c3a8a0 con 0x557be4ce7880 2023-10-11T16:40:08.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4c33d30 con 0x557be4cebcc0 2023-10-11T16:40:08.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94a5ffb700 1 -- 172.21.15.111:0/2634499264 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f9494014070 con 0x7f94a8101660 2023-10-11T16:40:08.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94acf9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4ce7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94acf9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4ce7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43014/0 (socket says 172.21.15.111:43014) 2023-10-11T16:40:08.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94acf9b700 1 -- 172.21.15.111:0/926062929 learned_addr learned my addr 172.21.15.111:0/926062929 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:08.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.094+0000 7f94ad79c700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f94980a12c0 0x7f94980a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94a7fff700 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4ceb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94ad79c700 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4cebcc0 0x557be4cec110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94acf9b700 1 -- 172.21.15.111:0/926062929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4cebcc0 msgr2=0x557be4cec110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:08.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94acf9b700 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4cebcc0 0x557be4cec110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94acf9b700 1 -- 172.21.15.111:0/926062929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 msgr2=0x557be4ceb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:08.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94acf9b700 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4ceb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94acf9b700 1 -- 172.21.15.111:0/926062929 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557be4ab5630 con 0x557be4ce6f70 2023-10-11T16:40:08.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94ad79c700 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4cebcc0 0x557be4cec110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:40:08.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94a7fff700 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4ceb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:08.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94acf9b700 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4ce7340 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f949c009990 tx=0x7f949c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d8f2ef75097f0a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:08.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f94ad79c700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f94980a12c0 0x7f94980a3770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f94a00058e0 tx=0x7f94a0005c40 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:08.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f9492ffd700 1 -- 172.21.15.111:0/926062929 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c003b90 con 0x557be4ce6f70 2023-10-11T16:40:08.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f9492ffd700 1 -- 172.21.15.111:0/926062929 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f949c00f040 con 0x557be4ce6f70 2023-10-11T16:40:08.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.095+0000 7f9492ffd700 1 -- 172.21.15.111:0/926062929 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0157e0 con 0x557be4ce6f70 2023-10-11T16:40:08.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 -- 172.21.15.111:0/926062929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 msgr2=0x557be4ce7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4ce7340 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f949c009990 tx=0x7f949c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 -- 172.21.15.111:0/926062929 shutdown_connections 2023-10-11T16:40:08.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4cebcc0 0x557be4cec110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4ce7340 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/926062929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4ceb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 -- 172.21.15.111:0/926062929 >> 172.21.15.111:0/926062929 conn(0x557be4cd8360 msgr2=0x557be4cddb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:08.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 -- 172.21.15.111:0/926062929 shutdown_connections 2023-10-11T16:40:08.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.096+0000 7f94c2a96b80 1 -- 172.21.15.111:0/926062929 wait complete. 2023-10-11T16:40:08.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.097+0000 7f94c2a96b80 1 -- start start 2023-10-11T16:40:08.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.097+0000 7f94c2a96b80 1 Processor -- start 2023-10-11T16:40:08.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.097+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.097+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.097+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.097+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4bd58e0 con 0x557be4cebcc0 2023-10-11T16:40:08.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.097+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4bd5580 con 0x557be4ce7880 2023-10-11T16:40:08.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.097+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4ab7920 con 0x557be4ce6f70 2023-10-11T16:40:08.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94c2a96b80 10 client.0 _ll_register_callbacks cb 0x557be4bf3780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:08.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94acf9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43030/0 (socket says 172.21.15.111:43030) 2023-10-11T16:40:08.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 -- 172.21.15.111:0/705900505 learned_addr learned my addr 172.21.15.111:0/705900505 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:08.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94a7fff700 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 -- 172.21.15.111:0/705900505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 msgr2=0x557be4cf3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:08.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 -- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 msgr2=0x557be4cf0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:08.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 -- 172.21.15.111:0/705900505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949c003cf0 con 0x557be4cebcc0 2023-10-11T16:40:08.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94a7fff700 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:08.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f94ad79c700 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf0be0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f94a00099c0 tx=0x7f94a0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3be576bbbc50351 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.098+0000 7f9487fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f9487fff700 1 -- 172.21.15.111:0/705900505 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a002b070 con 0x557be4cebcc0 2023-10-11T16:40:08.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f9487fff700 1 -- 172.21.15.111:0/705900505 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f94a000f460 con 0x557be4cebcc0 2023-10-11T16:40:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f94c2a96b80 1 -- 172.21.15.111:0/705900505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557be4cd5b20 con 0x557be4cebcc0 2023-10-11T16:40:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f9487fff700 1 -- 172.21.15.111:0/705900505 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a0026e60 con 0x557be4cebcc0 2023-10-11T16:40:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f94c2a96b80 10 client.16752 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f94c2a96b80 1 -- 172.21.15.111:0/705900505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557be4bfa190 con 0x557be4cebcc0 2023-10-11T16:40:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f9487fff700 1 -- 172.21.15.111:0/705900505 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557be4cd5b20 con 0x557be4cebcc0 2023-10-11T16:40:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f9487fff700 1 -- 172.21.15.111:0/705900505 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f94a0031770 con 0x557be4cebcc0 2023-10-11T16:40:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f94c2a96b80 10 client.16752 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f94c2a96b80 10 client.16752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f94c2a96b80 20 client.16752 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557be4d00bf0 0x557be4d03000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.099+0000 7f94c2a96b80 4 client.16752 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:08.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.100+0000 7f94c2a96b80 1 -- 172.21.15.111:0/705900505 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557be4bf60a0 con 0x557be4d00bf0 2023-10-11T16:40:08.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.100+0000 7f94acf9b700 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557be4d00bf0 0x557be4d03000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.101+0000 7f94acf9b700 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557be4d00bf0 0x557be4d03000 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48668002e123dabb server_cookie=3749fa5b567fecf6 in_seq=0 out_seq=0 2023-10-11T16:40:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.101+0000 7f9487fff700 10 client.16752 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.101+0000 7f9487fff700 1 -- 172.21.15.111:0/705900505 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557be4bf60a0 con 0x557be4d00bf0 2023-10-11T16:40:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.101+0000 7f9487fff700 10 client.16752 handle_command_reply: tid=0 2023-10-11T16:40:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.101+0000 7f94c2a96b80 1 client.16752 shutdown 2023-10-11T16:40:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 -- 172.21.15.111:0/705900505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 msgr2=0x557be4cf0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf0be0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f94a00099c0 tx=0x7f94a0009310 comp rx=0 tx=0).stop 2023-10-11T16:40:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 -- 172.21.15.111:0/705900505 shutdown_connections 2023-10-11T16:40:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557be4d00bf0 0x557be4d03000 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf0be0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f9487fff700 0 client.16752 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:08.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/705900505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 -- 172.21.15.111:0/705900505 >> 172.21.15.111:0/705900505 conn(0x557be4cd8360 msgr2=0x557be4ced170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:08.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 -- 172.21.15.111:0/705900505 shutdown_connections 2023-10-11T16:40:08.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 1 -- 172.21.15.111:0/705900505 wait complete. 2023-10-11T16:40:08.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.102+0000 7f94c2a96b80 20 client.16752 trim_cache size 0 max 16384 2023-10-11T16:40:08.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.115+0000 7f94c2a96b80 1 Processor -- start 2023-10-11T16:40:08.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.115+0000 7f94c2a96b80 1 -- start start 2023-10-11T16:40:08.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4cf7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce7880 0x557be4cf2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4cebcc0 0x557be4cf05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4cd5710 con 0x557be4ce6f70 2023-10-11T16:40:08.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4cfe230 con 0x557be4cebcc0 2023-10-11T16:40:08.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4ce7fd0 con 0x557be4ce7880 2023-10-11T16:40:08.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94acf9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4cf7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94acf9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4cf7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43034/0 (socket says 172.21.15.111:43034) 2023-10-11T16:40:08.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94acf9b700 1 -- 172.21.15.111:0/4261745146 learned_addr learned my addr 172.21.15.111:0/4261745146 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:08.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94ad79c700 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4cebcc0 0x557be4cf05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94a7fff700 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce7880 0x557be4cf2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94acf9b700 1 -- 172.21.15.111:0/4261745146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce7880 msgr2=0x557be4cf2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:08.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94acf9b700 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce7880 0x557be4cf2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94acf9b700 1 -- 172.21.15.111:0/4261745146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4cebcc0 msgr2=0x557be4cf05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:08.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94acf9b700 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4cebcc0 0x557be4cf05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94acf9b700 1 -- 172.21.15.111:0/4261745146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557be4cec360 con 0x557be4ce6f70 2023-10-11T16:40:08.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.116+0000 7f94a7fff700 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce7880 0x557be4cf2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:08.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94acf9b700 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4cf7db0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f949c009990 tx=0x7f949c0094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c210cd11b1ce931c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:08.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94937fe700 1 -- 172.21.15.111:0/4261745146 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c006000 con 0x557be4ce6f70 2023-10-11T16:40:08.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94937fe700 1 -- 172.21.15.111:0/4261745146 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f949c004e20 con 0x557be4ce6f70 2023-10-11T16:40:08.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94937fe700 1 -- 172.21.15.111:0/4261745146 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c007680 con 0x557be4ce6f70 2023-10-11T16:40:08.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4261745146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 msgr2=0x557be4cf7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4cf7db0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f949c009990 tx=0x7f949c0094f0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4261745146 shutdown_connections 2023-10-11T16:40:08.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4cebcc0 0x557be4cf05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4ce6f70 0x557be4cf7db0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4261745146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce7880 0x557be4cf2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.117+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4261745146 >> 172.21.15.111:0/4261745146 conn(0x557be4cd8360 msgr2=0x557be4ce9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:08.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.118+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4261745146 shutdown_connections 2023-10-11T16:40:08.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.118+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4261745146 wait complete. 2023-10-11T16:40:08.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.118+0000 7f94c2a96b80 1 -- start start 2023-10-11T16:40:08.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.118+0000 7f94c2a96b80 1 Processor -- start 2023-10-11T16:40:08.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94c2a96b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4d53c50 con 0x557be4cebcc0 2023-10-11T16:40:08.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4d53dc0 con 0x557be4ce7880 2023-10-11T16:40:08.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94c2a96b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557be4d53f80 con 0x557be4ce6f70 2023-10-11T16:40:08.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94c2a96b80 10 client.0 _ll_register_callbacks cb 0x7f94a000bee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:08.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94acf9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94ad79c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94acf9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:39024/0 (socket says 172.21.15.111:39024) 2023-10-11T16:40:08.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94acf9b700 1 -- 172.21.15.111:0/4057955876 learned_addr learned my addr 172.21.15.111:0/4057955876 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:08.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94a7fff700 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94acf9b700 1 -- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 msgr2=0x557be4cf42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:08.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94acf9b700 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94acf9b700 1 -- 172.21.15.111:0/4057955876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 msgr2=0x557be4cf4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:08.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94acf9b700 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94acf9b700 1 -- 172.21.15.111:0/4057955876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557be4ab5630 con 0x557be4ce6f70 2023-10-11T16:40:08.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94a7fff700 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:08.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.119+0000 7f94ad79c700 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:08.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.120+0000 7f94acf9b700 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f949c00d040 tx=0x7f949c006000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daca73e7d7d8a9fa server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:08.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.120+0000 7f9487fff700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:40:08.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.120+0000 7f9487fff700 1 -- 172.21.15.111:0/4057955876 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c025560 con 0x557be4ce6f70 2023-10-11T16:40:08.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.120+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4057955876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f949c00f040 con 0x557be4ce6f70 2023-10-11T16:40:08.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.120+0000 7f9487fff700 1 -- 172.21.15.111:0/4057955876 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f949c023040 con 0x557be4ce6f70 2023-10-11T16:40:08.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.120+0000 7f9487fff700 1 -- 172.21.15.111:0/4057955876 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f949c004960 con 0x557be4ce6f70 2023-10-11T16:40:08.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.120+0000 7f94c2a96b80 10 client.26881 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:08.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.120+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4057955876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x557be4a61de0 con 0x557be4ce6f70 2023-10-11T16:40:08.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f9487fff700 1 -- 172.21.15.111:0/4057955876 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f949c00f040 con 0x557be4ce6f70 2023-10-11T16:40:08.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f9487fff700 1 -- 172.21.15.111:0/4057955876 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f949c036640 con 0x557be4ce6f70 2023-10-11T16:40:08.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f94c2a96b80 10 client.26881 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:08.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f94c2a96b80 10 client.26881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:08.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f94c2a96b80 20 client.26881 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:08.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557be4d551e0 0x557be4d575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:08.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f94c2a96b80 4 client.26881 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:40:08.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4057955876 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557be4c02660 con 0x557be4d551e0 2023-10-11T16:40:08.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.121+0000 7f94a7fff700 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557be4d551e0 0x557be4d575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:08.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f9487fff700 10 client.26881 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94a7fff700 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557be4d551e0 0x557be4d575c0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6dd9c8423298282 server_cookie=e3d1b09427fc829a in_seq=0 out_seq=0 2023-10-11T16:40:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f9487fff700 1 -- 172.21.15.111:0/4057955876 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557be4c02660 con 0x557be4d551e0 2023-10-11T16:40:08.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f9487fff700 10 client.26881 handle_command_reply: tid=0 2023-10-11T16:40:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 client.26881 shutdown 2023-10-11T16:40:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4057955876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 msgr2=0x557be4cf3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f949c00d040 tx=0x7f949c006000 comp rx=0 tx=0).stop 2023-10-11T16:40:08.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4057955876 shutdown_connections 2023-10-11T16:40:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557be4d551e0 0x557be4d575c0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557be4cebcc0 0x557be4cf4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f9487fff700 0 client.26881 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:08.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557be4ce6f70 0x557be4cf3d90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 --2- 172.21.15.111:0/4057955876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557be4ce7880 0x557be4cf42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4057955876 >> 172.21.15.111:0/4057955876 conn(0x557be4cd8360 msgr2=0x557be4cf05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:08.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4057955876 shutdown_connections 2023-10-11T16:40:08.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.122+0000 7f94c2a96b80 1 -- 172.21.15.111:0/4057955876 wait complete. 2023-10-11T16:40:08.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94c2a96b80 20 client.26881 trim_cache size 0 max 16384 2023-10-11T16:40:08.163 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:40:08.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 -- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f94980a12c0 msgr2=0x7f94980a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f94980a12c0 0x7f94980a3770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f94a00058e0 tx=0x7f94a0005c40 comp rx=0 tx=0).stop 2023-10-11T16:40:08.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 -- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8101660 msgr2=0x7f94a8196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:08.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8101660 0x7f94a8196a10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f949400e910 tx=0x7f949400ec20 comp rx=0 tx=0).stop 2023-10-11T16:40:08.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 -- 172.21.15.111:0/2634499264 shutdown_connections 2023-10-11T16:40:08.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f94980a12c0 0x7f94980a3770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a8105620 0x7f94a819ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94a8100cc0 0x7f94a81964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 --2- 172.21.15.111:0/2634499264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a8101660 0x7f94a8196a10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:08.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 -- 172.21.15.111:0/2634499264 >> 172.21.15.111:0/2634499264 conn(0x7f94a8077560 msgr2=0x7f94a8103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:08.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 -- 172.21.15.111:0/2634499264 shutdown_connections 2023-10-11T16:40:08.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:08.123+0000 7f94af1ff700 1 -- 172.21.15.111:0/2634499264 wait complete. 2023-10-11T16:40:09.168 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:40:09.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:40:09.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 -- 172.21.15.111:0/3691606751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fbc074b00 msgr2=0x7f4fbc073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/3691606751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fbc074b00 0x7f4fbc073260 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4fac0098f0 tx=0x7f4fac009c00 comp rx=0 tx=0).stop 2023-10-11T16:40:09.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 -- 172.21.15.111:0/3691606751 shutdown_connections 2023-10-11T16:40:09.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/3691606751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fbc10cec0 0x7f4fbc10f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/3691606751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fbc0746a0 0x7f4fbc072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/3691606751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fbc074b00 0x7f4fbc073260 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 -- 172.21.15.111:0/3691606751 >> 172.21.15.111:0/3691606751 conn(0x7f4fbc077590 msgr2=0x7f4fbc077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 -- 172.21.15.111:0/3691606751 shutdown_connections 2023-10-11T16:40:09.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.381+0000 7f4fc43ff700 1 -- 172.21.15.111:0/3691606751 wait complete. 2023-10-11T16:40:09.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.382+0000 7f4fc43ff700 1 Processor -- start 2023-10-11T16:40:09.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc43ff700 1 -- start start 2023-10-11T16:40:09.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc43ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fbc0746a0 0x7f4fbc19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc43ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fbc074b00 0x7f4fbc19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc43ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fbc10cec0 0x7f4fbc198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc43ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc198d30 con 0x7f4fbc10cec0 2023-10-11T16:40:09.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc43ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc198ea0 con 0x7f4fbc074b00 2023-10-11T16:40:09.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc43ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fbc199060 con 0x7f4fbc0746a0 2023-10-11T16:40:09.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fbc10cec0 0x7f4fbc198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fbc10cec0 0x7f4fbc198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43050/0 (socket says 172.21.15.111:43050) 2023-10-11T16:40:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc299c700 1 -- 172.21.15.111:0/1040012410 learned_addr learned my addr 172.21.15.111:0/1040012410 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc199a700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fbc074b00 0x7f4fbc19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.383+0000 7f4fc219b700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fbc0746a0 0x7f4fbc19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fc299c700 1 -- 172.21.15.111:0/1040012410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fbc0746a0 msgr2=0x7f4fbc19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fc299c700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fbc0746a0 0x7f4fbc19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fc299c700 1 -- 172.21.15.111:0/1040012410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fbc074b00 msgr2=0x7f4fbc19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fc299c700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fbc074b00 0x7f4fbc19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fc299c700 1 -- 172.21.15.111:0/1040012410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac009520 con 0x7f4fbc10cec0 2023-10-11T16:40:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fc299c700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fbc10cec0 0x7f4fbc198560 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400ef90 tx=0x7f4fb400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fb37fe700 1 -- 172.21.15.111:0/1040012410 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4018070 con 0x7f4fbc10cec0 2023-10-11T16:40:09.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fc43ff700 1 -- 172.21.15.111:0/1040012410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fbc1125a0 con 0x7f4fbc10cec0 2023-10-11T16:40:09.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fb37fe700 1 -- 172.21.15.111:0/1040012410 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fb400f040 con 0x7f4fbc10cec0 2023-10-11T16:40:09.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fb37fe700 1 -- 172.21.15.111:0/1040012410 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb400cc90 con 0x7f4fbc10cec0 2023-10-11T16:40:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.384+0000 7f4fc43ff700 1 -- 172.21.15.111:0/1040012410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fbc112410 con 0x7f4fbc10cec0 2023-10-11T16:40:09.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.387+0000 7f4fb37fe700 1 -- 172.21.15.111:0/1040012410 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f4fb4003a00 con 0x7f4fbc10cec0 2023-10-11T16:40:09.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.388+0000 7f4fd7d70b80 1 Processor -- start 2023-10-11T16:40:09.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.388+0000 7f4fd7d70b80 1 -- start start 2023-10-11T16:40:09.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fb37fe700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4fa80a1340 0x7f4fa80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cebe0 0x55ecc80cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fb37fe700 1 -- 172.21.15.111:0/1040012410 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4fb40ccbf0 con 0x7f4fbc10cec0 2023-10-11T16:40:09.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80d3930 0x55ecc80d3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc7fd40e0 con 0x55ecc80cebe0 2023-10-11T16:40:09.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc8026920 con 0x55ecc80cf4f0 2023-10-11T16:40:09.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc801fdb0 con 0x55ecc80d3930 2023-10-11T16:40:09.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cebe0 0x55ecc80cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc199a700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4fa80a1340 0x7f4fa80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cebe0 0x55ecc80cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43066/0 (socket says 172.21.15.111:43066) 2023-10-11T16:40:09.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc219b700 1 -- 172.21.15.111:0/1781578305 learned_addr learned my addr 172.21.15.111:0/1781578305 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc199a700 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc299c700 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80d3930 0x55ecc80d3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc219b700 1 -- 172.21.15.111:0/1781578305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80d3930 msgr2=0x55ecc80d3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc219b700 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80d3930 0x55ecc80d3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc219b700 1 -- 172.21.15.111:0/1781578305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cf4f0 msgr2=0x55ecc80d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc219b700 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc219b700 1 -- 172.21.15.111:0/1781578305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecc7ea16b0 con 0x55ecc80cebe0 2023-10-11T16:40:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.389+0000 7f4fc299c700 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80d3930 0x55ecc80d3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.390+0000 7f4fc219b700 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cebe0 0x55ecc80cefb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009d80 tx=0x7f4fb8011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864c8529fa455171 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.390+0000 7f4fc199a700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4fa80a1340 0x7f4fa80a37f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4fac005e90 tx=0x7f4fac019040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.390+0000 7f4fb0bfe700 1 -- 172.21.15.111:0/1781578305 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8004230 con 0x55ecc80cebe0 2023-10-11T16:40:09.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.390+0000 7f4fb0bfe700 1 -- 172.21.15.111:0/1781578305 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fb8004390 con 0x55ecc80cebe0 2023-10-11T16:40:09.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.390+0000 7f4fb0bfe700 1 -- 172.21.15.111:0/1781578305 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8019860 con 0x55ecc80cebe0 2023-10-11T16:40:09.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1781578305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cebe0 msgr2=0x55ecc80cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cebe0 0x55ecc80cefb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009d80 tx=0x7f4fb8011720 comp rx=0 tx=0).stop 2023-10-11T16:40:09.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1781578305 shutdown_connections 2023-10-11T16:40:09.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80d3930 0x55ecc80d3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cebe0 0x55ecc80cefb0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/1781578305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1781578305 >> 172.21.15.111:0/1781578305 conn(0x55ecc80c43e0 msgr2=0x55ecc80c9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1781578305 shutdown_connections 2023-10-11T16:40:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1781578305 wait complete. 2023-10-11T16:40:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.391+0000 7f4fd7d70b80 1 -- start start 2023-10-11T16:40:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fd7d70b80 1 Processor -- start 2023-10-11T16:40:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80dafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80d3930 0x55ecc80d8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc7fc1960 con 0x55ecc80d3930 2023-10-11T16:40:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc7fc1600 con 0x55ecc80cebe0 2023-10-11T16:40:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc7ea39a0 con 0x55ecc80cf4f0 2023-10-11T16:40:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fd7d70b80 10 client.0 _ll_register_callbacks cb 0x55ecc7fdf800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fc299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80d3930 0x55ecc80d8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fc219b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80dafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fc219b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80dafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:38650/0 (socket says 172.21.15.111:38650) 2023-10-11T16:40:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fc299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80d3930 0x55ecc80d8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43080/0 (socket says 172.21.15.111:43080) 2023-10-11T16:40:09.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fc219b700 1 -- 172.21.15.111:0/3455629658 learned_addr learned my addr 172.21.15.111:0/3455629658 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.392+0000 7f4fc199a700 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fc219b700 1 -- 172.21.15.111:0/3455629658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 msgr2=0x55ecc80d8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fc219b700 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fc219b700 1 -- 172.21.15.111:0/3455629658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80d3930 msgr2=0x55ecc80d8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fc219b700 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80d3930 0x55ecc80d8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fc219b700 1 -- 172.21.15.111:0/3455629658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb80099e0 con 0x55ecc80cebe0 2023-10-11T16:40:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fc199a700 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fc219b700 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80dafe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8011b20 tx=0x7f4fb8003930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba9b8d038a229bd server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fa5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:40:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/3455629658 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8005450 con 0x55ecc80cebe0 2023-10-11T16:40:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/3455629658 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4fb8004500 con 0x55ecc80cebe0 2023-10-11T16:40:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.393+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/3455629658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ecc80c1ba0 con 0x55ecc80cebe0 2023-10-11T16:40:09.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/3455629658 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fb8003da0 con 0x55ecc80cebe0 2023-10-11T16:40:09.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fd7d70b80 10 client.26885 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:09.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/3455629658 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ecc7fe6210 con 0x55ecc80cebe0 2023-10-11T16:40:09.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/3455629658 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ecc80c1ba0 con 0x55ecc80cebe0 2023-10-11T16:40:09.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/3455629658 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f4fb8021040 con 0x55ecc80cebe0 2023-10-11T16:40:09.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fd7d70b80 10 client.26885 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:09.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fd7d70b80 10 client.26885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:09.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fd7d70b80 20 client.26885 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:09.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecc80e8830 0x55ecc80eac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fd7d70b80 4 client.26885 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:09.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/3455629658 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecc7fe2120 con 0x55ecc80e8830 2023-10-11T16:40:09.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.394+0000 7f4fc199a700 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecc80e8830 0x55ecc80eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.395+0000 7f4fa5ffb700 10 client.26885 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:09.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.395+0000 7f4fc199a700 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecc80e8830 0x55ecc80eac40 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38ed4de2e3746a4f server_cookie=794481fc9c53cec5 in_seq=0 out_seq=0 2023-10-11T16:40:09.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/3455629658 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ecc7fe2120 con 0x55ecc80e8830 2023-10-11T16:40:09.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fa5ffb700 10 client.26885 handle_command_reply: tid=0 2023-10-11T16:40:09.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 client.26885 shutdown 2023-10-11T16:40:09.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/3455629658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 msgr2=0x55ecc80dafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80dafe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8011b20 tx=0x7f4fb8003930 comp rx=0 tx=0).stop 2023-10-11T16:40:09.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/3455629658 shutdown_connections 2023-10-11T16:40:09.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecc80e8830 0x55ecc80eac40 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80d3930 0x55ecc80d8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80dafe0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fa5ffb700 0 client.26885 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:09.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/3455629658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80d8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.427 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/3455629658 >> 172.21.15.111:0/3455629658 conn(0x55ecc80c43e0 msgr2=0x55ecc80c8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/3455629658 shutdown_connections 2023-10-11T16:40:09.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.396+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/3455629658 wait complete. 2023-10-11T16:40:09.428 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.397+0000 7f4fd7d70b80 20 client.26885 trim_cache size 0 max 16384 2023-10-11T16:40:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fd7d70b80 1 Processor -- start 2023-10-11T16:40:09.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fd7d70b80 1 -- start start 2023-10-11T16:40:09.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80c43e0 0x55ecc80d1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80d1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc80d1cd0 con 0x55ecc80c43e0 2023-10-11T16:40:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc80d24d0 con 0x55ecc80cebe0 2023-10-11T16:40:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc80d9810 con 0x55ecc80cf4f0 2023-10-11T16:40:09.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fc219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80c43e0 0x55ecc80d1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fc219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80c43e0 0x55ecc80d1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43096/0 (socket says 172.21.15.111:43096) 2023-10-11T16:40:09.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fc219b700 1 -- 172.21.15.111:0/1747874352 learned_addr learned my addr 172.21.15.111:0/1747874352 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.414+0000 7f4fc199a700 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fc299c700 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fc219b700 1 -- 172.21.15.111:0/1747874352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 msgr2=0x55ecc80c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fc219b700 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fc219b700 1 -- 172.21.15.111:0/1747874352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 msgr2=0x55ecc80d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fc219b700 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fc219b700 1 -- 172.21.15.111:0/1747874352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecc7fe5f60 con 0x55ecc80c43e0 2023-10-11T16:40:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fc219b700 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80c43e0 0x55ecc80d1220 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8000c00 tx=0x7f4fb8011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b8106ed99e4469 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fb13ff700 1 -- 172.21.15.111:0/1747874352 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb80185d0 con 0x55ecc80c43e0 2023-10-11T16:40:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fb13ff700 1 -- 172.21.15.111:0/1747874352 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fb80237e0 con 0x55ecc80c43e0 2023-10-11T16:40:09.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.415+0000 7f4fb13ff700 1 -- 172.21.15.111:0/1747874352 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8005650 con 0x55ecc80c43e0 2023-10-11T16:40:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1747874352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80c43e0 msgr2=0x55ecc80d1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80c43e0 0x55ecc80d1220 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8000c00 tx=0x7f4fb8011720 comp rx=0 tx=0).stop 2023-10-11T16:40:09.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1747874352 shutdown_connections 2023-10-11T16:40:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80cf4f0 0x55ecc80c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80c43e0 0x55ecc80d1220 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/1747874352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1747874352 >> 172.21.15.111:0/1747874352 conn(0x55ecc80d6070 msgr2=0x55ecc80e4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1747874352 shutdown_connections 2023-10-11T16:40:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/1747874352 wait complete. 2023-10-11T16:40:09.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.416+0000 7f4fd7d70b80 1 -- start start 2023-10-11T16:40:09.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fd7d70b80 1 Processor -- start 2023-10-11T16:40:09.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80c43e0 0x55ecc80de0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80de5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fd7d70b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cf4f0 0x55ecc80deb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc813c700 con 0x55ecc80cf4f0 2023-10-11T16:40:09.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc813c840 con 0x55ecc80cebe0 2023-10-11T16:40:09.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fd7d70b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecc813c980 con 0x55ecc80c43e0 2023-10-11T16:40:09.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fd7d70b80 10 client.0 _ll_register_callbacks cb 0x7f4fb8009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:09.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fc299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cf4f0 0x55ecc80deb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fc219b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80c43e0 0x55ecc80de0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fc299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cf4f0 0x55ecc80deb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43112/0 (socket says 172.21.15.111:43112) 2023-10-11T16:40:09.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fc299c700 1 -- 172.21.15.111:0/2884804335 learned_addr learned my addr 172.21.15.111:0/2884804335 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.417+0000 7f4fc199a700 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80de5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fc299c700 1 -- 172.21.15.111:0/2884804335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80c43e0 msgr2=0x55ecc80de0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fc299c700 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80c43e0 0x55ecc80de0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fc299c700 1 -- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 msgr2=0x55ecc80de5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fc299c700 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80de5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fc299c700 1 -- 172.21.15.111:0/2884804335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecc7fe6210 con 0x55ecc80cf4f0 2023-10-11T16:40:09.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fc199a700 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80de5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:09.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fc299c700 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cf4f0 0x55ecc80deb20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4fb401c040 tx=0x7f4fb4099240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5b2fd425d6f749 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fa5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:09.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/2884804335 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb409d910 con 0x55ecc80cf4f0 2023-10-11T16:40:09.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/2884804335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4fb8004390 con 0x55ecc80cf4f0 2023-10-11T16:40:09.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/2884804335 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f4fb40a1ca0 con 0x55ecc80cf4f0 2023-10-11T16:40:09.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/2884804335 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fb4099b40 con 0x55ecc80cf4f0 2023-10-11T16:40:09.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fd7d70b80 10 client.16776 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:09.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.418+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/2884804335 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ecc7e36ce0 con 0x55ecc80cf4f0 2023-10-11T16:40:09.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/2884804335 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fb8004390 con 0x55ecc80cf4f0 2023-10-11T16:40:09.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/2884804335 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f4fb40b90b0 con 0x55ecc80cf4f0 2023-10-11T16:40:09.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fd7d70b80 10 client.16776 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:09.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fd7d70b80 10 client.16776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:09.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fd7d70b80 20 client.16776 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:09.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecc80d3930 0x55ecc81400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fd7d70b80 4 client.16776 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:40:09.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/2884804335 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ecc7fee6e0 con 0x55ecc80d3930 2023-10-11T16:40:09.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.419+0000 7f4fc219b700 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecc80d3930 0x55ecc81400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.420+0000 7f4fc219b700 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecc80d3930 0x55ecc81400f0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41d2ab6c549f1b55 server_cookie=abafc0c21073bb60 in_seq=0 out_seq=0 2023-10-11T16:40:09.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.420+0000 7f4fa5ffb700 10 client.16776 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:09.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.420+0000 7f4fa5ffb700 1 -- 172.21.15.111:0/2884804335 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ecc7fee6e0 con 0x55ecc80d3930 2023-10-11T16:40:09.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.420+0000 7f4fa5ffb700 10 client.16776 handle_command_reply: tid=0 2023-10-11T16:40:09.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.420+0000 7f4fd7d70b80 1 client.16776 shutdown 2023-10-11T16:40:09.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.420+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/2884804335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cf4f0 msgr2=0x55ecc80deb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.420+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cf4f0 0x55ecc80deb20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4fb401c040 tx=0x7f4fb4099240 comp rx=0 tx=0).stop 2023-10-11T16:40:09.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/2884804335 shutdown_connections 2023-10-11T16:40:09.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55ecc80d3930 0x55ecc81400f0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ecc80cf4f0 0x55ecc80deb20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ecc80c43e0 0x55ecc80de0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 1 --2- 172.21.15.111:0/2884804335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecc80cebe0 0x55ecc80de5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fa5ffb700 0 client.16776 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:09.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/2884804335 >> 172.21.15.111:0/2884804335 conn(0x55ecc80d6070 msgr2=0x55ecc80cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/2884804335 shutdown_connections 2023-10-11T16:40:09.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 1 -- 172.21.15.111:0/2884804335 wait complete. 2023-10-11T16:40:09.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fd7d70b80 20 client.16776 trim_cache size 0 max 16384 2023-10-11T16:40:09.460 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:40:09.460 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:40:09.461 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "613f22a5-242a-492e-b0e5-171a603641b1", 2023-10-11T16:40:09.461 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:40:09.461 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:40:09.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fc43ff700 1 -- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4fa80a1340 msgr2=0x7f4fa80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4fa80a1340 0x7f4fa80a37f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4fac005e90 tx=0x7f4fac019040 comp rx=0 tx=0).stop 2023-10-11T16:40:09.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fc43ff700 1 -- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fbc10cec0 msgr2=0x7f4fbc198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.421+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fbc10cec0 0x7f4fbc198560 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400ef90 tx=0x7f4fb400c3a0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.422+0000 7f4fc43ff700 1 -- 172.21.15.111:0/1040012410 shutdown_connections 2023-10-11T16:40:09.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.422+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f4fa80a1340 0x7f4fa80a37f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.422+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fbc10cec0 0x7f4fbc198560 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.422+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fbc0746a0 0x7f4fbc19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.422+0000 7f4fc43ff700 1 --2- 172.21.15.111:0/1040012410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fbc074b00 0x7f4fbc19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.422+0000 7f4fc43ff700 1 -- 172.21.15.111:0/1040012410 >> 172.21.15.111:0/1040012410 conn(0x7f4fbc077590 msgr2=0x7f4fbc104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.422+0000 7f4fc43ff700 1 -- 172.21.15.111:0/1040012410 shutdown_connections 2023-10-11T16:40:09.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.422+0000 7f4fc43ff700 1 -- 172.21.15.111:0/1040012410 wait complete. 2023-10-11T16:40:09.467 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:40:09.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 -- 172.21.15.111:0/1879585891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 msgr2=0x7f63b8100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1879585891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 0x7f63b8100ec0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f63a0009990 tx=0x7f63a0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 -- 172.21.15.111:0/1879585891 shutdown_connections 2023-10-11T16:40:09.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1879585891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63b8101400 0x7f63b810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1879585891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b81063f0 0x7f63b81067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1879585891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 0x7f63b8100ec0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 -- 172.21.15.111:0/1879585891 >> 172.21.15.111:0/1879585891 conn(0x7f63b8077590 msgr2=0x7f63b8077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 -- 172.21.15.111:0/1879585891 shutdown_connections 2023-10-11T16:40:09.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.666+0000 7f63be3ff700 1 -- 172.21.15.111:0/1879585891 wait complete. 2023-10-11T16:40:09.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63be3ff700 1 Processor -- start 2023-10-11T16:40:09.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63be3ff700 1 -- start start 2023-10-11T16:40:09.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63be3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 0x7f63b8198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63be3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63b8101400 0x7f63b8198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63be3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b81063f0 0x7f63b819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63be3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b819d5e0 con 0x7f63b8100a50 2023-10-11T16:40:09.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63be3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b819d750 con 0x7f63b81063f0 2023-10-11T16:40:09.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63be3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b819d910 con 0x7f63b8101400 2023-10-11T16:40:09.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.667+0000 7f63b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 0x7f63b8198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 0x7f63b8198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43134/0 (socket says 172.21.15.111:43134) 2023-10-11T16:40:09.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b7fff700 1 -- 172.21.15.111:0/1064030133 learned_addr learned my addr 172.21.15.111:0/1064030133 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63affff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63b8101400 0x7f63b8198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63bc99c700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b81063f0 0x7f63b819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b7fff700 1 -- 172.21.15.111:0/1064030133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63b8101400 msgr2=0x7f63b8198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b7fff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63b8101400 0x7f63b8198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b7fff700 1 -- 172.21.15.111:0/1064030133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b81063f0 msgr2=0x7f63b819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b7fff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b81063f0 0x7f63b819cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b7fff700 1 -- 172.21.15.111:0/1064030133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a00095f0 con 0x7f63b8100a50 2023-10-11T16:40:09.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63bc99c700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b81063f0 0x7f63b819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:09.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b7fff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 0x7f63b8198600 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f63a800d980 tx=0x7f63a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b5ffb700 1 -- 172.21.15.111:0/1064030133 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8004220 con 0x7f63b8100a50 2023-10-11T16:40:09.678 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b5ffb700 1 -- 172.21.15.111:0/1064030133 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63a8007640 con 0x7f63b8100a50 2023-10-11T16:40:09.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63be3ff700 1 -- 172.21.15.111:0/1064030133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b8110410 con 0x7f63b8100a50 2023-10-11T16:40:09.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.668+0000 7f63b5ffb700 1 -- 172.21.15.111:0/1064030133 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8005610 con 0x7f63b8100a50 2023-10-11T16:40:09.679 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.669+0000 7f63be3ff700 1 -- 172.21.15.111:0/1064030133 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b8110280 con 0x7f63b8100a50 2023-10-11T16:40:09.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.672+0000 7f63b5ffb700 1 -- 172.21.15.111:0/1064030133 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f63a8004750 con 0x7f63b8100a50 2023-10-11T16:40:09.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63d1cd2b80 1 Processor -- start 2023-10-11T16:40:09.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63d1cd2b80 1 -- start start 2023-10-11T16:40:09.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cab60 0x564da93caf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cb470 0x564da93cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da93cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da92d0060 con 0x564da93cb470 2023-10-11T16:40:09.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da93228a0 con 0x564da93cab60 2023-10-11T16:40:09.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da931bd30 con 0x564da93cf8b0 2023-10-11T16:40:09.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cb470 0x564da93cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.673+0000 7f63b5ffb700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f63a40a1340 0x7f63a40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cb470 0x564da93cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43142/0 (socket says 172.21.15.111:43142) 2023-10-11T16:40:09.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b7fff700 1 -- 172.21.15.111:0/2613978806 learned_addr learned my addr 172.21.15.111:0/2613978806 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63affff700 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da93cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63bc99c700 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cab60 0x564da93caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63bc99c700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f63a40a1340 0x7f63a40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b5ffb700 1 -- 172.21.15.111:0/1064030133 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f63a80ccef0 con 0x7f63b8100a50 2023-10-11T16:40:09.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b7fff700 1 -- 172.21.15.111:0/2613978806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 msgr2=0x564da93cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b7fff700 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da93cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b7fff700 1 -- 172.21.15.111:0/2613978806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cab60 msgr2=0x564da93caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b7fff700 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cab60 0x564da93caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b7fff700 1 -- 172.21.15.111:0/2613978806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564da919d630 con 0x564da93cb470 2023-10-11T16:40:09.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63affff700 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da93cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:09.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63b7fff700 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cb470 0x564da93cf370 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f63a800b540 tx=0x7f63a80050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3300371fda4606dd server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.674+0000 7f63bc99c700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f63a40a1340 0x7f63a40a37f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009fd0 tx=0x7f63b000b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63ae7fc700 1 -- 172.21.15.111:0/2613978806 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a800f3c0 con 0x564da93cb470 2023-10-11T16:40:09.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63ae7fc700 1 -- 172.21.15.111:0/2613978806 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63a8099b50 con 0x564da93cb470 2023-10-11T16:40:09.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63ae7fc700 1 -- 172.21.15.111:0/2613978806 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a80992a0 con 0x564da93cb470 2023-10-11T16:40:09.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2613978806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cb470 msgr2=0x564da93cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cb470 0x564da93cf370 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f63a800b540 tx=0x7f63a80050b0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2613978806 shutdown_connections 2023-10-11T16:40:09.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da93cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cab60 0x564da93caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/2613978806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cb470 0x564da93cf370 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2613978806 >> 172.21.15.111:0/2613978806 conn(0x564da93c0360 msgr2=0x564da93c5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2613978806 shutdown_connections 2023-10-11T16:40:09.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.675+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2613978806 wait complete. 2023-10-11T16:40:09.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.676+0000 7f63d1cd2b80 1 -- start start 2023-10-11T16:40:09.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.676+0000 7f63d1cd2b80 1 Processor -- start 2023-10-11T16:40:09.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cb470 0x564da93d4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cf8b0 0x564da93d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da92bd8e0 con 0x564da93cab60 2023-10-11T16:40:09.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da92bd580 con 0x564da93cf8b0 2023-10-11T16:40:09.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da919f920 con 0x564da93cb470 2023-10-11T16:40:09.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63d1cd2b80 10 client.0 _ll_register_callbacks cb 0x564da92db780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:09.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43148/0 (socket says 172.21.15.111:43148) 2023-10-11T16:40:09.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63b7fff700 1 -- 172.21.15.111:0/3402770351 learned_addr learned my addr 172.21.15.111:0/3402770351 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63bc99c700 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cf8b0 0x564da93d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63affff700 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cb470 0x564da93d4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63b7fff700 1 -- 172.21.15.111:0/3402770351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cb470 msgr2=0x564da93d4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63b7fff700 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cb470 0x564da93d4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63b7fff700 1 -- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cf8b0 msgr2=0x564da93d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63b7fff700 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cf8b0 0x564da93d4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63b7fff700 1 -- 172.21.15.111:0/3402770351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b8110280 con 0x564da93cab60 2023-10-11T16:40:09.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.677+0000 7f63bc99c700 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cf8b0 0x564da93d4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:09.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63b7fff700 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d70c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f63a8099a90 tx=0x7f63a8004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254b43254b1e013f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63977fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:09.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63977fe700 1 -- 172.21.15.111:0/3402770351 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a80a1fa0 con 0x564da93cab60 2023-10-11T16:40:09.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/3402770351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564da93bdb20 con 0x564da93cab60 2023-10-11T16:40:09.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63977fe700 1 -- 172.21.15.111:0/3402770351 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f63a809db80 con 0x564da93cab60 2023-10-11T16:40:09.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63977fe700 1 -- 172.21.15.111:0/3402770351 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63a8099db0 con 0x564da93cab60 2023-10-11T16:40:09.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63d1cd2b80 10 client.16788 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:09.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/3402770351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564da92e2190 con 0x564da93cab60 2023-10-11T16:40:09.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63977fe700 1 -- 172.21.15.111:0/3402770351 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564da93bdb20 con 0x564da93cab60 2023-10-11T16:40:09.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63977fe700 1 -- 172.21.15.111:0/3402770351 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f63a8098ab0 con 0x564da93cab60 2023-10-11T16:40:09.703 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63d1cd2b80 10 client.16788 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:09.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.678+0000 7f63d1cd2b80 10 client.16788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:09.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.679+0000 7f63d1cd2b80 20 client.16788 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:09.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.679+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564da93e4440 0x564da93e6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.679+0000 7f63d1cd2b80 4 client.16788 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:09.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.679+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/3402770351 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564da92de0a0 con 0x564da93e4440 2023-10-11T16:40:09.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.679+0000 7f63affff700 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564da93e4440 0x564da93e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.679+0000 7f63affff700 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564da93e4440 0x564da93e6820 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=579fa4d2857ebcd3 server_cookie=6cc39d850a284f41 in_seq=0 out_seq=0 2023-10-11T16:40:09.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.679+0000 7f63977fe700 10 client.16788 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:09.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63977fe700 1 -- 172.21.15.111:0/3402770351 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564da92de0a0 con 0x564da93e4440 2023-10-11T16:40:09.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63977fe700 10 client.16788 handle_command_reply: tid=0 2023-10-11T16:40:09.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 client.16788 shutdown 2023-10-11T16:40:09.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/3402770351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 msgr2=0x564da93d70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d70c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f63a8099a90 tx=0x7f63a8004400 comp rx=0 tx=0).stop 2023-10-11T16:40:09.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/3402770351 shutdown_connections 2023-10-11T16:40:09.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564da93e4440 0x564da93e6820 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cf8b0 0x564da93d4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d70c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63977fe700 0 client.16788 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:09.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/3402770351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cb470 0x564da93d4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/3402770351 >> 172.21.15.111:0/3402770351 conn(0x564da93c0360 msgr2=0x564da93d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/3402770351 shutdown_connections 2023-10-11T16:40:09.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.680+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/3402770351 wait complete. 2023-10-11T16:40:09.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.681+0000 7f63d1cd2b80 20 client.16788 trim_cache size 0 max 16384 2023-10-11T16:40:09.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.698+0000 7f63d1cd2b80 1 Processor -- start 2023-10-11T16:40:09.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.698+0000 7f63d1cd2b80 1 -- start start 2023-10-11T16:40:09.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cab60 0x564da93c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93d6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cf8b0 0x564da93d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da93bfee0 con 0x564da93cf8b0 2023-10-11T16:40:09.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da93ce470 con 0x564da93cb470 2023-10-11T16:40:09.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da93d4c40 con 0x564da93cab60 2023-10-11T16:40:09.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63bc99c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cf8b0 0x564da93d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63affff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93d6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63bc99c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cf8b0 0x564da93d1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43156/0 (socket says 172.21.15.111:43156) 2023-10-11T16:40:09.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63bc99c700 1 -- 172.21.15.111:0/2483017963 learned_addr learned my addr 172.21.15.111:0/2483017963 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63b7fff700 1 --2- 172.21.15.111:0/2483017963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cab60 0x564da93c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63affff700 1 -- 172.21.15.111:0/2483017963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cab60 msgr2=0x564da93c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63affff700 1 --2- 172.21.15.111:0/2483017963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cab60 0x564da93c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63affff700 1 -- 172.21.15.111:0/2483017963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cf8b0 msgr2=0x564da93d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63affff700 1 --2- 172.21.15.111:0/2483017963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cf8b0 0x564da93d1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.699+0000 7f63affff700 1 -- 172.21.15.111:0/2483017963 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564da93c1b00 con 0x564da93cb470 2023-10-11T16:40:09.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.700+0000 7f63affff700 1 --2- 172.21.15.111:0/2483017963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93d6c60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b2e0 tx=0x7f63a0005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1540585291cc16cc server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.700+0000 7f63aeffd700 1 -- 172.21.15.111:0/2483017963 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a001d040 con 0x564da93cb470 2023-10-11T16:40:09.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.700+0000 7f63aeffd700 1 -- 172.21.15.111:0/2483017963 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63a001e040 con 0x564da93cb470 2023-10-11T16:40:09.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.700+0000 7f63aeffd700 1 -- 172.21.15.111:0/2483017963 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a0005440 con 0x564da93cb470 2023-10-11T16:40:09.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2483017963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 msgr2=0x564da93d6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/2483017963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93d6c60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b2e0 tx=0x7f63a0005fb0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2483017963 shutdown_connections 2023-10-11T16:40:09.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/2483017963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cf8b0 0x564da93d1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/2483017963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cab60 0x564da93c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/2483017963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93d6c60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2483017963 >> 172.21.15.111:0/2483017963 conn(0x564da93c0360 msgr2=0x564da93cd970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2483017963 shutdown_connections 2023-10-11T16:40:09.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/2483017963 wait complete. 2023-10-11T16:40:09.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.701+0000 7f63d1cd2b80 1 -- start start 2023-10-11T16:40:09.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63d1cd2b80 1 Processor -- start 2023-10-11T16:40:09.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93da0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63d1cd2b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da9437290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da93da8c0 con 0x564da93cab60 2023-10-11T16:40:09.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da9437800 con 0x564da93cb470 2023-10-11T16:40:09.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63d1cd2b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da94379c0 con 0x564da93cf8b0 2023-10-11T16:40:09.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63d1cd2b80 10 client.0 _ll_register_callbacks cb 0x7f63a8099920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:09.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63affff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93da0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43158/0 (socket says 172.21.15.111:43158) 2023-10-11T16:40:09.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63affff700 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93da0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:38734/0 (socket says 172.21.15.111:38734) 2023-10-11T16:40:09.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63b7fff700 1 -- 172.21.15.111:0/1416121236 learned_addr learned my addr 172.21.15.111:0/1416121236 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:09.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.702+0000 7f63bc99c700 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da9437290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63affff700 1 -- 172.21.15.111:0/1416121236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 msgr2=0x564da9437290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63affff700 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da9437290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63affff700 1 -- 172.21.15.111:0/1416121236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 msgr2=0x564da93d9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:09.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63affff700 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63affff700 1 -- 172.21.15.111:0/1416121236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a000bb50 con 0x564da93cb470 2023-10-11T16:40:09.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63b7fff700 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:09.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63affff700 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93da0c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f63a00098e0 tx=0x7f63a0005950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1014e580577b6acb server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63977fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:40:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63977fe700 1 -- 172.21.15.111:0/1416121236 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a001d070 con 0x564da93cb470 2023-10-11T16:40:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/1416121236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63a8099b50 con 0x564da93cb470 2023-10-11T16:40:09.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63977fe700 1 -- 172.21.15.111:0/1416121236 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f63a0029360 con 0x564da93cb470 2023-10-11T16:40:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63977fe700 1 -- 172.21.15.111:0/1416121236 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63a0029520 con 0x564da93cb470 2023-10-11T16:40:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63d1cd2b80 10 client.26909 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:09.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.703+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/1416121236 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x564da9149de0 con 0x564da93cb470 2023-10-11T16:40:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63977fe700 1 -- 172.21.15.111:0/1416121236 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63a8099b50 con 0x564da93cb470 2023-10-11T16:40:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63977fe700 1 -- 172.21.15.111:0/1416121236 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f63a0004880 con 0x564da93cb470 2023-10-11T16:40:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63d1cd2b80 10 client.26909 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:09.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63d1cd2b80 10 client.26909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:09.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63d1cd2b80 20 client.26909 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:09.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564da9438d60 0x564da943b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:09.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63d1cd2b80 4 client.26909 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:40:09.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/1416121236 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564da92ea660 con 0x564da9438d60 2023-10-11T16:40:09.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.704+0000 7f63b7fff700 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564da9438d60 0x564da943b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:09.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63b7fff700 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564da9438d60 0x564da943b170 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e6df8b3320f3642 server_cookie=ddb2da851b88eefd in_seq=0 out_seq=0 2023-10-11T16:40:09.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63977fe700 10 client.26909 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63977fe700 1 -- 172.21.15.111:0/1416121236 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564da92ea660 con 0x564da9438d60 2023-10-11T16:40:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63977fe700 10 client.26909 handle_command_reply: tid=0 2023-10-11T16:40:09.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 client.26909 shutdown 2023-10-11T16:40:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 msgr2=0x564da93da0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93da0c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f63a00098e0 tx=0x7f63a0005950 comp rx=0 tx=0).stop 2023-10-11T16:40:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/1416121236 shutdown_connections 2023-10-11T16:40:09.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564da9438d60 0x564da943b170 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564da93cf8b0 0x564da9437290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564da93cab60 0x564da93d9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 --2- 172.21.15.111:0/1416121236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564da93cb470 0x564da93da0c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63977fe700 0 client.26909 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:09.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/1416121236 >> 172.21.15.111:0/1416121236 conn(0x564da93c0360 msgr2=0x564da93d1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/1416121236 shutdown_connections 2023-10-11T16:40:09.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 1 -- 172.21.15.111:0/1416121236 wait complete. 2023-10-11T16:40:09.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.705+0000 7f63d1cd2b80 20 client.26909 trim_cache size 0 max 16384 2023-10-11T16:40:09.740 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:40:09.740 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:40:09.741 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:40:09.741 INFO:teuthology.orchestra.run.smithi111.stdout: "613f22a5-242a-492e-b0e5-171a603641b1": { 2023-10-11T16:40:09.741 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:40:09.741 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "613f22a5-242a-492e-b0e5-171a603641b1", 2023-10-11T16:40:09.742 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:40:09.742 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:40:09.742 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:40:09.743 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:40:09.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 -- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f63a40a1340 msgr2=0x7f63a40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f63a40a1340 0x7f63a40a37f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009fd0 tx=0x7f63b000b040 comp rx=0 tx=0).stop 2023-10-11T16:40:09.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 -- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 msgr2=0x7f63b8198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:09.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 0x7f63b8198600 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f63a800d980 tx=0x7f63a800dc90 comp rx=0 tx=0).stop 2023-10-11T16:40:09.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 -- 172.21.15.111:0/1064030133 shutdown_connections 2023-10-11T16:40:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f63a40a1340 0x7f63a40a37f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63b81063f0 0x7f63b819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f63b8100a50 0x7f63b8198600 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 --2- 172.21.15.111:0/1064030133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63b8101400 0x7f63b8198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 -- 172.21.15.111:0/1064030133 >> 172.21.15.111:0/1064030133 conn(0x7f63b8077590 msgr2=0x7f63b810b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:09.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 -- 172.21.15.111:0/1064030133 shutdown_connections 2023-10-11T16:40:09.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:09.706+0000 7f63be3ff700 1 -- 172.21.15.111:0/1064030133 wait complete. 2023-10-11T16:40:09.748 INFO:tasks.cephfs.filesystem:scrub status for tag:613f22a5-242a-492e-b0e5-171a603641b1 - {'path': '/', 'tag': '613f22a5-242a-492e-b0e5-171a603641b1', 'options': 'recursive,force'} 2023-10-11T16:40:09.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:09 smithi111 ceph-mon[98527]: pgmap v230: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:40:09.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:09 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:40:09.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:09 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:09.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:09 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:40:09.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:09 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:09 smithi150 ceph-mon[132217]: pgmap v230: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:40:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:09 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:40:09.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:09 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:09.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:09 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:40:09.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:09 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:10.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:09 smithi130 ceph-mon[131859]: pgmap v230: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:40:10.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:09 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:40:10.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:09 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:10.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:09 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:40:10.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:09 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:10 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:40:10.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:10 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:10.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:10 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:40:10.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:10 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:40:10.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:10 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:10.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:10 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:40:11.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:10 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:40:11.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:10 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:11.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:10 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:40:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:11 smithi150 ceph-mon[132217]: pgmap v231: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:40:11.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:11 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:40:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:11 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:40:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:11 smithi150 ceph-mon[132217]: osdmap e95: 12 total, 12 up, 12 in 2023-10-11T16:40:11.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:11 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:40:12.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:11 smithi130 ceph-mon[131859]: pgmap v231: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:40:12.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:11 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:40:12.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:11 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:40:12.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:11 smithi130 ceph-mon[131859]: osdmap e95: 12 total, 12 up, 12 in 2023-10-11T16:40:12.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:11 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:40:12.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:11 smithi111 ceph-mon[98527]: pgmap v231: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:40:12.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:11 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:40:12.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:11 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:40:12.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:11 smithi111 ceph-mon[98527]: osdmap e95: 12 total, 12 up, 12 in 2023-10-11T16:40:12.139 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:11 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:40:13.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:12 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:40:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:12 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:40:13.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:12 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:40:13.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:12 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:40:13.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:12 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:40:13.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:12 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:40:13.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:12 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:40:13.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:12 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:40:13.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:12 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:40:13.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:12 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:40:13.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:12 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:40:13.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:12 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:40:14.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:13 smithi130 ceph-mon[131859]: pgmap v233: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:40:14.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:13 smithi111 ceph-mon[98527]: pgmap v233: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:40:14.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:13 smithi150 ceph-mon[132217]: pgmap v233: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:40:16.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:15 smithi130 ceph-mon[131859]: pgmap v234: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-11T16:40:16.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:15 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:40:16.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:15 smithi111 ceph-mon[98527]: pgmap v234: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-11T16:40:16.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:15 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:40:16.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:15 smithi150 ceph-mon[132217]: pgmap v234: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-11T16:40:16.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:15 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:40:17.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:16 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:40:17.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:16 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:40:17.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:16 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:40:18.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:17 smithi130 ceph-mon[131859]: pgmap v235: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:18.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:17 smithi111 ceph-mon[98527]: pgmap v235: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:18.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:17 smithi150 ceph-mon[132217]: pgmap v235: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:19.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:18 smithi130 ceph-mon[131859]: pgmap v236: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:19.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:18 smithi111 ceph-mon[98527]: pgmap v236: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:19.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:18 smithi150 ceph-mon[132217]: pgmap v236: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:21.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:21 smithi130 ceph-mon[131859]: pgmap v237: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:21.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:21 smithi130 ceph-mon[131859]: osdmap e96: 12 total, 12 up, 12 in 2023-10-11T16:40:21.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:21 smithi111 ceph-mon[98527]: pgmap v237: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:21.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:21 smithi111 ceph-mon[98527]: osdmap e96: 12 total, 12 up, 12 in 2023-10-11T16:40:21.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:21 smithi150 ceph-mon[132217]: pgmap v237: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:21.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:21 smithi150 ceph-mon[132217]: osdmap e96: 12 total, 12 up, 12 in 2023-10-11T16:40:23.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:23 smithi130 ceph-mon[131859]: pgmap v239: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:23.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:23 smithi111 ceph-mon[98527]: pgmap v239: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:23.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:23 smithi150 ceph-mon[132217]: pgmap v239: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:25.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:25 smithi130 ceph-mon[131859]: pgmap v240: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:25.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:25 smithi111 ceph-mon[98527]: pgmap v240: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:25.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:25 smithi150 ceph-mon[132217]: pgmap v240: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:40:27.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:27 smithi130 ceph-mon[131859]: pgmap v241: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:40:27.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:27 smithi111 ceph-mon[98527]: pgmap v241: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:40:27.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:27 smithi150 ceph-mon[132217]: pgmap v241: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:40:29.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:29 smithi130 ceph-mon[131859]: pgmap v242: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:40:29.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:29 smithi111 ceph-mon[98527]: pgmap v242: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:40:29.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:29 smithi150 ceph-mon[132217]: pgmap v242: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:40:31.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:31 smithi130 ceph-mon[131859]: pgmap v243: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:40:31.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:31 smithi111 ceph-mon[98527]: pgmap v243: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:40:31.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:31 smithi150 ceph-mon[132217]: pgmap v243: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:40:33.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:33 smithi130 ceph-mon[131859]: pgmap v244: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-11T16:40:33.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:33 smithi111 ceph-mon[98527]: pgmap v244: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-11T16:40:33.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:33 smithi150 ceph-mon[132217]: pgmap v244: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-11T16:40:35.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:35 smithi130 ceph-mon[131859]: pgmap v245: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:40:35.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:35 smithi111 ceph-mon[98527]: pgmap v245: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:40:35.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:35 smithi150 ceph-mon[132217]: pgmap v245: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:40:37.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:37 smithi130 ceph-mon[131859]: pgmap v246: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:40:37.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:37 smithi111 ceph-mon[98527]: pgmap v246: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:40:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:37 smithi150 ceph-mon[132217]: pgmap v246: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:40:39.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:39 smithi130 ceph-mon[131859]: pgmap v247: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:39.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:39 smithi111 ceph-mon[98527]: pgmap v247: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:39 smithi150 ceph-mon[132217]: pgmap v247: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:39.749 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:40:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 -- 172.21.15.111:0/636392758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 msgr2=0x7f3308101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 --2- 172.21.15.111:0/636392758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 0x7f3308101090 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0099c0 tx=0x7f32fc009cd0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 -- 172.21.15.111:0/636392758 shutdown_connections 2023-10-11T16:40:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 --2- 172.21.15.111:0/636392758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3308105620 0x7f330810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 --2- 172.21.15.111:0/636392758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 0x7f3308101090 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 --2- 172.21.15.111:0/636392758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3308101660 0x7f3308104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 -- 172.21.15.111:0/636392758 >> 172.21.15.111:0/636392758 conn(0x7f3308077560 msgr2=0x7f3308077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:39.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 -- 172.21.15.111:0/636392758 shutdown_connections 2023-10-11T16:40:39.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.961+0000 7f330f1ff700 1 -- 172.21.15.111:0/636392758 wait complete. 2023-10-11T16:40:39.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.962+0000 7f330f1ff700 1 Processor -- start 2023-10-11T16:40:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330f1ff700 1 -- start start 2023-10-11T16:40:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330f1ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 0x7f3308198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330f1ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3308101660 0x7f3308198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330f1ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3308105620 0x7f330819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330f1ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330819d660 con 0x7f3308100cc0 2023-10-11T16:40:39.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330f1ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330819d7d0 con 0x7f3308105620 2023-10-11T16:40:39.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330f1ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330819d990 con 0x7f3308101660 2023-10-11T16:40:39.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330cf9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 0x7f3308198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330cf9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 0x7f3308198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37344/0 (socket says 172.21.15.111:37344) 2023-10-11T16:40:39.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330cf9b700 1 -- 172.21.15.111:0/2471169378 learned_addr learned my addr 172.21.15.111:0/2471169378 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:39.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330d79c700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3308105620 0x7f330819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f3307fff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3308101660 0x7f3308198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330cf9b700 1 -- 172.21.15.111:0/2471169378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3308101660 msgr2=0x7f3308198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:39.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330cf9b700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3308101660 0x7f3308198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330cf9b700 1 -- 172.21.15.111:0/2471169378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3308105620 msgr2=0x7f330819d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:39.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330cf9b700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3308105620 0x7f330819d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.963+0000 7f330cf9b700 1 -- 172.21.15.111:0/2471169378 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32fc0095f0 con 0x7f3308100cc0 2023-10-11T16:40:39.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.964+0000 7f330cf9b700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 0x7f3308198680 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00b580 tx=0x7f32fc00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:39.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.964+0000 7f3305ffb700 1 -- 172.21.15.111:0/2471169378 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc01d070 con 0x7f3308100cc0 2023-10-11T16:40:39.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.964+0000 7f330f1ff700 1 -- 172.21.15.111:0/2471169378 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33081103d0 con 0x7f3308100cc0 2023-10-11T16:40:39.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.964+0000 7f3305ffb700 1 -- 172.21.15.111:0/2471169378 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32fc005510 con 0x7f3308100cc0 2023-10-11T16:40:39.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.964+0000 7f3305ffb700 1 -- 172.21.15.111:0/2471169378 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc004b30 con 0x7f3308100cc0 2023-10-11T16:40:39.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.964+0000 7f330f1ff700 1 -- 172.21.15.111:0/2471169378 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3308110240 con 0x7f3308100cc0 2023-10-11T16:40:39.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.967+0000 7f3305ffb700 1 -- 172.21.15.111:0/2471169378 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f32fc004cd0 con 0x7f3308100cc0 2023-10-11T16:40:39.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.968+0000 7f3305ffb700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f32f80a12c0 0x7f32f80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.968+0000 7f3322a5bb80 1 Processor -- start 2023-10-11T16:40:39.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.968+0000 7f3305ffb700 1 -- 172.21.15.111:0/2471169378 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f32fc0cdfe0 con 0x7f3308100cc0 2023-10-11T16:40:39.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f3307fff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f32f80a12c0 0x7f32f80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f3322a5bb80 1 -- start start 2023-10-11T16:40:39.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f5ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f64370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f648b0 0x560970f64d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970e65060 con 0x560970f648b0 2023-10-11T16:40:39.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970eb78a0 con 0x560970f5fb60 2023-10-11T16:40:39.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970eb0d30 con 0x560970f60470 2023-10-11T16:40:39.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f330cf9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f5ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f330d79c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f648b0 0x560970f64d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.983 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f330cf9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f5ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:56140/0 (socket says 172.21.15.111:56140) 2023-10-11T16:40:39.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.969+0000 7f330cf9b700 1 -- 172.21.15.111:0/3025425387 learned_addr learned my addr 172.21.15.111:0/3025425387 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:39.984 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f3307fff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f32f80a12c0 0x7f32f80a3770 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3308199c90 tx=0x7f32f4009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:39.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f3307fff700 1 --2- 172.21.15.111:0/3025425387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f64370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f330d79c700 1 -- 172.21.15.111:0/3025425387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 msgr2=0x560970f64370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:39.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f330d79c700 1 --2- 172.21.15.111:0/3025425387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f64370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.985 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f330d79c700 1 -- 172.21.15.111:0/3025425387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 msgr2=0x560970f5ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:39.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f330d79c700 1 --2- 172.21.15.111:0/3025425387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f5ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f330d79c700 1 -- 172.21.15.111:0/3025425387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560970d32630 con 0x560970f648b0 2023-10-11T16:40:39.986 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f330d79c700 1 --2- 172.21.15.111:0/3025425387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f648b0 0x560970f64d00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3300009310 tx=0x7f3300009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1aeec087f73c47 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:39.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f32f2ffd700 1 -- 172.21.15.111:0/3025425387 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330000e070 con 0x560970f648b0 2023-10-11T16:40:39.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f32f2ffd700 1 -- 172.21.15.111:0/3025425387 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f330000f040 con 0x560970f648b0 2023-10-11T16:40:39.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.970+0000 7f32f2ffd700 1 -- 172.21.15.111:0/3025425387 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33000147e0 con 0x560970f648b0 2023-10-11T16:40:39.987 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3025425387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f648b0 msgr2=0x560970f64d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:39.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3025425387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f648b0 0x560970f64d00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3300009310 tx=0x7f3300009920 comp rx=0 tx=0).stop 2023-10-11T16:40:39.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3025425387 shutdown_connections 2023-10-11T16:40:39.988 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3025425387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f648b0 0x560970f64d00 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3025425387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f5ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3025425387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f64370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3025425387 >> 172.21.15.111:0/3025425387 conn(0x560970f55360 msgr2=0x560970f5ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:39.989 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3025425387 shutdown_connections 2023-10-11T16:40:39.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.971+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3025425387 wait complete. 2023-10-11T16:40:39.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 1 -- start start 2023-10-11T16:40:39.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 1 Processor -- start 2023-10-11T16:40:39.990 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f6bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f69240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 0x560970f69870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.991 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970e528e0 con 0x560970f60470 2023-10-11T16:40:39.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970e52580 con 0x560970f5fb60 2023-10-11T16:40:39.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970d34920 con 0x560970f648b0 2023-10-11T16:40:39.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.972+0000 7f3322a5bb80 10 client.0 _ll_register_callbacks cb 0x560970e70780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:39.992 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f69240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f69240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37358/0 (socket says 172.21.15.111:37358) 2023-10-11T16:40:39.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 -- 172.21.15.111:0/3886854022 learned_addr learned my addr 172.21.15.111:0/3886854022 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:39.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f330cf9b700 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f6bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.993 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f330d79c700 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 0x560970f69870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:39.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 -- 172.21.15.111:0/3886854022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 msgr2=0x560970f69870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:39.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 0x560970f69870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.994 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 -- 172.21.15.111:0/3886854022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 msgr2=0x560970f6bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:39.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f6bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:39.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 -- 172.21.15.111:0/3886854022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3300009050 con 0x560970f60470 2023-10-11T16:40:39.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f3307fff700 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f69240 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3308078ca0 tx=0x7f32f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5eed2c50d22a65 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:39.995 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.973+0000 7f32e7fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:39.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f32e7fff700 1 -- 172.21.15.111:0/3886854022 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4004020 con 0x560970f60470 2023-10-11T16:40:39.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3886854022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560970f52b20 con 0x560970f60470 2023-10-11T16:40:39.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f32e7fff700 1 -- 172.21.15.111:0/3886854022 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f32f4018070 con 0x560970f60470 2023-10-11T16:40:39.996 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f32e7fff700 1 -- 172.21.15.111:0/3886854022 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32f4004180 con 0x560970f60470 2023-10-11T16:40:39.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f3322a5bb80 10 client.16806 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:39.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3886854022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560970e77190 con 0x560970f60470 2023-10-11T16:40:39.997 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f32e7fff700 1 -- 172.21.15.111:0/3886854022 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560970f52b20 con 0x560970f60470 2023-10-11T16:40:39.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f32e7fff700 1 -- 172.21.15.111:0/3886854022 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f32f4029510 con 0x560970f60470 2023-10-11T16:40:39.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f3322a5bb80 10 client.16806 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:39.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f3322a5bb80 10 client.16806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:39.998 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f3322a5bb80 20 client.16806 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:39.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.974+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560970f79860 0x560970f7bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:39.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.975+0000 7f3322a5bb80 4 client.16806 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:39.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.975+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3886854022 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560970e730a0 con 0x560970f79860 2023-10-11T16:40:39.999 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.975+0000 7f330cf9b700 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560970f79860 0x560970f7bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.975+0000 7f330cf9b700 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560970f79860 0x560970f7bc70 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d01c56842879f41d server_cookie=e7f6125cf88aa5f4 in_seq=0 out_seq=0 2023-10-11T16:40:40.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.975+0000 7f32e7fff700 10 client.16806 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:40.000 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.976+0000 7f32e7fff700 1 -- 172.21.15.111:0/3886854022 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560970e730a0 con 0x560970f79860 2023-10-11T16:40:40.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.976+0000 7f32e7fff700 10 client.16806 handle_command_reply: tid=0 2023-10-11T16:40:40.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.976+0000 7f3322a5bb80 1 client.16806 shutdown 2023-10-11T16:40:40.001 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3886854022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 msgr2=0x560970f69240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f69240 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3308078ca0 tx=0x7f32f40047c0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3886854022 shutdown_connections 2023-10-11T16:40:40.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560970f79860 0x560970f7bc70 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.002 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 0x560970f69870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f6bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f32e7fff700 0 client.16806 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:40.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3886854022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f69240 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3886854022 >> 172.21.15.111:0/3886854022 conn(0x560970f55360 msgr2=0x560970f65d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3886854022 shutdown_connections 2023-10-11T16:40:40.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3886854022 wait complete. 2023-10-11T16:40:40.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.977+0000 7f3322a5bb80 20 client.16806 trim_cache size 0 max 16384 2023-10-11T16:40:40.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.994+0000 7f3322a5bb80 1 Processor -- start 2023-10-11T16:40:40.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.994+0000 7f3322a5bb80 1 -- start start 2023-10-11T16:40:40.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.994+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f5fb60 0x560970f5ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.994+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.994+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f648b0 0x560970f6bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.994+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970f529c0 con 0x560970f5fb60 2023-10-11T16:40:40.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.994+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970f63140 con 0x560970f648b0 2023-10-11T16:40:40.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.994+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970f5c1c0 con 0x560970f60470 2023-10-11T16:40:40.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f5fb60 0x560970f5ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f5fb60 0x560970f5ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37364/0 (socket says 172.21.15.111:37364) 2023-10-11T16:40:40.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 -- 172.21.15.111:0/3427275418 learned_addr learned my addr 172.21.15.111:0/3427275418 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:40.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f3307fff700 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330d79c700 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f648b0 0x560970f6bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 -- 172.21.15.111:0/3427275418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 msgr2=0x560970f709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 -- 172.21.15.111:0/3427275418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f648b0 msgr2=0x560970f6bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f648b0 0x560970f6bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 -- 172.21.15.111:0/3427275418 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560970f59820 con 0x560970f5fb60 2023-10-11T16:40:40.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f3307fff700 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:40.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f330cf9b700 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f5fb60 0x560970f5ac80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009d70 tx=0x7f32fc0970b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689b9cfc5e524a09 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f32f37fe700 1 -- 172.21.15.111:0/3427275418 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc0a30b0 con 0x560970f5fb60 2023-10-11T16:40:40.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f32f37fe700 1 -- 172.21.15.111:0/3427275418 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32fc099090 con 0x560970f5fb60 2023-10-11T16:40:40.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.995+0000 7f32f37fe700 1 -- 172.21.15.111:0/3427275418 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc09a440 con 0x560970f5fb60 2023-10-11T16:40:40.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3427275418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f5fb60 msgr2=0x560970f5ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f5fb60 0x560970f5ac80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f32fc009d70 tx=0x7f32fc0970b0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3427275418 shutdown_connections 2023-10-11T16:40:40.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f648b0 0x560970f6bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f5fb60 0x560970f5ac80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/3427275418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f60470 0x560970f709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3427275418 >> 172.21.15.111:0/3427275418 conn(0x560970f55360 msgr2=0x560970f75b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3427275418 shutdown_connections 2023-10-11T16:40:40.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.996+0000 7f3322a5bb80 1 -- 172.21.15.111:0/3427275418 wait complete. 2023-10-11T16:40:40.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 1 -- start start 2023-10-11T16:40:40.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 1 Processor -- start 2023-10-11T16:40:40.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f70be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 0x560970f6ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970f6f2e0 con 0x560970f60470 2023-10-11T16:40:40.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970f6f450 con 0x560970f5fb60 2023-10-11T16:40:40.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560970f6f610 con 0x560970f648b0 2023-10-11T16:40:40.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3322a5bb80 10 client.0 _ll_register_callbacks cb 0x7f32f4012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:40.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3307fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f70be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3307fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f70be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37380/0 (socket says 172.21.15.111:37380) 2023-10-11T16:40:40.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.997+0000 7f3307fff700 1 -- 172.21.15.111:0/337903639 learned_addr learned my addr 172.21.15.111:0/337903639 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:40.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f330d79c700 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 0x560970f6ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f330cf9b700 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f3307fff700 1 -- 172.21.15.111:0/337903639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 msgr2=0x560970f6ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f3307fff700 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 0x560970f6ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f3307fff700 1 -- 172.21.15.111:0/337903639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 msgr2=0x560970f706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f3307fff700 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f3307fff700 1 -- 172.21.15.111:0/337903639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3308110240 con 0x560970f60470 2023-10-11T16:40:40.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f3307fff700 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f70be0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x560970f55300 tx=0x7f32f40117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20414095b6e9d019 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f32e7fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:40.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f32e7fff700 1 -- 172.21.15.111:0/337903639 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4029690 con 0x560970f60470 2023-10-11T16:40:40.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f32e7fff700 1 -- 172.21.15.111:0/337903639 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f32f4018040 con 0x560970f60470 2023-10-11T16:40:40.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.998+0000 7f3322a5bb80 1 -- 172.21.15.111:0/337903639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f330000f040 con 0x560970f60470 2023-10-11T16:40:40.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f32e7fff700 1 -- 172.21.15.111:0/337903639 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32f4029b10 con 0x560970f60470 2023-10-11T16:40:40.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f3322a5bb80 10 client.16812 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:40.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f3322a5bb80 1 -- 172.21.15.111:0/337903639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560970cdede0 con 0x560970f60470 2023-10-11T16:40:40.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f32e7fff700 1 -- 172.21.15.111:0/337903639 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f330000f040 con 0x560970f60470 2023-10-11T16:40:40.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f32e7fff700 1 -- 172.21.15.111:0/337903639 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f32f4003c40 con 0x560970f60470 2023-10-11T16:40:40.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f3322a5bb80 10 client.16812 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:40.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f3322a5bb80 10 client.16812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:40.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f3322a5bb80 20 client.16812 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:40.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560970fcea90 0x560970fd0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f3322a5bb80 4 client.16812 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:40:40.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f3322a5bb80 1 -- 172.21.15.111:0/337903639 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560970e7f660 con 0x560970fcea90 2023-10-11T16:40:40.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:39.999+0000 7f330cf9b700 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560970fcea90 0x560970fd0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f32e7fff700 10 client.16812 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:40.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f330cf9b700 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560970fcea90 0x560970fd0ea0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcca4f69d921a59b server_cookie=379eac52460cc3b3 in_seq=0 out_seq=0 2023-10-11T16:40:40.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f32e7fff700 1 -- 172.21.15.111:0/337903639 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560970e7f660 con 0x560970fcea90 2023-10-11T16:40:40.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f32e7fff700 10 client.16812 handle_command_reply: tid=0 2023-10-11T16:40:40.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 client.16812 shutdown 2023-10-11T16:40:40.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 -- 172.21.15.111:0/337903639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 msgr2=0x560970f70be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f70be0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x560970f55300 tx=0x7f32f40117f0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 -- 172.21.15.111:0/337903639 shutdown_connections 2023-10-11T16:40:40.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560970fcea90 0x560970fd0ea0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560970f648b0 0x560970f6ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560970f5fb60 0x560970f706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 --2- 172.21.15.111:0/337903639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560970f60470 0x560970f70be0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f32e7fff700 0 client.16812 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:40.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 -- 172.21.15.111:0/337903639 >> 172.21.15.111:0/337903639 conn(0x560970f55360 msgr2=0x560970f5e960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 -- 172.21.15.111:0/337903639 shutdown_connections 2023-10-11T16:40:40.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.000+0000 7f3322a5bb80 1 -- 172.21.15.111:0/337903639 wait complete. 2023-10-11T16:40:40.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f3322a5bb80 20 client.16812 trim_cache size 0 max 16384 2023-10-11T16:40:40.034 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:40:40.034 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:40:40.034 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:40:40.035 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:40:40.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 -- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f32f80a12c0 msgr2=0x7f32f80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f32f80a12c0 0x7f32f80a3770 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3308199c90 tx=0x7f32f4009260 comp rx=0 tx=0).stop 2023-10-11T16:40:40.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 -- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 msgr2=0x7f3308198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 0x7f3308198680 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00b580 tx=0x7f32fc00bfd0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 -- 172.21.15.111:0/2471169378 shutdown_connections 2023-10-11T16:40:40.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f32f80a12c0 0x7f32f80a3770 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3308105620 0x7f330819d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3308100cc0 0x7f3308198680 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 --2- 172.21.15.111:0/2471169378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3308101660 0x7f3308198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 -- 172.21.15.111:0/2471169378 >> 172.21.15.111:0/2471169378 conn(0x7f3308077560 msgr2=0x7f33080fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 -- 172.21.15.111:0/2471169378 shutdown_connections 2023-10-11T16:40:40.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.001+0000 7f330f1ff700 1 -- 172.21.15.111:0/2471169378 wait complete. 2023-10-11T16:40:40.041 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:40:40.042 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:40:40.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.242+0000 7fae6a5ff700 1 -- 172.21.15.111:0/1465892053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 msgr2=0x7fae64104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.242+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/1465892053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 0x7fae64104e60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0099c0 tx=0x7fae4c009cd0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.242+0000 7fae6a5ff700 1 -- 172.21.15.111:0/1465892053 shutdown_connections 2023-10-11T16:40:40.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.242+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/1465892053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae64105540 0x7fae6410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.242+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/1465892053 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae64100550 0x7fae64100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.242+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/1465892053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 0x7fae64104e60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.242+0000 7fae6a5ff700 1 -- 172.21.15.111:0/1465892053 >> 172.21.15.111:0/1465892053 conn(0x7fae64077020 msgr2=0x7fae64077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.243+0000 7fae6a5ff700 1 -- 172.21.15.111:0/1465892053 shutdown_connections 2023-10-11T16:40:40.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.243+0000 7fae6a5ff700 1 -- 172.21.15.111:0/1465892053 wait complete. 2023-10-11T16:40:40.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae6a5ff700 1 Processor -- start 2023-10-11T16:40:40.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae6a5ff700 1 -- start start 2023-10-11T16:40:40.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae6a5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae64100550 0x7fae641985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae6a5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 0x7fae64198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae6a5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae64105540 0x7fae6419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae6a5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6419d5b0 con 0x7fae64100ef0 2023-10-11T16:40:40.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae6a5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6419d720 con 0x7fae64105540 2023-10-11T16:40:40.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae6a5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6419d8e0 con 0x7fae64100550 2023-10-11T16:40:40.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae5bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 0x7fae64198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae63fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae64100550 0x7fae641985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae63fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae64100550 0x7fae641985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36258/0 (socket says 172.21.15.111:36258) 2023-10-11T16:40:40.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.244+0000 7fae63fff700 1 -- 172.21.15.111:0/2646080206 learned_addr learned my addr 172.21.15.111:0/2646080206 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:40.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae68b9c700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae64105540 0x7fae6419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae5bfff700 1 -- 172.21.15.111:0/2646080206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae64100550 msgr2=0x7fae641985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.253 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae5bfff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae64100550 0x7fae641985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae5bfff700 1 -- 172.21.15.111:0/2646080206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae64105540 msgr2=0x7fae6419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae5bfff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae64105540 0x7fae6419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae5bfff700 1 -- 172.21.15.111:0/2646080206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0095f0 con 0x7fae64100ef0 2023-10-11T16:40:40.254 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae5bfff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 0x7fae64198b10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0058e0 tx=0x7fae4c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae61ffb700 1 -- 172.21.15.111:0/2646080206 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c01d070 con 0x7fae64100ef0 2023-10-11T16:40:40.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.245+0000 7fae61ffb700 1 -- 172.21.15.111:0/2646080206 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae4c005050 con 0x7fae64100ef0 2023-10-11T16:40:40.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.246+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae64110310 con 0x7fae64100ef0 2023-10-11T16:40:40.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.246+0000 7fae61ffb700 1 -- 172.21.15.111:0/2646080206 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c004970 con 0x7fae64100ef0 2023-10-11T16:40:40.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.246+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae64110180 con 0x7fae64100ef0 2023-10-11T16:40:40.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.247+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae641104a0 con 0x7fae64100ef0 2023-10-11T16:40:40.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.248+0000 7fae61ffb700 1 -- 172.21.15.111:0/2646080206 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fae4c0051c0 con 0x7fae64100ef0 2023-10-11T16:40:40.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.251+0000 7fae61ffb700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fae500a12c0 0x7fae500a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.251+0000 7fae61ffb700 1 -- 172.21.15.111:0/2646080206 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fae4c0cdd80 con 0x7fae64100ef0 2023-10-11T16:40:40.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.251+0000 7fae63fff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fae500a12c0 0x7fae500a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.251+0000 7fae63fff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fae500a12c0 0x7fae500a3770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fae54009d20 tx=0x7fae54009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.259 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.255+0000 7fae61ffb700 1 -- 172.21.15.111:0/2646080206 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fae4c096db0 con 0x7fae64100ef0 2023-10-11T16:40:40.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.501+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae64067a60 con 0x7fae64100ef0 2023-10-11T16:40:40.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.502+0000 7fae61ffb700 1 -- 172.21.15.111:0/2646080206 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10829 (secure 0 0 0) 0x7fae4c09a3d0 con 0x7fae64100ef0 2023-10-11T16:40:40.504 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:40:40.504 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:36:18.517722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:40:40.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fae500a12c0 msgr2=0x7fae500a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fae500a12c0 0x7fae500a3770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fae54009d20 tx=0x7fae54009260 comp rx=0 tx=0).stop 2023-10-11T16:40:40.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 msgr2=0x7fae64198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 0x7fae64198b10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0058e0 tx=0x7fae4c00f690 comp rx=0 tx=0).stop 2023-10-11T16:40:40.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 shutdown_connections 2023-10-11T16:40:40.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fae500a12c0 0x7fae500a3770 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fae64105540 0x7fae6419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae64100550 0x7fae641985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 --2- 172.21.15.111:0/2646080206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae64100ef0 0x7fae64198b10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 >> 172.21.15.111:0/2646080206 conn(0x7fae64077020 msgr2=0x7fae640fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 shutdown_connections 2023-10-11T16:40:40.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.504+0000 7fae6a5ff700 1 -- 172.21.15.111:0/2646080206 wait complete. 2023-10-11T16:40:40.510 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-10-11T16:40:40.525 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:40:40.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 -- 172.21.15.111:0/4198401948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792c101390 msgr2=0x7f792c101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 --2- 172.21.15.111:0/4198401948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792c101390 0x7f792c101760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f791c009990 tx=0x7f791c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 -- 172.21.15.111:0/4198401948 shutdown_connections 2023-10-11T16:40:40.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 --2- 172.21.15.111:0/4198401948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f792c10a8b0 0x7f792c10ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 --2- 172.21.15.111:0/4198401948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792c101390 0x7f792c101760 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 --2- 172.21.15.111:0/4198401948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f792c101ca0 0x7f792c10a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 -- 172.21.15.111:0/4198401948 >> 172.21.15.111:0/4198401948 conn(0x7f792c0770b0 msgr2=0x7f792c0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 -- 172.21.15.111:0/4198401948 shutdown_connections 2023-10-11T16:40:40.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.724+0000 7f79321ff700 1 -- 172.21.15.111:0/4198401948 wait complete. 2023-10-11T16:40:40.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.726+0000 7f79321ff700 1 Processor -- start 2023-10-11T16:40:40.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.726+0000 7f79321ff700 1 -- start start 2023-10-11T16:40:40.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f79321ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f792c101390 0x7f792c0ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f79321ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f792c101ca0 0x7f792c0ff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f79321ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792c10a8b0 0x7f792c1a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f79321ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0fff30 con 0x7f792c101ca0 2023-10-11T16:40:40.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f79321ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c1a4ae0 con 0x7f792c10a8b0 2023-10-11T16:40:40.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f79321ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c1a4c20 con 0x7f792c101390 2023-10-11T16:40:40.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f792b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f792c101390 0x7f792c0ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f7923fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f792c101ca0 0x7f792c0ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f792b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f792c101390 0x7f792c0ff1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36282/0 (socket says 172.21.15.111:36282) 2023-10-11T16:40:40.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f792b7fe700 1 -- 172.21.15.111:0/1192994334 learned_addr learned my addr 172.21.15.111:0/1192994334 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:40.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f792bfff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792c10a8b0 0x7f792c1a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f7923fff700 1 -- 172.21.15.111:0/1192994334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f792c101390 msgr2=0x7f792c0ff1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f7923fff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f792c101390 0x7f792c0ff1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f7923fff700 1 -- 172.21.15.111:0/1192994334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792c10a8b0 msgr2=0x7f792c1a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f7923fff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792c10a8b0 0x7f792c1a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.727+0000 7f7923fff700 1 -- 172.21.15.111:0/1192994334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7924009c40 con 0x7f792c101ca0 2023-10-11T16:40:40.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.728+0000 7f7923fff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f792c101ca0 0x7f792c0ff700 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f791400e9f0 tx=0x7f791400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.728+0000 7f79297fa700 1 -- 172.21.15.111:0/1192994334 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791400c9c0 con 0x7f792c101ca0 2023-10-11T16:40:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.728+0000 7f79321ff700 1 -- 172.21.15.111:0/1192994334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791c0095f0 con 0x7f792c101ca0 2023-10-11T16:40:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.728+0000 7f79297fa700 1 -- 172.21.15.111:0/1192994334 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7914010430 con 0x7f792c101ca0 2023-10-11T16:40:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.728+0000 7f79297fa700 1 -- 172.21.15.111:0/1192994334 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7914018620 con 0x7f792c101ca0 2023-10-11T16:40:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.728+0000 7f79321ff700 1 -- 172.21.15.111:0/1192994334 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c110330 con 0x7f792c101ca0 2023-10-11T16:40:40.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.731+0000 7f79297fa700 1 -- 172.21.15.111:0/1192994334 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f7914010860 con 0x7f792c101ca0 2023-10-11T16:40:40.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.732+0000 7f79297fa700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f79180a12c0 0x7f79180a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.732+0000 7f7945b99b80 1 Processor -- start 2023-10-11T16:40:40.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.732+0000 7f79297fa700 1 -- 172.21.15.111:0/1192994334 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f7914014070 con 0x7f792c101ca0 2023-10-11T16:40:40.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.732+0000 7f792b7fe700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f79180a12c0 0x7f79180a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f7945b99b80 1 -- start start 2023-10-11T16:40:40.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7370 0x55ef08ae7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7c80 0x55ef08aebb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08aec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f7945b99b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef089e4060 con 0x55ef08ae7370 2023-10-11T16:40:40.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f7945b99b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef08a368a0 con 0x55ef08aec0c0 2023-10-11T16:40:40.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f7945b99b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef08a2fd30 con 0x55ef08ae7c80 2023-10-11T16:40:40.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f792b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7370 0x55ef08ae7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f7923fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7c80 0x55ef08aebb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f792b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7370 0x55ef08ae7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37420/0 (socket says 172.21.15.111:37420) 2023-10-11T16:40:40.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.733+0000 7f792b7fe700 1 -- 172.21.15.111:0/2341070971 learned_addr learned my addr 172.21.15.111:0/2341070971 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:40.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f792bfff700 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08aec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f792b7fe700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f79180a12c0 0x7f79180a3770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f791c006010 tx=0x7f791c005960 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7923fff700 1 -- 172.21.15.111:0/2341070971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 msgr2=0x55ef08aec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7923fff700 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08aec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7923fff700 1 -- 172.21.15.111:0/2341070971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7370 msgr2=0x55ef08ae7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7923fff700 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7370 0x55ef08ae7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7923fff700 1 -- 172.21.15.111:0/2341070971 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef088b1630 con 0x55ef08ae7c80 2023-10-11T16:40:40.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f792bfff700 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08aec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:40.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7923fff700 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7c80 0x55ef08aebb80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f79140ca700 tx=0x7f79140964f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af7e9ba88444c50 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7921ffb700 1 -- 172.21.15.111:0/2341070971 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791409e0b0 con 0x55ef08ae7c80 2023-10-11T16:40:40.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7921ffb700 1 -- 172.21.15.111:0/2341070971 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79140a24d0 con 0x55ef08ae7c80 2023-10-11T16:40:40.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.734+0000 7f7921ffb700 1 -- 172.21.15.111:0/2341070971 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7914099570 con 0x55ef08ae7c80 2023-10-11T16:40:40.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 -- 172.21.15.111:0/2341070971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7c80 msgr2=0x55ef08aebb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7c80 0x55ef08aebb80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f79140ca700 tx=0x7f79140964f0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 -- 172.21.15.111:0/2341070971 shutdown_connections 2023-10-11T16:40:40.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08aec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7370 0x55ef08ae7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2341070971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7c80 0x55ef08aebb80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 -- 172.21.15.111:0/2341070971 >> 172.21.15.111:0/2341070971 conn(0x55ef08ad4360 msgr2=0x55ef08ad9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 -- 172.21.15.111:0/2341070971 shutdown_connections 2023-10-11T16:40:40.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.735+0000 7f7945b99b80 1 -- 172.21.15.111:0/2341070971 wait complete. 2023-10-11T16:40:40.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 1 -- start start 2023-10-11T16:40:40.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 1 Processor -- start 2023-10-11T16:40:40.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7370 0x55ef08af3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08aec0c0 0x55ef08af1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef089d18e0 con 0x55ef08ae7c80 2023-10-11T16:40:40.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef089d1580 con 0x55ef08ae7370 2023-10-11T16:40:40.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef088b3920 con 0x55ef08aec0c0 2023-10-11T16:40:40.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.736+0000 7f7945b99b80 10 client.0 _ll_register_callbacks cb 0x55ef089ef780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:40.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37436/0 (socket says 172.21.15.111:37436) 2023-10-11T16:40:40.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 -- 172.21.15.111:0/3892708491 learned_addr learned my addr 172.21.15.111:0/3892708491 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:40.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f792bfff700 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08aec0c0 0x55ef08af1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f792b7fe700 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7370 0x55ef08af3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 -- 172.21.15.111:0/3892708491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08aec0c0 msgr2=0x55ef08af1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08aec0c0 0x55ef08af1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 -- 172.21.15.111:0/3892708491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7370 msgr2=0x55ef08af3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7370 0x55ef08af3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 -- 172.21.15.111:0/3892708491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c110330 con 0x55ef08ae7c80 2023-10-11T16:40:40.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f7923fff700 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af0b10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7914096af0 tx=0x7f7914095d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f44c289b96e4236 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.737+0000 7f790affd700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:40.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f790affd700 1 -- 172.21.15.111:0/3892708491 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791409e0b0 con 0x55ef08ae7c80 2023-10-11T16:40:40.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 1 -- 172.21.15.111:0/3892708491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ef08ad1b20 con 0x55ef08ae7c80 2023-10-11T16:40:40.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f790affd700 1 -- 172.21.15.111:0/3892708491 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f79140b82c0 con 0x55ef08ae7c80 2023-10-11T16:40:40.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f790affd700 1 -- 172.21.15.111:0/3892708491 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79140b8480 con 0x55ef08ae7c80 2023-10-11T16:40:40.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 10 client.16830 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:40.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 1 -- 172.21.15.111:0/3892708491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ef089f6190 con 0x55ef08ae7c80 2023-10-11T16:40:40.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f790affd700 1 -- 172.21.15.111:0/3892708491 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ef08ad1b20 con 0x55ef08ae7c80 2023-10-11T16:40:40.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f790affd700 1 -- 172.21.15.111:0/3892708491 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7914095e70 con 0x55ef08ae7c80 2023-10-11T16:40:40.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 10 client.16830 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:40.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 10 client.16830 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:40:40.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 20 client.16830 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:40.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55ef08b00d80 0x55ef08b03190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 4 client.16830 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:40.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.738+0000 7f7945b99b80 1 -- 172.21.15.111:0/3892708491 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef089f20a0 con 0x55ef08b00d80 2023-10-11T16:40:40.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.739+0000 7f792b7fe700 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55ef08b00d80 0x55ef08b03190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.739+0000 7f792b7fe700 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55ef08b00d80 0x55ef08b03190 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f44e0cf40f346743 server_cookie=faa7012769e15f29 in_seq=0 out_seq=0 2023-10-11T16:40:40.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.739+0000 7f790affd700 10 client.16830 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:40:40.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f790affd700 1 -- 172.21.15.111:0/3892708491 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ef089f20a0 con 0x55ef08b00d80 2023-10-11T16:40:40.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f790affd700 10 client.16830 handle_command_reply: tid=0 2023-10-11T16:40:40.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 client.16830 shutdown 2023-10-11T16:40:40.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 -- 172.21.15.111:0/3892708491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 msgr2=0x55ef08af0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af0b10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7914096af0 tx=0x7f7914095d90 comp rx=0 tx=0).stop 2023-10-11T16:40:40.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 -- 172.21.15.111:0/3892708491 shutdown_connections 2023-10-11T16:40:40.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55ef08b00d80 0x55ef08b03190 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08aec0c0 0x55ef08af1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7370 0x55ef08af3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3892708491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af0b10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 -- 172.21.15.111:0/3892708491 >> 172.21.15.111:0/3892708491 conn(0x55ef08ad4360 msgr2=0x55ef08aed570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f790affd700 0 client.16830 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:40:40.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 -- 172.21.15.111:0/3892708491 shutdown_connections 2023-10-11T16:40:40.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.740+0000 7f7945b99b80 1 -- 172.21.15.111:0/3892708491 wait complete. 2023-10-11T16:40:40.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.741+0000 7f7945b99b80 20 client.16830 trim_cache size 0 max 16384 2023-10-11T16:40:40.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7945b99b80 1 Processor -- start 2023-10-11T16:40:40.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7945b99b80 1 -- start start 2023-10-11T16:40:40.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08afd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7945b99b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef08af20f0 con 0x55ef08ae7c80 2023-10-11T16:40:40.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7945b99b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef08ad3ee0 con 0x55ef08aec0c0 2023-10-11T16:40:40.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7945b99b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef08ae8050 con 0x55ef08ae7370 2023-10-11T16:40:40.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7923fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f792b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7923fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37440/0 (socket says 172.21.15.111:37440) 2023-10-11T16:40:40.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f792bfff700 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08afd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.754+0000 7f7923fff700 1 -- 172.21.15.111:0/3239595111 learned_addr learned my addr 172.21.15.111:0/3239595111 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:40.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f792b7fe700 1 -- 172.21.15.111:0/3239595111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 msgr2=0x55ef08afd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f792b7fe700 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08afd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f792b7fe700 1 -- 172.21.15.111:0/3239595111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 msgr2=0x55ef08af02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f792b7fe700 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f792b7fe700 1 -- 172.21.15.111:0/3239595111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef08aec760 con 0x55ef08ae7370 2023-10-11T16:40:40.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f7923fff700 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f792b7fe700 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af81b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55ef08b03a60 tx=0x7f791c020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75eebc77c175a3e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.755+0000 7f79227fc700 1 -- 172.21.15.111:0/3239595111 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c034110 con 0x55ef08ae7370 2023-10-11T16:40:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f79227fc700 1 -- 172.21.15.111:0/3239595111 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f791c01e040 con 0x55ef08ae7370 2023-10-11T16:40:40.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f79227fc700 1 -- 172.21.15.111:0/3239595111 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c00f7c0 con 0x55ef08ae7370 2023-10-11T16:40:40.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 -- 172.21.15.111:0/3239595111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 msgr2=0x55ef08af81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af81b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55ef08b03a60 tx=0x7f791c020000 comp rx=0 tx=0).stop 2023-10-11T16:40:40.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 -- 172.21.15.111:0/3239595111 shutdown_connections 2023-10-11T16:40:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08aec0c0 0x55ef08afd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af81b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 --2- 172.21.15.111:0/3239595111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08ae7c80 0x55ef08af02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 -- 172.21.15.111:0/3239595111 >> 172.21.15.111:0/3239595111 conn(0x55ef08ad4360 msgr2=0x55ef08ad8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 -- 172.21.15.111:0/3239595111 shutdown_connections 2023-10-11T16:40:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.756+0000 7f7945b99b80 1 -- 172.21.15.111:0/3239595111 wait complete. 2023-10-11T16:40:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 1 -- start start 2023-10-11T16:40:40.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 1 Processor -- start 2023-10-11T16:40:40.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7c80 0x55ef08af66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08aec0c0 0x55ef08af6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef08b53ce0 con 0x55ef08aec0c0 2023-10-11T16:40:40.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef08b53e20 con 0x55ef08ae7c80 2023-10-11T16:40:40.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef08b53fe0 con 0x55ef08ae7370 2023-10-11T16:40:40.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.757+0000 7f7945b99b80 10 client.0 _ll_register_callbacks cb 0x55ef089ef780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:40.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08aec0c0 0x55ef08af6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08aec0c0 0x55ef08af6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37452/0 (socket says 172.21.15.111:37452) 2023-10-11T16:40:40.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792bfff700 1 -- 172.21.15.111:0/2496875758 learned_addr learned my addr 172.21.15.111:0/2496875758 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:40.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36314/0 (socket says 172.21.15.111:36314) 2023-10-11T16:40:40.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f7923fff700 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7c80 0x55ef08af66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792b7fe700 1 -- 172.21.15.111:0/2496875758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7c80 msgr2=0x55ef08af66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792b7fe700 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7c80 0x55ef08af66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792b7fe700 1 -- 172.21.15.111:0/2496875758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08aec0c0 msgr2=0x55ef08af6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:40.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792b7fe700 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08aec0c0 0x55ef08af6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792b7fe700 1 -- 172.21.15.111:0/2496875758 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c10d990 con 0x55ef08ae7370 2023-10-11T16:40:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792bfff700 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08aec0c0 0x55ef08af6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f792b7fe700 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af61a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f791c000c00 tx=0x7f791c009300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9b51aab9b032986 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:40.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.758+0000 7f790affd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:40:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f790affd700 1 -- 172.21.15.111:0/2496875758 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c0053e0 con 0x55ef08ae7370 2023-10-11T16:40:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 1 -- 172.21.15.111:0/2496875758 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79140a24d0 con 0x55ef08ae7370 2023-10-11T16:40:40.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f790affd700 1 -- 172.21.15.111:0/2496875758 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f791c034310 con 0x55ef08ae7370 2023-10-11T16:40:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f790affd700 1 -- 172.21.15.111:0/2496875758 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f791c0344d0 con 0x55ef08ae7370 2023-10-11T16:40:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 10 client.26959 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:40.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 1 -- 172.21.15.111:0/2496875758 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ef0885dde0 con 0x55ef08ae7370 2023-10-11T16:40:40.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f790affd700 1 -- 172.21.15.111:0/2496875758 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79140a24d0 con 0x55ef08ae7370 2023-10-11T16:40:40.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f790affd700 1 -- 172.21.15.111:0/2496875758 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f791c0046c0 con 0x55ef08ae7370 2023-10-11T16:40:40.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 10 client.26959 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:40.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 10 client.26959 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:40:40.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 20 client.26959 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:40.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55ef08b552b0 0x55ef08b57690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:40.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 4 client.26959 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:40:40.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.759+0000 7f7945b99b80 1 -- 172.21.15.111:0/2496875758 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ef089fe660 con 0x55ef08b552b0 2023-10-11T16:40:40.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.760+0000 7f7923fff700 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55ef08b552b0 0x55ef08b57690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:40.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.760+0000 7f790affd700 10 client.26959 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:40:40.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.760+0000 7f7923fff700 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55ef08b552b0 0x55ef08b57690 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef8759f6802cec96 server_cookie=ddccd7bbef9f0458 in_seq=0 out_seq=0 2023-10-11T16:40:40.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.760+0000 7f790affd700 1 -- 172.21.15.111:0/2496875758 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ef089fe660 con 0x55ef08b552b0 2023-10-11T16:40:40.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.760+0000 7f790affd700 10 client.26959 handle_command_reply: tid=0 2023-10-11T16:40:40.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.760+0000 7f7945b99b80 1 client.26959 shutdown 2023-10-11T16:40:40.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 -- 172.21.15.111:0/2496875758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 msgr2=0x55ef08af61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af61a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f791c000c00 tx=0x7f791c009300 comp rx=0 tx=0).stop 2023-10-11T16:40:40.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 -- 172.21.15.111:0/2496875758 shutdown_connections 2023-10-11T16:40:40.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55ef08b552b0 0x55ef08b57690 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef08aec0c0 0x55ef08af6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef08ae7370 0x55ef08af61a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f790affd700 0 client.26959 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:40:40.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 --2- 172.21.15.111:0/2496875758 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef08ae7c80 0x55ef08af66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 -- 172.21.15.111:0/2496875758 >> 172.21.15.111:0/2496875758 conn(0x55ef08ad4360 msgr2=0x55ef08b02d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 -- 172.21.15.111:0/2496875758 shutdown_connections 2023-10-11T16:40:40.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 1 -- 172.21.15.111:0/2496875758 wait complete. 2023-10-11T16:40:40.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f7945b99b80 20 client.26959 trim_cache size 0 max 16384 2023-10-11T16:40:40.798 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:40:40.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f79321ff700 1 -- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f79180a12c0 msgr2=0x7f79180a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f79321ff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f79180a12c0 0x7f79180a3770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f791c006010 tx=0x7f791c005960 comp rx=0 tx=0).stop 2023-10-11T16:40:40.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f79321ff700 1 -- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f792c101ca0 msgr2=0x7f792c0ff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:40.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.761+0000 7f79321ff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f792c101ca0 0x7f792c0ff700 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f791400e9f0 tx=0x7f791400ed00 comp rx=0 tx=0).stop 2023-10-11T16:40:40.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.762+0000 7f79321ff700 1 -- 172.21.15.111:0/1192994334 shutdown_connections 2023-10-11T16:40:40.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.762+0000 7f79321ff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f79180a12c0 0x7f79180a3770 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.762+0000 7f79321ff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792c10a8b0 0x7f792c1a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.762+0000 7f79321ff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f792c101390 0x7f792c0ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.762+0000 7f79321ff700 1 --2- 172.21.15.111:0/1192994334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f792c101ca0 0x7f792c0ff700 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:40.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.762+0000 7f79321ff700 1 -- 172.21.15.111:0/1192994334 >> 172.21.15.111:0/1192994334 conn(0x7f792c0770b0 msgr2=0x7f792c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:40.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.762+0000 7f79321ff700 1 -- 172.21.15.111:0/1192994334 shutdown_connections 2023-10-11T16:40:40.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:40.762+0000 7f79321ff700 1 -- 172.21.15.111:0/1192994334 wait complete. 2023-10-11T16:40:40.803 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:40:41.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 -- 172.21.15.111:0/2689261092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88101660 msgr2=0x7f1c88104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.003 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/2689261092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88101660 0x7f1c88104ef0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1c740099c0 tx=0x7f1c74009cd0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 -- 172.21.15.111:0/2689261092 shutdown_connections 2023-10-11T16:40:41.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/2689261092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c88105620 0x7f1c8810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/2689261092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c88100cc0 0x7f1c88101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.004 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/2689261092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88101660 0x7f1c88104ef0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 -- 172.21.15.111:0/2689261092 >> 172.21.15.111:0/2689261092 conn(0x7f1c88077560 msgr2=0x7f1c88077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 -- 172.21.15.111:0/2689261092 shutdown_connections 2023-10-11T16:40:41.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.001+0000 7f1c8ebff700 1 -- 172.21.15.111:0/2689261092 wait complete. 2023-10-11T16:40:41.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8ebff700 1 Processor -- start 2023-10-11T16:40:41.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8ebff700 1 -- start start 2023-10-11T16:40:41.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8ebff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88100cc0 0x7f1c881964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8ebff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c88101660 0x7f1c881969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8ebff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c88105620 0x7f1c8819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8ebff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8819b490 con 0x7f1c88100cc0 2023-10-11T16:40:41.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8ebff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8819b600 con 0x7f1c88101660 2023-10-11T16:40:41.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8ebff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8819b7c0 con 0x7f1c88105620 2023-10-11T16:40:41.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.002+0000 7f1c8c99b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88100cc0 0x7f1c881964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8c99b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88100cc0 0x7f1c881964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37478/0 (socket says 172.21.15.111:37478) 2023-10-11T16:40:41.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8c99b700 1 -- 172.21.15.111:0/3383507268 learned_addr learned my addr 172.21.15.111:0/3383507268 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c87fff700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c88101660 0x7f1c881969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8d19c700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c88105620 0x7f1c8819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8c99b700 1 -- 172.21.15.111:0/3383507268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c88105620 msgr2=0x7f1c8819ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c88105620 0x7f1c8819ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.012 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8c99b700 1 -- 172.21.15.111:0/3383507268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c88101660 msgr2=0x7f1c881969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c88101660 0x7f1c881969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8c99b700 1 -- 172.21.15.111:0/3383507268 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c740095f0 con 0x7f1c88100cc0 2023-10-11T16:40:41.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88100cc0 0x7f1c881964b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00e9a0 tx=0x7f1c7c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c85ffb700 1 -- 172.21.15.111:0/3383507268 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c009790 con 0x7f1c88100cc0 2023-10-11T16:40:41.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c85ffb700 1 -- 172.21.15.111:0/3383507268 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c7c00cbd0 con 0x7f1c88100cc0 2023-10-11T16:40:41.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c8ebff700 1 -- 172.21.15.111:0/3383507268 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c88110370 con 0x7f1c88100cc0 2023-10-11T16:40:41.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.003+0000 7f1c85ffb700 1 -- 172.21.15.111:0/3383507268 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c018680 con 0x7f1c88100cc0 2023-10-11T16:40:41.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.004+0000 7f1c8ebff700 1 -- 172.21.15.111:0/3383507268 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c881101e0 con 0x7f1c88100cc0 2023-10-11T16:40:41.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.007+0000 7f1c85ffb700 1 -- 172.21.15.111:0/3383507268 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f1c7c00cd40 con 0x7f1c88100cc0 2023-10-11T16:40:41.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.008+0000 7f1ca2580b80 1 Processor -- start 2023-10-11T16:40:41.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.008+0000 7f1c85ffb700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1c780a12c0 0x7f1c780a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.008+0000 7f1ca2580b80 1 -- start start 2023-10-11T16:40:41.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.008+0000 7f1c85ffb700 1 -- 172.21.15.111:0/3383507268 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f1c7c014070 con 0x7f1c88100cc0 2023-10-11T16:40:41.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.008+0000 7f1c8d19c700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1c780a12c0 0x7f1c780a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f251c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f251c880 0x5571f2520780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f2520cc0 0x5571f2521110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f241d060 con 0x5571f251bf70 2023-10-11T16:40:41.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f246f8a0 con 0x5571f2520cc0 2023-10-11T16:40:41.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f2468d30 con 0x5571f251c880 2023-10-11T16:40:41.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f251c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8d19c700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1c780a12c0 0x7f1c780a3770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009380 tx=0x7f1c8000b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f251c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37486/0 (socket says 172.21.15.111:37486) 2023-10-11T16:40:41.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 -- 172.21.15.111:0/1062627376 learned_addr learned my addr 172.21.15.111:0/1062627376 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8d19c700 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f2520cc0 0x5571f2521110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c87fff700 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f251c880 0x5571f2520780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 -- 172.21.15.111:0/1062627376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f251c880 msgr2=0x5571f2520780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f251c880 0x5571f2520780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 -- 172.21.15.111:0/1062627376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f2520cc0 msgr2=0x5571f2521110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f2520cc0 0x5571f2521110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 -- 172.21.15.111:0/1062627376 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571f22ea630 con 0x5571f251bf70 2023-10-11T16:40:41.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.009+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f251c340 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c096ab0 tx=0x7f1c7c099a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772d78b737ec8e09 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.028 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.010+0000 7f1c72ffd700 1 -- 172.21.15.111:0/1062627376 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c09e070 con 0x5571f251bf70 2023-10-11T16:40:41.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.010+0000 7f1c72ffd700 1 -- 172.21.15.111:0/1062627376 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c7c0a2490 con 0x5571f251bf70 2023-10-11T16:40:41.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.010+0000 7f1c72ffd700 1 -- 172.21.15.111:0/1062627376 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0a2000 con 0x5571f251bf70 2023-10-11T16:40:41.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.010+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1062627376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 msgr2=0x5571f251c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.010+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f251c340 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c096ab0 tx=0x7f1c7c099a20 comp rx=0 tx=0).stop 2023-10-11T16:40:41.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1062627376 shutdown_connections 2023-10-11T16:40:41.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f2520cc0 0x5571f2521110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f251c340 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/1062627376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f251c880 0x5571f2520780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1062627376 >> 172.21.15.111:0/1062627376 conn(0x5571f250d360 msgr2=0x5571f2512b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.031 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1062627376 shutdown_connections 2023-10-11T16:40:41.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1062627376 wait complete. 2023-10-11T16:40:41.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 -- start start 2023-10-11T16:40:41.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.011+0000 7f1ca2580b80 1 Processor -- start 2023-10-11T16:40:41.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 0x5571f2528500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f25257f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f2525f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f240a8e0 con 0x5571f251c880 2023-10-11T16:40:41.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f240a580 con 0x5571f251bf70 2023-10-11T16:40:41.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f22ec920 con 0x5571f2520cc0 2023-10-11T16:40:41.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1ca2580b80 10 client.0 _ll_register_callbacks cb 0x5571f2428780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:41.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c87fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f25257f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c87fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f25257f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37496/0 (socket says 172.21.15.111:37496) 2023-10-11T16:40:41.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c87fff700 1 -- 172.21.15.111:0/2180329035 learned_addr learned my addr 172.21.15.111:0/2180329035 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 0x5571f2528500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c8d19c700 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f2525f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c87fff700 1 -- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 msgr2=0x5571f2525f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c87fff700 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f2525f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c87fff700 1 -- 172.21.15.111:0/2180329035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 msgr2=0x5571f2528500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c87fff700 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 0x5571f2528500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c87fff700 1 -- 172.21.15.111:0/2180329035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c881101e0 con 0x5571f251c880 2023-10-11T16:40:41.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.012+0000 7f1c8d19c700 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f2525f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:41.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1c87fff700 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f25257f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1c740098e0 tx=0x7f1c74003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a589ba1756b4de server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1c67fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:41.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1c67fff700 1 -- 172.21.15.111:0/2180329035 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74005610 con 0x5571f251c880 2023-10-11T16:40:41.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2180329035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571f250ab20 con 0x5571f251c880 2023-10-11T16:40:41.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1c67fff700 1 -- 172.21.15.111:0/2180329035 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f1c7400b790 con 0x5571f251c880 2023-10-11T16:40:41.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1c67fff700 1 -- 172.21.15.111:0/2180329035 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c7400b990 con 0x5571f251c880 2023-10-11T16:40:41.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1ca2580b80 10 client.16848 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:41.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2180329035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5571f242f190 con 0x5571f251c880 2023-10-11T16:40:41.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.013+0000 7f1c67fff700 1 -- 172.21.15.111:0/2180329035 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5571f250ab20 con 0x5571f251c880 2023-10-11T16:40:41.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.014+0000 7f1c67fff700 1 -- 172.21.15.111:0/2180329035 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f1c7400fb90 con 0x5571f251c880 2023-10-11T16:40:41.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.014+0000 7f1ca2580b80 10 client.16848 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:41.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.014+0000 7f1ca2580b80 10 client.16848 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:40:41.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.014+0000 7f1ca2580b80 20 client.16848 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:41.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.014+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x5571f2535d70 0x5571f2538180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.014+0000 7f1ca2580b80 4 client.16848 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:41.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.014+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2180329035 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5571f242b0a0 con 0x5571f2535d70 2023-10-11T16:40:41.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.014+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x5571f2535d70 0x5571f2538180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.015+0000 7f1c67fff700 10 client.16848 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:40:41.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.015+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x5571f2535d70 0x5571f2538180 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ac9098d1e008888 server_cookie=d8eb58e0a2864309 in_seq=0 out_seq=0 2023-10-11T16:40:41.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.015+0000 7f1c67fff700 1 -- 172.21.15.111:0/2180329035 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5571f242b0a0 con 0x5571f2535d70 2023-10-11T16:40:41.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.015+0000 7f1c67fff700 10 client.16848 handle_command_reply: tid=0 2023-10-11T16:40:41.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.015+0000 7f1ca2580b80 1 client.16848 shutdown 2023-10-11T16:40:41.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2180329035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 msgr2=0x5571f25257f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f25257f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1c740098e0 tx=0x7f1c74003ed0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2180329035 shutdown_connections 2023-10-11T16:40:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x5571f2535d70 0x5571f2538180 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f2525f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 0x5571f2528500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2180329035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f25257f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2180329035 >> 172.21.15.111:0/2180329035 conn(0x5571f250d360 msgr2=0x5571f2522170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1c67fff700 0 client.16848 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:40:41.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2180329035 shutdown_connections 2023-10-11T16:40:41.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2180329035 wait complete. 2023-10-11T16:40:41.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.016+0000 7f1ca2580b80 20 client.16848 trim_cache size 0 max 16384 2023-10-11T16:40:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.029+0000 7f1ca2580b80 1 Processor -- start 2023-10-11T16:40:41.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.029+0000 7f1ca2580b80 1 -- start start 2023-10-11T16:40:41.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f25250e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251c880 0x5571f2511550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f25280a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f2526d60 con 0x5571f251bf70 2023-10-11T16:40:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f250cee0 con 0x5571f251c880 2023-10-11T16:40:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f25141c0 con 0x5571f2520cc0 2023-10-11T16:40:41.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f25250e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f25250e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37508/0 (socket says 172.21.15.111:37508) 2023-10-11T16:40:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 -- 172.21.15.111:0/1392172948 learned_addr learned my addr 172.21.15.111:0/1392172948 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c87fff700 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251c880 0x5571f2511550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8d19c700 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f25280a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 -- 172.21.15.111:0/1392172948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 msgr2=0x5571f25280a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f25280a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 -- 172.21.15.111:0/1392172948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251c880 msgr2=0x5571f2511550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251c880 0x5571f2511550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 -- 172.21.15.111:0/1392172948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571f242eee0 con 0x5571f251bf70 2023-10-11T16:40:41.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c87fff700 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251c880 0x5571f2511550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.030+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f25250e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c5b0 tx=0x7f1c7c0cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e663cc34aa5ba322 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1c737fe700 1 -- 172.21.15.111:0/1392172948 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0983e0 con 0x5571f251bf70 2023-10-11T16:40:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1c737fe700 1 -- 172.21.15.111:0/1392172948 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c7c0b82f0 con 0x5571f251bf70 2023-10-11T16:40:41.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1c737fe700 1 -- 172.21.15.111:0/1392172948 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c099790 con 0x5571f251bf70 2023-10-11T16:40:41.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1392172948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 msgr2=0x5571f25250e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f25250e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c5b0 tx=0x7f1c7c0cc040 comp rx=0 tx=0).stop 2023-10-11T16:40:41.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1392172948 shutdown_connections 2023-10-11T16:40:41.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f25280a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251bf70 0x5571f25250e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/1392172948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251c880 0x5571f2511550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1392172948 >> 172.21.15.111:0/1392172948 conn(0x5571f250d360 msgr2=0x5571f25320f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1392172948 shutdown_connections 2023-10-11T16:40:41.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.031+0000 7f1ca2580b80 1 -- 172.21.15.111:0/1392172948 wait complete. 2023-10-11T16:40:41.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 1 -- start start 2023-10-11T16:40:41.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 1 Processor -- start 2023-10-11T16:40:41.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 0x5571f252cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f252ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f252b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f252bba0 con 0x5571f251c880 2023-10-11T16:40:41.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f2588c30 con 0x5571f251bf70 2023-10-11T16:40:41.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571f2588d70 con 0x5571f2520cc0 2023-10-11T16:40:41.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.032+0000 7f1ca2580b80 10 client.0 _ll_register_callbacks cb 0x7f1c74003ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:41.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f252ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f252ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37510/0 (socket says 172.21.15.111:37510) 2023-10-11T16:40:41.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 -- 172.21.15.111:0/2462568102 learned_addr learned my addr 172.21.15.111:0/2462568102 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 0x5571f252cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c8d19c700 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f252b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 -- 172.21.15.111:0/2462568102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 msgr2=0x5571f252b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f252b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 -- 172.21.15.111:0/2462568102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 msgr2=0x5571f252cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 0x5571f252cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 -- 172.21.15.111:0/2462568102 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7c0b1040 con 0x5571f251c880 2023-10-11T16:40:41.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c87fff700 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f252ada0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1c740058e0 tx=0x7f1c74005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87587d7067c72a65 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c67fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:41.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c67fff700 1 -- 172.21.15.111:0/2462568102 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74005940 con 0x5571f251c880 2023-10-11T16:40:41.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1c67fff700 1 -- 172.21.15.111:0/2462568102 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f1c74000e40 con 0x5571f251c880 2023-10-11T16:40:41.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.033+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2462568102 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c7c0a2490 con 0x5571f251c880 2023-10-11T16:40:41.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1c67fff700 1 -- 172.21.15.111:0/2462568102 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c74005bc0 con 0x5571f251c880 2023-10-11T16:40:41.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1ca2580b80 10 client.16854 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:41.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2462568102 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5571f227fc60 con 0x5571f251c880 2023-10-11T16:40:41.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1c67fff700 1 -- 172.21.15.111:0/2462568102 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c7c0a2490 con 0x5571f251c880 2023-10-11T16:40:41.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1c67fff700 1 -- 172.21.15.111:0/2462568102 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f1c7400b260 con 0x5571f251c880 2023-10-11T16:40:41.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1ca2580b80 10 client.16854 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:41.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1ca2580b80 10 client.16854 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:40:41.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1ca2580b80 20 client.16854 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:41.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x5571f258a040 0x5571f258c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1ca2580b80 4 client.16854 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:40:41.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.034+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2462568102 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5571f2437660 con 0x5571f258a040 2023-10-11T16:40:41.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x5571f258a040 0x5571f258c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1c67fff700 10 client.16854 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:40:41.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1c8c99b700 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x5571f258a040 0x5571f258c450 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a17bf9469e67d4b6 server_cookie=6f8aab05b39afabf in_seq=0 out_seq=0 2023-10-11T16:40:41.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1c67fff700 1 -- 172.21.15.111:0/2462568102 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5571f2437660 con 0x5571f258a040 2023-10-11T16:40:41.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1c67fff700 10 client.16854 handle_command_reply: tid=0 2023-10-11T16:40:41.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 client.16854 shutdown 2023-10-11T16:40:41.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2462568102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 msgr2=0x5571f252ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f252ada0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f1c740058e0 tx=0x7f1c74005740 comp rx=0 tx=0).stop 2023-10-11T16:40:41.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2462568102 shutdown_connections 2023-10-11T16:40:41.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x5571f258a040 0x5571f258c450 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5571f2520cc0 0x5571f252b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5571f251bf70 0x5571f252cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 --2- 172.21.15.111:0/2462568102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5571f251c880 0x5571f252ada0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1c67fff700 0 client.16854 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:40:41.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.035+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2462568102 >> 172.21.15.111:0/2462568102 conn(0x5571f250d360 msgr2=0x5571f25320f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2462568102 shutdown_connections 2023-10-11T16:40:41.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1ca2580b80 1 -- 172.21.15.111:0/2462568102 wait complete. 2023-10-11T16:40:41.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1ca2580b80 20 client.16854 trim_cache size 0 max 16384 2023-10-11T16:40:41.079 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:40:41.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 -- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1c780a12c0 msgr2=0x7f1c780a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1c780a12c0 0x7f1c780a3770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009380 tx=0x7f1c8000b040 comp rx=0 tx=0).stop 2023-10-11T16:40:41.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 -- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88100cc0 msgr2=0x7f1c881964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88100cc0 0x7f1c881964b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00e9a0 tx=0x7f1c7c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 -- 172.21.15.111:0/3383507268 shutdown_connections 2023-10-11T16:40:41.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1c780a12c0 0x7f1c780a3770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c88105620 0x7f1c8819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1c88100cc0 0x7f1c881964b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 --2- 172.21.15.111:0/3383507268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c88101660 0x7f1c881969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 -- 172.21.15.111:0/3383507268 >> 172.21.15.111:0/3383507268 conn(0x7f1c88077560 msgr2=0x7f1c880ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.036+0000 7f1c8ebff700 1 -- 172.21.15.111:0/3383507268 shutdown_connections 2023-10-11T16:40:41.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.037+0000 7f1c8ebff700 1 -- 172.21.15.111:0/3383507268 wait complete. 2023-10-11T16:40:41.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:40:41.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1199768340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c0074b00 msgr2=0x7fc6c0073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1199768340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c0074b00 0x7fc6c0073260 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009990 tx=0x7fc6a8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1199768340 shutdown_connections 2023-10-11T16:40:41.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1199768340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c010cec0 0x7fc6c010f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1199768340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6c00746a0 0x7fc6c0072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1199768340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c0074b00 0x7fc6c0073260 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1199768340 >> 172.21.15.111:0/1199768340 conn(0x7fc6c0077590 msgr2=0x7fc6c0077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1199768340 shutdown_connections 2023-10-11T16:40:41.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.284+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1199768340 wait complete. 2023-10-11T16:40:41.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.285+0000 7fc6c49ff700 1 Processor -- start 2023-10-11T16:40:41.292 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.285+0000 7fc6c49ff700 1 -- start start 2023-10-11T16:40:41.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.285+0000 7fc6c49ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c00746a0 0x7fc6c0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.285+0000 7fc6c49ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6c0074b00 0x7fc6c0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.285+0000 7fc6c49ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c010cec0 0x7fc6c019d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.293 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.285+0000 7fc6c49ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c019d660 con 0x7fc6c00746a0 2023-10-11T16:40:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.285+0000 7fc6c49ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c019d7d0 con 0x7fc6c0074b00 2023-10-11T16:40:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.285+0000 7fc6c49ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c019d990 con 0x7fc6c010cec0 2023-10-11T16:40:41.294 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c00746a0 0x7fc6c0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c00746a0 0x7fc6c0198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37526/0 (socket says 172.21.15.111:37526) 2023-10-11T16:40:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 -- 172.21.15.111:0/1658456075 learned_addr learned my addr 172.21.15.111:0/1658456075 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c010cec0 0x7fc6c019d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.295 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6c0074b00 0x7fc6c0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 -- 172.21.15.111:0/1658456075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c010cec0 msgr2=0x7fc6c019d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c010cec0 0x7fc6c019d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.296 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 -- 172.21.15.111:0/1658456075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6c0074b00 msgr2=0x7fc6c0198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6c0074b00 0x7fc6c0198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 -- 172.21.15.111:0/1658456075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b0009c40 con 0x7fc6c00746a0 2023-10-11T16:40:41.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6c0074b00 0x7fc6c0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:41.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6be59c700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c00746a0 0x7fc6c0198680 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000c370 tx=0x7fc6b000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6b77fe700 1 -- 172.21.15.111:0/1658456075 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0017070 con 0x7fc6c00746a0 2023-10-11T16:40:41.298 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1658456075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a80095f0 con 0x7fc6c00746a0 2023-10-11T16:40:41.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.286+0000 7fc6b77fe700 1 -- 172.21.15.111:0/1658456075 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6b000f040 con 0x7fc6c00746a0 2023-10-11T16:40:41.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.287+0000 7fc6b77fe700 1 -- 172.21.15.111:0/1658456075 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b000e050 con 0x7fc6c00746a0 2023-10-11T16:40:41.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.287+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1658456075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c01125a0 con 0x7fc6c00746a0 2023-10-11T16:40:41.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.291+0000 7fc6b77fe700 1 -- 172.21.15.111:0/1658456075 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fc6b00090d0 con 0x7fc6c00746a0 2023-10-11T16:40:41.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.292+0000 7fc6d83c9b80 1 Processor -- start 2023-10-11T16:40:41.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.292+0000 7fc6d83c9b80 1 -- start start 2023-10-11T16:40:41.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.292+0000 7fc6b77fe700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc6ac0a12c0 0x7fc6ac0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.292+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b8bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.292+0000 7fc6b77fe700 1 -- 172.21.15.111:0/1658456075 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fc6b00ccd00 con 0x7fc6c00746a0 2023-10-11T16:40:41.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b90370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b908b0 0x55fa01b90d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01a91060 con 0x55fa01b8c470 2023-10-11T16:40:41.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01ae38a0 con 0x55fa01b908b0 2023-10-11T16:40:41.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01adcd30 con 0x55fa01b8bb60 2023-10-11T16:40:41.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc6ac0a12c0 0x7fc6ac0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6be59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b8bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6be59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b8bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36392/0 (socket says 172.21.15.111:36392) 2023-10-11T16:40:41.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6be59c700 1 -- 172.21.15.111:0/795105623 learned_addr learned my addr 172.21.15.111:0/795105623 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b90370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b908b0 0x55fa01b90d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6be59c700 1 -- 172.21.15.111:0/795105623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b908b0 msgr2=0x55fa01b90d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6be59c700 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b908b0 0x55fa01b90d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6be59c700 1 -- 172.21.15.111:0/795105623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 msgr2=0x55fa01b90370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6be59c700 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b90370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6be59c700 1 -- 172.21.15.111:0/795105623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa0195e630 con 0x55fa01b8bb60 2023-10-11T16:40:41.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.293+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc6ac0a12c0 0x7fc6ac0a3770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc6c006a100 tx=0x7fc6b8009f90 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.294+0000 7fc6be59c700 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b8bf30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc6b001e030 tx=0x7fc6b0099520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ddfbeb44144f4fa server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.294+0000 7fc6b4ff9700 1 -- 172.21.15.111:0/795105623 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b009d990 con 0x55fa01b8bb60 2023-10-11T16:40:41.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.294+0000 7fc6b4ff9700 1 -- 172.21.15.111:0/795105623 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6b00a1db0 con 0x55fa01b8bb60 2023-10-11T16:40:41.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.294+0000 7fc6b4ff9700 1 -- 172.21.15.111:0/795105623 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b00a1920 con 0x55fa01b8bb60 2023-10-11T16:40:41.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.294+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/795105623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 msgr2=0x55fa01b8bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.294+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b8bf30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc6b001e030 tx=0x7fc6b0099520 comp rx=0 tx=0).stop 2023-10-11T16:40:41.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.294+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/795105623 shutdown_connections 2023-10-11T16:40:41.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.295+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b908b0 0x55fa01b90d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.295+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b8bf30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.295+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/795105623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b90370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.295+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/795105623 >> 172.21.15.111:0/795105623 conn(0x55fa01b81360 msgr2=0x55fa01b86b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.295+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/795105623 shutdown_connections 2023-10-11T16:40:41.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.295+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/795105623 wait complete. 2023-10-11T16:40:41.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.295+0000 7fc6d83c9b80 1 -- start start 2023-10-11T16:40:41.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.295+0000 7fc6d83c9b80 1 Processor -- start 2023-10-11T16:40:41.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8bb60 0x55fa01b98070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b952f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b908b0 0x55fa01b95a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01a7e8e0 con 0x55fa01b8c470 2023-10-11T16:40:41.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01a7e580 con 0x55fa01b8bb60 2023-10-11T16:40:41.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01960920 con 0x55fa01b908b0 2023-10-11T16:40:41.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6d83c9b80 10 client.0 _ll_register_callbacks cb 0x55fa01a9c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:41.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b952f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37548/0 (socket says 172.21.15.111:37548) 2023-10-11T16:40:41.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6b7fff700 1 -- 172.21.15.111:0/3193782909 learned_addr learned my addr 172.21.15.111:0/3193782909 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6be59c700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8bb60 0x55fa01b98070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b908b0 0x55fa01b95a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6b7fff700 1 -- 172.21.15.111:0/3193782909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b908b0 msgr2=0x55fa01b95a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b908b0 0x55fa01b95a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6b7fff700 1 -- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8bb60 msgr2=0x55fa01b98070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8bb60 0x55fa01b98070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.296+0000 7fc6b7fff700 1 -- 172.21.15.111:0/3193782909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b0084040 con 0x55fa01b8c470 2023-10-11T16:40:41.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b908b0 0x55fa01b95a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:41.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6be59c700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8bb60 0x55fa01b98070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:41.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b952f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009990 tx=0x7fc6a800ba80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7dc367015bf684 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:41.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3193782909 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a800f930 con 0x55fa01b8c470 2023-10-11T16:40:41.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3193782909 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fc6a800fa90 con 0x55fa01b8c470 2023-10-11T16:40:41.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3193782909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa01b7eb20 con 0x55fa01b8c470 2023-10-11T16:40:41.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3193782909 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6a800fc50 con 0x55fa01b8c470 2023-10-11T16:40:41.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6d83c9b80 10 client.16866 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:41.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.297+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3193782909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55fa01aa3190 con 0x55fa01b8c470 2023-10-11T16:40:41.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3193782909 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fa01b7eb20 con 0x55fa01b8c470 2023-10-11T16:40:41.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3193782909 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc6a80173f0 con 0x55fa01b8c470 2023-10-11T16:40:41.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6d83c9b80 10 client.16866 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:41.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6d83c9b80 10 client.16866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:41.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6d83c9b80 20 client.16866 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:41.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fa01ba58d0 0x55fa01ba7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6d83c9b80 4 client.16866 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:41.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3193782909 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa01a9f0a0 con 0x55fa01ba58d0 2023-10-11T16:40:41.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.298+0000 7fc6be59c700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fa01ba58d0 0x55fa01ba7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.299+0000 7fc6a5ffb700 10 client.16866 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:41.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.299+0000 7fc6be59c700 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fa01ba58d0 0x55fa01ba7ce0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e58a3b8e9672e047 server_cookie=aaff7df72b528afd in_seq=0 out_seq=0 2023-10-11T16:40:41.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.299+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3193782909 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55fa01a9f0a0 con 0x55fa01ba58d0 2023-10-11T16:40:41.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6a5ffb700 10 client.16866 handle_command_reply: tid=0 2023-10-11T16:40:41.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 client.16866 shutdown 2023-10-11T16:40:41.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3193782909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 msgr2=0x55fa01b952f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b952f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009990 tx=0x7fc6a800ba80 comp rx=0 tx=0).stop 2023-10-11T16:40:41.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3193782909 shutdown_connections 2023-10-11T16:40:41.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fa01ba58d0 0x55fa01ba7ce0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b908b0 0x55fa01b95a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8bb60 0x55fa01b98070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3193782909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b952f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6a5ffb700 0 client.16866 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:41.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3193782909 >> 172.21.15.111:0/3193782909 conn(0x55fa01b81360 msgr2=0x55fa01b91d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3193782909 shutdown_connections 2023-10-11T16:40:41.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3193782909 wait complete. 2023-10-11T16:40:41.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.300+0000 7fc6d83c9b80 20 client.16866 trim_cache size 0 max 16384 2023-10-11T16:40:41.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.313+0000 7fc6d83c9b80 1 Processor -- start 2023-10-11T16:40:41.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6d83c9b80 1 -- start start 2023-10-11T16:40:41.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b81360 0x55fa01b9c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b97c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b8e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01b80ee0 con 0x55fa01b8c470 2023-10-11T16:40:41.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01b93850 con 0x55fa01b81360 2023-10-11T16:40:41.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01b8f4e0 con 0x55fa01b8bb60 2023-10-11T16:40:41.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6bed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b8e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6bed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b8e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37558/0 (socket says 172.21.15.111:37558) 2023-10-11T16:40:41.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6bed9d700 1 -- 172.21.15.111:0/1903208107 learned_addr learned my addr 172.21.15.111:0/1903208107 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6be59c700 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b81360 0x55fa01b9c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b97c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6bed9d700 1 -- 172.21.15.111:0/1903208107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 msgr2=0x55fa01b97c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b97c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6bed9d700 1 -- 172.21.15.111:0/1903208107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b81360 msgr2=0x55fa01b9c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b81360 0x55fa01b9c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6bed9d700 1 -- 172.21.15.111:0/1903208107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa01b82b00 con 0x55fa01b8c470 2023-10-11T16:40:41.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.314+0000 7fc6be59c700 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b81360 0x55fa01b9c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:41.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.315+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b8e9d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800f790 tx=0x7fc6b801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbca5cfef618c03d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.315+0000 7fc6b57fa700 1 -- 172.21.15.111:0/1903208107 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8028610 con 0x55fa01b8c470 2023-10-11T16:40:41.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.315+0000 7fc6b57fa700 1 -- 172.21.15.111:0/1903208107 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6b8028c30 con 0x55fa01b8c470 2023-10-11T16:40:41.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.315+0000 7fc6b57fa700 1 -- 172.21.15.111:0/1903208107 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8020dc0 con 0x55fa01b8c470 2023-10-11T16:40:41.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/1903208107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 msgr2=0x55fa01b8e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b8e9d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800f790 tx=0x7fc6b801d000 comp rx=0 tx=0).stop 2023-10-11T16:40:41.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/1903208107 shutdown_connections 2023-10-11T16:40:41.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8c470 0x55fa01b8e9d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b81360 0x55fa01b9c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/1903208107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b8bb60 0x55fa01b97c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/1903208107 >> 172.21.15.111:0/1903208107 conn(0x55fa01b86230 msgr2=0x55fa01ba1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/1903208107 shutdown_connections 2023-10-11T16:40:41.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/1903208107 wait complete. 2023-10-11T16:40:41.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 -- start start 2023-10-11T16:40:41.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.316+0000 7fc6d83c9b80 1 Processor -- start 2023-10-11T16:40:41.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b81360 0x55fa01b9a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8bb60 0x55fa01b9ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6d83c9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8c470 0x55fa01b9b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01bf87b0 con 0x55fa01b8bb60 2023-10-11T16:40:41.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01bf88f0 con 0x55fa01b8c470 2023-10-11T16:40:41.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6d83c9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa01bf8ab0 con 0x55fa01b81360 2023-10-11T16:40:41.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6d83c9b80 10 client.0 _ll_register_callbacks cb 0x7fc6a8009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:41.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8bb60 0x55fa01b9ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6bed9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8c470 0x55fa01b9b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8bb60 0x55fa01b9ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37568/0 (socket says 172.21.15.111:37568) 2023-10-11T16:40:41.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6b7fff700 1 -- 172.21.15.111:0/3349761913 learned_addr learned my addr 172.21.15.111:0/3349761913 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:41.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6be59c700 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b81360 0x55fa01b9a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6b7fff700 1 -- 172.21.15.111:0/3349761913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b81360 msgr2=0x55fa01b9a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b81360 0x55fa01b9a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6b7fff700 1 -- 172.21.15.111:0/3349761913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8c470 msgr2=0x55fa01b9b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:41.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8c470 0x55fa01b9b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6b7fff700 1 -- 172.21.15.111:0/3349761913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa0195e630 con 0x55fa01b8bb60 2023-10-11T16:40:41.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6be59c700 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b81360 0x55fa01b9a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:41.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.317+0000 7fc6bed9d700 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8c470 0x55fa01b9b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:40:41.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6b7fff700 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8bb60 0x55fa01b9ae70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8005f50 tx=0x7fc6a8005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32eeaafd5a7ddd52 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:41.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:41.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3349761913 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a802d5b0 con 0x55fa01b8bb60 2023-10-11T16:40:41.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3349761913 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fc6a8000e40 con 0x55fa01b8bb60 2023-10-11T16:40:41.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3349761913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6b00a1db0 con 0x55fa01b8bb60 2023-10-11T16:40:41.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3349761913 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6a8017600 con 0x55fa01b8bb60 2023-10-11T16:40:41.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6d83c9b80 10 client.16872 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:41.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3349761913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7fc6b800f480 con 0x55fa01b8bb60 2023-10-11T16:40:41.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3349761913 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6b00a1db0 con 0x55fa01b8bb60 2023-10-11T16:40:41.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3349761913 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc6a802d8d0 con 0x55fa01b8bb60 2023-10-11T16:40:41.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6d83c9b80 10 client.16872 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:41.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6d83c9b80 10 client.16872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:41.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.318+0000 7fc6d83c9b80 20 client.16872 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:41.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fa01b908b0 0x55fa01bfbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:41.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6d83c9b80 4 client.16872 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:40:41.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3349761913 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fa01aab660 con 0x55fa01b908b0 2023-10-11T16:40:41.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6be59c700 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fa01b908b0 0x55fa01bfbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:41.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6a5ffb700 10 client.16872 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:41.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6be59c700 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fa01b908b0 0x55fa01bfbe40 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52751b76c2e93094 server_cookie=1cf18c5878a43a6c in_seq=0 out_seq=0 2023-10-11T16:40:41.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6a5ffb700 1 -- 172.21.15.111:0/3349761913 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fa01aab660 con 0x55fa01b908b0 2023-10-11T16:40:41.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6a5ffb700 10 client.16872 handle_command_reply: tid=0 2023-10-11T16:40:41.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.319+0000 7fc6d83c9b80 1 client.16872 shutdown 2023-10-11T16:40:41.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3349761913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8bb60 msgr2=0x55fa01b9ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8bb60 0x55fa01b9ae70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8005f50 tx=0x7fc6a8005c80 comp rx=0 tx=0).stop 2023-10-11T16:40:41.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3349761913 shutdown_connections 2023-10-11T16:40:41.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55fa01b908b0 0x55fa01bfbe40 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa01b8c470 0x55fa01b9b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6a5ffb700 0 client.16872 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:41.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fa01b81360 0x55fa01b9a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 --2- 172.21.15.111:0/3349761913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55fa01b8bb60 0x55fa01b9ae70 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3349761913 >> 172.21.15.111:0/3349761913 conn(0x55fa01b86230 msgr2=0x55fa01b92c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3349761913 shutdown_connections 2023-10-11T16:40:41.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 1 -- 172.21.15.111:0/3349761913 wait complete. 2023-10-11T16:40:41.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6d83c9b80 20 client.16872 trim_cache size 0 max 16384 2023-10-11T16:40:41.361 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:40:41.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc6ac0a12c0 msgr2=0x7fc6ac0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc6ac0a12c0 0x7fc6ac0a3770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc6c006a100 tx=0x7fc6b8009f90 comp rx=0 tx=0).stop 2023-10-11T16:40:41.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c00746a0 msgr2=0x7fc6c0198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:41.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.320+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c00746a0 0x7fc6c0198680 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000c370 tx=0x7fc6b000c730 comp rx=0 tx=0).stop 2023-10-11T16:40:41.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.321+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1658456075 shutdown_connections 2023-10-11T16:40:41.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.321+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc6ac0a12c0 0x7fc6ac0a3770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.321+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6c010cec0 0x7fc6c019d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.321+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c00746a0 0x7fc6c0198680 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.321+0000 7fc6c49ff700 1 --2- 172.21.15.111:0/1658456075 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc6c0074b00 0x7fc6c0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:41.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.321+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1658456075 >> 172.21.15.111:0/1658456075 conn(0x7fc6c0077590 msgr2=0x7fc6c0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:41.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.321+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1658456075 shutdown_connections 2023-10-11T16:40:41.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:41.321+0000 7fc6c49ff700 1 -- 172.21.15.111:0/1658456075 wait complete. 2023-10-11T16:40:41.365 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:41 smithi111 ceph-mon[98527]: pgmap v248: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:40:41.365 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:41 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2646080206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:40:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:41 smithi130 ceph-mon[131859]: pgmap v248: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:40:41.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:41 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2646080206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:40:41.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:41 smithi150 ceph-mon[132217]: pgmap v248: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:40:41.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:41 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2646080206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:40:42.367 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:40:42.368 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:40:42.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.578+0000 7f60bafff700 1 -- 172.21.15.111:0/1521286995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4105620 msgr2=0x7f60b410d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.578+0000 7f60bafff700 1 --2- 172.21.15.111:0/1521286995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4105620 0x7f60b410d1a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00b0d0 tx=0x7f60ac00b3e0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.578+0000 7f60bafff700 1 -- 172.21.15.111:0/1521286995 shutdown_connections 2023-10-11T16:40:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.578+0000 7f60bafff700 1 --2- 172.21.15.111:0/1521286995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4105620 0x7f60b410d1a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.578+0000 7f60bafff700 1 --2- 172.21.15.111:0/1521286995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b4100cc0 0x7f60b4101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.578+0000 7f60bafff700 1 --2- 172.21.15.111:0/1521286995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b4101660 0x7f60b4104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.578+0000 7f60bafff700 1 -- 172.21.15.111:0/1521286995 >> 172.21.15.111:0/1521286995 conn(0x7f60b4077560 msgr2=0x7f60b4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.582 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.579+0000 7f60bafff700 1 -- 172.21.15.111:0/1521286995 shutdown_connections 2023-10-11T16:40:42.583 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.579+0000 7f60bafff700 1 -- 172.21.15.111:0/1521286995 wait complete. 2023-10-11T16:40:42.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.579+0000 7f60bafff700 1 Processor -- start 2023-10-11T16:40:42.585 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60bafff700 1 -- start start 2023-10-11T16:40:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60bafff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4100cc0 0x7f60b4072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60bafff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b4101660 0x7f60b4073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60bafff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b4105620 0x7f60b41a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.586 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60bafff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4073b60 con 0x7f60b4100cc0 2023-10-11T16:40:42.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60bafff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41a4d50 con 0x7f60b4105620 2023-10-11T16:40:42.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60bafff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41a4e90 con 0x7f60b4101660 2023-10-11T16:40:42.587 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60b8d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4100cc0 0x7f60b4072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60b8d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4100cc0 0x7f60b4072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37586/0 (socket says 172.21.15.111:37586) 2023-10-11T16:40:42.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60b8d9b700 1 -- 172.21.15.111:0/3118222886 learned_addr learned my addr 172.21.15.111:0/3118222886 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60b3fff700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b4101660 0x7f60b4073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.588 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.580+0000 7f60b959c700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b4105620 0x7f60b41a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b8d9b700 1 -- 172.21.15.111:0/3118222886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b4101660 msgr2=0x7f60b4073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b4101660 0x7f60b4073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.589 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b8d9b700 1 -- 172.21.15.111:0/3118222886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b4105620 msgr2=0x7f60b41a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b4105620 0x7f60b41a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b8d9b700 1 -- 172.21.15.111:0/3118222886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009d00 con 0x7f60b4100cc0 2023-10-11T16:40:42.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b959c700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b4105620 0x7f60b41a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:40:42.590 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4100cc0 0x7f60b4072df0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f60a800e9a0 tx=0x7f60a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b1ffb700 1 -- 172.21.15.111:0/3118222886 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8009790 con 0x7f60b4100cc0 2023-10-11T16:40:42.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b1ffb700 1 -- 172.21.15.111:0/3118222886 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a800cbd0 con 0x7f60b4100cc0 2023-10-11T16:40:42.591 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60bafff700 1 -- 172.21.15.111:0/3118222886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b41103e0 con 0x7f60b4100cc0 2023-10-11T16:40:42.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60b1ffb700 1 -- 172.21.15.111:0/3118222886 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8018680 con 0x7f60b4100cc0 2023-10-11T16:40:42.592 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.581+0000 7f60bafff700 1 -- 172.21.15.111:0/3118222886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b4110250 con 0x7f60b4100cc0 2023-10-11T16:40:42.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.584+0000 7f60b1ffb700 1 -- 172.21.15.111:0/3118222886 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f60a800cd40 con 0x7f60b4100cc0 2023-10-11T16:40:42.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.586+0000 7f60ce8c6b80 1 Processor -- start 2023-10-11T16:40:42.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.586+0000 7f60ce8c6b80 1 -- start start 2023-10-11T16:40:42.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 0x560162346fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x56016234b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56016234b930 0x56016234bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016224c0e0 con 0x5601623474f0 2023-10-11T16:40:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b1ffb700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60a40a12c0 0x7f60a40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016229e920 con 0x560162346be0 2023-10-11T16:40:42.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560162297db0 con 0x56016234b930 2023-10-11T16:40:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56016234b930 0x56016234bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b1ffb700 1 -- 172.21.15.111:0/3118222886 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f60a8014070 con 0x7f60b4100cc0 2023-10-11T16:40:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b8d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x56016234b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56016234b930 0x56016234bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36440/0 (socket says 172.21.15.111:36440) 2023-10-11T16:40:42.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b3fff700 1 -- 172.21.15.111:0/1648804707 learned_addr learned my addr 172.21.15.111:0/1648804707 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b959c700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60a40a12c0 0x7f60a40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b959c700 1 --2- 172.21.15.111:0/1648804707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 0x560162346fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b3fff700 1 -- 172.21.15.111:0/1648804707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 msgr2=0x560162346fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b3fff700 1 --2- 172.21.15.111:0/1648804707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 0x560162346fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b3fff700 1 -- 172.21.15.111:0/1648804707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 msgr2=0x56016234b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b3fff700 1 --2- 172.21.15.111:0/1648804707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x56016234b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.587+0000 7f60b3fff700 1 -- 172.21.15.111:0/1648804707 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601621196b0 con 0x56016234b930 2023-10-11T16:40:42.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.588+0000 7f60b3fff700 1 --2- 172.21.15.111:0/1648804707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56016234b930 0x56016234bd80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009cb0 tx=0x7f60a00058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcc4925927b9efc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.588+0000 7f60b959c700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60a40a12c0 0x7f60a40a3770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009cd0 tx=0x7f60ac006210 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.588+0000 7f609effd700 1 -- 172.21.15.111:0/1648804707 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0011ce0 con 0x56016234b930 2023-10-11T16:40:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.588+0000 7f609effd700 1 -- 172.21.15.111:0/1648804707 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a0011e40 con 0x56016234b930 2023-10-11T16:40:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.588+0000 7f609effd700 1 -- 172.21.15.111:0/1648804707 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0005280 con 0x56016234b930 2023-10-11T16:40:42.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1648804707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56016234b930 msgr2=0x56016234bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/1648804707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56016234b930 0x56016234bd80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009cb0 tx=0x7f60a00058e0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1648804707 shutdown_connections 2023-10-11T16:40:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/1648804707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56016234b930 0x56016234bd80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/1648804707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 0x560162346fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/1648804707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x56016234b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1648804707 >> 172.21.15.111:0/1648804707 conn(0x56016233c3e0 msgr2=0x560162341bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1648804707 shutdown_connections 2023-10-11T16:40:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1648804707 wait complete. 2023-10-11T16:40:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 -- start start 2023-10-11T16:40:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.589+0000 7f60ce8c6b80 1 Processor -- start 2023-10-11T16:40:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 0x5601623531b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601623474f0 0x5601623503c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016234b930 0x560162350b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560162239960 con 0x56016234b930 2023-10-11T16:40:42.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560162239600 con 0x5601623474f0 2023-10-11T16:40:42.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016211b9a0 con 0x560162346be0 2023-10-11T16:40:42.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60ce8c6b80 10 client.0 _ll_register_callbacks cb 0x560162257800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:42.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60b959c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016234b930 0x560162350b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60b959c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016234b930 0x560162350b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37612/0 (socket says 172.21.15.111:37612) 2023-10-11T16:40:42.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60b959c700 1 -- 172.21.15.111:0/721104446 learned_addr learned my addr 172.21.15.111:0/721104446 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60b3fff700 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601623474f0 0x5601623503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.590+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 0x5601623531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60b959c700 1 -- 172.21.15.111:0/721104446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 msgr2=0x5601623531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60b959c700 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 0x5601623531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60b959c700 1 -- 172.21.15.111:0/721104446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601623474f0 msgr2=0x5601623503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60b959c700 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601623474f0 0x5601623503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60b959c700 1 -- 172.21.15.111:0/721104446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a00098e0 con 0x56016234b930 2023-10-11T16:40:42.613 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60b959c700 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016234b930 0x560162350b00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f60ac015000 tx=0x7f60ac00b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15ff3509f41ee06 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f6093fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:42.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f6093fff700 1 -- 172.21.15.111:0/721104446 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac027070 con 0x56016234b930 2023-10-11T16:40:42.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f6093fff700 1 -- 172.21.15.111:0/721104446 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f60ac0077b0 con 0x56016234b930 2023-10-11T16:40:42.614 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/721104446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560162339ba0 con 0x56016234b930 2023-10-11T16:40:42.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f6093fff700 1 -- 172.21.15.111:0/721104446 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60ac00e040 con 0x56016234b930 2023-10-11T16:40:42.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60ce8c6b80 10 client.16884 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:42.615 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.591+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/721104446 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56016225e210 con 0x56016234b930 2023-10-11T16:40:42.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f6093fff700 1 -- 172.21.15.111:0/721104446 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560162339ba0 con 0x56016234b930 2023-10-11T16:40:42.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f6093fff700 1 -- 172.21.15.111:0/721104446 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f60ac020450 con 0x56016234b930 2023-10-11T16:40:42.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f60ce8c6b80 10 client.16884 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:42.616 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f60ce8c6b80 10 client.16884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:42.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f60ce8c6b80 20 client.16884 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:42.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5601623609b0 0x560162362dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.617 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f60ce8c6b80 4 client.16884 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:42.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/721104446 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56016225a120 con 0x5601623609b0 2023-10-11T16:40:42.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.592+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5601623609b0 0x560162362dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.593+0000 7f6093fff700 10 client.16884 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:42.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.593+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5601623609b0 0x560162362dc0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a20b72cdb3110b4f server_cookie=9c89751b16ab5890 in_seq=0 out_seq=0 2023-10-11T16:40:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.593+0000 7f6093fff700 1 -- 172.21.15.111:0/721104446 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56016225a120 con 0x5601623609b0 2023-10-11T16:40:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f6093fff700 10 client.16884 handle_command_reply: tid=0 2023-10-11T16:40:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 client.16884 shutdown 2023-10-11T16:40:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/721104446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016234b930 msgr2=0x560162350b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016234b930 0x560162350b00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f60ac015000 tx=0x7f60ac00b3e0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/721104446 shutdown_connections 2023-10-11T16:40:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5601623609b0 0x560162362dc0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016234b930 0x560162350b00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 0x5601623531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/721104446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601623474f0 0x5601623503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f6093fff700 0 client.16884 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/721104446 >> 172.21.15.111:0/721104446 conn(0x56016233c3e0 msgr2=0x56016234cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/721104446 shutdown_connections 2023-10-11T16:40:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/721104446 wait complete. 2023-10-11T16:40:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.594+0000 7f60ce8c6b80 20 client.16884 trim_cache size 0 max 16384 2023-10-11T16:40:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60ce8c6b80 1 Processor -- start 2023-10-11T16:40:42.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60ce8c6b80 1 -- start start 2023-10-11T16:40:42.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56016233c3e0 0x560162341d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 0x560162357a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x560162352ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560162351100 con 0x5601623474f0 2023-10-11T16:40:42.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016233c1f0 con 0x56016233c3e0 2023-10-11T16:40:42.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016235cf50 con 0x560162346be0 2023-10-11T16:40:42.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60b959c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x560162352ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60b959c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x560162352ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37628/0 (socket says 172.21.15.111:37628) 2023-10-11T16:40:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60b959c700 1 -- 172.21.15.111:0/1599639730 learned_addr learned my addr 172.21.15.111:0/1599639730 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.612+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56016233c3e0 0x560162341d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f60b3fff700 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 0x560162357a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f60b959c700 1 -- 172.21.15.111:0/1599639730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 msgr2=0x560162357a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f60b959c700 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 0x560162357a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f60b959c700 1 -- 172.21.15.111:0/1599639730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56016233c3e0 msgr2=0x560162341d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f60b959c700 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56016233c3e0 0x560162341d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f60b959c700 1 -- 172.21.15.111:0/1599639730 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56016233db80 con 0x5601623474f0 2023-10-11T16:40:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f60b959c700 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x560162352ce0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0062a0 tx=0x7f60ac006160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf600dce98d80e65 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f609f7fe700 1 -- 172.21.15.111:0/1599639730 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac027020 con 0x5601623474f0 2023-10-11T16:40:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f609f7fe700 1 -- 172.21.15.111:0/1599639730 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60ac0077b0 con 0x5601623474f0 2023-10-11T16:40:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.613+0000 7f609f7fe700 1 -- 172.21.15.111:0/1599639730 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00e030 con 0x5601623474f0 2023-10-11T16:40:42.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1599639730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 msgr2=0x560162352ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x560162352ce0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0062a0 tx=0x7f60ac006160 comp rx=0 tx=0).stop 2023-10-11T16:40:42.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1599639730 shutdown_connections 2023-10-11T16:40:42.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5601623474f0 0x560162352ce0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56016233c3e0 0x560162341d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/1599639730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560162346be0 0x560162357a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1599639730 >> 172.21.15.111:0/1599639730 conn(0x560162341210 msgr2=0x560162341600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1599639730 shutdown_connections 2023-10-11T16:40:42.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/1599639730 wait complete. 2023-10-11T16:40:42.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.614+0000 7f60ce8c6b80 1 -- start start 2023-10-11T16:40:42.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60ce8c6b80 1 Processor -- start 2023-10-11T16:40:42.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016233c3e0 0x560162356750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 0x560162356c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60ce8c6b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601623474f0 0x5601623b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560162357430 con 0x56016233c3e0 2023-10-11T16:40:42.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601623b4df0 con 0x560162346be0 2023-10-11T16:40:42.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60ce8c6b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601623b4fb0 con 0x5601623474f0 2023-10-11T16:40:42.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60ce8c6b80 10 client.0 _ll_register_callbacks cb 0x7f60ac00bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:42.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60b8d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016233c3e0 0x560162356750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60b8d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016233c3e0 0x560162356750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37634/0 (socket says 172.21.15.111:37634) 2023-10-11T16:40:42.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60b8d9b700 1 -- 172.21.15.111:0/2825680452 learned_addr learned my addr 172.21.15.111:0/2825680452 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60b959c700 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601623474f0 0x5601623b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.615+0000 7f60b3fff700 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 0x560162356c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60b8d9b700 1 -- 172.21.15.111:0/2825680452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601623474f0 msgr2=0x5601623b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601623474f0 0x5601623b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60b8d9b700 1 -- 172.21.15.111:0/2825680452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 msgr2=0x560162356c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 0x560162356c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60b8d9b700 1 -- 172.21.15.111:0/2825680452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56016225e210 con 0x56016233c3e0 2023-10-11T16:40:42.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60b8d9b700 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016233c3e0 0x560162356750 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f60a8099d40 tx=0x7f60a8096380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf039204aac3e7a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f6093fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:42.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f6093fff700 1 -- 172.21.15.111:0/2825680452 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a809e070 con 0x56016233c3e0 2023-10-11T16:40:42.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f6093fff700 1 -- 172.21.15.111:0/2825680452 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f60a80a2490 con 0x56016233c3e0 2023-10-11T16:40:42.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/2825680452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60a0011e40 con 0x56016233c3e0 2023-10-11T16:40:42.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f6093fff700 1 -- 172.21.15.111:0/2825680452 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60a80b82b0 con 0x56016233c3e0 2023-10-11T16:40:42.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60ce8c6b80 10 client.16890 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:42.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/2825680452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5601620c5e60 con 0x56016233c3e0 2023-10-11T16:40:42.644 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.616+0000 7f6093fff700 1 -- 172.21.15.111:0/2825680452 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a80ca080 con 0x56016233c3e0 2023-10-11T16:40:42.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f6093fff700 1 -- 172.21.15.111:0/2825680452 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f60a8014d30 con 0x56016233c3e0 2023-10-11T16:40:42.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f60ce8c6b80 10 client.16890 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:42.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f60ce8c6b80 10 client.16890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:42.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f60ce8c6b80 20 client.16890 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:42.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56016234b930 0x5601623b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f60ce8c6b80 4 client.16890 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:40:42.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/2825680452 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5601622666e0 con 0x56016234b930 2023-10-11T16:40:42.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f60b3fff700 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56016234b930 0x5601623b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f6093fff700 10 client.16890 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:42.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.617+0000 7f60b3fff700 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56016234b930 0x5601623b87c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49adfaca53a12e90 server_cookie=daa5d3000eeb6ec4 in_seq=0 out_seq=0 2023-10-11T16:40:42.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f6093fff700 1 -- 172.21.15.111:0/2825680452 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5601622666e0 con 0x56016234b930 2023-10-11T16:40:42.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f6093fff700 10 client.16890 handle_command_reply: tid=0 2023-10-11T16:40:42.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 client.16890 shutdown 2023-10-11T16:40:42.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/2825680452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016233c3e0 msgr2=0x560162356750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016233c3e0 0x560162356750 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f60a8099d40 tx=0x7f60a8096380 comp rx=0 tx=0).stop 2023-10-11T16:40:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/2825680452 shutdown_connections 2023-10-11T16:40:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x56016234b930 0x5601623b87c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601623474f0 0x5601623b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f6093fff700 0 client.16890 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x56016233c3e0 0x560162356750 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 --2- 172.21.15.111:0/2825680452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560162346be0 0x560162356c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.618+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/2825680452 >> 172.21.15.111:0/2825680452 conn(0x560162341210 msgr2=0x56016234a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.619+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/2825680452 shutdown_connections 2023-10-11T16:40:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.619+0000 7f60ce8c6b80 1 -- 172.21.15.111:0/2825680452 wait complete. 2023-10-11T16:40:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.619+0000 7f60ce8c6b80 20 client.16890 trim_cache size 0 max 16384 2023-10-11T16:40:42.652 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:40:42.652 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:40:42.653 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "71508ae2-294f-4d72-b0da-a098629c8467", 2023-10-11T16:40:42.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:40:42.653 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:40:42.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.619+0000 7f60bafff700 1 -- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60a40a12c0 msgr2=0x7f60a40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.619+0000 7f60bafff700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60a40a12c0 0x7f60a40a3770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009cd0 tx=0x7f60ac006210 comp rx=0 tx=0).stop 2023-10-11T16:40:42.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.619+0000 7f60bafff700 1 -- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4100cc0 msgr2=0x7f60b4072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.619+0000 7f60bafff700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4100cc0 0x7f60b4072df0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f60a800e9a0 tx=0x7f60a800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.620+0000 7f60bafff700 1 -- 172.21.15.111:0/3118222886 shutdown_connections 2023-10-11T16:40:42.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.620+0000 7f60bafff700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f60a40a12c0 0x7f60a40a3770 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.620+0000 7f60bafff700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60b4105620 0x7f60b41a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.620+0000 7f60bafff700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b4100cc0 0x7f60b4072df0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.620+0000 7f60bafff700 1 --2- 172.21.15.111:0/3118222886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60b4101660 0x7f60b4073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.620+0000 7f60bafff700 1 -- 172.21.15.111:0/3118222886 >> 172.21.15.111:0/3118222886 conn(0x7f60b4077560 msgr2=0x7f60b4103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.620+0000 7f60bafff700 1 -- 172.21.15.111:0/3118222886 shutdown_connections 2023-10-11T16:40:42.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.620+0000 7f60bafff700 1 -- 172.21.15.111:0/3118222886 wait complete. 2023-10-11T16:40:42.658 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:40:42.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.857+0000 7f7803bff700 1 -- 172.21.15.111:0/390421237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77fc100a50 msgr2=0x7f77fc100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.857+0000 7f7803bff700 1 --2- 172.21.15.111:0/390421237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77fc100a50 0x7f77fc100ec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0099c0 tx=0x7f77ec009cd0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.857+0000 7f7803bff700 1 -- 172.21.15.111:0/390421237 shutdown_connections 2023-10-11T16:40:42.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.857+0000 7f7803bff700 1 --2- 172.21.15.111:0/390421237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77fc101400 0x7f77fc10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.857+0000 7f7803bff700 1 --2- 172.21.15.111:0/390421237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77fc1063f0 0x7f77fc1067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.857+0000 7f7803bff700 1 --2- 172.21.15.111:0/390421237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77fc100a50 0x7f77fc100ec0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.857+0000 7f7803bff700 1 -- 172.21.15.111:0/390421237 >> 172.21.15.111:0/390421237 conn(0x7f77fc077590 msgr2=0x7f77fc077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.858+0000 7f7803bff700 1 -- 172.21.15.111:0/390421237 shutdown_connections 2023-10-11T16:40:42.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.858+0000 7f7803bff700 1 -- 172.21.15.111:0/390421237 wait complete. 2023-10-11T16:40:42.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f7803bff700 1 Processor -- start 2023-10-11T16:40:42.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f7803bff700 1 -- start start 2023-10-11T16:40:42.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f7803bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77fc100a50 0x7f77fc072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f7803bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77fc101400 0x7f77fc073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f7803bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77fc1063f0 0x7f77fc1a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f7803bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc073bd0 con 0x7f77fc1063f0 2023-10-11T16:40:42.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f7803bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1a4b40 con 0x7f77fc101400 2023-10-11T16:40:42.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f7803bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77fc1a4c80 con 0x7f77fc100a50 2023-10-11T16:40:42.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f780219c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77fc1063f0 0x7f77fc1a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f780219c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77fc1063f0 0x7f77fc1a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37648/0 (socket says 172.21.15.111:37648) 2023-10-11T16:40:42.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f780219c700 1 -- 172.21.15.111:0/3811183316 learned_addr learned my addr 172.21.15.111:0/3811183316 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f780119a700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77fc101400 0x7f77fc073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.859+0000 7f780199b700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77fc100a50 0x7f77fc072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f780219c700 1 -- 172.21.15.111:0/3811183316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77fc100a50 msgr2=0x7f77fc072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f780219c700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77fc100a50 0x7f77fc072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f780219c700 1 -- 172.21.15.111:0/3811183316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77fc101400 msgr2=0x7f77fc073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f780219c700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77fc101400 0x7f77fc073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f780219c700 1 -- 172.21.15.111:0/3811183316 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec0095f0 con 0x7f77fc1063f0 2023-10-11T16:40:42.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f780219c700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77fc1063f0 0x7f77fc1a4600 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f77f400ef60 tx=0x7f77f400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f77f2ffd700 1 -- 172.21.15.111:0/3811183316 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4018070 con 0x7f77fc1063f0 2023-10-11T16:40:42.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f7803bff700 1 -- 172.21.15.111:0/3811183316 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77fc110410 con 0x7f77fc1063f0 2023-10-11T16:40:42.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f77f2ffd700 1 -- 172.21.15.111:0/3811183316 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77f400f040 con 0x7f77fc1063f0 2023-10-11T16:40:42.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f77f2ffd700 1 -- 172.21.15.111:0/3811183316 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f400cc90 con 0x7f77fc1063f0 2023-10-11T16:40:42.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.860+0000 7f7803bff700 1 -- 172.21.15.111:0/3811183316 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc110280 con 0x7f77fc1063f0 2023-10-11T16:40:42.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.863+0000 7f77f2ffd700 1 -- 172.21.15.111:0/3811183316 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f77f4003a00 con 0x7f77fc1063f0 2023-10-11T16:40:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.864+0000 7f7817430b80 1 Processor -- start 2023-10-11T16:40:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.864+0000 7f77f2ffd700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77e80a12c0 0x7f77e80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f7817430b80 1 -- start start 2023-10-11T16:40:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f77f2ffd700 1 -- 172.21.15.111:0/3811183316 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f77f40ccb90 con 0x7f77fc1063f0 2023-10-11T16:40:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f780119a700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77e80a12c0 0x7f77e80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f7817430b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd00cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f7817430b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 0x558cbd011370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f7817430b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd0118b0 0x558cbd011d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f7817430b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbcf12060 con 0x558cbd00cb60 2023-10-11T16:40:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f7817430b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbcf648a0 con 0x558cbd00d470 2023-10-11T16:40:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f7817430b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbcf5dd30 con 0x558cbd0118b0 2023-10-11T16:40:42.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f780199b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd00cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f780199b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd00cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37652/0 (socket says 172.21.15.111:37652) 2023-10-11T16:40:42.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f780199b700 1 -- 172.21.15.111:0/1947792047 learned_addr learned my addr 172.21.15.111:0/1947792047 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.865+0000 7f780119a700 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 0x558cbd011370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f780219c700 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd0118b0 0x558cbd011d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f780119a700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77e80a12c0 0x7f77e80a3770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009990 tx=0x7f77ec004960 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f780199b700 1 -- 172.21.15.111:0/1947792047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd0118b0 msgr2=0x558cbd011d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f780199b700 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd0118b0 0x558cbd011d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f780199b700 1 -- 172.21.15.111:0/1947792047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 msgr2=0x558cbd011370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f780199b700 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 0x558cbd011370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f780199b700 1 -- 172.21.15.111:0/1947792047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cbcddf630 con 0x558cbd00cb60 2023-10-11T16:40:42.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f780199b700 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd00cf30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009d00 tx=0x7f77f8011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3f20a0edf39834 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f77e7fff700 1 -- 172.21.15.111:0/1947792047 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8004230 con 0x558cbd00cb60 2023-10-11T16:40:42.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f77e7fff700 1 -- 172.21.15.111:0/1947792047 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77f8004390 con 0x558cbd00cb60 2023-10-11T16:40:42.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.866+0000 7f77e7fff700 1 -- 172.21.15.111:0/1947792047 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f80197a0 con 0x558cbd00cb60 2023-10-11T16:40:42.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 -- 172.21.15.111:0/1947792047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 msgr2=0x558cbd00cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd00cf30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009d00 tx=0x7f77f8011720 comp rx=0 tx=0).stop 2023-10-11T16:40:42.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 -- 172.21.15.111:0/1947792047 shutdown_connections 2023-10-11T16:40:42.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd0118b0 0x558cbd011d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd00cf30 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 --2- 172.21.15.111:0/1947792047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 0x558cbd011370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 -- 172.21.15.111:0/1947792047 >> 172.21.15.111:0/1947792047 conn(0x558cbd002360 msgr2=0x558cbd007b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 -- 172.21.15.111:0/1947792047 shutdown_connections 2023-10-11T16:40:42.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.867+0000 7f7817430b80 1 -- 172.21.15.111:0/1947792047 wait complete. 2023-10-11T16:40:42.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 1 -- start start 2023-10-11T16:40:42.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 1 Processor -- start 2023-10-11T16:40:42.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd018f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00d470 0x558cbd0161c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd0118b0 0x558cbd016740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbceff8e0 con 0x558cbd00d470 2023-10-11T16:40:42.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbceff580 con 0x558cbd0118b0 2023-10-11T16:40:42.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbcde1920 con 0x558cbd00cb60 2023-10-11T16:40:42.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.868+0000 7f7817430b80 10 client.0 _ll_register_callbacks cb 0x558cbcf1d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:42.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780119a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00d470 0x558cbd0161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780199b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd018f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780119a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00d470 0x558cbd0161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37662/0 (socket says 172.21.15.111:37662) 2023-10-11T16:40:42.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780119a700 1 -- 172.21.15.111:0/4194295517 learned_addr learned my addr 172.21.15.111:0/4194295517 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780199b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd018f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36502/0 (socket says 172.21.15.111:36502) 2023-10-11T16:40:42.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780219c700 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd0118b0 0x558cbd016740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780199b700 1 -- 172.21.15.111:0/4194295517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd0118b0 msgr2=0x558cbd016740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780199b700 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd0118b0 0x558cbd016740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780199b700 1 -- 172.21.15.111:0/4194295517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00d470 msgr2=0x558cbd0161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780199b700 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00d470 0x558cbd0161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780199b700 1 -- 172.21.15.111:0/4194295517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f8009960 con 0x558cbd00cb60 2023-10-11T16:40:42.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f780199b700 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd018f40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f77f8003f20 tx=0x7f77f80038b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bee23794eff870d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.869+0000 7f77e57fa700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:40:42.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f77e57fa700 1 -- 172.21.15.111:0/4194295517 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8005250 con 0x558cbd00cb60 2023-10-11T16:40:42.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 1 -- 172.21.15.111:0/4194295517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558cbcfffb20 con 0x558cbd00cb60 2023-10-11T16:40:42.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f77e57fa700 1 -- 172.21.15.111:0/4194295517 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f77f80053b0 con 0x558cbd00cb60 2023-10-11T16:40:42.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f77e57fa700 1 -- 172.21.15.111:0/4194295517 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77f8005570 con 0x558cbd00cb60 2023-10-11T16:40:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 10 client.27025 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:42.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 1 -- 172.21.15.111:0/4194295517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558cbcf24190 con 0x558cbd00cb60 2023-10-11T16:40:42.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f77e57fa700 1 -- 172.21.15.111:0/4194295517 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558cbcfffb20 con 0x558cbd00cb60 2023-10-11T16:40:42.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f77e57fa700 1 -- 172.21.15.111:0/4194295517 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f77f80225f0 con 0x558cbd00cb60 2023-10-11T16:40:42.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 10 client.27025 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 10 client.27025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 20 client.27025 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558cbd026770 0x558cbd028b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 4 client.27025 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:40:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.870+0000 7f7817430b80 1 -- 172.21.15.111:0/4194295517 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558cbcf200a0 con 0x558cbd026770 2023-10-11T16:40:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.871+0000 7f780119a700 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558cbd026770 0x558cbd028b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.871+0000 7f77e57fa700 10 client.27025 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:42.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.871+0000 7f780119a700 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558cbd026770 0x558cbd028b80 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a1f6f28e2f9e25d server_cookie=be7fc091f07b30cc in_seq=0 out_seq=0 2023-10-11T16:40:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f77e57fa700 1 -- 172.21.15.111:0/4194295517 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558cbcf200a0 con 0x558cbd026770 2023-10-11T16:40:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f77e57fa700 10 client.27025 handle_command_reply: tid=0 2023-10-11T16:40:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 client.27025 shutdown 2023-10-11T16:40:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 -- 172.21.15.111:0/4194295517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 msgr2=0x558cbd018f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd018f40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f77f8003f20 tx=0x7f77f80038b0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 -- 172.21.15.111:0/4194295517 shutdown_connections 2023-10-11T16:40:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558cbd026770 0x558cbd028b80 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd0118b0 0x558cbd016740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f77e57fa700 0 client.27025 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:42.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd018f40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 --2- 172.21.15.111:0/4194295517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00d470 0x558cbd0161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 -- 172.21.15.111:0/4194295517 >> 172.21.15.111:0/4194295517 conn(0x558cbd002360 msgr2=0x558cbd006de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 -- 172.21.15.111:0/4194295517 shutdown_connections 2023-10-11T16:40:42.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 1 -- 172.21.15.111:0/4194295517 wait complete. 2023-10-11T16:40:42.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.872+0000 7f7817430b80 20 client.27025 trim_cache size 0 max 16384 2023-10-11T16:40:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f7817430b80 1 Processor -- start 2023-10-11T16:40:42.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f7817430b80 1 -- start start 2023-10-11T16:40:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f7817430b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd002360 0x558cbd00fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f7817430b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd00f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f7817430b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 0x558cbd00f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f7817430b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbd0091c0 con 0x558cbd002360 2023-10-11T16:40:42.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f7817430b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbd023e80 con 0x558cbd00d470 2023-10-11T16:40:42.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f7817430b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbd019520 con 0x558cbd00cb60 2023-10-11T16:40:42.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f780199b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd002360 0x558cbd00fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f780199b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd002360 0x558cbd00fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37678/0 (socket says 172.21.15.111:37678) 2023-10-11T16:40:42.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f780199b700 1 -- 172.21.15.111:0/1616980850 learned_addr learned my addr 172.21.15.111:0/1616980850 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.890+0000 7f780119a700 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd00f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f780219c700 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 0x558cbd00f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f780199b700 1 -- 172.21.15.111:0/1616980850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 msgr2=0x558cbd00f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f780199b700 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd00f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f780199b700 1 -- 172.21.15.111:0/1616980850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 msgr2=0x558cbd00f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f780199b700 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 0x558cbd00f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f780199b700 1 -- 172.21.15.111:0/1616980850 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cbcf23ee0 con 0x558cbd002360 2023-10-11T16:40:42.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f780199b700 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd002360 0x558cbd00fdd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f77f8005e90 tx=0x7f77f8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e378d65a971a05 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f77f0bff700 1 -- 172.21.15.111:0/1616980850 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8022c40 con 0x558cbd002360 2023-10-11T16:40:42.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f77f0bff700 1 -- 172.21.15.111:0/1616980850 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77f8022430 con 0x558cbd002360 2023-10-11T16:40:42.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.891+0000 7f77f0bff700 1 -- 172.21.15.111:0/1616980850 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f8005770 con 0x558cbd002360 2023-10-11T16:40:42.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 -- 172.21.15.111:0/1616980850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd002360 msgr2=0x558cbd00fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd002360 0x558cbd00fdd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f77f8005e90 tx=0x7f77f8009f10 comp rx=0 tx=0).stop 2023-10-11T16:40:42.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 -- 172.21.15.111:0/1616980850 shutdown_connections 2023-10-11T16:40:42.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd00d470 0x558cbd00f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd002360 0x558cbd00fdd0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 --2- 172.21.15.111:0/1616980850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00cb60 0x558cbd00f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 -- 172.21.15.111:0/1616980850 >> 172.21.15.111:0/1616980850 conn(0x558cbd013ff0 msgr2=0x558cbd022b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 -- 172.21.15.111:0/1616980850 shutdown_connections 2023-10-11T16:40:42.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 -- 172.21.15.111:0/1616980850 wait complete. 2023-10-11T16:40:42.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 -- start start 2023-10-11T16:40:42.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.892+0000 7f7817430b80 1 Processor -- start 2023-10-11T16:40:42.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f7817430b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd002360 0x558cbd01bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f7817430b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd01c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f7817430b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00d470 0x558cbd0795f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f7817430b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbd079b30 con 0x558cbd00cb60 2023-10-11T16:40:42.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f7817430b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbd079c70 con 0x558cbd002360 2023-10-11T16:40:42.920 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f7817430b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cbd079db0 con 0x558cbd00d470 2023-10-11T16:40:42.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f7817430b80 10 client.0 _ll_register_callbacks cb 0x7f77ec005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:40:42.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f780119a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd01c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f780119a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd01c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37682/0 (socket says 172.21.15.111:37682) 2023-10-11T16:40:42.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f780219c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00d470 0x558cbd0795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f780119a700 1 -- 172.21.15.111:0/3990478683 learned_addr learned my addr 172.21.15.111:0/3990478683 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:40:42.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.893+0000 7f780199b700 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd002360 0x558cbd01bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f780119a700 1 -- 172.21.15.111:0/3990478683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00d470 msgr2=0x558cbd0795f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f780119a700 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00d470 0x558cbd0795f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.925 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f780119a700 1 -- 172.21.15.111:0/3990478683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd002360 msgr2=0x558cbd01bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:40:42.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f780119a700 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd002360 0x558cbd01bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f780119a700 1 -- 172.21.15.111:0/3990478683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558cbcf24190 con 0x558cbd00cb60 2023-10-11T16:40:42.926 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f780119a700 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd01c0c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0058e0 tx=0x7f77ec021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e2b8516fdec64f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:40:42.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f77e57fa700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:40:42.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f77e57fa700 1 -- 172.21.15.111:0/3990478683 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec005630 con 0x558cbd00cb60 2023-10-11T16:40:42.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f77e57fa700 1 -- 172.21.15.111:0/3990478683 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f77ec0377f0 con 0x558cbd00cb60 2023-10-11T16:40:42.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f7817430b80 1 -- 172.21.15.111:0/3990478683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77f8004390 con 0x558cbd00cb60 2023-10-11T16:40:42.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f77e57fa700 1 -- 172.21.15.111:0/3990478683 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77ec0379d0 con 0x558cbd00cb60 2023-10-11T16:40:42.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f7817430b80 10 client.16908 fetch_fsmap learned FSMap version 33 2023-10-11T16:40:42.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.894+0000 7f7817430b80 1 -- 172.21.15.111:0/3990478683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558cbcd74c60 con 0x558cbd00cb60 2023-10-11T16:40:42.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f77e57fa700 1 -- 172.21.15.111:0/3990478683 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77f8004390 con 0x558cbd00cb60 2023-10-11T16:40:42.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f77e57fa700 1 -- 172.21.15.111:0/3990478683 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f77ec00bd50 con 0x558cbd00cb60 2023-10-11T16:40:42.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f7817430b80 10 client.16908 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:40:42.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f7817430b80 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:40:42.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f7817430b80 20 client.16908 populate_metadata read hostname 'smithi111' 2023-10-11T16:40:42.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f7817430b80 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558cbd0118b0 0x558cbd07d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:40:42.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f7817430b80 4 client.16908 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:40:42.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f7817430b80 1 -- 172.21.15.111:0/3990478683 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558cbcf2c660 con 0x558cbd0118b0 2023-10-11T16:40:42.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.895+0000 7f780199b700 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558cbd0118b0 0x558cbd07d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:40:42.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f77e57fa700 10 client.16908 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:42.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f780199b700 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558cbd0118b0 0x558cbd07d420 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eff347cbda13ed3c server_cookie=e881f524b6064aae in_seq=0 out_seq=0 2023-10-11T16:40:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f77e57fa700 1 -- 172.21.15.111:0/3990478683 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558cbcf2c660 con 0x558cbd0118b0 2023-10-11T16:40:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f77e57fa700 10 client.16908 handle_command_reply: tid=0 2023-10-11T16:40:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 client.16908 shutdown 2023-10-11T16:40:42.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 -- 172.21.15.111:0/3990478683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 msgr2=0x558cbd01c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd01c0c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0058e0 tx=0x7f77ec021000 comp rx=0 tx=0).stop 2023-10-11T16:40:42.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 -- 172.21.15.111:0/3990478683 shutdown_connections 2023-10-11T16:40:42.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x558cbd0118b0 0x558cbd07d420 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x558cbd00d470 0x558cbd0795f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f77e57fa700 0 client.16908 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:40:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558cbd002360 0x558cbd01bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 --2- 172.21.15.111:0/3990478683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x558cbd00cb60 0x558cbd01c0c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 -- 172.21.15.111:0/3990478683 >> 172.21.15.111:0/3990478683 conn(0x558cbd013ff0 msgr2=0x558cbd00fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 -- 172.21.15.111:0/3990478683 shutdown_connections 2023-10-11T16:40:42.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.896+0000 7f7817430b80 1 -- 172.21.15.111:0/3990478683 wait complete. 2023-10-11T16:40:42.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7817430b80 20 client.16908 trim_cache size 0 max 16384 2023-10-11T16:40:42.937 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:40:42.938 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:40:42.938 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:40:42.938 INFO:teuthology.orchestra.run.smithi111.stdout: "71508ae2-294f-4d72-b0da-a098629c8467": { 2023-10-11T16:40:42.939 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:40:42.939 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "71508ae2-294f-4d72-b0da-a098629c8467", 2023-10-11T16:40:42.939 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:40:42.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:40:42.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:40:42.940 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:40:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 -- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77e80a12c0 msgr2=0x7f77e80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77e80a12c0 0x7f77e80a3770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f77ec009990 tx=0x7f77ec004960 comp rx=0 tx=0).stop 2023-10-11T16:40:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 -- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77fc1063f0 msgr2=0x7f77fc1a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:40:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77fc1063f0 0x7f77fc1a4600 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f77f400ef60 tx=0x7f77f400ef90 comp rx=0 tx=0).stop 2023-10-11T16:40:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 -- 172.21.15.111:0/3811183316 shutdown_connections 2023-10-11T16:40:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f77e80a12c0 0x7f77e80a3770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77fc1063f0 0x7f77fc1a4600 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77fc100a50 0x7f77fc072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 --2- 172.21.15.111:0/3811183316 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77fc101400 0x7f77fc073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:40:42.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 -- 172.21.15.111:0/3811183316 >> 172.21.15.111:0/3811183316 conn(0x7f77fc077590 msgr2=0x7f77fc0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:40:42.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 -- 172.21.15.111:0/3811183316 shutdown_connections 2023-10-11T16:40:42.945 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:40:42.897+0000 7f7803bff700 1 -- 172.21.15.111:0/3811183316 wait complete. 2023-10-11T16:40:42.946 INFO:tasks.cephfs.filesystem:scrub status for tag:71508ae2-294f-4d72-b0da-a098629c8467 - {'path': '/', 'tag': '71508ae2-294f-4d72-b0da-a098629c8467', 'options': 'recursive,force'} 2023-10-11T16:40:43.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:43 smithi130 ceph-mon[131859]: pgmap v249: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:43.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:43 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:40:43.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:43 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:43.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:43 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:40:43.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:43 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:40:43.584 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:43 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:43.585 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:43 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:43.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:43 smithi111 ceph-mon[98527]: pgmap v249: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:43.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:43 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:40:43.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:43 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:43.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:43 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:40:43.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:43 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:40:43.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:43 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:43.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:43 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:43 smithi150 ceph-mon[132217]: pgmap v249: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:43 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:40:43.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:43 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:43.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:43 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:40:43.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:43 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:40:43.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:43 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:43.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:43 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:40:45.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:45 smithi130 ceph-mon[131859]: pgmap v250: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2023-10-11T16:40:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:45 smithi111 ceph-mon[98527]: pgmap v250: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2023-10-11T16:40:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:45 smithi150 ceph-mon[132217]: pgmap v250: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2023-10-11T16:40:47.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:47 smithi130 ceph-mon[131859]: pgmap v251: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:40:47.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:47 smithi111 ceph-mon[98527]: pgmap v251: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:40:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:47 smithi150 ceph-mon[132217]: pgmap v251: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:40:49.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:49 smithi130 ceph-mon[131859]: pgmap v252: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:40:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:49 smithi111 ceph-mon[98527]: pgmap v252: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:40:49.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:49 smithi150 ceph-mon[132217]: pgmap v252: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:40:51.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:51 smithi130 ceph-mon[131859]: pgmap v253: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:40:51.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:51 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:40:51.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:51 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:40:51.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:51 smithi111 ceph-mon[98527]: pgmap v253: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:40:51.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:51 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:40:51.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:51 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:40:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:51 smithi150 ceph-mon[132217]: pgmap v253: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:40:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:51 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:40:51.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:51 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:40:53.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:53 smithi130 ceph-mon[131859]: pgmap v254: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:40:53.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:53 smithi111 ceph-mon[98527]: pgmap v254: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:40:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:53 smithi150 ceph-mon[132217]: pgmap v254: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:40:55.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:55 smithi130 ceph-mon[131859]: pgmap v255: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-11T16:40:55.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:55 smithi111 ceph-mon[98527]: pgmap v255: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-11T16:40:55.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:55 smithi150 ceph-mon[132217]: pgmap v255: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-11T16:40:57.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:57 smithi130 ceph-mon[131859]: pgmap v256: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:40:57.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:57 smithi111 ceph-mon[98527]: pgmap v256: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:40:57.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:57 smithi150 ceph-mon[132217]: pgmap v256: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-11T16:40:59.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:40:59 smithi130 ceph-mon[131859]: pgmap v257: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:59.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:40:59 smithi111 ceph-mon[98527]: pgmap v257: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:40:59.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:40:59 smithi150 ceph-mon[132217]: pgmap v257: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:01.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:01 smithi130 ceph-mon[131859]: pgmap v258: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:01.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:01 smithi111 ceph-mon[98527]: pgmap v258: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:01.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:01 smithi150 ceph-mon[132217]: pgmap v258: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:03.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:03 smithi130 ceph-mon[131859]: pgmap v259: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:03.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:03 smithi111 ceph-mon[98527]: pgmap v259: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:03.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:03 smithi150 ceph-mon[132217]: pgmap v259: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:05.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:05 smithi130 ceph-mon[131859]: pgmap v260: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-11T16:41:05.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:05 smithi111 ceph-mon[98527]: pgmap v260: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-11T16:41:05.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:05 smithi150 ceph-mon[132217]: pgmap v260: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-11T16:41:07.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:07 smithi130 ceph-mon[131859]: pgmap v261: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-10-11T16:41:07.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:07 smithi111 ceph-mon[98527]: pgmap v261: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-10-11T16:41:07.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:07 smithi150 ceph-mon[132217]: pgmap v261: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-10-11T16:41:09.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:09 smithi130 ceph-mon[131859]: pgmap v262: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:41:09.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:09 smithi111 ceph-mon[98527]: pgmap v262: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:41:09.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:09 smithi150 ceph-mon[132217]: pgmap v262: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:41:11.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:11 smithi111 ceph-mon[98527]: pgmap v263: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:41:11.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:11 smithi150 ceph-mon[132217]: pgmap v263: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:41:11.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:11 smithi130 ceph-mon[131859]: pgmap v263: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:41:12.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:12 smithi111 ceph-mon[98527]: osdmap e97: 12 total, 12 up, 12 in 2023-10-11T16:41:12.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:12 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:41:12.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:12 smithi150 ceph-mon[132217]: osdmap e97: 12 total, 12 up, 12 in 2023-10-11T16:41:12.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:12 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:41:12.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:12 smithi130 ceph-mon[131859]: osdmap e97: 12 total, 12 up, 12 in 2023-10-11T16:41:12.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:12 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:41:12.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:41:13.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.338+0000 7f05a37ff700 1 -- 172.21.15.111:0/1785813276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 msgr2=0x7f059c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.338+0000 7f05a37ff700 1 --2- 172.21.15.111:0/1785813276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 0x7f059c104f50 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f05980099c0 tx=0x7f0598009cd0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.339+0000 7f05a37ff700 1 -- 172.21.15.111:0/1785813276 shutdown_connections 2023-10-11T16:41:13.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.339+0000 7f05a37ff700 1 --2- 172.21.15.111:0/1785813276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059c105630 0x7f059c10d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.339+0000 7f05a37ff700 1 --2- 172.21.15.111:0/1785813276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f059c100c50 0x7f059c101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.339+0000 7f05a37ff700 1 --2- 172.21.15.111:0/1785813276 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 0x7f059c104f50 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.339+0000 7f05a37ff700 1 -- 172.21.15.111:0/1785813276 >> 172.21.15.111:0/1785813276 conn(0x7f059c0775c0 msgr2=0x7f059c0779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:13.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.340+0000 7f05a37ff700 1 -- 172.21.15.111:0/1785813276 shutdown_connections 2023-10-11T16:41:13.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.340+0000 7f05a37ff700 1 -- 172.21.15.111:0/1785813276 wait complete. 2023-10-11T16:41:13.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.343+0000 7f05a37ff700 1 Processor -- start 2023-10-11T16:41:13.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.344+0000 7f05a37ff700 1 -- start start 2023-10-11T16:41:13.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.344+0000 7f05a37ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059c100c50 0x7f059c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.344+0000 7f05a37ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 0x7f059c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a0d9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 0x7f059c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059c100c50 0x7f059c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059c100c50 0x7f059c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36204/0 (socket says 172.21.15.111:36204) 2023-10-11T16:41:13.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a0d9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 0x7f059c198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60968/0 (socket says 172.21.15.111:60968) 2023-10-11T16:41:13.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a37ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f059c105630 0x7f059c19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a37ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c19d620 con 0x7f059c1015f0 2023-10-11T16:41:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a37ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c19d790 con 0x7f059c105630 2023-10-11T16:41:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a37ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c19d950 con 0x7f059c100c50 2023-10-11T16:41:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a159b700 1 -- 172.21.15.111:0/458859900 learned_addr learned my addr 172.21.15.111:0/458859900 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.345+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f059c105630 0x7f059c19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.346+0000 7f05a159b700 1 -- 172.21.15.111:0/458859900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f059c105630 msgr2=0x7f059c19cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.346+0000 7f05a159b700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f059c105630 0x7f059c19cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.346+0000 7f05a159b700 1 -- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 msgr2=0x7f059c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.346+0000 7f05a159b700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 0x7f059c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.346+0000 7f05a159b700 1 -- 172.21.15.111:0/458859900 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05980095f0 con 0x7f059c100c50 2023-10-11T16:41:13.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.346+0000 7f05a159b700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059c100c50 0x7f059c1986b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f058c00d980 tx=0x7f058c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:13.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.347+0000 7f05927fc700 1 -- 172.21.15.111:0/458859900 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c004020 con 0x7f059c100c50 2023-10-11T16:41:13.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.347+0000 7f05927fc700 1 -- 172.21.15.111:0/458859900 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c00f690 con 0x7f059c100c50 2023-10-11T16:41:13.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.347+0000 7f05927fc700 1 -- 172.21.15.111:0/458859900 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c003a60 con 0x7f059c100c50 2023-10-11T16:41:13.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.347+0000 7f05a37ff700 1 -- 172.21.15.111:0/458859900 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c1103e0 con 0x7f059c100c50 2023-10-11T16:41:13.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.348+0000 7f05a37ff700 1 -- 172.21.15.111:0/458859900 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c110250 con 0x7f059c100c50 2023-10-11T16:41:13.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.355+0000 7f05927fc700 1 -- 172.21.15.111:0/458859900 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f058c003bc0 con 0x7f059c100c50 2023-10-11T16:41:13.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.357+0000 7f05b7180b80 1 Processor -- start 2023-10-11T16:41:13.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.357+0000 7f05b7180b80 1 -- start start 2023-10-11T16:41:13.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.358+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90ef740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.358+0000 7f05a1d9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.358+0000 7f05a1d9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90ef740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36214/0 (socket says 172.21.15.111:36214) 2023-10-11T16:41:13.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.358+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f90f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.358+0000 7f05a1d9c700 1 -- 172.21.15.111:0/3335583016 learned_addr learned my addr 172.21.15.111:0/3335583016 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:13.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.358+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 0x5616f90f4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.359+0000 7f05b7180b80 1 -- 172.21.15.111:0/3335583016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f8fec060 con 0x5616f90f40c0 2023-10-11T16:41:13.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.359+0000 7f05b7180b80 1 -- 172.21.15.111:0/3335583016 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f903e8a0 con 0x5616f90efc80 2023-10-11T16:41:13.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.359+0000 7f05b7180b80 1 -- 172.21.15.111:0/3335583016 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f9037d30 con 0x5616f90ef370 2023-10-11T16:41:13.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.359+0000 7f05a159b700 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f90f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.359+0000 7f05927fc700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05880a1490 0x7f05880a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.359+0000 7f05a0d9a700 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 0x5616f90f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.360+0000 7f05a159b700 1 -- 172.21.15.111:0/3335583016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 msgr2=0x5616f90ef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:13.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.360+0000 7f05a159b700 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90ef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.360+0000 7f05a159b700 1 -- 172.21.15.111:0/3335583016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 msgr2=0x5616f90f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.360+0000 7f05a159b700 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 0x5616f90f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.360+0000 7f05a159b700 1 -- 172.21.15.111:0/3335583016 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616f8eb9630 con 0x5616f90efc80 2023-10-11T16:41:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.360+0000 7f05927fc700 1 -- 172.21.15.111:0/458859900 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f058c0ccbc0 con 0x7f059c100c50 2023-10-11T16:41:13.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.360+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05880a1490 0x7f05880a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.361+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05880a1490 0x7f05880a3940 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f059400a620 tx=0x7f0594004be0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:13.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.361+0000 7f05a159b700 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f90f3b80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f058c0c9e50 tx=0x7f058c08f160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce52b5c376eef923 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:13.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.363+0000 7f05877fe700 1 -- 172.21.15.111:0/3335583016 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c00ff00 con 0x5616f90efc80 2023-10-11T16:41:13.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.363+0000 7f05877fe700 1 -- 172.21.15.111:0/3335583016 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c004180 con 0x5616f90efc80 2023-10-11T16:41:13.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.363+0000 7f05877fe700 1 -- 172.21.15.111:0/3335583016 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c0232f0 con 0x5616f90efc80 2023-10-11T16:41:13.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.364+0000 7f05b7180b80 1 -- 172.21.15.111:0/3335583016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 msgr2=0x5616f90f3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.364+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f90f3b80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f058c0c9e50 tx=0x7f058c08f160 comp rx=0 tx=0).stop 2023-10-11T16:41:13.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.365+0000 7f05b7180b80 1 -- 172.21.15.111:0/3335583016 shutdown_connections 2023-10-11T16:41:13.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.365+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 0x5616f90f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.365+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.365+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3335583016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f90f3b80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.365+0000 7f05b7180b80 1 -- 172.21.15.111:0/3335583016 >> 172.21.15.111:0/3335583016 conn(0x5616f90dc360 msgr2=0x5616f90e1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:13.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.366+0000 7f05b7180b80 1 -- 172.21.15.111:0/3335583016 shutdown_connections 2023-10-11T16:41:13.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.366+0000 7f05b7180b80 1 -- 172.21.15.111:0/3335583016 wait complete. 2023-10-11T16:41:13.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.366+0000 7f05b7180b80 1 -- start start 2023-10-11T16:41:13.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.367+0000 7f05b7180b80 1 Processor -- start 2023-10-11T16:41:13.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.368+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90ef370 0x5616f90fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.368+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90efc80 0x5616f90f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90f9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05a159b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90ef370 0x5616f90fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05a159b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90ef370 0x5616f90fb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:58108/0 (socket says 172.21.15.111:58108) 2023-10-11T16:41:13.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05a159b700 1 -- 172.21.15.111:0/3200908967 learned_addr learned my addr 172.21.15.111:0/3200908967 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:13.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f8fd98e0 con 0x5616f90efc80 2023-10-11T16:41:13.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f8fd9580 con 0x5616f90ef370 2023-10-11T16:41:13.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f8ebb920 con 0x5616f90f40c0 2023-10-11T16:41:13.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90f9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.369+0000 7f05b7180b80 10 client.0 _ll_register_callbacks cb 0x5616f8ff7780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:13.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.370+0000 7f05a1d9c700 1 -- 172.21.15.111:0/3200908967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90ef370 msgr2=0x5616f90fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:13.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.370+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90ef370 0x5616f90fb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.370+0000 7f05a1d9c700 1 -- 172.21.15.111:0/3200908967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90efc80 msgr2=0x5616f90f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:13.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.370+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90efc80 0x5616f90f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.370+0000 7f05a1d9c700 1 -- 172.21.15.111:0/3200908967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c110250 con 0x5616f90f40c0 2023-10-11T16:41:13.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.370+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90f9090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0594009d80 tx=0x7f059400d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e6620966c33c605 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:13.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.370+0000 7f0584ff9700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:41:13.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.371+0000 7f0584ff9700 1 -- 172.21.15.111:0/3200908967 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05940215b0 con 0x5616f90f40c0 2023-10-11T16:41:13.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.372+0000 7f0584ff9700 1 -- 172.21.15.111:0/3200908967 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f0594006e40 con 0x5616f90f40c0 2023-10-11T16:41:13.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.372+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616f90d9b20 con 0x5616f90f40c0 2023-10-11T16:41:13.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.373+0000 7f0584ff9700 1 -- 172.21.15.111:0/3200908967 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0594021710 con 0x5616f90f40c0 2023-10-11T16:41:13.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.373+0000 7f05b7180b80 10 client.27040 fetch_fsmap learned FSMap version 33 2023-10-11T16:41:13.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.373+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5616f8ffe190 con 0x5616f90f40c0 2023-10-11T16:41:13.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.375+0000 7f0584ff9700 1 -- 172.21.15.111:0/3200908967 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5616f90d9b20 con 0x5616f90f40c0 2023-10-11T16:41:13.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.375+0000 7f0584ff9700 1 -- 172.21.15.111:0/3200908967 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0594022ba0 con 0x5616f90f40c0 2023-10-11T16:41:13.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.377+0000 7f05b7180b80 10 client.27040 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:41:13.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.377+0000 7f05b7180b80 10 client.27040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:13.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.377+0000 7f05b7180b80 20 client.27040 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:13.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.377+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5616f91090b0 0x5616f910b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.377+0000 7f05b7180b80 4 client.27040 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:13.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.377+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616f8ffa0a0 con 0x5616f91090b0 2023-10-11T16:41:13.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.377+0000 7f05a159b700 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5616f91090b0 0x5616f910b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.378+0000 7f05a159b700 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5616f91090b0 0x5616f910b4c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd4ae5ea0b5044a5 server_cookie=1796243f25f352b4 in_seq=0 out_seq=0 2023-10-11T16:41:13.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.378+0000 7f0584ff9700 10 client.27040 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:13.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.379+0000 7f0584ff9700 1 -- 172.21.15.111:0/3200908967 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5616f8ffa0a0 con 0x5616f91090b0 2023-10-11T16:41:13.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.379+0000 7f0584ff9700 10 client.27040 handle_command_reply: tid=0 2023-10-11T16:41:13.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.379+0000 7f05b7180b80 1 client.27040 shutdown 2023-10-11T16:41:13.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 msgr2=0x5616f90f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90f9090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0594009d80 tx=0x7f059400d040 comp rx=0 tx=0).stop 2023-10-11T16:41:13.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 shutdown_connections 2023-10-11T16:41:13.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5616f91090b0 0x5616f910b4c0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90f9090 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90ef370 0x5616f90fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f05b7180b80 1 --2- 172.21.15.111:0/3200908967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90efc80 0x5616f90f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f0584ff9700 0 client.27040 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.380+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 >> 172.21.15.111:0/3200908967 conn(0x5616f90dc360 msgr2=0x5616f90ee170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:13.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.381+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 shutdown_connections 2023-10-11T16:41:13.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.383+0000 7f05b7180b80 1 -- 172.21.15.111:0/3200908967 wait complete. 2023-10-11T16:41:13.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.385+0000 7f05b7180b80 20 client.27040 trim_cache size 0 max 16384 2023-10-11T16:41:13.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.430+0000 7f05b7180b80 1 Processor -- start 2023-10-11T16:41:13.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.431+0000 7f05b7180b80 1 -- start start 2023-10-11T16:41:13.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.431+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90f8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f91053c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 0x5616f90fb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05b7180b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f90f7020 con 0x5616f90f40c0 2023-10-11T16:41:13.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05b7180b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f90f2c30 con 0x5616f90efc80 2023-10-11T16:41:13.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05b7180b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f90f8780 con 0x5616f90ef370 2023-10-11T16:41:13.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05a159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05a159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90f8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36232/0 (socket says 172.21.15.111:36232) 2023-10-11T16:41:13.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05a159b700 1 -- 172.21.15.111:0/509642595 learned_addr learned my addr 172.21.15.111:0/509642595 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:13.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05a0d9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f91053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.432+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/509642595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 0x5616f90fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.433+0000 7f05a159b700 1 -- 172.21.15.111:0/509642595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 msgr2=0x5616f91053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:13.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.433+0000 7f05a159b700 1 --2- 172.21.15.111:0/509642595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f91053c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.433+0000 7f05a159b700 1 -- 172.21.15.111:0/509642595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 msgr2=0x5616f90fb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:13.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.433+0000 7f05a159b700 1 --2- 172.21.15.111:0/509642595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 0x5616f90fb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.433+0000 7f05a159b700 1 -- 172.21.15.111:0/509642595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616f8ffdee0 con 0x5616f90ef370 2023-10-11T16:41:13.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.433+0000 7f05a159b700 1 --2- 172.21.15.111:0/509642595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90f8240 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f058c095550 tx=0x7f058c099440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5a40a80b300bb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:13.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.433+0000 7f0587fff700 1 -- 172.21.15.111:0/509642595 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c0232f0 con 0x5616f90ef370 2023-10-11T16:41:13.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.434+0000 7f0587fff700 1 -- 172.21.15.111:0/509642595 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c0997d0 con 0x5616f90ef370 2023-10-11T16:41:13.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.434+0000 7f0587fff700 1 -- 172.21.15.111:0/509642595 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c097b70 con 0x5616f90ef370 2023-10-11T16:41:13.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.435+0000 7f05b7180b80 1 -- 172.21.15.111:0/509642595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 msgr2=0x5616f90f8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.435+0000 7f05b7180b80 1 --2- 172.21.15.111:0/509642595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90f8240 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f058c095550 tx=0x7f058c099440 comp rx=0 tx=0).stop 2023-10-11T16:41:13.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.436+0000 7f05b7180b80 1 -- 172.21.15.111:0/509642595 shutdown_connections 2023-10-11T16:41:13.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.436+0000 7f05b7180b80 1 --2- 172.21.15.111:0/509642595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90f40c0 0x5616f90fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.436+0000 7f05b7180b80 1 --2- 172.21.15.111:0/509642595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90ef370 0x5616f90f8240 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.436+0000 7f05b7180b80 1 --2- 172.21.15.111:0/509642595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f91053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.436+0000 7f05b7180b80 1 -- 172.21.15.111:0/509642595 >> 172.21.15.111:0/509642595 conn(0x5616f90dc360 msgr2=0x5616f90e0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:13.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.436+0000 7f05b7180b80 1 -- 172.21.15.111:0/509642595 shutdown_connections 2023-10-11T16:41:13.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.436+0000 7f05b7180b80 1 -- 172.21.15.111:0/509642595 wait complete. 2023-10-11T16:41:13.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.437+0000 7f05b7180b80 1 -- start start 2023-10-11T16:41:13.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.437+0000 7f05b7180b80 1 Processor -- start 2023-10-11T16:41:13.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.438+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90ef370 0x5616f90fe170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.438+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f90fe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.438+0000 7f05b7180b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.438+0000 7f05b7180b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f915bf70 con 0x5616f90ef370 2023-10-11T16:41:13.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.438+0000 7f05b7180b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f915c0b0 con 0x5616f90efc80 2023-10-11T16:41:13.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.438+0000 7f05b7180b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616f915c270 con 0x5616f90f40c0 2023-10-11T16:41:13.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.438+0000 7f05b7180b80 10 client.0 _ll_register_callbacks cb 0x7f059400add0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:13.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a1d9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a1d9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36234/0 (socket says 172.21.15.111:36234) 2023-10-11T16:41:13.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a1d9c700 1 -- 172.21.15.111:0/904651580 learned_addr learned my addr 172.21.15.111:0/904651580 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:13.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a159b700 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90ef370 0x5616f90fe170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a1d9c700 1 -- 172.21.15.111:0/904651580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 msgr2=0x5616f90fe6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:13.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f90fe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a1d9c700 1 -- 172.21.15.111:0/904651580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90ef370 msgr2=0x5616f90fe170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:13.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90ef370 0x5616f90fe170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.439+0000 7f05a1d9c700 1 -- 172.21.15.111:0/904651580 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616f8eb9630 con 0x5616f90f40c0 2023-10-11T16:41:13.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.440+0000 7f05a1d9c700 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90febf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f059400a450 tx=0x7f0594016e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9c0efe6cc8b0771 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:13.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.440+0000 7f0584ff9700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:41:13.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.441+0000 7f0584ff9700 1 -- 172.21.15.111:0/904651580 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594009390 con 0x5616f90f40c0 2023-10-11T16:41:13.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.441+0000 7f0584ff9700 1 -- 172.21.15.111:0/904651580 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f0594016550 con 0x5616f90f40c0 2023-10-11T16:41:13.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.441+0000 7f05b7180b80 1 -- 172.21.15.111:0/904651580 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f058c004180 con 0x5616f90f40c0 2023-10-11T16:41:13.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.444+0000 7f0584ff9700 1 -- 172.21.15.111:0/904651580 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0594009650 con 0x5616f90f40c0 2023-10-11T16:41:13.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.444+0000 7f05b7180b80 10 client.27046 fetch_fsmap learned FSMap version 33 2023-10-11T16:41:13.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.444+0000 7f05b7180b80 1 -- 172.21.15.111:0/904651580 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5616f8e4ec60 con 0x5616f90f40c0 2023-10-11T16:41:13.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.444+0000 7f0584ff9700 1 -- 172.21.15.111:0/904651580 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f058c004180 con 0x5616f90f40c0 2023-10-11T16:41:13.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.444+0000 7f0584ff9700 1 -- 172.21.15.111:0/904651580 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f059403d440 con 0x5616f90f40c0 2023-10-11T16:41:13.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.445+0000 7f05b7180b80 10 client.27046 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:41:13.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.445+0000 7f05b7180b80 10 client.27046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:13.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.445+0000 7f05b7180b80 20 client.27046 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:13.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.445+0000 7f05b7180b80 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5616f915d890 0x5616f915fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.445+0000 7f05b7180b80 4 client.27046 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:41:13.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.445+0000 7f05b7180b80 1 -- 172.21.15.111:0/904651580 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5616f9006660 con 0x5616f915d890 2023-10-11T16:41:13.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.445+0000 7f05a159b700 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5616f915d890 0x5616f915fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.446+0000 7f05a159b700 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5616f915d890 0x5616f915fca0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bca6b7cbd8477b4 server_cookie=1a322df0c003f730 in_seq=0 out_seq=0 2023-10-11T16:41:13.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.446+0000 7f0584ff9700 10 client.27046 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:13.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.446+0000 7f0584ff9700 1 -- 172.21.15.111:0/904651580 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5616f9006660 con 0x5616f915d890 2023-10-11T16:41:13.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.446+0000 7f0584ff9700 10 client.27046 handle_command_reply: tid=0 2023-10-11T16:41:13.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.446+0000 7f05b7180b80 1 client.27046 shutdown 2023-10-11T16:41:13.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 -- 172.21.15.111:0/904651580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 msgr2=0x5616f90febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90febf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f059400a450 tx=0x7f0594016e70 comp rx=0 tx=0).stop 2023-10-11T16:41:13.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 -- 172.21.15.111:0/904651580 shutdown_connections 2023-10-11T16:41:13.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5616f915d890 0x5616f915fca0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5616f90f40c0 0x5616f90febf0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5616f90ef370 0x5616f90fe170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 --2- 172.21.15.111:0/904651580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5616f90efc80 0x5616f90fe6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 -- 172.21.15.111:0/904651580 >> 172.21.15.111:0/904651580 conn(0x5616f90dc360 msgr2=0x5616f9104930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:13.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f0584ff9700 0 client.27046 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:13.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.447+0000 7f05b7180b80 1 -- 172.21.15.111:0/904651580 shutdown_connections 2023-10-11T16:41:13.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.448+0000 7f05b7180b80 1 -- 172.21.15.111:0/904651580 wait complete. 2023-10-11T16:41:13.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.448+0000 7f05b7180b80 20 client.27046 trim_cache size 0 max 16384 2023-10-11T16:41:13.475 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:41:13.476 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:41:13.476 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:41:13.476 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:41:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 -- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05880a1490 msgr2=0x7f05880a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05880a1490 0x7f05880a3940 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f059400a620 tx=0x7f0594004be0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 -- 172.21.15.111:0/458859900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059c100c50 msgr2=0x7f059c1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059c100c50 0x7f059c1986b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f058c00d980 tx=0x7f058c00dc90 comp rx=0 tx=0).stop 2023-10-11T16:41:13.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 -- 172.21.15.111:0/458859900 shutdown_connections 2023-10-11T16:41:13.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f05880a1490 0x7f05880a3940 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f059c105630 0x7f059c19cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059c100c50 0x7f059c1986b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 --2- 172.21.15.111:0/458859900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f059c1015f0 0x7f059c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.449+0000 7f05a37ff700 1 -- 172.21.15.111:0/458859900 >> 172.21.15.111:0/458859900 conn(0x7f059c0775c0 msgr2=0x7f059c0fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:13.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.450+0000 7f05a37ff700 1 -- 172.21.15.111:0/458859900 shutdown_connections 2023-10-11T16:41:13.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.450+0000 7f05a37ff700 1 -- 172.21.15.111:0/458859900 wait complete. 2023-10-11T16:41:13.486 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:41:13.487 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:41:13.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:13 smithi111 ceph-mon[98527]: pgmap v265: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:41:13.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:13 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:41:13.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:13 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:41:13.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:13 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:41:13.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:13 smithi150 ceph-mon[132217]: pgmap v265: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:41:13.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:13 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:41:13.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:13 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:41:13.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:13 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:41:13.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:13 smithi130 ceph-mon[131859]: pgmap v265: 129 pgs: 129 active+clean; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:41:13.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:13 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:41:13.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:13 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:41:13.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:13 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:41:13.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.848+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/2962966564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb80ff2b0 msgr2=0x7f0fb80ff680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.848+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/2962966564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb80ff2b0 0x7f0fb80ff680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009990 tx=0x7f0fb0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.848+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/2962966564 shutdown_connections 2023-10-11T16:41:13.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.848+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/2962966564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb81045d0 0x7f0fb8104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.848+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/2962966564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb80ff2b0 0x7f0fb80ff680 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.848+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/2962966564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb80ffbc0 0x7f0fb8104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.848+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/2962966564 >> 172.21.15.111:0/2962966564 conn(0x7f0fb8076310 msgr2=0x7f0fb8076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:13.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.849+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/2962966564 shutdown_connections 2023-10-11T16:41:13.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.849+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/2962966564 wait complete. 2023-10-11T16:41:13.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.851+0000 7f0fbd3ff700 1 Processor -- start 2023-10-11T16:41:13.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.852+0000 7f0fbd3ff700 1 -- start start 2023-10-11T16:41:13.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.852+0000 7f0fbd3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb80ff2b0 0x7f0fb806fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.852+0000 7f0fbd3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb80ffbc0 0x7f0fb80702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.852+0000 7f0fbd3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb81045d0 0x7f0fb806af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.852+0000 7f0fbd3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb806b5d0 con 0x7f0fb80ffbc0 2023-10-11T16:41:13.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.852+0000 7f0fbd3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb806b740 con 0x7f0fb81045d0 2023-10-11T16:41:13.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.852+0000 7f0fbd3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb806b900 con 0x7f0fb80ff2b0 2023-10-11T16:41:13.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.853+0000 7f0fb6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb80ff2b0 0x7f0fb806fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.853+0000 7f0fb6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb80ff2b0 0x7f0fb806fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:36264/0 (socket says 172.21.15.111:36264) 2023-10-11T16:41:13.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.853+0000 7f0fb6ffd700 1 -- 172.21.15.111:0/4102232169 learned_addr learned my addr 172.21.15.111:0/4102232169 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:13.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.853+0000 7f0fb67fc700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb80ffbc0 0x7f0fb80702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.853+0000 7f0fb77fe700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb81045d0 0x7f0fb806af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.854+0000 7f0fb77fe700 1 -- 172.21.15.111:0/4102232169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb80ff2b0 msgr2=0x7f0fb806fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.854+0000 7f0fb77fe700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb80ff2b0 0x7f0fb806fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.854+0000 7f0fb77fe700 1 -- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb80ffbc0 msgr2=0x7f0fb80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:13.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.854+0000 7f0fb77fe700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb80ffbc0 0x7f0fb80702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:13.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.854+0000 7f0fb77fe700 1 -- 172.21.15.111:0/4102232169 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00095f0 con 0x7f0fb81045d0 2023-10-11T16:41:13.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.854+0000 7f0fb77fe700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb81045d0 0x7f0fb806af20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000e940 tx=0x7f0fa000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:13.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.855+0000 7f0faffff700 1 -- 172.21.15.111:0/4102232169 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0004220 con 0x7f0fb81045d0 2023-10-11T16:41:13.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.855+0000 7f0faffff700 1 -- 172.21.15.111:0/4102232169 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fa0004cf0 con 0x7f0fb81045d0 2023-10-11T16:41:13.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.855+0000 7f0faffff700 1 -- 172.21.15.111:0/4102232169 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0004220 con 0x7f0fb81045d0 2023-10-11T16:41:13.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.856+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb8116960 con 0x7f0fb81045d0 2023-10-11T16:41:13.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.856+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb81167d0 con 0x7f0fb81045d0 2023-10-11T16:41:13.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.859+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb8116af0 con 0x7f0fb81045d0 2023-10-11T16:41:13.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.862+0000 7f0faffff700 1 -- 172.21.15.111:0/4102232169 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f0fa0004380 con 0x7f0fb81045d0 2023-10-11T16:41:13.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.866+0000 7f0faffff700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fa40a13d0 0x7f0fa40a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:13.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.867+0000 7f0faffff700 1 -- 172.21.15.111:0/4102232169 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f0fa0014070 con 0x7f0fb81045d0 2023-10-11T16:41:13.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.867+0000 7f0fb6ffd700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fa40a13d0 0x7f0fa40a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:13.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.868+0000 7f0fb6ffd700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fa40a13d0 0x7f0fa40a3880 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000b2e0 tx=0x7f0fb0005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:13.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:13.872+0000 7f0faffff700 1 -- 172.21.15.111:0/4102232169 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0fa0095980 con 0x7f0fb81045d0 2023-10-11T16:41:14.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.368+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fb806c520 con 0x7f0fb81045d0 2023-10-11T16:41:14.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.375+0000 7f0faffff700 1 -- 172.21.15.111:0/4102232169 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10829 (secure 0 0 0) 0x7f0fa0098fa0 con 0x7f0fb81045d0 2023-10-11T16:41:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:41:14.380 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:36:18.517722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:41:14.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.382+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fa40a13d0 msgr2=0x7f0fa40a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.382+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fa40a13d0 0x7f0fa40a3880 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000b2e0 tx=0x7f0fb0005fb0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.382+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb81045d0 msgr2=0x7f0fb806af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.382+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb81045d0 0x7f0fb806af20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000e940 tx=0x7f0fa000ec50 comp rx=0 tx=0).stop 2023-10-11T16:41:14.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.383+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 shutdown_connections 2023-10-11T16:41:14.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.383+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fa40a13d0 0x7f0fa40a3880 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.383+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fb81045d0 0x7f0fb806af20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.383+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb80ff2b0 0x7f0fb806fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.383+0000 7f0fbd3ff700 1 --2- 172.21.15.111:0/4102232169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb80ffbc0 0x7f0fb80702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.383+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 >> 172.21.15.111:0/4102232169 conn(0x7f0fb8076310 msgr2=0x7f0fb81021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:14.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.383+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 shutdown_connections 2023-10-11T16:41:14.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.384+0000 7f0fbd3ff700 1 -- 172.21.15.111:0/4102232169 wait complete. 2023-10-11T16:41:14.388 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2023-10-11T16:41:14.419 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:41:14.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:14 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:41:14.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:14 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:41:14.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:14 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:41:14.638 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:14 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:41:14.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:14 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:41:14.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:14 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:41:14.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:14 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:41:14.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:14 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:41:14.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.788+0000 7f48e37ff700 1 -- 172.21.15.111:0/1383052402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc101660 msgr2=0x7f48dc104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.788+0000 7f48e37ff700 1 --2- 172.21.15.111:0/1383052402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc101660 0x7f48dc104ef0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f48d00099c0 tx=0x7f48d0009cd0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.789+0000 7f48e37ff700 1 -- 172.21.15.111:0/1383052402 shutdown_connections 2023-10-11T16:41:14.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.789+0000 7f48e37ff700 1 --2- 172.21.15.111:0/1383052402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48dc105620 0x7f48dc10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.789+0000 7f48e37ff700 1 --2- 172.21.15.111:0/1383052402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48dc100cc0 0x7f48dc101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.789+0000 7f48e37ff700 1 --2- 172.21.15.111:0/1383052402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc101660 0x7f48dc104ef0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.789+0000 7f48e37ff700 1 -- 172.21.15.111:0/1383052402 >> 172.21.15.111:0/1383052402 conn(0x7f48dc077560 msgr2=0x7f48dc077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:14.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.789+0000 7f48e37ff700 1 -- 172.21.15.111:0/1383052402 shutdown_connections 2023-10-11T16:41:14.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.789+0000 7f48e37ff700 1 -- 172.21.15.111:0/1383052402 wait complete. 2023-10-11T16:41:14.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.792+0000 7f48e37ff700 1 Processor -- start 2023-10-11T16:41:14.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.793+0000 7f48e37ff700 1 -- start start 2023-10-11T16:41:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e37ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc100cc0 0x7f48dc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e37ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48dc101660 0x7f48dc198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e37ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48dc105620 0x7f48dc19bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e37ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc19c5b0 con 0x7f48dc105620 2023-10-11T16:41:14.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e37ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc19c720 con 0x7f48dc100cc0 2023-10-11T16:41:14.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e37ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc19c8e0 con 0x7f48dc101660 2023-10-11T16:41:14.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e159b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc100cc0 0x7f48dc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e0d9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48dc101660 0x7f48dc198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e0d9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48dc101660 0x7f48dc198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48646/0 (socket says 172.21.15.111:48646) 2023-10-11T16:41:14.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.794+0000 7f48e0d9a700 1 -- 172.21.15.111:0/579329662 learned_addr learned my addr 172.21.15.111:0/579329662 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:14.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48e159b700 1 -- 172.21.15.111:0/579329662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48dc101660 msgr2=0x7f48dc198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48e159b700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48dc101660 0x7f48dc198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48e159b700 1 -- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48dc105620 msgr2=0x7f48dc19bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48e159b700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48dc105620 0x7f48dc19bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48e159b700 1 -- 172.21.15.111:0/579329662 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d4009520 con 0x7f48dc100cc0 2023-10-11T16:41:14.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48e159b700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc100cc0 0x7f48dc1986c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f48d400e9f0 tx=0x7f48d400edb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:14.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48da7fc700 1 -- 172.21.15.111:0/579329662 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d400cb70 con 0x7f48dc100cc0 2023-10-11T16:41:14.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48da7fc700 1 -- 172.21.15.111:0/579329662 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d40044e0 con 0x7f48dc100cc0 2023-10-11T16:41:14.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.795+0000 7f48e37ff700 1 -- 172.21.15.111:0/579329662 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d00095f0 con 0x7f48dc100cc0 2023-10-11T16:41:14.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.796+0000 7f48e37ff700 1 -- 172.21.15.111:0/579329662 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc110370 con 0x7f48dc100cc0 2023-10-11T16:41:14.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.796+0000 7f48da7fc700 1 -- 172.21.15.111:0/579329662 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4003e70 con 0x7f48dc100cc0 2023-10-11T16:41:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.809+0000 7f48da7fc700 1 -- 172.21.15.111:0/579329662 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f48d400ccd0 con 0x7f48dc100cc0 2023-10-11T16:41:14.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.809+0000 7f48f719ab80 1 Processor -- start 2023-10-11T16:41:14.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.810+0000 7f48f719ab80 1 -- start start 2023-10-11T16:41:14.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54577b60 0x55cc54577f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54578470 0x55cc5457c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc5457cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48f719ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5447d060 con 0x55cc5457c8b0 2023-10-11T16:41:14.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48f719ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc544cf8a0 con 0x55cc54577b60 2023-10-11T16:41:14.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48f719ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc544c8d30 con 0x55cc54578470 2023-10-11T16:41:14.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48e159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54578470 0x55cc5457c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48e159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54578470 0x55cc5457c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48660/0 (socket says 172.21.15.111:48660) 2023-10-11T16:41:14.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48e159b700 1 -- 172.21.15.111:0/2436517242 learned_addr learned my addr 172.21.15.111:0/2436517242 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:14.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.811+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/2436517242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc5457cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.813+0000 7f48e159b700 1 -- 172.21.15.111:0/2436517242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54577b60 msgr2=0x55cc54577f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:41:14.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.813+0000 7f48e159b700 1 --2- 172.21.15.111:0/2436517242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54577b60 0x55cc54577f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.813+0000 7f48e159b700 1 -- 172.21.15.111:0/2436517242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 msgr2=0x55cc5457cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.813+0000 7f48e159b700 1 --2- 172.21.15.111:0/2436517242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc5457cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.813+0000 7f48e159b700 1 -- 172.21.15.111:0/2436517242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc5434a630 con 0x55cc54578470 2023-10-11T16:41:14.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.813+0000 7f48da7fc700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f48c40a1340 0x7f48c40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.813+0000 7f48e159b700 1 --2- 172.21.15.111:0/2436517242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54578470 0x55cc5457c370 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f48d40ca7d0 tx=0x7f48d40d53e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6101ef160814e40 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:14.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.815+0000 7f48da7fc700 1 -- 172.21.15.111:0/579329662 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f48d4014070 con 0x7f48dc100cc0 2023-10-11T16:41:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.815+0000 7f48c37fe700 1 -- 172.21.15.111:0/2436517242 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4096520 con 0x55cc54578470 2023-10-11T16:41:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.815+0000 7f48c37fe700 1 -- 172.21.15.111:0/2436517242 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d4099fb0 con 0x55cc54578470 2023-10-11T16:41:14.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.815+0000 7f48c37fe700 1 -- 172.21.15.111:0/2436517242 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d409e280 con 0x55cc54578470 2023-10-11T16:41:14.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.815+0000 7f48e1d9c700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f48c40a1340 0x7f48c40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48e1d9c700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f48c40a1340 0x7f48c40a37f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00e2d0 tx=0x7f48cc00d040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:14.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48f719ab80 1 -- 172.21.15.111:0/2436517242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54578470 msgr2=0x55cc5457c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48f719ab80 1 --2- 172.21.15.111:0/2436517242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54578470 0x55cc5457c370 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f48d40ca7d0 tx=0x7f48d40d53e0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48f719ab80 1 -- 172.21.15.111:0/2436517242 shutdown_connections 2023-10-11T16:41:14.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48f719ab80 1 --2- 172.21.15.111:0/2436517242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc5457cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48f719ab80 1 --2- 172.21.15.111:0/2436517242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54577b60 0x55cc54577f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48f719ab80 1 --2- 172.21.15.111:0/2436517242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54578470 0x55cc5457c370 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48f719ab80 1 -- 172.21.15.111:0/2436517242 >> 172.21.15.111:0/2436517242 conn(0x55cc5456d360 msgr2=0x55cc54572b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:14.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.818+0000 7f48f719ab80 1 -- 172.21.15.111:0/2436517242 shutdown_connections 2023-10-11T16:41:14.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.819+0000 7f48f719ab80 1 -- 172.21.15.111:0/2436517242 wait complete. 2023-10-11T16:41:14.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.819+0000 7f48f719ab80 1 -- start start 2023-10-11T16:41:14.824 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.820+0000 7f48f719ab80 1 Processor -- start 2023-10-11T16:41:14.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54583fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc54578470 0x55cc54581220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc5457c8b0 0x55cc54581800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48f719ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5446a8e0 con 0x55cc54578470 2023-10-11T16:41:14.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48f719ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5446a580 con 0x55cc5457c8b0 2023-10-11T16:41:14.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48f719ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5434c920 con 0x55cc54577b60 2023-10-11T16:41:14.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48e159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54583fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48f719ab80 10 client.0 _ll_register_callbacks cb 0x55cc54488780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:14.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48e159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54583fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48670/0 (socket says 172.21.15.111:48670) 2023-10-11T16:41:14.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48e159b700 1 -- 172.21.15.111:0/876920186 learned_addr learned my addr 172.21.15.111:0/876920186 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:14.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc54578470 0x55cc54581220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.822+0000 7f48e1d9c700 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc5457c8b0 0x55cc54581800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.823+0000 7f48e0d9a700 1 -- 172.21.15.111:0/876920186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 msgr2=0x55cc54583fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:14.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.823+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54583fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.823+0000 7f48e0d9a700 1 -- 172.21.15.111:0/876920186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc5457c8b0 msgr2=0x55cc54581800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:14.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.823+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc5457c8b0 0x55cc54581800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.823+0000 7f48e0d9a700 1 -- 172.21.15.111:0/876920186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc110370 con 0x55cc54578470 2023-10-11T16:41:14.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.823+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc54578470 0x55cc54581220 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f48d00098e0 tx=0x7f48d00056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c553f3167fac22e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:14.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.823+0000 7f48c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:14.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.824+0000 7f48c0ff9700 1 -- 172.21.15.111:0/876920186 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d001d050 con 0x55cc54578470 2023-10-11T16:41:14.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.824+0000 7f48c0ff9700 1 -- 172.21.15.111:0/876920186 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f48d0003be0 con 0x55cc54578470 2023-10-11T16:41:14.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.824+0000 7f48f719ab80 1 -- 172.21.15.111:0/876920186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc5456ab20 con 0x55cc54578470 2023-10-11T16:41:14.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.825+0000 7f48c0ff9700 1 -- 172.21.15.111:0/876920186 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48d0003da0 con 0x55cc54578470 2023-10-11T16:41:14.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.825+0000 7f48f719ab80 10 client.16935 fetch_fsmap learned FSMap version 33 2023-10-11T16:41:14.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.825+0000 7f48f719ab80 1 -- 172.21.15.111:0/876920186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cc5448f190 con 0x55cc54578470 2023-10-11T16:41:14.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.825+0000 7f48c0ff9700 1 -- 172.21.15.111:0/876920186 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cc5456ab20 con 0x55cc54578470 2023-10-11T16:41:14.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.826+0000 7f48c0ff9700 1 -- 172.21.15.111:0/876920186 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f48d0023880 con 0x55cc54578470 2023-10-11T16:41:14.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.826+0000 7f48f719ab80 10 client.16935 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:41:14.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.826+0000 7f48f719ab80 10 client.16935 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:41:14.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.826+0000 7f48f719ab80 20 client.16935 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:14.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.826+0000 7f48f719ab80 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55cc5459a1b0 0x55cc5459c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.826+0000 7f48f719ab80 4 client.16935 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:14.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.826+0000 7f48f719ab80 1 -- 172.21.15.111:0/876920186 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc5448b0a0 con 0x55cc5459a1b0 2023-10-11T16:41:14.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.826+0000 7f48e159b700 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55cc5459a1b0 0x55cc5459c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.827+0000 7f48e159b700 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55cc5459a1b0 0x55cc5459c5c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=db4974e9fa12fb10 server_cookie=d65c80014c41314e in_seq=0 out_seq=0 2023-10-11T16:41:14.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.827+0000 7f48c0ff9700 10 client.16935 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:41:14.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.828+0000 7f48c0ff9700 1 -- 172.21.15.111:0/876920186 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cc5448b0a0 con 0x55cc5459a1b0 2023-10-11T16:41:14.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.828+0000 7f48c0ff9700 10 client.16935 handle_command_reply: tid=0 2023-10-11T16:41:14.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.828+0000 7f48f719ab80 1 client.16935 shutdown 2023-10-11T16:41:14.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.828+0000 7f48f719ab80 1 -- 172.21.15.111:0/876920186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc54578470 msgr2=0x55cc54581220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.828+0000 7f48f719ab80 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc54578470 0x55cc54581220 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f48d00098e0 tx=0x7f48d00056f0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.829+0000 7f48f719ab80 1 -- 172.21.15.111:0/876920186 shutdown_connections 2023-10-11T16:41:14.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.829+0000 7f48f719ab80 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55cc5459a1b0 0x55cc5459c5c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.829+0000 7f48f719ab80 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc5457c8b0 0x55cc54581800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.829+0000 7f48f719ab80 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54583fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.829+0000 7f48f719ab80 1 --2- 172.21.15.111:0/876920186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc54578470 0x55cc54581220 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.829+0000 7f48c0ff9700 0 client.16935 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:41:14.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.829+0000 7f48f719ab80 1 -- 172.21.15.111:0/876920186 >> 172.21.15.111:0/876920186 conn(0x55cc5456d360 msgr2=0x55cc5457dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:14.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.829+0000 7f48f719ab80 1 -- 172.21.15.111:0/876920186 shutdown_connections 2023-10-11T16:41:14.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.830+0000 7f48f719ab80 1 -- 172.21.15.111:0/876920186 wait complete. 2023-10-11T16:41:14.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:14 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:41:14.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:14 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:41:14.841 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:14 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:41:14.842 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:14 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:41:14.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.830+0000 7f48f719ab80 20 client.16935 trim_cache size 0 max 16384 2023-10-11T16:41:14.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.858+0000 7f48f719ab80 1 Processor -- start 2023-10-11T16:41:14.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.859+0000 7f48f719ab80 1 -- start start 2023-10-11T16:41:14.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.859+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc545912c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.859+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54580a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.859+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc54571c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.859+0000 7f48f719ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5456cee0 con 0x55cc5457c8b0 2023-10-11T16:41:14.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.859+0000 7f48f719ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5456d170 con 0x55cc54578470 2023-10-11T16:41:14.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.859+0000 7f48f719ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5457f7a0 con 0x55cc54577b60 2023-10-11T16:41:14.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.859+0000 7f48e159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc545912c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e0d9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54580a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e0d9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54580a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45356/0 (socket says 172.21.15.111:45356) 2023-10-11T16:41:14.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e0d9a700 1 -- 172.21.15.111:0/270164706 learned_addr learned my addr 172.21.15.111:0/270164706 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:14.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc545912c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48680/0 (socket says 172.21.15.111:48680) 2023-10-11T16:41:14.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e1d9c700 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc54571c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e0d9a700 1 -- 172.21.15.111:0/270164706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 msgr2=0x55cc545912c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:14.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc545912c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e0d9a700 1 -- 172.21.15.111:0/270164706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 msgr2=0x55cc54571c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:14.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc54571c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e0d9a700 1 -- 172.21.15.111:0/270164706 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc5448eee0 con 0x55cc54578470 2023-10-11T16:41:14.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.860+0000 7f48e159b700 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc545912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:14.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.861+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54580a00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f48d0023c10 tx=0x7f48d000f4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=689b737849077c3b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:14.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.861+0000 7f48c3fff700 1 -- 172.21.15.111:0/270164706 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d001d040 con 0x55cc54578470 2023-10-11T16:41:14.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.861+0000 7f48c3fff700 1 -- 172.21.15.111:0/270164706 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d001e040 con 0x55cc54578470 2023-10-11T16:41:14.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.861+0000 7f48c3fff700 1 -- 172.21.15.111:0/270164706 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d001d040 con 0x55cc54578470 2023-10-11T16:41:14.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.862+0000 7f48f719ab80 1 -- 172.21.15.111:0/270164706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 msgr2=0x55cc54580a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.862+0000 7f48f719ab80 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54580a00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f48d0023c10 tx=0x7f48d000f4e0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.863+0000 7f48f719ab80 1 -- 172.21.15.111:0/270164706 shutdown_connections 2023-10-11T16:41:14.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.863+0000 7f48f719ab80 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc54571c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.863+0000 7f48f719ab80 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc545912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.863+0000 7f48f719ab80 1 --2- 172.21.15.111:0/270164706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54580a00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.863+0000 7f48f719ab80 1 -- 172.21.15.111:0/270164706 >> 172.21.15.111:0/270164706 conn(0x55cc5456d360 msgr2=0x55cc54596480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:14.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.863+0000 7f48f719ab80 1 -- 172.21.15.111:0/270164706 shutdown_connections 2023-10-11T16:41:14.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.863+0000 7f48f719ab80 1 -- 172.21.15.111:0/270164706 wait complete. 2023-10-11T16:41:14.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.864+0000 7f48f719ab80 1 -- start start 2023-10-11T16:41:14.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.864+0000 7f48f719ab80 1 Processor -- start 2023-10-11T16:41:14.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54572720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54572c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48f719ab80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc5457f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48e159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54572720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48e159b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54572720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48686/0 (socket says 172.21.15.111:48686) 2023-10-11T16:41:14.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48f719ab80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5457ff70 con 0x55cc5457c8b0 2023-10-11T16:41:14.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48f719ab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc545800e0 con 0x55cc54578470 2023-10-11T16:41:14.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48f719ab80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc545802a0 con 0x55cc54577b60 2023-10-11T16:41:14.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.865+0000 7f48e0d9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54572c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48f719ab80 10 client.0 _ll_register_callbacks cb 0x7f48d4005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:14.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48e159b700 1 -- 172.21.15.111:0/454505866 learned_addr learned my addr 172.21.15.111:0/454505866 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:14.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48e159b700 1 -- 172.21.15.111:0/454505866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 msgr2=0x55cc54572c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:14.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48e159b700 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54572c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48e159b700 1 -- 172.21.15.111:0/454505866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 msgr2=0x55cc5457f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:14.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48e159b700 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc5457f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48e159b700 1 -- 172.21.15.111:0/454505866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc5448f190 con 0x55cc54577b60 2023-10-11T16:41:14.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54572c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:14.878 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48e159b700 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54572720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f48d4099ee0 tx=0x7f48d40d58e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f094b086a153a34a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:14.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.866+0000 7f48c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:41:14.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.867+0000 7f48c0ff9700 1 -- 172.21.15.111:0/454505866 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4099d20 con 0x55cc54577b60 2023-10-11T16:41:14.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.867+0000 7f48c0ff9700 1 -- 172.21.15.111:0/454505866 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f48d40a2630 con 0x55cc54577b60 2023-10-11T16:41:14.879 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.867+0000 7f48f719ab80 1 -- 172.21.15.111:0/454505866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48d4099fb0 con 0x55cc54577b60 2023-10-11T16:41:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.867+0000 7f48c0ff9700 1 -- 172.21.15.111:0/454505866 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48d4099b40 con 0x55cc54577b60 2023-10-11T16:41:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.867+0000 7f48f719ab80 10 client.27070 fetch_fsmap learned FSMap version 33 2023-10-11T16:41:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.867+0000 7f48f719ab80 1 -- 172.21.15.111:0/454505866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55cc542dfc60 con 0x55cc54577b60 2023-10-11T16:41:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48c0ff9700 1 -- 172.21.15.111:0/454505866 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d4099fb0 con 0x55cc54577b60 2023-10-11T16:41:14.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48c0ff9700 1 -- 172.21.15.111:0/454505866 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f48d40968f0 con 0x55cc54577b60 2023-10-11T16:41:14.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48f719ab80 10 client.27070 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:41:14.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48f719ab80 10 client.27070 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:41:14.881 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48f719ab80 20 client.27070 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:14.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48f719ab80 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55cc545eecf0 0x55cc545f1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:14.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48f719ab80 4 client.27070 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:41:14.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48f719ab80 1 -- 172.21.15.111:0/454505866 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cc54497660 con 0x55cc545eecf0 2023-10-11T16:41:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.868+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55cc545eecf0 0x55cc545f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.870+0000 7f48c0ff9700 10 client.27070 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:41:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.870+0000 7f48e0d9a700 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55cc545eecf0 0x55cc545f1100 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b8f240816d2dd02 server_cookie=310b5d3dfe006387 in_seq=0 out_seq=0 2023-10-11T16:41:14.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.870+0000 7f48c0ff9700 1 -- 172.21.15.111:0/454505866 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cc54497660 con 0x55cc545eecf0 2023-10-11T16:41:14.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.870+0000 7f48c0ff9700 10 client.27070 handle_command_reply: tid=0 2023-10-11T16:41:14.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.870+0000 7f48f719ab80 1 client.27070 shutdown 2023-10-11T16:41:14.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.870+0000 7f48f719ab80 1 -- 172.21.15.111:0/454505866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 msgr2=0x55cc54572720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.870+0000 7f48f719ab80 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54572720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f48d4099ee0 tx=0x7f48d40d58e0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.871+0000 7f48f719ab80 1 -- 172.21.15.111:0/454505866 shutdown_connections 2023-10-11T16:41:14.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.871+0000 7f48f719ab80 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55cc545eecf0 0x55cc545f1100 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.871+0000 7f48f719ab80 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55cc5457c8b0 0x55cc5457f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.871+0000 7f48f719ab80 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cc54577b60 0x55cc54572720 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.871+0000 7f48f719ab80 1 --2- 172.21.15.111:0/454505866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cc54578470 0x55cc54572c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.871+0000 7f48f719ab80 1 -- 172.21.15.111:0/454505866 >> 172.21.15.111:0/454505866 conn(0x55cc5456d360 msgr2=0x55cc5457a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:14.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.871+0000 7f48c0ff9700 0 client.27070 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:41:14.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.872+0000 7f48f719ab80 1 -- 172.21.15.111:0/454505866 shutdown_connections 2023-10-11T16:41:14.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.872+0000 7f48f719ab80 1 -- 172.21.15.111:0/454505866 wait complete. 2023-10-11T16:41:14.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.872+0000 7f48f719ab80 20 client.27070 trim_cache size 0 max 16384 2023-10-11T16:41:14.888 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:41:14.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.873+0000 7f48e37ff700 1 -- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f48c40a1340 msgr2=0x7f48c40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.873+0000 7f48e37ff700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f48c40a1340 0x7f48c40a37f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00e2d0 tx=0x7f48cc00d040 comp rx=0 tx=0).stop 2023-10-11T16:41:14.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.874+0000 7f48e37ff700 1 -- 172.21.15.111:0/579329662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc100cc0 msgr2=0x7f48dc1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:14.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.874+0000 7f48e37ff700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc100cc0 0x7f48dc1986c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f48d400e9f0 tx=0x7f48d400edb0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.874+0000 7f48e37ff700 1 -- 172.21.15.111:0/579329662 shutdown_connections 2023-10-11T16:41:14.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.874+0000 7f48e37ff700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f48c40a1340 0x7f48c40a37f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.874+0000 7f48e37ff700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48dc105620 0x7f48dc19bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.874+0000 7f48e37ff700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48dc100cc0 0x7f48dc1986c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.874+0000 7f48e37ff700 1 --2- 172.21.15.111:0/579329662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48dc101660 0x7f48dc198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:14.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.874+0000 7f48e37ff700 1 -- 172.21.15.111:0/579329662 >> 172.21.15.111:0/579329662 conn(0x7f48dc077560 msgr2=0x7f48dc0feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:14.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.876+0000 7f48e37ff700 1 -- 172.21.15.111:0/579329662 shutdown_connections 2023-10-11T16:41:14.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:14.876+0000 7f48e37ff700 1 -- 172.21.15.111:0/579329662 wait complete. 2023-10-11T16:41:14.912 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:41:15.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.278+0000 7f0d189ff700 1 -- 172.21.15.111:0/1810681801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101390 msgr2=0x7f0d14101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.278+0000 7f0d189ff700 1 --2- 172.21.15.111:0/1810681801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101390 0x7f0d14101760 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009990 tx=0x7f0d04009ca0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.280 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.278+0000 7f0d189ff700 1 -- 172.21.15.111:0/1810681801 shutdown_connections 2023-10-11T16:41:15.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.278+0000 7f0d189ff700 1 --2- 172.21.15.111:0/1810681801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d1410a8b0 0x7f0d1410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.278+0000 7f0d189ff700 1 --2- 172.21.15.111:0/1810681801 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101390 0x7f0d14101760 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.278+0000 7f0d189ff700 1 --2- 172.21.15.111:0/1810681801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14101ca0 0x7f0d1410a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.281 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.278+0000 7f0d189ff700 1 -- 172.21.15.111:0/1810681801 >> 172.21.15.111:0/1810681801 conn(0x7f0d140770b0 msgr2=0x7f0d140774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.279+0000 7f0d189ff700 1 -- 172.21.15.111:0/1810681801 shutdown_connections 2023-10-11T16:41:15.282 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.279+0000 7f0d189ff700 1 -- 172.21.15.111:0/1810681801 wait complete. 2023-10-11T16:41:15.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.280+0000 7f0d189ff700 1 Processor -- start 2023-10-11T16:41:15.283 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.282+0000 7f0d189ff700 1 -- start start 2023-10-11T16:41:15.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.283+0000 7f0d189ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d14101390 0x7f0d14109e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.284 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.283+0000 7f0d189ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101ca0 0x7f0d14104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.283+0000 7f0d1259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d14101390 0x7f0d14109e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.285 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.283+0000 7f0d189ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d1410a8b0 0x7f0d141053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.283+0000 7f0d189ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14105900 con 0x7f0d1410a8b0 2023-10-11T16:41:15.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.283+0000 7f0d1259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d14101390 0x7f0d14109e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48710/0 (socket says 172.21.15.111:48710) 2023-10-11T16:41:15.286 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.283+0000 7f0d1259c700 1 -- 172.21.15.111:0/4049567196 learned_addr learned my addr 172.21.15.111:0/4049567196 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d1410a8b0 0x7f0d141053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14105a70 con 0x7f0d14101ca0 2023-10-11T16:41:15.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14105c30 con 0x7f0d14101390 2023-10-11T16:41:15.287 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101ca0 0x7f0d14104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.288 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d11d9b700 1 -- 172.21.15.111:0/4049567196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d14101390 msgr2=0x7f0d14109e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d14101390 0x7f0d14109e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d11d9b700 1 -- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d1410a8b0 msgr2=0x7f0d141053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.289 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d1410a8b0 0x7f0d141053c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.284+0000 7f0d11d9b700 1 -- 172.21.15.111:0/4049567196 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d040095f0 con 0x7f0d14101ca0 2023-10-11T16:41:15.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.285+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101ca0 0x7f0d14104e80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800b450 tx=0x7f0d0800b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.285+0000 7f0d037fe700 1 -- 172.21.15.111:0/4049567196 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08011830 con 0x7f0d14101ca0 2023-10-11T16:41:15.290 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.285+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d14110400 con 0x7f0d14101ca0 2023-10-11T16:41:15.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.285+0000 7f0d037fe700 1 -- 172.21.15.111:0/4049567196 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d0800be60 con 0x7f0d14101ca0 2023-10-11T16:41:15.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.285+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d14110270 con 0x7f0d14101ca0 2023-10-11T16:41:15.291 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.285+0000 7f0d037fe700 1 -- 172.21.15.111:0/4049567196 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08010d20 con 0x7f0d14101ca0 2023-10-11T16:41:15.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.293+0000 7f0d037fe700 1 -- 172.21.15.111:0/4049567196 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f0d08010380 con 0x7f0d14101ca0 2023-10-11T16:41:15.297 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.295+0000 7f0d2c365b80 1 Processor -- start 2023-10-11T16:41:15.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.296+0000 7f0d2c365b80 1 -- start start 2023-10-11T16:41:15.299 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.296+0000 7f0d037fe700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0cfc0a1340 0x7f0cfc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.297+0000 7f0d037fe700 1 -- 172.21.15.111:0/4049567196 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f0d080cca10 con 0x7f0d14101ca0 2023-10-11T16:41:15.300 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.298+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0cfc0a1340 0x7f0cfc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9961f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e9966370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.302 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 0x55d8e9966d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e9867060 con 0x55d8e9961b60 2023-10-11T16:41:15.303 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e98b98a0 con 0x55d8e9962470 2023-10-11T16:41:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e98b2d30 con 0x55d8e99668b0 2023-10-11T16:41:15.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e9966370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e9966370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45392/0 (socket says 172.21.15.111:45392) 2023-10-11T16:41:15.306 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.299+0000 7f0d11d9b700 1 -- 172.21.15.111:0/2186531304 learned_addr learned my addr 172.21.15.111:0/2186531304 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.300+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0cfc0a1340 0x7f0cfc0a37f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0d1406a4a0 tx=0x7f0d0c00b040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.300+0000 7f0d1259c700 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9961f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.307 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.300+0000 7f0d11d9b700 1 -- 172.21.15.111:0/2186531304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 msgr2=0x55d8e9966d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.308 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.301+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 0x55d8e9966d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.301+0000 7f0d11d9b700 1 -- 172.21.15.111:0/2186531304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 msgr2=0x55d8e9961f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.301+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9961f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.301+0000 7f0d11d9b700 1 -- 172.21.15.111:0/2186531304 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8e9734630 con 0x55d8e9962470 2023-10-11T16:41:15.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.301+0000 7f0d1259c700 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9961f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:41:15.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.301+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e9966370 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0d080961a0 tx=0x7f0d08099090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39c71e2117fa571a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.302+0000 7f0d00ff9700 1 -- 172.21.15.111:0/2186531304 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d080a1b80 con 0x55d8e9962470 2023-10-11T16:41:15.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.302+0000 7f0d00ff9700 1 -- 172.21.15.111:0/2186531304 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d08010e80 con 0x55d8e9962470 2023-10-11T16:41:15.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.302+0000 7f0d00ff9700 1 -- 172.21.15.111:0/2186531304 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08099bf0 con 0x55d8e9962470 2023-10-11T16:41:15.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.303+0000 7f0d2c365b80 1 -- 172.21.15.111:0/2186531304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 msgr2=0x55d8e9966370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.303+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e9966370 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0d080961a0 tx=0x7f0d08099090 comp rx=0 tx=0).stop 2023-10-11T16:41:15.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.304+0000 7f0d2c365b80 1 -- 172.21.15.111:0/2186531304 shutdown_connections 2023-10-11T16:41:15.313 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.304+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 0x55d8e9966d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.304+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9961f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.304+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/2186531304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e9966370 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.304+0000 7f0d2c365b80 1 -- 172.21.15.111:0/2186531304 >> 172.21.15.111:0/2186531304 conn(0x55d8e9957360 msgr2=0x55d8e995cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.305+0000 7f0d2c365b80 1 -- 172.21.15.111:0/2186531304 shutdown_connections 2023-10-11T16:41:15.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.305+0000 7f0d2c365b80 1 -- 172.21.15.111:0/2186531304 wait complete. 2023-10-11T16:41:15.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.306+0000 7f0d2c365b80 1 -- start start 2023-10-11T16:41:15.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.307+0000 7f0d2c365b80 1 Processor -- start 2023-10-11T16:41:15.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.310+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e996b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.310+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e996b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 0x55d8e996be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e996b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e996b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45404/0 (socket says 172.21.15.111:45404) 2023-10-11T16:41:15.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d12d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 0x55d8e996be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d12d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 0x55d8e996be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48734/0 (socket says 172.21.15.111:48734) 2023-10-11T16:41:15.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d11d9b700 1 -- 172.21.15.111:0/293749140 learned_addr learned my addr 172.21.15.111:0/293749140 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e98548e0 con 0x55d8e9961b60 2023-10-11T16:41:15.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e9854580 con 0x55d8e9962470 2023-10-11T16:41:15.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e9736920 con 0x55d8e99668b0 2023-10-11T16:41:15.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.311+0000 7f0d2c365b80 10 client.0 _ll_register_callbacks cb 0x55d8e9872780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:15.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.312+0000 7f0d1259c700 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e996b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.312+0000 7f0d11d9b700 1 -- 172.21.15.111:0/293749140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 msgr2=0x55d8e996be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.312+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 0x55d8e996be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.312+0000 7f0d11d9b700 1 -- 172.21.15.111:0/293749140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 msgr2=0x55d8e996b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.312+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e996b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.323 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.313+0000 7f0d11d9b700 1 -- 172.21.15.111:0/293749140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d14110270 con 0x55d8e9962470 2023-10-11T16:41:15.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.313+0000 7f0d1259c700 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e996b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:15.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.313+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e996b920 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009fd0 tx=0x7f0d080b9270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cab253afd87abc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.313+0000 7f0cf9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:41:15.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.314+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/293749140 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08018520 con 0x55d8e9962470 2023-10-11T16:41:15.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.314+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/293749140 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f0d08098c20 con 0x55d8e9962470 2023-10-11T16:41:15.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.315+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d8e9954b20 con 0x55d8e9962470 2023-10-11T16:41:15.325 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.315+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/293749140 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d08018520 con 0x55d8e9962470 2023-10-11T16:41:15.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.316+0000 7f0d2c365b80 10 client.27071 fetch_fsmap learned FSMap version 33 2023-10-11T16:41:15.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.316+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d8e9879190 con 0x55d8e9962470 2023-10-11T16:41:15.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.316+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/293749140 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d8e9954b20 con 0x55d8e9962470 2023-10-11T16:41:15.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.316+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/293749140 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0d080a1780 con 0x55d8e9962470 2023-10-11T16:41:15.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.316+0000 7f0d2c365b80 10 client.27071 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:41:15.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.316+0000 7f0d2c365b80 10 client.27071 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:41:15.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.316+0000 7f0d2c365b80 20 client.27071 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:15.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.316+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55d8e997b450 0x55d8e997d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.317+0000 7f0d2c365b80 4 client.27071 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.317+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d8e98750a0 con 0x55d8e997b450 2023-10-11T16:41:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.317+0000 7f0d1259c700 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55d8e997b450 0x55d8e997d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.318+0000 7f0cf9ffb700 10 client.27071 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:41:15.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.318+0000 7f0d1259c700 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55d8e997b450 0x55d8e997d830 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2047a708ff308803 server_cookie=3b0e5070654d91e5 in_seq=0 out_seq=0 2023-10-11T16:41:15.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.319+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/293749140 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d8e98750a0 con 0x55d8e997b450 2023-10-11T16:41:15.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.319+0000 7f0cf9ffb700 10 client.27071 handle_command_reply: tid=0 2023-10-11T16:41:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.319+0000 7f0d2c365b80 1 client.27071 shutdown 2023-10-11T16:41:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 msgr2=0x55d8e996b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e996b920 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009fd0 tx=0x7f0d080b9270 comp rx=0 tx=0).stop 2023-10-11T16:41:15.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 shutdown_connections 2023-10-11T16:41:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55d8e997b450 0x55d8e997d830 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e99668b0 0x55d8e996be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e996b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/293749140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e9962470 0x55d8e996b920 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 >> 172.21.15.111:0/293749140 conn(0x55d8e9957360 msgr2=0x55d8e9967d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0cf9ffb700 0 client.27071 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:41:15.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 shutdown_connections 2023-10-11T16:41:15.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.320+0000 7f0d2c365b80 1 -- 172.21.15.111:0/293749140 wait complete. 2023-10-11T16:41:15.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.321+0000 7f0d2c365b80 20 client.27071 trim_cache size 0 max 16384 2023-10-11T16:41:15.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.347+0000 7f0d2c365b80 1 Processor -- start 2023-10-11T16:41:15.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.348+0000 7f0d2c365b80 1 -- start start 2023-10-11T16:41:15.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.348+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9972a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.348+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9962470 0x55d8e996b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.348+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e996be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.348+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e996e730 con 0x55d8e9961b60 2023-10-11T16:41:15.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.348+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e9965500 con 0x55d8e99668b0 2023-10-11T16:41:15.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.348+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e9969400 con 0x55d8e9962470 2023-10-11T16:41:15.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.349+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9962470 0x55d8e996b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.349+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9962470 0x55d8e996b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48748/0 (socket says 172.21.15.111:48748) 2023-10-11T16:41:15.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.349+0000 7f0d11d9b700 1 -- 172.21.15.111:0/313885555 learned_addr learned my addr 172.21.15.111:0/313885555 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.349+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e996be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.349+0000 7f0d12d9d700 1 -- 172.21.15.111:0/313885555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9962470 msgr2=0x55d8e996b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.349+0000 7f0d1259c700 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9972a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.350+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9962470 0x55d8e996b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.350+0000 7f0d12d9d700 1 -- 172.21.15.111:0/313885555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 msgr2=0x55d8e9972a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.350+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9972a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.350+0000 7f0d12d9d700 1 -- 172.21.15.111:0/313885555 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8e9878ee0 con 0x55d8e99668b0 2023-10-11T16:41:15.356 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.350+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9962470 0x55d8e996b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:15.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.350+0000 7f0d1259c700 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9972a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:41:15.357 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.351+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e996be90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009fd0 tx=0x7f0d0c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d71f940b5806ac6 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.351+0000 7f0d017fa700 1 -- 172.21.15.111:0/313885555 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c027070 con 0x55d8e99668b0 2023-10-11T16:41:15.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.351+0000 7f0d017fa700 1 -- 172.21.15.111:0/313885555 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d0c00f040 con 0x55d8e99668b0 2023-10-11T16:41:15.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.351+0000 7f0d017fa700 1 -- 172.21.15.111:0/313885555 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c01c040 con 0x55d8e99668b0 2023-10-11T16:41:15.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.352+0000 7f0d2c365b80 1 -- 172.21.15.111:0/313885555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 msgr2=0x55d8e996be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.353+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e996be90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009fd0 tx=0x7f0d0c004b40 comp rx=0 tx=0).stop 2023-10-11T16:41:15.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.353+0000 7f0d2c365b80 1 -- 172.21.15.111:0/313885555 shutdown_connections 2023-10-11T16:41:15.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.353+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e996be90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.353+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9961b60 0x55d8e9972a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.353+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/313885555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9962470 0x55d8e996b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.353+0000 7f0d2c365b80 1 -- 172.21.15.111:0/313885555 >> 172.21.15.111:0/313885555 conn(0x55d8e9957360 msgr2=0x55d8e9964970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.354+0000 7f0d2c365b80 1 -- 172.21.15.111:0/313885555 shutdown_connections 2023-10-11T16:41:15.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.354+0000 7f0d2c365b80 1 -- 172.21.15.111:0/313885555 wait complete. 2023-10-11T16:41:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.354+0000 7f0d2c365b80 1 -- start start 2023-10-11T16:41:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.355+0000 7f0d2c365b80 1 Processor -- start 2023-10-11T16:41:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.355+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9961b60 0x55d8e99c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.355+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9962470 0x55d8e99c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.356+0000 7f0d2c365b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e9972080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.356+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e9972680 con 0x55d8e9962470 2023-10-11T16:41:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.356+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e99727f0 con 0x55d8e99668b0 2023-10-11T16:41:15.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.356+0000 7f0d2c365b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e99729b0 con 0x55d8e9961b60 2023-10-11T16:41:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.356+0000 7f0d2c365b80 10 client.0 _ll_register_callbacks cb 0x7f0d08099270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.356+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9962470 0x55d8e99c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.357+0000 7f0d12d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e9972080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.357+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9962470 0x55d8e99c82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36370/0 (socket says 172.21.15.111:36370) 2023-10-11T16:41:15.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.357+0000 7f0d11d9b700 1 -- 172.21.15.111:0/3593305206 learned_addr learned my addr 172.21.15.111:0/3593305206 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.357+0000 7f0d1259c700 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9961b60 0x55d8e99c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.357+0000 7f0d12d9d700 1 -- 172.21.15.111:0/3593305206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9961b60 msgr2=0x55d8e99c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.357+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9961b60 0x55d8e99c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.357+0000 7f0d12d9d700 1 -- 172.21.15.111:0/3593305206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9962470 msgr2=0x55d8e99c82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.357+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9962470 0x55d8e99c82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.358+0000 7f0d12d9d700 1 -- 172.21.15.111:0/3593305206 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8e9879190 con 0x55d8e99668b0 2023-10-11T16:41:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.358+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9962470 0x55d8e99c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:15.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.358+0000 7f0d12d9d700 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e9972080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c004b10 tx=0x7f0d0c00ff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5df26bc43566fbc server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.358+0000 7f0cf9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:41:15.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.359+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/3593305206 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c027070 con 0x55d8e99668b0 2023-10-11T16:41:15.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.359+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/3593305206 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f0d0c004db0 con 0x55d8e99668b0 2023-10-11T16:41:15.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.359+0000 7f0d2c365b80 1 -- 172.21.15.111:0/3593305206 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d08010e80 con 0x55d8e99668b0 2023-10-11T16:41:15.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.360+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/3593305206 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d0c00e040 con 0x55d8e99668b0 2023-10-11T16:41:15.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.360+0000 7f0d2c365b80 10 client.27077 fetch_fsmap learned FSMap version 33 2023-10-11T16:41:15.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.360+0000 7f0d2c365b80 1 -- 172.21.15.111:0/3593305206 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f0d0c009040 con 0x55d8e99668b0 2023-10-11T16:41:15.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.361+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/3593305206 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d08010e80 con 0x55d8e99668b0 2023-10-11T16:41:15.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.361+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/3593305206 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0d0c0239a0 con 0x55d8e99668b0 2023-10-11T16:41:15.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.361+0000 7f0d2c365b80 10 client.27077 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:41:15.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.361+0000 7f0d2c365b80 10 client.27077 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:41:15.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.361+0000 7f0d2c365b80 20 client.27077 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:15.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.361+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55d8e99cfb30 0x55d8e99d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.361+0000 7f0d2c365b80 4 client.27077 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:41:15.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.361+0000 7f0d2c365b80 1 -- 172.21.15.111:0/3593305206 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d8e9881660 con 0x55d8e99cfb30 2023-10-11T16:41:15.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.362+0000 7f0d1259c700 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55d8e99cfb30 0x55d8e99d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.363+0000 7f0cf9ffb700 10 client.27077 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:41:15.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.363+0000 7f0d1259c700 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55d8e99cfb30 0x55d8e99d1f40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=45866ae0618cd2a9 server_cookie=63f60701cab579b6 in_seq=0 out_seq=0 2023-10-11T16:41:15.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.364+0000 7f0cf9ffb700 1 -- 172.21.15.111:0/3593305206 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d8e9881660 con 0x55d8e99cfb30 2023-10-11T16:41:15.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.364+0000 7f0cf9ffb700 10 client.27077 handle_command_reply: tid=0 2023-10-11T16:41:15.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.364+0000 7f0d2c365b80 1 client.27077 shutdown 2023-10-11T16:41:15.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0d2c365b80 1 -- 172.21.15.111:0/3593305206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 msgr2=0x55d8e9972080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e9972080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c004b10 tx=0x7f0d0c00ff50 comp rx=0 tx=0).stop 2023-10-11T16:41:15.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0d2c365b80 1 -- 172.21.15.111:0/3593305206 shutdown_connections 2023-10-11T16:41:15.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55d8e99cfb30 0x55d8e99d1f40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8e99668b0 0x55d8e9972080 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d8e9961b60 0x55d8e99c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0d2c365b80 1 --2- 172.21.15.111:0/3593305206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d8e9962470 0x55d8e99c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0d2c365b80 1 -- 172.21.15.111:0/3593305206 >> 172.21.15.111:0/3593305206 conn(0x55d8e9957360 msgr2=0x55d8e996b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.365+0000 7f0cf9ffb700 0 client.27077 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:41:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.366+0000 7f0d2c365b80 1 -- 172.21.15.111:0/3593305206 shutdown_connections 2023-10-11T16:41:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.366+0000 7f0d2c365b80 1 -- 172.21.15.111:0/3593305206 wait complete. 2023-10-11T16:41:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.367+0000 7f0d2c365b80 20 client.27077 trim_cache size 0 max 16384 2023-10-11T16:41:15.380 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:41:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0cfc0a1340 msgr2=0x7f0cfc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0cfc0a1340 0x7f0cfc0a37f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0d1406a4a0 tx=0x7f0d0c00b040 comp rx=0 tx=0).stop 2023-10-11T16:41:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101ca0 msgr2=0x7f0d14104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101ca0 0x7f0d14104e80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800b450 tx=0x7f0d0800b760 comp rx=0 tx=0).stop 2023-10-11T16:41:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 shutdown_connections 2023-10-11T16:41:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0cfc0a1340 0x7f0cfc0a37f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d1410a8b0 0x7f0d141053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d14101390 0x7f0d14109e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 --2- 172.21.15.111:0/4049567196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d14101ca0 0x7f0d14104e80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.368+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 >> 172.21.15.111:0/4049567196 conn(0x7f0d140770b0 msgr2=0x7f0d140fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.369+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 shutdown_connections 2023-10-11T16:41:15.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.369+0000 7f0d189ff700 1 -- 172.21.15.111:0/4049567196 wait complete. 2023-10-11T16:41:15.384 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:15 smithi111 ceph-mon[98527]: pgmap v266: 129 pgs: 10 active+clean+snaptrim, 86 active+clean, 33 active+clean+snaptrim_wait; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:41:15.403 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:41:15.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:15 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/4102232169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:41:15.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:15 smithi150 ceph-mon[132217]: pgmap v266: 129 pgs: 10 active+clean+snaptrim, 86 active+clean, 33 active+clean+snaptrim_wait; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:41:15.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:15 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/4102232169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:41:15.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.765+0000 7fba8a3ff700 1 -- 172.21.15.111:0/2573146085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba84102520 msgr2=0x7fba84102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.765+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/2573146085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba84102520 0x7fba84102990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fba78009990 tx=0x7fba78009ca0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.765+0000 7fba8a3ff700 1 -- 172.21.15.111:0/2573146085 shutdown_connections 2023-10-11T16:41:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.765+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/2573146085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba84102ed0 0x7fba8410f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.765+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/2573146085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84108550 0x7fba84108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.765+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/2573146085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba84102520 0x7fba84102990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.765+0000 7fba8a3ff700 1 -- 172.21.15.111:0/2573146085 >> 172.21.15.111:0/2573146085 conn(0x7fba840fdfc0 msgr2=0x7fba841003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.766+0000 7fba8a3ff700 1 -- 172.21.15.111:0/2573146085 shutdown_connections 2023-10-11T16:41:15.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.766+0000 7fba8a3ff700 1 -- 172.21.15.111:0/2573146085 wait complete. 2023-10-11T16:41:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.767+0000 7fba8a3ff700 1 Processor -- start 2023-10-11T16:41:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba8a3ff700 1 -- start start 2023-10-11T16:41:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba8a3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba84102520 0x7fba8410f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba8a3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84102ed0 0x7fba8410d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba8a3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba84108550 0x7fba8410d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba8a3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8410e0e0 con 0x7fba84108550 2023-10-11T16:41:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba8a3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba841a8ef0 con 0x7fba84102ed0 2023-10-11T16:41:15.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba83fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba84102520 0x7fba8410f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba8a3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba841a9060 con 0x7fba84102520 2023-10-11T16:41:15.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba837fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84102ed0 0x7fba8410d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba83fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba84102520 0x7fba8410f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48776/0 (socket says 172.21.15.111:48776) 2023-10-11T16:41:15.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba83fff700 1 -- 172.21.15.111:0/691653167 learned_addr learned my addr 172.21.15.111:0/691653167 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.768+0000 7fba837fe700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84102ed0 0x7fba8410d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45440/0 (socket says 172.21.15.111:45440) 2023-10-11T16:41:15.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.769+0000 7fba8899c700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba84108550 0x7fba8410d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.769+0000 7fba837fe700 1 -- 172.21.15.111:0/691653167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba84102520 msgr2=0x7fba8410f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.769+0000 7fba837fe700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba84102520 0x7fba8410f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.769+0000 7fba837fe700 1 -- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba84108550 msgr2=0x7fba8410d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.769+0000 7fba837fe700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba84108550 0x7fba8410d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.769+0000 7fba837fe700 1 -- 172.21.15.111:0/691653167 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780095f0 con 0x7fba84102ed0 2023-10-11T16:41:15.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.769+0000 7fba83fff700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba84102520 0x7fba8410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:15.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.769+0000 7fba837fe700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84102ed0 0x7fba8410d2a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fba780098e0 tx=0x7fba7800f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.771+0000 7fba817fa700 1 -- 172.21.15.111:0/691653167 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7801d070 con 0x7fba84102ed0 2023-10-11T16:41:15.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.771+0000 7fba817fa700 1 -- 172.21.15.111:0/691653167 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba78021410 con 0x7fba84102ed0 2023-10-11T16:41:15.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.771+0000 7fba817fa700 1 -- 172.21.15.111:0/691653167 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78017650 con 0x7fba84102ed0 2023-10-11T16:41:15.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.771+0000 7fba8a3ff700 1 -- 172.21.15.111:0/691653167 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba84075b90 con 0x7fba84102ed0 2023-10-11T16:41:15.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.771+0000 7fba8a3ff700 1 -- 172.21.15.111:0/691653167 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba84075a00 con 0x7fba84102ed0 2023-10-11T16:41:15.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.779+0000 7fba9dd47b80 1 Processor -- start 2023-10-11T16:41:15.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.779+0000 7fba9dd47b80 1 -- start start 2023-10-11T16:41:15.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c37b60 0x560592c37f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c38470 0x560592c3c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 0x560592c3cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba9dd47b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592b3d060 con 0x560592c38470 2023-10-11T16:41:15.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba9dd47b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592b8f8a0 con 0x560592c3c8b0 2023-10-11T16:41:15.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba9dd47b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592b88d30 con 0x560592c37b60 2023-10-11T16:41:15.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba8899c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c37b60 0x560592c37f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba8899c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c37b60 0x560592c37f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48778/0 (socket says 172.21.15.111:48778) 2023-10-11T16:41:15.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba8899c700 1 -- 172.21.15.111:0/4027297766 learned_addr learned my addr 172.21.15.111:0/4027297766 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.780+0000 7fba817fa700 1 -- 172.21.15.111:0/691653167 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fba78021580 con 0x7fba84102ed0 2023-10-11T16:41:15.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.781+0000 7fba8899c700 1 -- 172.21.15.111:0/4027297766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 msgr2=0x560592c3cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.781+0000 7fba8899c700 1 --2- 172.21.15.111:0/4027297766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 0x560592c3cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.781+0000 7fba83fff700 1 --2- 172.21.15.111:0/4027297766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c38470 0x560592c3c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.781+0000 7fba8899c700 1 -- 172.21.15.111:0/4027297766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c38470 msgr2=0x560592c3c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.781+0000 7fba8899c700 1 --2- 172.21.15.111:0/4027297766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c38470 0x560592c3c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.781+0000 7fba8899c700 1 -- 172.21.15.111:0/4027297766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560592a0a630 con 0x560592c37b60 2023-10-11T16:41:15.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.781+0000 7fba8899c700 1 --2- 172.21.15.111:0/4027297766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c37b60 0x560592c37f30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009dd0 tx=0x7fba7c0077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=493a9ee5aa42b08d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.782+0000 7fba6a7fc700 1 -- 172.21.15.111:0/4027297766 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c018070 con 0x560592c37b60 2023-10-11T16:41:15.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.782+0000 7fba6a7fc700 1 -- 172.21.15.111:0/4027297766 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba7c00f050 con 0x560592c37b60 2023-10-11T16:41:15.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.782+0000 7fba6a7fc700 1 -- 172.21.15.111:0/4027297766 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c014410 con 0x560592c37b60 2023-10-11T16:41:15.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.783+0000 7fba9dd47b80 1 -- 172.21.15.111:0/4027297766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c37b60 msgr2=0x560592c37f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.783+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/4027297766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c37b60 0x560592c37f30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009dd0 tx=0x7fba7c0077e0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba817fa700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fba6c0a1450 0x7fba6c0a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba9dd47b80 1 -- 172.21.15.111:0/4027297766 shutdown_connections 2023-10-11T16:41:15.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/4027297766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 0x560592c3cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba817fa700 1 -- 172.21.15.111:0/691653167 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fba780ce150 con 0x7fba84102ed0 2023-10-11T16:41:15.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/4027297766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c37b60 0x560592c37f30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/4027297766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c38470 0x560592c3c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba9dd47b80 1 -- 172.21.15.111:0/4027297766 >> 172.21.15.111:0/4027297766 conn(0x560592c2d360 msgr2=0x560592c32b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba9dd47b80 1 -- 172.21.15.111:0/4027297766 shutdown_connections 2023-10-11T16:41:15.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.784+0000 7fba9dd47b80 1 -- 172.21.15.111:0/4027297766 wait complete. 2023-10-11T16:41:15.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.785+0000 7fba8899c700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fba6c0a1450 0x7fba6c0a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.785+0000 7fba9dd47b80 1 -- start start 2023-10-11T16:41:15.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.786+0000 7fba9dd47b80 1 Processor -- start 2023-10-11T16:41:15.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.787+0000 7fba8899c700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fba6c0a1450 0x7fba6c0a3900 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fba7c007be0 tx=0x7fba7c004550 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.787+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c37b60 0x560592c44070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c412f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 0x560592c41990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba83fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c37b60 0x560592c44070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba83fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c37b60 0x560592c44070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36406/0 (socket says 172.21.15.111:36406) 2023-10-11T16:41:15.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba83fff700 1 -- 172.21.15.111:0/19166208 learned_addr learned my addr 172.21.15.111:0/19166208 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592b2a8e0 con 0x560592c37b60 2023-10-11T16:41:15.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592b2a580 con 0x560592c3c8b0 2023-10-11T16:41:15.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592a0c920 con 0x560592c38470 2023-10-11T16:41:15.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba9dd47b80 10 client.0 _ll_register_callbacks cb 0x560592b48780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:15.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba8899c700 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 0x560592c41990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.788+0000 7fba837fe700 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.789+0000 7fba83fff700 1 -- 172.21.15.111:0/19166208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 msgr2=0x560592c412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.789+0000 7fba83fff700 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.789+0000 7fba83fff700 1 -- 172.21.15.111:0/19166208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 msgr2=0x560592c41990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.789+0000 7fba83fff700 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 0x560592c41990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.789+0000 7fba83fff700 1 -- 172.21.15.111:0/19166208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c009a00 con 0x560592c37b60 2023-10-11T16:41:15.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.789+0000 7fba837fe700 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:15.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.790+0000 7fba83fff700 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c37b60 0x560592c44070 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fba740106a0 tx=0x7fba740109b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd7bca53be6dfe server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.790+0000 7fba637fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:15.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.790+0000 7fba637fe700 1 -- 172.21.15.111:0/19166208 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74012860 con 0x560592c37b60 2023-10-11T16:41:15.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.790+0000 7fba637fe700 1 -- 172.21.15.111:0/19166208 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fba74016070 con 0x560592c37b60 2023-10-11T16:41:15.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.790+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560592c2ab20 con 0x560592c37b60 2023-10-11T16:41:15.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.790+0000 7fba637fe700 1 -- 172.21.15.111:0/19166208 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba740129c0 con 0x560592c37b60 2023-10-11T16:41:15.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.791+0000 7fba9dd47b80 10 client.16953 fetch_fsmap learned FSMap version 33 2023-10-11T16:41:15.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.791+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560592b4f190 con 0x560592c37b60 2023-10-11T16:41:15.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.791+0000 7fba637fe700 1 -- 172.21.15.111:0/19166208 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba7401b480 con 0x560592c37b60 2023-10-11T16:41:15.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.791+0000 7fba637fe700 1 -- 172.21.15.111:0/19166208 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fba740113e0 con 0x560592c37b60 2023-10-11T16:41:15.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.791+0000 7fba9dd47b80 10 client.16953 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:41:15.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.791+0000 7fba9dd47b80 10 client.16953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:15.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.791+0000 7fba9dd47b80 20 client.16953 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:15.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.792+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560592c51880 0x560592c53cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.792+0000 7fba9dd47b80 4 client.16953 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:15.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.792+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560592c2ab20 con 0x560592c51880 2023-10-11T16:41:15.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.792+0000 7fba837fe700 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560592c51880 0x560592c53cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.793+0000 7fba637fe700 10 client.16953 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:15.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.793+0000 7fba837fe700 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560592c51880 0x560592c53cd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c86231922bdf9d9 server_cookie=26d4ed472ef896a7 in_seq=0 out_seq=0 2023-10-11T16:41:15.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.794+0000 7fba637fe700 1 -- 172.21.15.111:0/19166208 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560592c2ab20 con 0x560592c51880 2023-10-11T16:41:15.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.794+0000 7fba637fe700 10 client.16953 handle_command_reply: tid=0 2023-10-11T16:41:15.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.794+0000 7fba9dd47b80 1 client.16953 shutdown 2023-10-11T16:41:15.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c37b60 msgr2=0x560592c44070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c37b60 0x560592c44070 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fba740106a0 tx=0x7fba740109b0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 shutdown_connections 2023-10-11T16:41:15.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560592c51880 0x560592c53cd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c3c8b0 0x560592c41990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c37b60 0x560592c44070 secure :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fba740106a0 tx=0x7fba740109b0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/19166208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 >> 172.21.15.111:0/19166208 conn(0x560592c2d360 msgr2=0x560592c3dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba637fe700 0 client.16953 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:15.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 shutdown_connections 2023-10-11T16:41:15.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.795+0000 7fba9dd47b80 1 -- 172.21.15.111:0/19166208 wait complete. 2023-10-11T16:41:15.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.796+0000 7fba9dd47b80 20 client.16953 trim_cache size 0 max 16384 2023-10-11T16:41:15.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.825+0000 7fba9dd47b80 1 Processor -- start 2023-10-11T16:41:15.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.825+0000 7fba9dd47b80 1 -- start start 2023-10-11T16:41:15.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.825+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c2d360 0x560592c489e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.825+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c43c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.825+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c3c8b0 0x560592c3a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.825+0000 7fba9dd47b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592c3f390 con 0x560592c3c8b0 2023-10-11T16:41:15.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.825+0000 7fba9dd47b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592c3b510 con 0x560592c2d360 2023-10-11T16:41:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.826+0000 7fba837fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c43c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.826+0000 7fba837fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c43c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48800/0 (socket says 172.21.15.111:48800) 2023-10-11T16:41:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.826+0000 7fba837fe700 1 -- 172.21.15.111:0/2561137350 learned_addr learned my addr 172.21.15.111:0/2561137350 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.826+0000 7fba9dd47b80 1 -- 172.21.15.111:0/2561137350 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592c42640 con 0x560592c38470 2023-10-11T16:41:15.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.827+0000 7fba8899c700 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c3c8b0 0x560592c3a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.827+0000 7fba837fe700 1 -- 172.21.15.111:0/2561137350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c2d360 msgr2=0x560592c489e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-11T16:41:15.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.827+0000 7fba837fe700 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c2d360 0x560592c489e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.827+0000 7fba837fe700 1 -- 172.21.15.111:0/2561137350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c3c8b0 msgr2=0x560592c3a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.827+0000 7fba837fe700 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c3c8b0 0x560592c3a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.827+0000 7fba837fe700 1 -- 172.21.15.111:0/2561137350 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560592c3cf50 con 0x560592c38470 2023-10-11T16:41:15.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.827+0000 7fba8899c700 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c3c8b0 0x560592c3a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:41:15.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.827+0000 7fba837fe700 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c43c10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fba78097860 tx=0x7fba7801a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d48aea9a89039 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.828+0000 7fba6affd700 1 -- 172.21.15.111:0/2561137350 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba780b2070 con 0x560592c38470 2023-10-11T16:41:15.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.828+0000 7fba6affd700 1 -- 172.21.15.111:0/2561137350 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba7809abc0 con 0x560592c38470 2023-10-11T16:41:15.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.828+0000 7fba6affd700 1 -- 172.21.15.111:0/2561137350 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7809b2b0 con 0x560592c38470 2023-10-11T16:41:15.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:15 smithi130 ceph-mon[131859]: pgmap v266: 129 pgs: 10 active+clean+snaptrim, 86 active+clean, 33 active+clean+snaptrim_wait; 663 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:41:15.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:15 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/4102232169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:41:15.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 -- 172.21.15.111:0/2561137350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 msgr2=0x560592c43c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c43c10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fba78097860 tx=0x7fba7801a040 comp rx=0 tx=0).stop 2023-10-11T16:41:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 -- 172.21.15.111:0/2561137350 shutdown_connections 2023-10-11T16:41:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c3c8b0 0x560592c3a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c2d360 0x560592c489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/2561137350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c38470 0x560592c43c10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 -- 172.21.15.111:0/2561137350 >> 172.21.15.111:0/2561137350 conn(0x560592c32230 msgr2=0x560592c53890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 -- 172.21.15.111:0/2561137350 shutdown_connections 2023-10-11T16:41:15.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.830+0000 7fba9dd47b80 1 -- 172.21.15.111:0/2561137350 wait complete. 2023-10-11T16:41:15.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.831+0000 7fba9dd47b80 1 -- start start 2023-10-11T16:41:15.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.832+0000 7fba9dd47b80 1 Processor -- start 2023-10-11T16:41:15.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.832+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c2d360 0x560592ca0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.832+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c38470 0x560592ca0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.832+0000 7fba9dd47b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c3c8b0 0x560592ca11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.832+0000 7fba9dd47b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592c469c0 con 0x560592c2d360 2023-10-11T16:41:15.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.832+0000 7fba9dd47b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592c46b00 con 0x560592c38470 2023-10-11T16:41:15.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.832+0000 7fba9dd47b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560592c46cc0 con 0x560592c3c8b0 2023-10-11T16:41:15.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.832+0000 7fba9dd47b80 10 client.0 _ll_register_callbacks cb 0x7fba74007c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:15.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.833+0000 7fba8899c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c3c8b0 0x560592ca11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.833+0000 7fba837fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c38470 0x560592ca0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.833+0000 7fba8899c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c3c8b0 0x560592ca11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:48814/0 (socket says 172.21.15.111:48814) 2023-10-11T16:41:15.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.833+0000 7fba8899c700 1 -- 172.21.15.111:0/522367519 learned_addr learned my addr 172.21.15.111:0/522367519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:15.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.834+0000 7fba83fff700 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c2d360 0x560592ca0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.834+0000 7fba8899c700 1 -- 172.21.15.111:0/522367519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c38470 msgr2=0x560592ca0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.834+0000 7fba8899c700 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c38470 0x560592ca0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.834+0000 7fba8899c700 1 -- 172.21.15.111:0/522367519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c2d360 msgr2=0x560592ca0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:15.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.834+0000 7fba8899c700 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c2d360 0x560592ca0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.834+0000 7fba8899c700 1 -- 172.21.15.111:0/522367519 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba84075a00 con 0x560592c3c8b0 2023-10-11T16:41:15.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.834+0000 7fba83fff700 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c2d360 0x560592ca0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:41:15.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.834+0000 7fba8899c700 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c3c8b0 0x560592ca11e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0045c0 tx=0x7fba7c0077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a011a2a54b7152d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:15.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.835+0000 7fba637fe700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:41:15.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.835+0000 7fba637fe700 1 -- 172.21.15.111:0/522367519 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c018070 con 0x560592c3c8b0 2023-10-11T16:41:15.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.835+0000 7fba637fe700 1 -- 172.21.15.111:0/522367519 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fba7c01e8b0 con 0x560592c3c8b0 2023-10-11T16:41:15.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.835+0000 7fba9dd47b80 1 -- 172.21.15.111:0/522367519 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba7c00f050 con 0x560592c3c8b0 2023-10-11T16:41:15.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.837+0000 7fba637fe700 1 -- 172.21.15.111:0/522367519 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba7c004e80 con 0x560592c3c8b0 2023-10-11T16:41:15.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.838+0000 7fba9dd47b80 10 client.27103 fetch_fsmap learned FSMap version 33 2023-10-11T16:41:15.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.838+0000 7fba9dd47b80 1 -- 172.21.15.111:0/522367519 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5605929b6de0 con 0x560592c3c8b0 2023-10-11T16:41:15.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.839+0000 7fba637fe700 1 -- 172.21.15.111:0/522367519 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba7c00f050 con 0x560592c3c8b0 2023-10-11T16:41:15.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.839+0000 7fba637fe700 1 -- 172.21.15.111:0/522367519 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 33) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fba7c0056c0 con 0x560592c3c8b0 2023-10-11T16:41:15.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.840+0000 7fba9dd47b80 10 client.27103 fetch_fsmap finished waiting for FSMap version 33 2023-10-11T16:41:15.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.840+0000 7fba9dd47b80 10 client.27103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:15.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.840+0000 7fba9dd47b80 20 client.27103 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:15.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.840+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560592cb70e0 0x560592cb94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:15.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.840+0000 7fba9dd47b80 4 client.27103 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:41:15.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.840+0000 7fba9dd47b80 1 -- 172.21.15.111:0/522367519 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560592b4b0a0 con 0x560592cb70e0 2023-10-11T16:41:15.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.842+0000 7fba83fff700 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560592cb70e0 0x560592cb94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:15.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.842+0000 7fba637fe700 10 client.27103 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:15.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.842+0000 7fba83fff700 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560592cb70e0 0x560592cb94f0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec7ded144cbb887f server_cookie=f5b8ecd228a7b8e3 in_seq=0 out_seq=0 2023-10-11T16:41:15.853 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba637fe700 1 -- 172.21.15.111:0/522367519 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560592b4b0a0 con 0x560592cb70e0 2023-10-11T16:41:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba637fe700 10 client.27103 handle_command_reply: tid=0 2023-10-11T16:41:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 client.27103 shutdown 2023-10-11T16:41:15.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 -- 172.21.15.111:0/522367519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c3c8b0 msgr2=0x560592ca11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c3c8b0 0x560592ca11e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0045c0 tx=0x7fba7c0077e0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 -- 172.21.15.111:0/522367519 shutdown_connections 2023-10-11T16:41:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560592cb70e0 0x560592cb94f0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560592c3c8b0 0x560592ca11e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560592c2d360 0x560592ca0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 --2- 172.21.15.111:0/522367519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560592c38470 0x560592ca0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba9dd47b80 1 -- 172.21.15.111:0/522367519 >> 172.21.15.111:0/522367519 conn(0x560592c32230 msgr2=0x560592c3a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.843+0000 7fba637fe700 0 client.27103 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:15.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.844+0000 7fba9dd47b80 1 -- 172.21.15.111:0/522367519 shutdown_connections 2023-10-11T16:41:15.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.844+0000 7fba9dd47b80 1 -- 172.21.15.111:0/522367519 wait complete. 2023-10-11T16:41:15.858 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.844+0000 7fba9dd47b80 20 client.27103 trim_cache size 0 max 16384 2023-10-11T16:41:15.858 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:41:15.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 -- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fba6c0a1450 msgr2=0x7fba6c0a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.859 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fba6c0a1450 0x7fba6c0a3900 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fba7c007be0 tx=0x7fba7c004550 comp rx=0 tx=0).stop 2023-10-11T16:41:15.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 -- 172.21.15.111:0/691653167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84102ed0 msgr2=0x7fba8410d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:15.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84102ed0 0x7fba8410d2a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fba780098e0 tx=0x7fba7800f740 comp rx=0 tx=0).stop 2023-10-11T16:41:15.860 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 -- 172.21.15.111:0/691653167 shutdown_connections 2023-10-11T16:41:15.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fba6c0a1450 0x7fba6c0a3900 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba84108550 0x7fba8410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba84102520 0x7fba8410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.861 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 --2- 172.21.15.111:0/691653167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84102ed0 0x7fba8410d2a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:15.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 -- 172.21.15.111:0/691653167 >> 172.21.15.111:0/691653167 conn(0x7fba840fdfc0 msgr2=0x7fba840ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:15.862 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.845+0000 7fba8a3ff700 1 -- 172.21.15.111:0/691653167 shutdown_connections 2023-10-11T16:41:15.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:15.846+0000 7fba8a3ff700 1 -- 172.21.15.111:0/691653167 wait complete. 2023-10-11T16:41:16.881 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:41:16.881 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:41:17.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 -- 172.21.15.111:0/1533012328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c100cc0 msgr2=0x7fc46c101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 --2- 172.21.15.111:0/1533012328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c100cc0 0x7fc46c101090 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc46800b0d0 tx=0x7fc46800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 -- 172.21.15.111:0/1533012328 shutdown_connections 2023-10-11T16:41:17.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 --2- 172.21.15.111:0/1533012328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc46c105620 0x7fc46c10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 --2- 172.21.15.111:0/1533012328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c100cc0 0x7fc46c101090 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 --2- 172.21.15.111:0/1533012328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc46c101660 0x7fc46c104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 -- 172.21.15.111:0/1533012328 >> 172.21.15.111:0/1533012328 conn(0x7fc46c077560 msgr2=0x7fc46c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 -- 172.21.15.111:0/1533012328 shutdown_connections 2023-10-11T16:41:17.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.085+0000 7fc4739ff700 1 -- 172.21.15.111:0/1533012328 wait complete. 2023-10-11T16:41:17.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.087+0000 7fc4739ff700 1 Processor -- start 2023-10-11T16:41:17.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.087+0000 7fc4739ff700 1 -- start start 2023-10-11T16:41:17.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.087+0000 7fc4739ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc46c100cc0 0x7fc46c196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.087+0000 7fc4739ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc46c101660 0x7fc46c196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc4739ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c105620 0x7fc46c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc4739ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c19a5e0 con 0x7fc46c105620 2023-10-11T16:41:17.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc4739ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c19a750 con 0x7fc46c101660 2023-10-11T16:41:17.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc4739ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c19a910 con 0x7fc46c100cc0 2023-10-11T16:41:17.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c105620 0x7fc46c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c105620 0x7fc46c199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36432/0 (socket says 172.21.15.111:36432) 2023-10-11T16:41:17.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 -- 172.21.15.111:0/3732671698 learned_addr learned my addr 172.21.15.111:0/3732671698 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc47179b700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc46c100cc0 0x7fc46c196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc470f9a700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc46c101660 0x7fc46c196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 -- 172.21.15.111:0/3732671698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc46c100cc0 msgr2=0x7fc46c196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc46c100cc0 0x7fc46c196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 -- 172.21.15.111:0/3732671698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc46c101660 msgr2=0x7fc46c196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc46c101660 0x7fc46c196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 -- 172.21.15.111:0/3732671698 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468009d00 con 0x7fc46c105620 2023-10-11T16:41:17.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.088+0000 7fc471f9c700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c105620 0x7fc46c199fe0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e940 tx=0x7fc46400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.089+0000 7fc4627fc700 1 -- 172.21.15.111:0/3732671698 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4640097b0 con 0x7fc46c105620 2023-10-11T16:41:17.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.089+0000 7fc4627fc700 1 -- 172.21.15.111:0/3732671698 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc464005020 con 0x7fc46c105620 2023-10-11T16:41:17.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.089+0000 7fc4627fc700 1 -- 172.21.15.111:0/3732671698 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46400cc40 con 0x7fc46c105620 2023-10-11T16:41:17.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.089+0000 7fc4739ff700 1 -- 172.21.15.111:0/3732671698 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46c1103d0 con 0x7fc46c105620 2023-10-11T16:41:17.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.089+0000 7fc4739ff700 1 -- 172.21.15.111:0/3732671698 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c110240 con 0x7fc46c105620 2023-10-11T16:41:17.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.093+0000 7fc4627fc700 1 -- 172.21.15.111:0/3732671698 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fc464005400 con 0x7fc46c105620 2023-10-11T16:41:17.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.094+0000 7fc4872efb80 1 Processor -- start 2023-10-11T16:41:17.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.094+0000 7fc4627fc700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc4580a1340 0x7fc4580a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc4872efb80 1 -- start start 2023-10-11T16:41:17.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc4627fc700 1 -- 172.21.15.111:0/3732671698 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fc464014070 con 0x7fc46c105620 2023-10-11T16:41:17.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc470f9a700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc4580a1340 0x7fc4580a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a964323c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96432900 0x560a96436800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a96437190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc4872efb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a963330e0 con 0x560a96431ff0 2023-10-11T16:41:17.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc4872efb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a96385920 con 0x560a96436d40 2023-10-11T16:41:17.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc4872efb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a9637edb0 con 0x560a96432900 2023-10-11T16:41:17.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc47179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a964323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc47179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a964323c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36438/0 (socket says 172.21.15.111:36438) 2023-10-11T16:41:17.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc47179b700 1 -- 172.21.15.111:0/884190720 learned_addr learned my addr 172.21.15.111:0/884190720 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc471f9c700 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a96437190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.095+0000 7fc470f9a700 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96432900 0x560a96436800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc470f9a700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc4580a1340 0x7fc4580a37f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc46c197d00 tx=0x7fc45c007f80 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc471f9c700 1 -- 172.21.15.111:0/884190720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96432900 msgr2=0x560a96436800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc471f9c700 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96432900 0x560a96436800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc471f9c700 1 -- 172.21.15.111:0/884190720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 msgr2=0x560a964323c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc471f9c700 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a964323c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc471f9c700 1 -- 172.21.15.111:0/884190720 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a962006b0 con 0x560a96436d40 2023-10-11T16:41:17.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc47179b700 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a964323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:17.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc471f9c700 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a96437190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc4640ca520 tx=0x7fc4640998f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d5a7ad7c90570cb server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc4577fe700 1 -- 172.21.15.111:0/884190720 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4640b1070 con 0x560a96436d40 2023-10-11T16:41:17.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc4577fe700 1 -- 172.21.15.111:0/884190720 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4640a22f0 con 0x560a96436d40 2023-10-11T16:41:17.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.096+0000 7fc4577fe700 1 -- 172.21.15.111:0/884190720 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46409a360 con 0x560a96436d40 2023-10-11T16:41:17.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 -- 172.21.15.111:0/884190720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 msgr2=0x560a96437190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a96437190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc4640ca520 tx=0x7fc4640998f0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 -- 172.21.15.111:0/884190720 shutdown_connections 2023-10-11T16:41:17.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a96437190 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a964323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 --2- 172.21.15.111:0/884190720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96432900 0x560a96436800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 -- 172.21.15.111:0/884190720 >> 172.21.15.111:0/884190720 conn(0x560a964233e0 msgr2=0x560a96428bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 -- 172.21.15.111:0/884190720 shutdown_connections 2023-10-11T16:41:17.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 -- 172.21.15.111:0/884190720 wait complete. 2023-10-11T16:41:17.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.097+0000 7fc4872efb80 1 -- start start 2023-10-11T16:41:17.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc4872efb80 1 Processor -- start 2023-10-11T16:41:17.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96431ff0 0x560a9643e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a9643bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc4872efb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a96320960 con 0x560a96432900 2023-10-11T16:41:17.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc4872efb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a96320600 con 0x560a96436d40 2023-10-11T16:41:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc4872efb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a962029a0 con 0x560a96431ff0 2023-10-11T16:41:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc4872efb80 10 client.0 _ll_register_callbacks cb 0x560a9633e800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc470f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc471f9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a9643bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc470f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36446/0 (socket says 172.21.15.111:36446) 2023-10-11T16:41:17.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc470f9a700 1 -- 172.21.15.111:0/2864370553 learned_addr learned my addr 172.21.15.111:0/2864370553 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.098+0000 7fc47179b700 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96431ff0 0x560a9643e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc471f9c700 1 -- 172.21.15.111:0/2864370553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96431ff0 msgr2=0x560a9643e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc471f9c700 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96431ff0 0x560a9643e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc471f9c700 1 -- 172.21.15.111:0/2864370553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 msgr2=0x560a9643b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc471f9c700 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc471f9c700 1 -- 172.21.15.111:0/2864370553 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c110240 con 0x560a96436d40 2023-10-11T16:41:17.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc470f9a700 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:17.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc47179b700 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96431ff0 0x560a9643e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:17.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc471f9c700 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a9643bf20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x560a9643eab0 tx=0x7fc464005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f0c18d4aa369b42 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc454ff9700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:41:17.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc454ff9700 1 -- 172.21.15.111:0/2864370553 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4640b1030 con 0x560a96436d40 2023-10-11T16:41:17.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc454ff9700 1 -- 172.21.15.111:0/2864370553 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fc4640b2070 con 0x560a96436d40 2023-10-11T16:41:17.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc4872efb80 1 -- 172.21.15.111:0/2864370553 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a96420ba0 con 0x560a96436d40 2023-10-11T16:41:17.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc454ff9700 1 -- 172.21.15.111:0/2864370553 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc46400ce80 con 0x560a96436d40 2023-10-11T16:41:17.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc4872efb80 10 client.27101 fetch_fsmap learned FSMap version 34 2023-10-11T16:41:17.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.099+0000 7fc4872efb80 1 -- 172.21.15.111:0/2864370553 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x560a96345210 con 0x560a96436d40 2023-10-11T16:41:17.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc454ff9700 1 -- 172.21.15.111:0/2864370553 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560a96420ba0 con 0x560a96436d40 2023-10-11T16:41:17.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc454ff9700 1 -- 172.21.15.111:0/2864370553 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 34) v1 ==== 7490+0+0 (secure 0 0 0) 0x7fc4640a1e60 con 0x560a96436d40 2023-10-11T16:41:17.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc4872efb80 10 client.27101 fetch_fsmap finished waiting for FSMap version 34 2023-10-11T16:41:17.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc4872efb80 10 client.27101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:17.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc4872efb80 20 client.27101 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:17.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a9644b940 0x560a9644dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc4872efb80 4 client.27101 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:17.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc4872efb80 1 -- 172.21.15.111:0/2864370553 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a96341120 con 0x560a9644b940 2023-10-11T16:41:17.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.100+0000 7fc47179b700 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a9644b940 0x560a9644dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.101+0000 7fc47179b700 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a9644b940 0x560a9644dd20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fadba0ede6da2240 server_cookie=6099587f649e3d0a in_seq=0 out_seq=0 2023-10-11T16:41:17.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.101+0000 7fc454ff9700 10 client.27101 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:17.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.101+0000 7fc454ff9700 1 -- 172.21.15.111:0/2864370553 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560a96341120 con 0x560a9644b940 2023-10-11T16:41:17.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.101+0000 7fc454ff9700 10 client.27101 handle_command_reply: tid=0 2023-10-11T16:41:17.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.101+0000 7fc4872efb80 1 client.27101 shutdown 2023-10-11T16:41:17.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 -- 172.21.15.111:0/2864370553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 msgr2=0x560a9643bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a9643bf20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x560a9643eab0 tx=0x7fc464005770 comp rx=0 tx=0).stop 2023-10-11T16:41:17.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 -- 172.21.15.111:0/2864370553 shutdown_connections 2023-10-11T16:41:17.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a9644b940 0x560a9644dd20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96436d40 0x560a9643bf20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96431ff0 0x560a9643e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2864370553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc454ff9700 0 client.27101 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:17.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 -- 172.21.15.111:0/2864370553 >> 172.21.15.111:0/2864370553 conn(0x560a964233e0 msgr2=0x560a964381f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 -- 172.21.15.111:0/2864370553 shutdown_connections 2023-10-11T16:41:17.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 1 -- 172.21.15.111:0/2864370553 wait complete. 2023-10-11T16:41:17.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.102+0000 7fc4872efb80 20 client.27101 trim_cache size 0 max 16384 2023-10-11T16:41:17.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc4872efb80 1 Processor -- start 2023-10-11T16:41:17.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc4872efb80 1 -- start start 2023-10-11T16:41:17.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96431ff0 0x560a96428c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96435d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc4872efb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a96433050 con 0x560a96432900 2023-10-11T16:41:17.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc4872efb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a964398a0 con 0x560a96431ff0 2023-10-11T16:41:17.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc4872efb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a9643ebc0 con 0x560a96436d40 2023-10-11T16:41:17.140 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc470f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc470f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36460/0 (socket says 172.21.15.111:36460) 2023-10-11T16:41:17.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc470f9a700 1 -- 172.21.15.111:0/636284904 learned_addr learned my addr 172.21.15.111:0/636284904 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc47179b700 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96431ff0 0x560a96428c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.120+0000 7fc471f9c700 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96435d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc470f9a700 1 -- 172.21.15.111:0/636284904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 msgr2=0x560a96435d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc470f9a700 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96435d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc470f9a700 1 -- 172.21.15.111:0/636284904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96431ff0 msgr2=0x560a96428c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc470f9a700 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96431ff0 0x560a96428c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc470f9a700 1 -- 172.21.15.111:0/636284904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a9643b350 con 0x560a96432900 2023-10-11T16:41:17.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc470f9a700 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643e090 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc45c010aa0 tx=0x7fc45c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a0baefec213c59 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc457fff700 1 -- 172.21.15.111:0/636284904 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c024070 con 0x560a96432900 2023-10-11T16:41:17.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc457fff700 1 -- 172.21.15.111:0/636284904 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc45c010db0 con 0x560a96432900 2023-10-11T16:41:17.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.121+0000 7fc457fff700 1 -- 172.21.15.111:0/636284904 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c011ce0 con 0x560a96432900 2023-10-11T16:41:17.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 -- 172.21.15.111:0/636284904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 msgr2=0x560a9643e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643e090 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc45c010aa0 tx=0x7fc45c015000 comp rx=0 tx=0).stop 2023-10-11T16:41:17.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 -- 172.21.15.111:0/636284904 shutdown_connections 2023-10-11T16:41:17.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96435d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96431ff0 0x560a96428c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 --2- 172.21.15.111:0/636284904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96432900 0x560a9643e090 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 -- 172.21.15.111:0/636284904 >> 172.21.15.111:0/636284904 conn(0x560a964233e0 msgr2=0x560a96434e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 -- 172.21.15.111:0/636284904 shutdown_connections 2023-10-11T16:41:17.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 -- 172.21.15.111:0/636284904 wait complete. 2023-10-11T16:41:17.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.122+0000 7fc4872efb80 1 -- start start 2023-10-11T16:41:17.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.123+0000 7fc4872efb80 1 Processor -- start 2023-10-11T16:41:17.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.123+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a96442ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.123+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96432900 0x560a96441460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.123+0000 7fc4872efb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96441ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.123+0000 7fc4872efb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a964422b0 con 0x560a96431ff0 2023-10-11T16:41:17.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.123+0000 7fc4872efb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a9649f7a0 con 0x560a96432900 2023-10-11T16:41:17.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.123+0000 7fc4872efb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a9649f8e0 con 0x560a96436d40 2023-10-11T16:41:17.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.123+0000 7fc4872efb80 10 client.0 _ll_register_callbacks cb 0x7fc464005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:17.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a96442ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a96442ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36462/0 (socket says 172.21.15.111:36462) 2023-10-11T16:41:17.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 -- 172.21.15.111:0/2350008305 learned_addr learned my addr 172.21.15.111:0/2350008305 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc471f9c700 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96441ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc470f9a700 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96432900 0x560a96441460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 -- 172.21.15.111:0/2350008305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 msgr2=0x560a96441ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96441ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 -- 172.21.15.111:0/2350008305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96432900 msgr2=0x560a96441460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96432900 0x560a96441460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 -- 172.21.15.111:0/2350008305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c004490 con 0x560a96431ff0 2023-10-11T16:41:17.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc471f9c700 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96441ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:17.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc47179b700 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a96442ff0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc468009610 tx=0x7fc468024220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78927def50cbb9a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc454ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:17.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.124+0000 7fc454ff9700 1 -- 172.21.15.111:0/2350008305 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc468028030 con 0x560a96431ff0 2023-10-11T16:41:17.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 1 -- 172.21.15.111:0/2350008305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4640a22f0 con 0x560a96431ff0 2023-10-11T16:41:17.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc454ff9700 1 -- 172.21.15.111:0/2350008305 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fc468029070 con 0x560a96431ff0 2023-10-11T16:41:17.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc454ff9700 1 -- 172.21.15.111:0/2350008305 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc468022040 con 0x560a96431ff0 2023-10-11T16:41:17.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 10 client.16971 fetch_fsmap learned FSMap version 34 2023-10-11T16:41:17.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 1 -- 172.21.15.111:0/2350008305 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x560a961ace60 con 0x560a96431ff0 2023-10-11T16:41:17.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc454ff9700 1 -- 172.21.15.111:0/2350008305 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4640a22f0 con 0x560a96431ff0 2023-10-11T16:41:17.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc454ff9700 1 -- 172.21.15.111:0/2350008305 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 34) v1 ==== 7490+0+0 (secure 0 0 0) 0x7fc46800b100 con 0x560a96431ff0 2023-10-11T16:41:17.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 10 client.16971 fetch_fsmap finished waiting for FSMap version 34 2023-10-11T16:41:17.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 10 client.16971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:17.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 20 client.16971 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:17.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a964a0f90 0x560a964a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 4 client.16971 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:41:17.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.125+0000 7fc4872efb80 1 -- 172.21.15.111:0/2350008305 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560a9634d6e0 con 0x560a964a0f90 2023-10-11T16:41:17.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.126+0000 7fc470f9a700 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a964a0f90 0x560a964a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.126+0000 7fc454ff9700 10 client.16971 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:17.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.126+0000 7fc470f9a700 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a964a0f90 0x560a964a33a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d15b3c5634663da server_cookie=470327d3244bcb6e in_seq=0 out_seq=0 2023-10-11T16:41:17.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc454ff9700 1 -- 172.21.15.111:0/2350008305 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560a9634d6e0 con 0x560a964a0f90 2023-10-11T16:41:17.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc454ff9700 10 client.16971 handle_command_reply: tid=0 2023-10-11T16:41:17.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 client.16971 shutdown 2023-10-11T16:41:17.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 -- 172.21.15.111:0/2350008305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 msgr2=0x560a96442ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a96442ff0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc468009610 tx=0x7fc468024220 comp rx=0 tx=0).stop 2023-10-11T16:41:17.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 -- 172.21.15.111:0/2350008305 shutdown_connections 2023-10-11T16:41:17.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x560a964a0f90 0x560a964a33a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x560a96436d40 0x560a96441ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x560a96431ff0 0x560a96442ff0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 --2- 172.21.15.111:0/2350008305 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560a96432900 0x560a96441460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 -- 172.21.15.111:0/2350008305 >> 172.21.15.111:0/2350008305 conn(0x560a964233e0 msgr2=0x560a96435d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc454ff9700 0 client.16971 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:17.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 -- 172.21.15.111:0/2350008305 shutdown_connections 2023-10-11T16:41:17.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 1 -- 172.21.15.111:0/2350008305 wait complete. 2023-10-11T16:41:17.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.127+0000 7fc4872efb80 20 client.16971 trim_cache size 0 max 16384 2023-10-11T16:41:17.167 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:41:17.167 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:41:17.168 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "0f997ce5-5f2f-48b0-b949-4c07d21ace62", 2023-10-11T16:41:17.168 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:41:17.168 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:41:17.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 -- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc4580a1340 msgr2=0x7fc4580a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc4580a1340 0x7fc4580a37f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc46c197d00 tx=0x7fc45c007f80 comp rx=0 tx=0).stop 2023-10-11T16:41:17.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 -- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c105620 msgr2=0x7fc46c199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c105620 0x7fc46c199fe0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e940 tx=0x7fc46400ec50 comp rx=0 tx=0).stop 2023-10-11T16:41:17.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 -- 172.21.15.111:0/3732671698 shutdown_connections 2023-10-11T16:41:17.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc4580a1340 0x7fc4580a37f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc46c105620 0x7fc46c199fe0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.171 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc46c100cc0 0x7fc46c196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 --2- 172.21.15.111:0/3732671698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc46c101660 0x7fc46c196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 -- 172.21.15.111:0/3732671698 >> 172.21.15.111:0/3732671698 conn(0x7fc46c077560 msgr2=0x7fc46c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.172 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 -- 172.21.15.111:0/3732671698 shutdown_connections 2023-10-11T16:41:17.173 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.128+0000 7fc4739ff700 1 -- 172.21.15.111:0/3732671698 wait complete. 2023-10-11T16:41:17.173 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:41:17.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 -- 172.21.15.111:0/3750818348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0100cc0 msgr2=0x7f82a0101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 --2- 172.21.15.111:0/3750818348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0100cc0 0x7f82a0101090 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f829c0099c0 tx=0x7f829c009cd0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 -- 172.21.15.111:0/3750818348 shutdown_connections 2023-10-11T16:41:17.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 --2- 172.21.15.111:0/3750818348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82a0105620 0x7f82a010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 --2- 172.21.15.111:0/3750818348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0100cc0 0x7f82a0101090 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 --2- 172.21.15.111:0/3750818348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a0101660 0x7f82a0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 -- 172.21.15.111:0/3750818348 >> 172.21.15.111:0/3750818348 conn(0x7f82a0077560 msgr2=0x7f82a0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 -- 172.21.15.111:0/3750818348 shutdown_connections 2023-10-11T16:41:17.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.372+0000 7f82a75ff700 1 -- 172.21.15.111:0/3750818348 wait complete. 2023-10-11T16:41:17.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.373+0000 7f82a75ff700 1 Processor -- start 2023-10-11T16:41:17.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.373+0000 7f82a75ff700 1 -- start start 2023-10-11T16:41:17.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.373+0000 7f82a75ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a0100cc0 0x7f82a0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.373+0000 7f82a75ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82a0101660 0x7f82a0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.373+0000 7f82a75ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0105620 0x7f82a01a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.373+0000 7f82a75ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a0073b60 con 0x7f82a0105620 2023-10-11T16:41:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.373+0000 7f82a75ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a01a4c30 con 0x7f82a0101660 2023-10-11T16:41:17.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.373+0000 7f82a75ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a01a4d70 con 0x7f82a0100cc0 2023-10-11T16:41:17.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0105620 0x7f82a01a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a4b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82a0101660 0x7f82a0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0105620 0x7f82a01a46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36480/0 (socket says 172.21.15.111:36480) 2023-10-11T16:41:17.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2771842369 learned_addr learned my addr 172.21.15.111:0/2771842369 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a539b700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a0100cc0 0x7f82a0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2771842369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a0100cc0 msgr2=0x7f82a0072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a0100cc0 0x7f82a0072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2771842369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82a0101660 msgr2=0x7f82a0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82a0101660 0x7f82a0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2771842369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0095f0 con 0x7f82a0105620 2023-10-11T16:41:17.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0105620 0x7f82a01a46f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f829800ef30 tx=0x7f829800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82967fc700 1 -- 172.21.15.111:0/2771842369 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298018070 con 0x7f82a0105620 2023-10-11T16:41:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82967fc700 1 -- 172.21.15.111:0/2771842369 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f829800f040 con 0x7f82a0105620 2023-10-11T16:41:17.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82a75ff700 1 -- 172.21.15.111:0/2771842369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a01103d0 con 0x7f82a0105620 2023-10-11T16:41:17.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.374+0000 7f82967fc700 1 -- 172.21.15.111:0/2771842369 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829800cd00 con 0x7f82a0105620 2023-10-11T16:41:17.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.375+0000 7f82a75ff700 1 -- 172.21.15.111:0/2771842369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a0110240 con 0x7f82a0105620 2023-10-11T16:41:17.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.378+0000 7f82967fc700 1 -- 172.21.15.111:0/2771842369 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f8298003a00 con 0x7f82a0105620 2023-10-11T16:41:17.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82bae31b80 1 Processor -- start 2023-10-11T16:41:17.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82967fc700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f828c0a12c0 0x7f828c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82bae31b80 1 -- start start 2023-10-11T16:41:17.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82967fc700 1 -- 172.21.15.111:0/2771842369 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f82980ccc90 con 0x7f82a0105620 2023-10-11T16:41:17.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f828c0a12c0 0x7f828c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e68f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e69470 0x55f8a2e6d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82bae31b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2d6e060 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82bae31b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2dc08a0 con 0x55f8a2e68b60 2023-10-11T16:41:17.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.379+0000 7f82bae31b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2db9d30 con 0x55f8a2e69470 2023-10-11T16:41:17.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a539b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e68f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36482/0 (socket says 172.21.15.111:36482) 2023-10-11T16:41:17.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2956265314 learned_addr learned my addr 172.21.15.111:0/2956265314 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/2956265314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e69470 0x55f8a2e6d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f828c0a12c0 0x7f828c0a3770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f82a00745e0 tx=0x7f8290011440 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2956265314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e69470 msgr2=0x55f8a2e6d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2956265314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e69470 0x55f8a2e6d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2956265314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 msgr2=0x55f8a2e68f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2956265314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e68f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2956265314 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8a2c3b630 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.380+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2956265314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6dd00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f829800ef00 tx=0x7f8298099800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd70324cac2ec0c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f828b7fe700 1 -- 172.21.15.111:0/2956265314 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298084040 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f828b7fe700 1 -- 172.21.15.111:0/2956265314 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f829800ce60 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f828b7fe700 1 -- 172.21.15.111:0/2956265314 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298099df0 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f82bae31b80 1 -- 172.21.15.111:0/2956265314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 msgr2=0x55f8a2e6dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2956265314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6dd00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f829800ef00 tx=0x7f8298099800 comp rx=0 tx=0).stop 2023-10-11T16:41:17.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f82bae31b80 1 -- 172.21.15.111:0/2956265314 shutdown_connections 2023-10-11T16:41:17.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2956265314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6dd00 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2956265314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e68f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2956265314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e69470 0x55f8a2e6d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.381+0000 7f82bae31b80 1 -- 172.21.15.111:0/2956265314 >> 172.21.15.111:0/2956265314 conn(0x55f8a2e5e360 msgr2=0x55f8a2e63b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.382+0000 7f82bae31b80 1 -- 172.21.15.111:0/2956265314 shutdown_connections 2023-10-11T16:41:17.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.382+0000 7f82bae31b80 1 -- 172.21.15.111:0/2956265314 wait complete. 2023-10-11T16:41:17.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.382+0000 7f82bae31b80 1 -- start start 2023-10-11T16:41:17.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.382+0000 7f82bae31b80 1 Processor -- start 2023-10-11T16:41:17.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e72350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e72890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82bae31b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2d5b8e0 con 0x55f8a2e69470 2023-10-11T16:41:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82bae31b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2d5b580 con 0x55f8a2e68b60 2023-10-11T16:41:17.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82bae31b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2c3d920 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82bae31b80 10 client.0 _ll_register_callbacks cb 0x55f8a2d79780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:17.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82a4b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e72890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82a539b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e72350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82a539b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e72350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:45580/0 (socket says 172.21.15.111:45580) 2023-10-11T16:41:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82a539b700 1 -- 172.21.15.111:0/3747125409 learned_addr learned my addr 172.21.15.111:0/3747125409 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e72890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36488/0 (socket says 172.21.15.111:36488) 2023-10-11T16:41:17.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.383+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.384+0000 7f82a4b9a700 1 -- 172.21.15.111:0/3747125409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e6d8b0 msgr2=0x55f8a2e72dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.384+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.384+0000 7f82a4b9a700 1 -- 172.21.15.111:0/3747125409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 msgr2=0x55f8a2e72350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.384+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e72350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.384+0000 7f82a4b9a700 1 -- 172.21.15.111:0/3747125409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a0110240 con 0x55f8a2e69470 2023-10-11T16:41:17.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.384+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:17.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.384+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e72890 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f82a006a350 tx=0x7f82900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f973cba7dcb3a893 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.384+0000 7f8288ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.387+0000 7f8288ff9700 1 -- 172.21.15.111:0/3747125409 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000f7d0 con 0x55f8a2e69470 2023-10-11T16:41:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.387+0000 7f82bae31b80 1 -- 172.21.15.111:0/3747125409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8a2e5bb20 con 0x55f8a2e69470 2023-10-11T16:41:17.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.387+0000 7f8288ff9700 1 -- 172.21.15.111:0/3747125409 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f829000f930 con 0x55f8a2e69470 2023-10-11T16:41:17.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.387+0000 7f8288ff9700 1 -- 172.21.15.111:0/3747125409 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f829000faf0 con 0x55f8a2e69470 2023-10-11T16:41:17.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82bae31b80 10 client.16983 fetch_fsmap learned FSMap version 34 2023-10-11T16:41:17.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82bae31b80 1 -- 172.21.15.111:0/3747125409 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55f8a2d80190 con 0x55f8a2e69470 2023-10-11T16:41:17.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f8288ff9700 1 -- 172.21.15.111:0/3747125409 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f8a2e5bb20 con 0x55f8a2e69470 2023-10-11T16:41:17.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f8288ff9700 1 -- 172.21.15.111:0/3747125409 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 34) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f8290028bc0 con 0x55f8a2e69470 2023-10-11T16:41:17.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82bae31b80 10 client.16983 fetch_fsmap finished waiting for FSMap version 34 2023-10-11T16:41:17.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82bae31b80 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82bae31b80 20 client.16983 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82bae31b80 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8a2e829f0 0x55f8a2e84e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82bae31b80 4 client.16983 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:17.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82bae31b80 1 -- 172.21.15.111:0/3747125409 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8a2d7c0a0 con 0x55f8a2e829f0 2023-10-11T16:41:17.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.388+0000 7f82a539b700 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8a2e829f0 0x55f8a2e84e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.389+0000 7f82a539b700 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8a2e829f0 0x55f8a2e84e00 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcaf681a8cfd1a04 server_cookie=6aadbf543f22fff3 in_seq=0 out_seq=0 2023-10-11T16:41:17.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.389+0000 7f8288ff9700 10 client.16983 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:17.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f8288ff9700 1 -- 172.21.15.111:0/3747125409 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f8a2d7c0a0 con 0x55f8a2e829f0 2023-10-11T16:41:17.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f8288ff9700 10 client.16983 handle_command_reply: tid=0 2023-10-11T16:41:17.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 client.16983 shutdown 2023-10-11T16:41:17.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 -- 172.21.15.111:0/3747125409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 msgr2=0x55f8a2e72890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e72890 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f82a006a350 tx=0x7f82900047c0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 -- 172.21.15.111:0/3747125409 shutdown_connections 2023-10-11T16:41:17.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8a2e829f0 0x55f8a2e84e00 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e68b60 0x55f8a2e72350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 --2- 172.21.15.111:0/3747125409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e72890 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 -- 172.21.15.111:0/3747125409 >> 172.21.15.111:0/3747125409 conn(0x55f8a2e5e360 msgr2=0x55f8a2e6ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f8288ff9700 0 client.16983 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 -- 172.21.15.111:0/3747125409 shutdown_connections 2023-10-11T16:41:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.390+0000 7f82bae31b80 1 -- 172.21.15.111:0/3747125409 wait complete. 2023-10-11T16:41:17.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.391+0000 7f82bae31b80 20 client.16983 trim_cache size 0 max 16384 2023-10-11T16:41:17.430 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82bae31b80 1 Processor -- start 2023-10-11T16:41:17.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82bae31b80 1 -- start start 2023-10-11T16:41:17.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 0x55f8a2e799a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.431 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e6c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82bae31b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2e70900 con 0x55f8a2e69470 2023-10-11T16:41:17.432 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82bae31b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2e6c480 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82bae31b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2e69900 con 0x55f8a2e68b60 2023-10-11T16:41:17.433 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82a4b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e6c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82a4b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e6c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36500/0 (socket says 172.21.15.111:36500) 2023-10-11T16:41:17.434 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82a4b9a700 1 -- 172.21.15.111:0/942764065 learned_addr learned my addr 172.21.15.111:0/942764065 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.408+0000 7f82a539b700 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 0x55f8a2e799a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f82a4b9a700 1 -- 172.21.15.111:0/942764065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 msgr2=0x55f8a2e799a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 0x55f8a2e799a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.436 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f82a4b9a700 1 -- 172.21.15.111:0/942764065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e6d8b0 msgr2=0x55f8a2e6cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f82a4b9a700 1 -- 172.21.15.111:0/942764065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8a2e5fb00 con 0x55f8a2e69470 2023-10-11T16:41:17.437 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:17.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e6c890 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8290004790 tx=0x7f8290011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34988ee545397817 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f828bfff700 1 -- 172.21.15.111:0/942764065 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000ea00 con 0x55f8a2e69470 2023-10-11T16:41:17.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f828bfff700 1 -- 172.21.15.111:0/942764065 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8290011b30 con 0x55f8a2e69470 2023-10-11T16:41:17.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.409+0000 7f828bfff700 1 -- 172.21.15.111:0/942764065 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829000fcf0 con 0x55f8a2e69470 2023-10-11T16:41:17.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 -- 172.21.15.111:0/942764065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 msgr2=0x55f8a2e6c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e6c890 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8290004790 tx=0x7f8290011de0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 -- 172.21.15.111:0/942764065 shutdown_connections 2023-10-11T16:41:17.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e6cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 0x55f8a2e799a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 --2- 172.21.15.111:0/942764065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e69470 0x55f8a2e6c890 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 -- 172.21.15.111:0/942764065 >> 172.21.15.111:0/942764065 conn(0x55f8a2e5e360 msgr2=0x55f8a2e7ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 -- 172.21.15.111:0/942764065 shutdown_connections 2023-10-11T16:41:17.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 -- 172.21.15.111:0/942764065 wait complete. 2023-10-11T16:41:17.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.410+0000 7f82bae31b80 1 -- start start 2023-10-11T16:41:17.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82bae31b80 1 Processor -- start 2023-10-11T16:41:17.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 0x55f8a2e71c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e69470 0x55f8a2e72150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82bae31b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82bae31b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2e72bd0 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82bae31b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2e72d40 con 0x55f8a2e69470 2023-10-11T16:41:17.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82bae31b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8a2e72f00 con 0x55f8a2e68b60 2023-10-11T16:41:17.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82bae31b80 10 client.0 _ll_register_callbacks cb 0x7f829001e760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:17.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.411+0000 7f82a5b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a5b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36516/0 (socket says 172.21.15.111:36516) 2023-10-11T16:41:17.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2813628010 learned_addr learned my addr 172.21.15.111:0/2813628010 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:17.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a539b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 0x55f8a2e71c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a4b9a700 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e69470 0x55f8a2e72150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2813628010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 msgr2=0x55f8a2e71c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 0x55f8a2e71c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2813628010 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e69470 msgr2=0x55f8a2e72150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:17.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e69470 0x55f8a2e72150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a5b9c700 1 -- 172.21.15.111:0/2813628010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8a2c3b630 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82a5b9c700 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72690 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8298003f90 tx=0x7f82980092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbcdeeccaad2c9e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:17.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f8288ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:17.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f8288ff9700 1 -- 172.21.15.111:0/2813628010 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298084040 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f8288ff9700 1 -- 172.21.15.111:0/2813628010 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f829809d960 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82bae31b80 1 -- 172.21.15.111:0/2813628010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8a2d88660 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f8288ff9700 1 -- 172.21.15.111:0/2813628010 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8298099930 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82bae31b80 10 client.16989 fetch_fsmap learned FSMap version 34 2023-10-11T16:41:17.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.412+0000 7f82bae31b80 1 -- 172.21.15.111:0/2813628010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55f8a2d80190 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f8288ff9700 1 -- 172.21.15.111:0/2813628010 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f8a2d88660 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f8288ff9700 1 -- 172.21.15.111:0/2813628010 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 34) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f829809dbc0 con 0x55f8a2e6d8b0 2023-10-11T16:41:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f82bae31b80 10 client.16989 fetch_fsmap finished waiting for FSMap version 34 2023-10-11T16:41:17.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f82bae31b80 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f82bae31b80 20 client.16989 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8a2ed7310 0x55f8a2ed96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:17.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f82bae31b80 4 client.16989 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:41:17.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f82bae31b80 1 -- 172.21.15.111:0/2813628010 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f8a2c08f70 con 0x55f8a2ed7310 2023-10-11T16:41:17.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.413+0000 7f82a539b700 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8a2ed7310 0x55f8a2ed96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:17.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f8288ff9700 10 client.16989 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:17.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82a539b700 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8a2ed7310 0x55f8a2ed96f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ccfa1cb53d0a900 server_cookie=6b70b2c73a1138f9 in_seq=0 out_seq=0 2023-10-11T16:41:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f8288ff9700 1 -- 172.21.15.111:0/2813628010 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f8a2c08f70 con 0x55f8a2ed7310 2023-10-11T16:41:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f8288ff9700 10 client.16989 handle_command_reply: tid=0 2023-10-11T16:41:17.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 client.16989 shutdown 2023-10-11T16:41:17.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 -- 172.21.15.111:0/2813628010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 msgr2=0x55f8a2e72690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72690 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8298003f90 tx=0x7f82980092e0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 -- 172.21.15.111:0/2813628010 shutdown_connections 2023-10-11T16:41:17.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55f8a2ed7310 0x55f8a2ed96f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f8a2e6d8b0 0x55f8a2e72690 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.462 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f8288ff9700 0 client.16989 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:17.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f8a2e68b60 0x55f8a2e71c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 --2- 172.21.15.111:0/2813628010 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f8a2e69470 0x55f8a2e72150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 -- 172.21.15.111:0/2813628010 >> 172.21.15.111:0/2813628010 conn(0x55f8a2e5e360 msgr2=0x55f8a2e67960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 -- 172.21.15.111:0/2813628010 shutdown_connections 2023-10-11T16:41:17.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 1 -- 172.21.15.111:0/2813628010 wait complete. 2023-10-11T16:41:17.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.414+0000 7f82bae31b80 20 client.16989 trim_cache size 0 max 16384 2023-10-11T16:41:17.465 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:41:17.465 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:41:17.466 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:41:17.466 INFO:teuthology.orchestra.run.smithi111.stdout: "0f997ce5-5f2f-48b0-b949-4c07d21ace62": { 2023-10-11T16:41:17.466 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:41:17.467 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "0f997ce5-5f2f-48b0-b949-4c07d21ace62", 2023-10-11T16:41:17.467 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:41:17.467 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:41:17.467 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:41:17.468 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:41:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 -- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f828c0a12c0 msgr2=0x7f828c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f828c0a12c0 0x7f828c0a3770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f82a00745e0 tx=0x7f8290011440 comp rx=0 tx=0).stop 2023-10-11T16:41:17.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 -- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0105620 msgr2=0x7f82a01a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:17.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0105620 0x7f82a01a46f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f829800ef30 tx=0x7f829800c3a0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 -- 172.21.15.111:0/2771842369 shutdown_connections 2023-10-11T16:41:17.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f828c0a12c0 0x7f828c0a3770 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82a0105620 0x7f82a01a46f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82a0100cc0 0x7f82a0072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 --2- 172.21.15.111:0/2771842369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82a0101660 0x7f82a0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 -- 172.21.15.111:0/2771842369 >> 172.21.15.111:0/2771842369 conn(0x7f82a0077560 msgr2=0x7f82a00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 -- 172.21.15.111:0/2771842369 shutdown_connections 2023-10-11T16:41:17.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:17.415+0000 7f82a75ff700 1 -- 172.21.15.111:0/2771842369 wait complete. 2023-10-11T16:41:17.473 INFO:tasks.cephfs.filesystem:scrub status for tag:0f997ce5-5f2f-48b0-b949-4c07d21ace62 - {'path': '/', 'tag': '0f997ce5-5f2f-48b0-b949-4c07d21ace62', 'options': 'recursive,force'} 2023-10-11T16:41:17.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:17 smithi111 ceph-mon[98527]: pgmap v267: 129 pgs: 21 active+clean+snaptrim, 42 active+clean, 66 active+clean+snaptrim_wait; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:41:17.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:17 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:17.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:17 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:41:17.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:17 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:17.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:17 smithi150 ceph-mon[132217]: pgmap v267: 129 pgs: 21 active+clean+snaptrim, 42 active+clean, 66 active+clean+snaptrim_wait; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:41:17.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:17 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:17.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:17 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:41:17.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:17 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:17.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:17 smithi130 ceph-mon[131859]: pgmap v267: 129 pgs: 21 active+clean+snaptrim, 42 active+clean, 66 active+clean+snaptrim_wait; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:41:17.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:17 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:17.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:17 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:41:17.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:17 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:18.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:18 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:41:18.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:18 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:18.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:18 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:41:18.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:18 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:18.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:18 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:41:18.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:18 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:18.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:18 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:41:18.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:18 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:18.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:18 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:41:18.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:18 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:18.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:18 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:41:18.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:18 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:19.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:19 smithi150 ceph-mon[132217]: pgmap v268: 129 pgs: 21 active+clean+snaptrim, 42 active+clean, 66 active+clean+snaptrim_wait; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:41:19.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:19 smithi130 ceph-mon[131859]: pgmap v268: 129 pgs: 21 active+clean+snaptrim, 42 active+clean, 66 active+clean+snaptrim_wait; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:41:19.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:19 smithi111 ceph-mon[98527]: pgmap v268: 129 pgs: 21 active+clean+snaptrim, 42 active+clean, 66 active+clean+snaptrim_wait; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:41:21.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:21 smithi150 ceph-mon[132217]: pgmap v269: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:41:21.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:21 smithi130 ceph-mon[131859]: pgmap v269: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:41:21.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:21 smithi111 ceph-mon[98527]: pgmap v269: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 204 B/s wr, 7 op/s 2023-10-11T16:41:23.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:23 smithi150 ceph-mon[132217]: pgmap v270: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 194 B/s wr, 7 op/s 2023-10-11T16:41:23.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:23 smithi130 ceph-mon[131859]: pgmap v270: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 194 B/s wr, 7 op/s 2023-10-11T16:41:23.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:23 smithi111 ceph-mon[98527]: pgmap v270: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 194 B/s wr, 7 op/s 2023-10-11T16:41:25.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:25 smithi150 ceph-mon[132217]: pgmap v271: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-10-11T16:41:25.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:25 smithi150 ceph-mon[132217]: osdmap e98: 12 total, 12 up, 12 in 2023-10-11T16:41:25.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:25 smithi130 ceph-mon[131859]: pgmap v271: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-10-11T16:41:25.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:25 smithi130 ceph-mon[131859]: osdmap e98: 12 total, 12 up, 12 in 2023-10-11T16:41:25.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:25 smithi111 ceph-mon[98527]: pgmap v271: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-10-11T16:41:25.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:25 smithi111 ceph-mon[98527]: osdmap e98: 12 total, 12 up, 12 in 2023-10-11T16:41:26.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:26 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:41:26.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:26 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:41:26.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:26 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:41:26.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:26 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:41:26.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:26 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:41:26.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:26 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:41:27.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:27 smithi150 ceph-mon[132217]: pgmap v273: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 7 op/s 2023-10-11T16:41:27.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:27 smithi130 ceph-mon[131859]: pgmap v273: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 7 op/s 2023-10-11T16:41:27.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:27 smithi111 ceph-mon[98527]: pgmap v273: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 7 op/s 2023-10-11T16:41:29.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:29 smithi150 ceph-mon[132217]: pgmap v274: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 7 op/s 2023-10-11T16:41:29.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:29 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:29.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:29 smithi130 ceph-mon[131859]: pgmap v274: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 7 op/s 2023-10-11T16:41:29.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:29 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:29.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:29 smithi111 ceph-mon[98527]: pgmap v274: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 7 op/s 2023-10-11T16:41:29.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:29 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:31.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:31 smithi150 ceph-mon[132217]: pgmap v275: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:31.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:31 smithi130 ceph-mon[131859]: pgmap v275: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:31.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:31 smithi111 ceph-mon[98527]: pgmap v275: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:33.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:33 smithi150 ceph-mon[132217]: pgmap v276: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:33.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:33 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:33.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:33 smithi130 ceph-mon[131859]: pgmap v276: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:33.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:33 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:33.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:33 smithi111 ceph-mon[98527]: pgmap v276: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:33.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:33 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:35.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:35 smithi150 ceph-mon[132217]: pgmap v277: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:35.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:35 smithi130 ceph-mon[131859]: pgmap v277: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:35.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:35 smithi111 ceph-mon[98527]: pgmap v277: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:41:37.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:37 smithi130 ceph-mon[131859]: pgmap v278: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-11T16:41:37.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:37 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:37.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:37 smithi111 ceph-mon[98527]: pgmap v278: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-11T16:41:37.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:37 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:37 smithi150 ceph-mon[132217]: pgmap v278: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-11T16:41:37.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:37 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:38.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:38 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:38.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:38 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:38.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:38 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:39.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:39 smithi130 ceph-mon[131859]: pgmap v279: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:39.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:39 smithi111 ceph-mon[98527]: pgmap v279: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:39.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:39 smithi150 ceph-mon[132217]: pgmap v279: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:41.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:41 smithi130 ceph-mon[131859]: pgmap v280: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:41.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:41 smithi111 ceph-mon[98527]: pgmap v280: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:41.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:41 smithi150 ceph-mon[132217]: pgmap v280: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:43.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:43 smithi130 ceph-mon[131859]: pgmap v281: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:43.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:43 smithi111 ceph-mon[98527]: pgmap v281: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:43.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:43 smithi150 ceph-mon[132217]: pgmap v281: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:45.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:45 smithi130 ceph-mon[131859]: pgmap v282: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:41:45.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:45 smithi111 ceph-mon[98527]: pgmap v282: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:41:45.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:45 smithi150 ceph-mon[132217]: pgmap v282: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-11T16:41:47.475 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:41:47.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2619807548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 msgr2=0x7f0fe01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2619807548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 0x7f0fe01067c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00099c0 tx=0x7f0fd0009cd0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2619807548 shutdown_connections 2023-10-11T16:41:47.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2619807548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fe0101400 0x7f0fe010d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2619807548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 0x7f0fe01067c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2619807548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe0100a50 0x7f0fe0100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2619807548 >> 172.21.15.111:0/2619807548 conn(0x7f0fe0077590 msgr2=0x7f0fe0077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:47.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2619807548 shutdown_connections 2023-10-11T16:41:47.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.686+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2619807548 wait complete. 2023-10-11T16:41:47.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.687+0000 7f0fe61ff700 1 Processor -- start 2023-10-11T16:41:47.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.687+0000 7f0fe61ff700 1 -- start start 2023-10-11T16:41:47.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.687+0000 7f0fe61ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fe0100a50 0x7f0fe0072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.693 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.687+0000 7f0fe61ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe0101400 0x7f0fe0073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.687+0000 7f0fe61ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 0x7f0fe01a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.687+0000 7f0fe61ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe0073bd0 con 0x7f0fe0100a50 2023-10-11T16:41:47.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.687+0000 7f0fe61ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe01a4b70 con 0x7f0fe01063f0 2023-10-11T16:41:47.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.687+0000 7f0fe61ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe01a4cb0 con 0x7f0fe0101400 2023-10-11T16:41:47.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fe0100a50 0x7f0fe0072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 0x7f0fe01a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 0x7f0fe01a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40848/0 (socket says 172.21.15.111:40848) 2023-10-11T16:41:47.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 -- 172.21.15.111:0/2279425358 learned_addr learned my addr 172.21.15.111:0/2279425358 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:47.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe0101400 0x7f0fe0073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 -- 172.21.15.111:0/2279425358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe0101400 msgr2=0x7f0fe0073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe0101400 0x7f0fe0073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 -- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fe0100a50 msgr2=0x7f0fe0072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fe0100a50 0x7f0fe0072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 -- 172.21.15.111:0/2279425358 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00095f0 con 0x7f0fe01063f0 2023-10-11T16:41:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fe0100a50 0x7f0fe0072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:47.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.688+0000 7f0fdffff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 0x7f0fe01a4630 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800ef60 tx=0x7f0fd800ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:47.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.689+0000 7f0fdcff9700 1 -- 172.21.15.111:0/2279425358 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8018070 con 0x7f0fe01063f0 2023-10-11T16:41:47.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.689+0000 7f0fdcff9700 1 -- 172.21.15.111:0/2279425358 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fd800f040 con 0x7f0fe01063f0 2023-10-11T16:41:47.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.689+0000 7f0fdcff9700 1 -- 172.21.15.111:0/2279425358 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd800ccc0 con 0x7f0fe01063f0 2023-10-11T16:41:47.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.689+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2279425358 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe01103d0 con 0x7f0fe01063f0 2023-10-11T16:41:47.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.689+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2279425358 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe0110240 con 0x7f0fe01063f0 2023-10-11T16:41:47.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.692+0000 7f0fdcff9700 1 -- 172.21.15.111:0/2279425358 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f0fd8003a00 con 0x7f0fe01063f0 2023-10-11T16:41:47.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.693+0000 7f0ff9bb3b80 1 Processor -- start 2023-10-11T16:41:47.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.693+0000 7f0ff9bb3b80 1 -- start start 2023-10-11T16:41:47.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.693+0000 7f0fdcff9700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fc80a12c0 0x7f0fc80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdcff9700 1 -- 172.21.15.111:0/2279425358 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f0fd80ccb70 con 0x7f0fe01063f0 2023-10-11T16:41:47.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52db370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c51dc060 con 0x55d3c52db8b0 2023-10-11T16:41:47.707 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c522e8a0 con 0x55d3c52d6b60 2023-10-11T16:41:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c5227d30 con 0x55d3c52d7470 2023-10-11T16:41:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fc80a12c0 0x7f0fc80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.708 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdf7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdf7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40860/0 (socket says 172.21.15.111:40860) 2023-10-11T16:41:47.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/141889427 learned_addr learned my addr 172.21.15.111:0/141889427 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:47.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdffff700 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.709 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/141889427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 msgr2=0x55d3c52db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:47.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52db370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/141889427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 msgr2=0x55d3c52dbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:47.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/141889427 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3c50a9630 con 0x55d3c52d6b60 2023-10-11T16:41:47.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.694+0000 7f0fdffff700 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:41:47.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.695+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fc80a12c0 0x7f0fc80a3770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0fe00745e0 tx=0x7f0fd4011440 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:47.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.695+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d6f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00058e0 tx=0x7f0fd0005ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=374306edc5ec86c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.695+0000 7f0fc5ffb700 1 -- 172.21.15.111:0/141889427 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000be40 con 0x55d3c52d6b60 2023-10-11T16:41:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.695+0000 7f0fc5ffb700 1 -- 172.21.15.111:0/141889427 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fd001e040 con 0x55d3c52d6b60 2023-10-11T16:41:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.695+0000 7f0fc5ffb700 1 -- 172.21.15.111:0/141889427 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0003e80 con 0x55d3c52d6b60 2023-10-11T16:41:47.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/141889427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 msgr2=0x55d3c52d6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d6f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00058e0 tx=0x7f0fd0005ee0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/141889427 shutdown_connections 2023-10-11T16:41:47.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d6f30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/141889427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52db370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/141889427 >> 172.21.15.111:0/141889427 conn(0x55d3c52cc360 msgr2=0x55d3c52d1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:47.714 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/141889427 shutdown_connections 2023-10-11T16:41:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/141889427 wait complete. 2023-10-11T16:41:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.696+0000 7f0ff9bb3b80 1 -- start start 2023-10-11T16:41:47.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.697+0000 7f0ff9bb3b80 1 Processor -- start 2023-10-11T16:41:47.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.697+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.697+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.697+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52e0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.697+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c51c98e0 con 0x55d3c52db8b0 2023-10-11T16:41:47.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.697+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c51c9580 con 0x55d3c52d6b60 2023-10-11T16:41:47.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.697+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c50ab920 con 0x55d3c52d7470 2023-10-11T16:41:47.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.697+0000 7f0ff9bb3b80 10 client.0 _ll_register_callbacks cb 0x55d3c51e7780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:47.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52e0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56196/0 (socket says 172.21.15.111:56196) 2023-10-11T16:41:47.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdeffd700 1 -- 172.21.15.111:0/1392321949 learned_addr learned my addr 172.21.15.111:0/1392321949 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:47.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdeffd700 1 -- 172.21.15.111:0/1392321949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 msgr2=0x55d3c52e0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:47.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdeffd700 1 -- 172.21.15.111:0/1392321949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 msgr2=0x55d3c52e0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:47.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52e0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.698+0000 7f0fdeffd700 1 -- 172.21.15.111:0/1392321949 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe010da50 con 0x55d3c52d7470 2023-10-11T16:41:47.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.699+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e08c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4011fd0 tx=0x7f0fd40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c91327e4ed0157e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:47.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.699+0000 7f0fbeffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:41:47.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.699+0000 7f0fbeffd700 1 -- 172.21.15.111:0/1392321949 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4004830 con 0x55d3c52d7470 2023-10-11T16:41:47.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.699+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/1392321949 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3c52c9b20 con 0x55d3c52d7470 2023-10-11T16:41:47.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.699+0000 7f0fbeffd700 1 -- 172.21.15.111:0/1392321949 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f0fd4004990 con 0x55d3c52d7470 2023-10-11T16:41:47.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.699+0000 7f0fbeffd700 1 -- 172.21.15.111:0/1392321949 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fd4004b50 con 0x55d3c52d7470 2023-10-11T16:41:47.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.699+0000 7f0ff9bb3b80 10 client.27148 fetch_fsmap learned FSMap version 38 2023-10-11T16:41:47.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.699+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/1392321949 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55d3c51ee190 con 0x55d3c52d7470 2023-10-11T16:41:47.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0fbeffd700 1 -- 172.21.15.111:0/1392321949 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d3c52c9b20 con 0x55d3c52d7470 2023-10-11T16:41:47.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0fbeffd700 1 -- 172.21.15.111:0/1392321949 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 38) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f0fd401e850 con 0x55d3c52d7470 2023-10-11T16:41:47.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0ff9bb3b80 10 client.27148 fetch_fsmap finished waiting for FSMap version 38 2023-10-11T16:41:47.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0ff9bb3b80 10 client.27148 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:47.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0ff9bb3b80 20 client.27148 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:47.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d3c52f9140 0x55d3c52fb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0ff9bb3b80 4 client.27148 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:47.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/1392321949 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3c51ea0a0 con 0x55d3c52f9140 2023-10-11T16:41:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.700+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d3c52f9140 0x55d3c52fb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.701+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d3c52f9140 0x55d3c52fb550 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a72e70ea97375bf6 server_cookie=39e02118ebba1ae4 in_seq=0 out_seq=0 2023-10-11T16:41:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.701+0000 7f0fbeffd700 10 client.27148 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0fbeffd700 1 -- 172.21.15.111:0/1392321949 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d3c51ea0a0 con 0x55d3c52f9140 2023-10-11T16:41:47.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0fbeffd700 10 client.27148 handle_command_reply: tid=0 2023-10-11T16:41:47.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 client.27148 shutdown 2023-10-11T16:41:47.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/1392321949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 msgr2=0x55d3c52e08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e08c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4011fd0 tx=0x7f0fd40047c0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/1392321949 shutdown_connections 2023-10-11T16:41:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d3c52f9140 0x55d3c52fb550 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52e0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0fbeffd700 0 client.27148 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/1392321949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e08c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/1392321949 >> 172.21.15.111:0/1392321949 conn(0x55d3c52cc360 msgr2=0x55d3c52dcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:47.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/1392321949 shutdown_connections 2023-10-11T16:41:47.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/1392321949 wait complete. 2023-10-11T16:41:47.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.702+0000 7f0ff9bb3b80 20 client.27148 trim_cache size 0 max 16384 2023-10-11T16:41:47.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.720+0000 7f0ff9bb3b80 1 Processor -- start 2023-10-11T16:41:47.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.720+0000 7f0ff9bb3b80 1 -- start start 2023-10-11T16:41:47.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c52c9a60 con 0x55d3c52db8b0 2023-10-11T16:41:47.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c52d7bf0 con 0x55d3c52d6b60 2023-10-11T16:41:47.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c52da500 con 0x55d3c52d7470 2023-10-11T16:41:47.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdf7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56208/0 (socket says 172.21.15.111:56208) 2023-10-11T16:41:47.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdeffd700 1 -- 172.21.15.111:0/633265252 learned_addr learned my addr 172.21.15.111:0/633265252 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:47.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/633265252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 msgr2=0x55d3c52e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:47.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/633265252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/633265252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 msgr2=0x55d3c52dfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:47.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/633265252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/633265252 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3c52cdb00 con 0x55d3c52d6b60 2023-10-11T16:41:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.721+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/633265252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.722+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/633265252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d1be0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0005cd0 tx=0x7f0fd0005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8a56fefa81d4808 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.722+0000 7f0fc67fc700 1 -- 172.21.15.111:0/633265252 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0036070 con 0x55d3c52d6b60 2023-10-11T16:41:47.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.722+0000 7f0fc67fc700 1 -- 172.21.15.111:0/633265252 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fd00321f0 con 0x55d3c52d6b60 2023-10-11T16:41:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.722+0000 7f0fc67fc700 1 -- 172.21.15.111:0/633265252 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0004e60 con 0x55d3c52d6b60 2023-10-11T16:41:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/633265252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 msgr2=0x55d3c52d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/633265252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d1be0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0005cd0 tx=0x7f0fd0005a20 comp rx=0 tx=0).stop 2023-10-11T16:41:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/633265252 shutdown_connections 2023-10-11T16:41:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/633265252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52db8b0 0x55d3c52dfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/633265252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52d1be0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/633265252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52d7470 0x55d3c52e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/633265252 >> 172.21.15.111:0/633265252 conn(0x55d3c52cc360 msgr2=0x55d3c52ecbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/633265252 shutdown_connections 2023-10-11T16:41:47.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/633265252 wait complete. 2023-10-11T16:41:47.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 -- start start 2023-10-11T16:41:47.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.723+0000 7f0ff9bb3b80 1 Processor -- start 2023-10-11T16:41:47.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52d7470 0x55d3c52e4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0ff9bb3b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52db8b0 0x55d3c52e5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c52e5820 con 0x55d3c52d7470 2023-10-11T16:41:47.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c52e5990 con 0x55d3c52d6b60 2023-10-11T16:41:47.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0ff9bb3b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3c534bfc0 con 0x55d3c52db8b0 2023-10-11T16:41:47.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0ff9bb3b80 10 client.0 _ll_register_callbacks cb 0x7f0fd400f3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:47.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0fdf7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0fdeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52d7470 0x55d3c52e4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0fdf7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40902/0 (socket says 172.21.15.111:40902) 2023-10-11T16:41:47.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/415162239 learned_addr learned my addr 172.21.15.111:0/415162239 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:47.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0fdffff700 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52db8b0 0x55d3c52e5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/415162239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52db8b0 msgr2=0x55d3c52e5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:47.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52db8b0 0x55d3c52e5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.724+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/415162239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52d7470 msgr2=0x55d3c52e4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:47.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52d7470 0x55d3c52e4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0fdf7fe700 1 -- 172.21.15.111:0/415162239 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3c51ee190 con 0x55d3c52d6b60 2023-10-11T16:41:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0fdf7fe700 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e7920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000b600 tx=0x7f0fd0032360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec3f9b2ed6cfad0e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0fbeffd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:41:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0fbeffd700 1 -- 172.21.15.111:0/415162239 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55d3c52d6b60 2023-10-11T16:41:47.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/415162239 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fd001e040 con 0x55d3c52d6b60 2023-10-11T16:41:47.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0fbeffd700 1 -- 172.21.15.111:0/415162239 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f0fd0032030 con 0x55d3c52d6b60 2023-10-11T16:41:47.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0fbeffd700 1 -- 172.21.15.111:0/415162239 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fd0032610 con 0x55d3c52d6b60 2023-10-11T16:41:47.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0ff9bb3b80 10 client.27143 fetch_fsmap learned FSMap version 38 2023-10-11T16:41:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.725+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/415162239 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55d3c50a9630 con 0x55d3c52d6b60 2023-10-11T16:41:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0fbeffd700 1 -- 172.21.15.111:0/415162239 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fd001e040 con 0x55d3c52d6b60 2023-10-11T16:41:47.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0fbeffd700 1 -- 172.21.15.111:0/415162239 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 38) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f0fd000f620 con 0x55d3c52d6b60 2023-10-11T16:41:47.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0ff9bb3b80 10 client.27143 fetch_fsmap finished waiting for FSMap version 38 2023-10-11T16:41:47.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0ff9bb3b80 10 client.27143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:47.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0ff9bb3b80 20 client.27143 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:47.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d3c52f9140 0x55d3c534f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0ff9bb3b80 4 client.27143 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:41:47.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/415162239 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d3c51f6660 con 0x55d3c52f9140 2023-10-11T16:41:47.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.726+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d3c52f9140 0x55d3c534f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0fbeffd700 10 client.27143 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:47.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0fdeffd700 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d3c52f9140 0x55d3c534f0d0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41e0f62fa902c267 server_cookie=7a8aac70e255719b in_seq=0 out_seq=0 2023-10-11T16:41:47.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0fbeffd700 1 -- 172.21.15.111:0/415162239 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d3c51f6660 con 0x55d3c52f9140 2023-10-11T16:41:47.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0fbeffd700 10 client.27143 handle_command_reply: tid=0 2023-10-11T16:41:47.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 client.27143 shutdown 2023-10-11T16:41:47.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/415162239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 msgr2=0x55d3c52e7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e7920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0fd000b600 tx=0x7f0fd0032360 comp rx=0 tx=0).stop 2023-10-11T16:41:47.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/415162239 shutdown_connections 2023-10-11T16:41:47.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55d3c52f9140 0x55d3c534f0d0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3c52db8b0 0x55d3c52e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0fbeffd700 0 client.27143 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:47.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d3c52d6b60 0x55d3c52e7920 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 --2- 172.21.15.111:0/415162239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55d3c52d7470 0x55d3c52e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/415162239 >> 172.21.15.111:0/415162239 conn(0x55d3c52cc360 msgr2=0x55d3c52e0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:47.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/415162239 shutdown_connections 2023-10-11T16:41:47.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 1 -- 172.21.15.111:0/415162239 wait complete. 2023-10-11T16:41:47.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.727+0000 7f0ff9bb3b80 20 client.27143 trim_cache size 0 max 16384 2023-10-11T16:41:47.759 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:41:47.759 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:41:47.759 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:41:47.760 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:41:47.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fc80a12c0 msgr2=0x7f0fc80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fc80a12c0 0x7f0fc80a3770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0fe00745e0 tx=0x7f0fd4011440 comp rx=0 tx=0).stop 2023-10-11T16:41:47.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2279425358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 msgr2=0x7f0fe01a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 0x7f0fe01a4630 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800ef60 tx=0x7f0fd800ef90 comp rx=0 tx=0).stop 2023-10-11T16:41:47.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2279425358 shutdown_connections 2023-10-11T16:41:47.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0fc80a12c0 0x7f0fc80a3770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0fe01063f0 0x7f0fe01a4630 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fe0100a50 0x7f0fe0072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 --2- 172.21.15.111:0/2279425358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe0101400 0x7f0fe0073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2279425358 >> 172.21.15.111:0/2279425358 conn(0x7f0fe0077590 msgr2=0x7f0fe00ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:47.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2279425358 shutdown_connections 2023-10-11T16:41:47.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.728+0000 7f0fe61ff700 1 -- 172.21.15.111:0/2279425358 wait complete. 2023-10-11T16:41:47.764 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:47 smithi111 ceph-mon[98527]: pgmap v283: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:41:47.765 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:41:47.766 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:41:47.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:47 smithi130 ceph-mon[131859]: pgmap v283: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:41:47.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:47 smithi150 ceph-mon[132217]: pgmap v283: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:41:47.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 -- 172.21.15.111:0/788528980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 msgr2=0x7f6790104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 --2- 172.21.15.111:0/788528980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 0x7f6790104e70 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f67800099c0 tx=0x7f6780009cd0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 -- 172.21.15.111:0/788528980 shutdown_connections 2023-10-11T16:41:47.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 --2- 172.21.15.111:0/788528980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6790105550 0x7f679010d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 --2- 172.21.15.111:0/788528980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6790100510 0x7f67901008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 --2- 172.21.15.111:0/788528980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 0x7f6790104e70 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 -- 172.21.15.111:0/788528980 >> 172.21.15.111:0/788528980 conn(0x7f6790077050 msgr2=0x7f6790077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:47.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 -- 172.21.15.111:0/788528980 shutdown_connections 2023-10-11T16:41:47.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.962+0000 7f67981ff700 1 -- 172.21.15.111:0/788528980 wait complete. 2023-10-11T16:41:47.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.963+0000 7f67981ff700 1 Processor -- start 2023-10-11T16:41:47.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.963+0000 7f67981ff700 1 -- start start 2023-10-11T16:41:47.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.963+0000 7f67981ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6790100510 0x7f6790072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f67981ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 0x7f6790073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f67981ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6790105550 0x7f67901a4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f67981ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6790073b60 con 0x7f6790100eb0 2023-10-11T16:41:47.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f67981ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67901a4a80 con 0x7f6790100510 2023-10-11T16:41:47.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f67981ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67901a4bc0 con 0x7f6790105550 2023-10-11T16:41:47.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 0x7f6790073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 0x7f6790073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39870/0 (socket says 172.21.15.111:39870) 2023-10-11T16:41:47.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 -- 172.21.15.111:0/1034679683 learned_addr learned my addr 172.21.15.111:0/1034679683 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:47.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679679c700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6790105550 0x7f67901a4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f6795f9b700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6790100510 0x7f6790072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 -- 172.21.15.111:0/1034679683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6790105550 msgr2=0x7f67901a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6790105550 0x7f67901a4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 -- 172.21.15.111:0/1034679683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6790100510 msgr2=0x7f6790072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:47.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6790100510 0x7f6790072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:47.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 -- 172.21.15.111:0/1034679683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67800095f0 con 0x7f6790100eb0 2023-10-11T16:41:47.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.964+0000 7f679579a700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 0x7f6790073330 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6780009990 tx=0x7f678000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:47.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.965+0000 7f6786ffd700 1 -- 172.21.15.111:0/1034679683 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678001d070 con 0x7f6790100eb0 2023-10-11T16:41:47.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.965+0000 7f6786ffd700 1 -- 172.21.15.111:0/1034679683 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6780005020 con 0x7f6790100eb0 2023-10-11T16:41:47.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.965+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6790110320 con 0x7f6790100eb0 2023-10-11T16:41:47.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.965+0000 7f6786ffd700 1 -- 172.21.15.111:0/1034679683 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780004a70 con 0x7f6790100eb0 2023-10-11T16:41:47.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.965+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6790110190 con 0x7f6790100eb0 2023-10-11T16:41:47.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.966+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67901104b0 con 0x7f6790100eb0 2023-10-11T16:41:47.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.968+0000 7f6786ffd700 1 -- 172.21.15.111:0/1034679683 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f6780005400 con 0x7f6790100eb0 2023-10-11T16:41:47.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.970+0000 7f6786ffd700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f677c0a1340 0x7f677c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:47.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.970+0000 7f6786ffd700 1 -- 172.21.15.111:0/1034679683 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f67800cdae0 con 0x7f6790100eb0 2023-10-11T16:41:47.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.970+0000 7f6795f9b700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f677c0a1340 0x7f677c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:47.978 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.971+0000 7f6795f9b700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f677c0a1340 0x7f677c0a37f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f678c009c60 tx=0x7f678c009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:47.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:47.975+0000 7f6786ffd700 1 -- 172.21.15.111:0/1034679683 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6780096b10 con 0x7f6790100eb0 2023-10-11T16:41:48.217 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.216+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6790067a60 con 0x7f6790100eb0 2023-10-11T16:41:48.218 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.216+0000 7f6786ffd700 1 -- 172.21.15.111:0/1034679683 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+10821 (secure 0 0 0) 0x7f6780005190 con 0x7f6790100eb0 2023-10-11T16:41:48.218 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:41:48.219 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:41:37.452624+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:41:48.220 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f677c0a1340 msgr2=0x7f677c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f677c0a1340 0x7f677c0a37f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f678c009c60 tx=0x7f678c009260 comp rx=0 tx=0).stop 2023-10-11T16:41:48.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 msgr2=0x7f6790073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.221 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 0x7f6790073330 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6780009990 tx=0x7f678000f690 comp rx=0 tx=0).stop 2023-10-11T16:41:48.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 shutdown_connections 2023-10-11T16:41:48.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f677c0a1340 0x7f677c0a37f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.222 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6790105550 0x7f67901a4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6790100510 0x7f6790072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 --2- 172.21.15.111:0/1034679683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6790100eb0 0x7f6790073330 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 >> 172.21.15.111:0/1034679683 conn(0x7f6790077050 msgr2=0x7f67900fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.223 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 shutdown_connections 2023-10-11T16:41:48.224 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.219+0000 7f67981ff700 1 -- 172.21.15.111:0/1034679683 wait complete. 2023-10-11T16:41:48.224 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 38 2023-10-11T16:41:48.240 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:41:48.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 -- 172.21.15.111:0/1653633203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4101660 msgr2=0x7f78c4104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1653633203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4101660 0x7f78c4104ef0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0099c0 tx=0x7f78ac009cd0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 -- 172.21.15.111:0/1653633203 shutdown_connections 2023-10-11T16:41:48.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1653633203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4105620 0x7f78c410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1653633203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c4100cc0 0x7f78c4101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1653633203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4101660 0x7f78c4104ef0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 -- 172.21.15.111:0/1653633203 >> 172.21.15.111:0/1653633203 conn(0x7f78c4077560 msgr2=0x7f78c4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 -- 172.21.15.111:0/1653633203 shutdown_connections 2023-10-11T16:41:48.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.441+0000 7f78c95ff700 1 -- 172.21.15.111:0/1653633203 wait complete. 2023-10-11T16:41:48.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.442+0000 7f78c95ff700 1 Processor -- start 2023-10-11T16:41:48.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.442+0000 7f78c95ff700 1 -- start start 2023-10-11T16:41:48.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c95ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4100cc0 0x7f78c41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c95ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4101660 0x7f78c4198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c95ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c4105620 0x7f78c419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c95ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c419c5a0 con 0x7f78c4100cc0 2023-10-11T16:41:48.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c95ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c419c710 con 0x7f78c4101660 2023-10-11T16:41:48.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c95ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c419c8d0 con 0x7f78c4105620 2023-10-11T16:41:48.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4100cc0 0x7f78c41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4101660 0x7f78c4198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4100cc0 0x7f78c41986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39892/0 (socket says 172.21.15.111:39892) 2023-10-11T16:41:48.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c2ffd700 1 -- 172.21.15.111:0/1741909791 learned_addr learned my addr 172.21.15.111:0/1741909791 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c37fe700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c4105620 0x7f78c419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c2ffd700 1 -- 172.21.15.111:0/1741909791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c4105620 msgr2=0x7f78c419bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c4105620 0x7f78c419bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c2ffd700 1 -- 172.21.15.111:0/1741909791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4101660 msgr2=0x7f78c4198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4101660 0x7f78c4198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78c2ffd700 1 -- 172.21.15.111:0/1741909791 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4009520 con 0x7f78c4100cc0 2023-10-11T16:41:48.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.443+0000 7f78bbfff700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4101660 0x7f78c4198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:48.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.444+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4100cc0 0x7f78c41986b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f78b400eb30 tx=0x7f78b400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.444+0000 7f78c0ff9700 1 -- 172.21.15.111:0/1741909791 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4009880 con 0x7f78c4100cc0 2023-10-11T16:41:48.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.444+0000 7f78c0ff9700 1 -- 172.21.15.111:0/1741909791 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78b40044e0 con 0x7f78c4100cc0 2023-10-11T16:41:48.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.444+0000 7f78c95ff700 1 -- 172.21.15.111:0/1741909791 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac0095f0 con 0x7f78c4100cc0 2023-10-11T16:41:48.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.444+0000 7f78c0ff9700 1 -- 172.21.15.111:0/1741909791 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4003d80 con 0x7f78c4100cc0 2023-10-11T16:41:48.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.444+0000 7f78c95ff700 1 -- 172.21.15.111:0/1741909791 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c4110370 con 0x7f78c4100cc0 2023-10-11T16:41:48.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.447+0000 7f78c0ff9700 1 -- 172.21.15.111:0/1741909791 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f78b4003f20 con 0x7f78c4100cc0 2023-10-11T16:41:48.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.448+0000 7f78dceecb80 1 Processor -- start 2023-10-11T16:41:48.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.448+0000 7f78c0ff9700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f78b00a1340 0x7f78b00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.448+0000 7f78dceecb80 1 -- start start 2023-10-11T16:41:48.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.448+0000 7f78c0ff9700 1 -- 172.21.15.111:0/1741909791 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f78b4014070 con 0x7f78c4100cc0 2023-10-11T16:41:48.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7780b60 0x5605f7780f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7781470 0x5605f7785370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c37fe700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f78b00a1340 0x7f78b00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f7785d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78dceecb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f7686060 con 0x5605f7780b60 2023-10-11T16:41:48.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78dceecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f76d88a0 con 0x5605f77858b0 2023-10-11T16:41:48.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78dceecb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f76d1d30 con 0x5605f7781470 2023-10-11T16:41:48.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7780b60 0x5605f7780f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7780b60 0x5605f7780f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39894/0 (socket says 172.21.15.111:39894) 2023-10-11T16:41:48.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c2ffd700 1 -- 172.21.15.111:0/2961907440 learned_addr learned my addr 172.21.15.111:0/2961907440 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c37fe700 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f7785d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78bbfff700 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7781470 0x5605f7785370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c2ffd700 1 -- 172.21.15.111:0/2961907440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7781470 msgr2=0x5605f7785370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7781470 0x5605f7785370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c2ffd700 1 -- 172.21.15.111:0/2961907440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 msgr2=0x5605f7785d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f7785d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c2ffd700 1 -- 172.21.15.111:0/2961907440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605f7553630 con 0x5605f7780b60 2023-10-11T16:41:48.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78bbfff700 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7781470 0x5605f7785370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:41:48.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.449+0000 7f78c37fe700 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f7785d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:48.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.450+0000 7f78c37fe700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f78b00a1340 0x7f78b00a37f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009ec0 tx=0x7f78bc00fc40 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.450+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7780b60 0x5605f7780f30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f78b40a21e0 tx=0x7f78b4099c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321bb37528d77584 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.450+0000 7f78b97fa700 1 -- 172.21.15.111:0/2961907440 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40cd050 con 0x5605f7780b60 2023-10-11T16:41:48.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.450+0000 7f78b97fa700 1 -- 172.21.15.111:0/2961907440 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78b400cbd0 con 0x5605f7780b60 2023-10-11T16:41:48.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.450+0000 7f78b97fa700 1 -- 172.21.15.111:0/2961907440 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b400cdd0 con 0x5605f7780b60 2023-10-11T16:41:48.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.450+0000 7f78dceecb80 1 -- 172.21.15.111:0/2961907440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7780b60 msgr2=0x5605f7780f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.450+0000 7f78dceecb80 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7780b60 0x5605f7780f30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f78b40a21e0 tx=0x7f78b4099c20 comp rx=0 tx=0).stop 2023-10-11T16:41:48.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 -- 172.21.15.111:0/2961907440 shutdown_connections 2023-10-11T16:41:48.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f7785d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7780b60 0x5605f7780f30 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 --2- 172.21.15.111:0/2961907440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7781470 0x5605f7785370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 -- 172.21.15.111:0/2961907440 >> 172.21.15.111:0/2961907440 conn(0x5605f7776360 msgr2=0x5605f777bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 -- 172.21.15.111:0/2961907440 shutdown_connections 2023-10-11T16:41:48.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 -- 172.21.15.111:0/2961907440 wait complete. 2023-10-11T16:41:48.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 -- start start 2023-10-11T16:41:48.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.451+0000 7f78dceecb80 1 Processor -- start 2023-10-11T16:41:48.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 0x5605f778d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f778a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78dceecb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f76738e0 con 0x5605f7781470 2023-10-11T16:41:48.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78dceecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f7673580 con 0x5605f77858b0 2023-10-11T16:41:48.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78dceecb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f7555920 con 0x5605f7780b60 2023-10-11T16:41:48.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78dceecb80 10 client.0 _ll_register_callbacks cb 0x5605f7691780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:48.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39908/0 (socket says 172.21.15.111:39908) 2023-10-11T16:41:48.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78bbfff700 1 -- 172.21.15.111:0/480668683 learned_addr learned my addr 172.21.15.111:0/480668683 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78c37fe700 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f778a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 0x5605f778d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78bbfff700 1 -- 172.21.15.111:0/480668683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 msgr2=0x5605f778d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78bbfff700 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 0x5605f778d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78bbfff700 1 -- 172.21.15.111:0/480668683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 msgr2=0x5605f778a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78bbfff700 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f778a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78bbfff700 1 -- 172.21.15.111:0/480668683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c4110370 con 0x5605f7781470 2023-10-11T16:41:48.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.452+0000 7f78c37fe700 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f778a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:48.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.453+0000 7f78bbfff700 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778a2a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f78ac000c00 tx=0x7f78ac00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f545a8398ecb3e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.453+0000 7f78aa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:48.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.453+0000 7f78aa7fc700 1 -- 172.21.15.111:0/480668683 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac01d050 con 0x5605f7781470 2023-10-11T16:41:48.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.453+0000 7f78aa7fc700 1 -- 172.21.15.111:0/480668683 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f78ac00fd70 con 0x5605f7781470 2023-10-11T16:41:48.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.453+0000 7f78dceecb80 1 -- 172.21.15.111:0/480668683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5605f7773b20 con 0x5605f7781470 2023-10-11T16:41:48.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.453+0000 7f78aa7fc700 1 -- 172.21.15.111:0/480668683 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78ac005070 con 0x5605f7781470 2023-10-11T16:41:48.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.453+0000 7f78dceecb80 10 client.17022 fetch_fsmap learned FSMap version 38 2023-10-11T16:41:48.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.453+0000 7f78dceecb80 1 -- 172.21.15.111:0/480668683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5605f7698190 con 0x5605f7781470 2023-10-11T16:41:48.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78aa7fc700 1 -- 172.21.15.111:0/480668683 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5605f7773b20 con 0x5605f7781470 2023-10-11T16:41:48.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78aa7fc700 1 -- 172.21.15.111:0/480668683 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 38) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f78ac003c20 con 0x5605f7781470 2023-10-11T16:41:48.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78dceecb80 10 client.17022 fetch_fsmap finished waiting for FSMap version 38 2023-10-11T16:41:48.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78dceecb80 10 client.17022 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:41:48.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78dceecb80 20 client.17022 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:48.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78dceecb80 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5605f779a850 0x5605f779cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78dceecb80 4 client.17022 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:48.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78dceecb80 1 -- 172.21.15.111:0/480668683 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605f76940a0 con 0x5605f779a850 2023-10-11T16:41:48.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.454+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5605f779a850 0x5605f779cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.487 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.455+0000 7f78aa7fc700 10 client.17022 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:41:48.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.455+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5605f779a850 0x5605f779cc60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=854514df5fa879b4 server_cookie=e26044c9330a7771 in_seq=0 out_seq=0 2023-10-11T16:41:48.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.455+0000 7f78aa7fc700 1 -- 172.21.15.111:0/480668683 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5605f76940a0 con 0x5605f779a850 2023-10-11T16:41:48.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.455+0000 7f78aa7fc700 10 client.17022 handle_command_reply: tid=0 2023-10-11T16:41:48.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.455+0000 7f78dceecb80 1 client.17022 shutdown 2023-10-11T16:41:48.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 -- 172.21.15.111:0/480668683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 msgr2=0x5605f778a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778a2a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f78ac000c00 tx=0x7f78ac00f740 comp rx=0 tx=0).stop 2023-10-11T16:41:48.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 -- 172.21.15.111:0/480668683 shutdown_connections 2023-10-11T16:41:48.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5605f779a850 0x5605f779cc60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f77858b0 0x5605f778a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 0x5605f778d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78aa7fc700 0 client.17022 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:41:48.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 --2- 172.21.15.111:0/480668683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778a2a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 -- 172.21.15.111:0/480668683 >> 172.21.15.111:0/480668683 conn(0x5605f7776360 msgr2=0x5605f7786d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.497 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 -- 172.21.15.111:0/480668683 shutdown_connections 2023-10-11T16:41:48.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 1 -- 172.21.15.111:0/480668683 wait complete. 2023-10-11T16:41:48.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.456+0000 7f78dceecb80 20 client.17022 trim_cache size 0 max 16384 2023-10-11T16:41:48.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.469+0000 7f78dceecb80 1 Processor -- start 2023-10-11T16:41:48.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.469+0000 7f78dceecb80 1 -- start start 2023-10-11T16:41:48.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7780b60 0x5605f77919a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f77858b0 0x5605f7783970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78dceecb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f7784480 con 0x5605f7781470 2023-10-11T16:41:48.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78dceecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f7791ee0 con 0x5605f7780b60 2023-10-11T16:41:48.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78dceecb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f7788580 con 0x5605f77858b0 2023-10-11T16:41:48.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78c2ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7780b60 0x5605f77919a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39918/0 (socket says 172.21.15.111:39918) 2023-10-11T16:41:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78bbfff700 1 -- 172.21.15.111:0/1919237035 learned_addr learned my addr 172.21.15.111:0/1919237035 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78c37fe700 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f77858b0 0x5605f7783970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78c2ffd700 1 -- 172.21.15.111:0/1919237035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f77858b0 msgr2=0x5605f7783970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f77858b0 0x5605f7783970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.507 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78c2ffd700 1 -- 172.21.15.111:0/1919237035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 msgr2=0x5605f778cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78c2ffd700 1 -- 172.21.15.111:0/1919237035 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605f7697ee0 con 0x5605f7780b60 2023-10-11T16:41:48.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.470+0000 7f78bbfff700 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:48.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.471+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7780b60 0x5605f77919a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f78b40a21e0 tx=0x7f78b40caec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85c114207720d4e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.471+0000 7f78b9ffb700 1 -- 172.21.15.111:0/1919237035 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40cd040 con 0x5605f7780b60 2023-10-11T16:41:48.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.471+0000 7f78b9ffb700 1 -- 172.21.15.111:0/1919237035 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78b409a340 con 0x5605f7780b60 2023-10-11T16:41:48.509 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.471+0000 7f78b9ffb700 1 -- 172.21.15.111:0/1919237035 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b409a520 con 0x5605f7780b60 2023-10-11T16:41:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.471+0000 7f78dceecb80 1 -- 172.21.15.111:0/1919237035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7780b60 msgr2=0x5605f77919a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.471+0000 7f78dceecb80 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7780b60 0x5605f77919a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f78b40a21e0 tx=0x7f78b40caec0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 -- 172.21.15.111:0/1919237035 shutdown_connections 2023-10-11T16:41:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f77858b0 0x5605f7783970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7780b60 0x5605f77919a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 --2- 172.21.15.111:0/1919237035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f7781470 0x5605f778cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 -- 172.21.15.111:0/1919237035 >> 172.21.15.111:0/1919237035 conn(0x5605f7776360 msgr2=0x5605f777a550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 -- 172.21.15.111:0/1919237035 shutdown_connections 2023-10-11T16:41:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 -- 172.21.15.111:0/1919237035 wait complete. 2023-10-11T16:41:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 -- start start 2023-10-11T16:41:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.472+0000 7f78dceecb80 1 Processor -- start 2023-10-11T16:41:48.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 0x5605f778db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7781470 0x5605f778e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78dceecb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f77858b0 0x5605f77e7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78dceecb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f77e7bd0 con 0x5605f77858b0 2023-10-11T16:41:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78dceecb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f77e7d40 con 0x5605f7781470 2023-10-11T16:41:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78dceecb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605f77e7f00 con 0x5605f7780b60 2023-10-11T16:41:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78dceecb80 10 client.0 _ll_register_callbacks cb 0x7f78ac009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:48.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7781470 0x5605f778e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f77858b0 0x5605f77e7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78bbfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7781470 0x5605f778e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:40956/0 (socket says 172.21.15.111:40956) 2023-10-11T16:41:48.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78bbfff700 1 -- 172.21.15.111:0/4054785679 learned_addr learned my addr 172.21.15.111:0/4054785679 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 0x5605f778db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78c37fe700 1 -- 172.21.15.111:0/4054785679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 msgr2=0x5605f778db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78c37fe700 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 0x5605f778db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78c37fe700 1 -- 172.21.15.111:0/4054785679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7781470 msgr2=0x5605f778e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78c37fe700 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7781470 0x5605f778e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.473+0000 7f78c37fe700 1 -- 172.21.15.111:0/4054785679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605f7698190 con 0x5605f77858b0 2023-10-11T16:41:48.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78c37fe700 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f77858b0 0x5605f77e7690 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x5605f778b6a0 tx=0x7f78bc004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d44a5b316a32010 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78aa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:48.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78aa7fc700 1 -- 172.21.15.111:0/4054785679 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc00e040 con 0x5605f77858b0 2023-10-11T16:41:48.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78dceecb80 1 -- 172.21.15.111:0/4054785679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78b400cbd0 con 0x5605f77858b0 2023-10-11T16:41:48.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78aa7fc700 1 -- 172.21.15.111:0/4054785679 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f78bc018e50 con 0x5605f77858b0 2023-10-11T16:41:48.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78aa7fc700 1 -- 172.21.15.111:0/4054785679 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78bc0093f0 con 0x5605f77858b0 2023-10-11T16:41:48.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78dceecb80 10 client.17028 fetch_fsmap learned FSMap version 38 2023-10-11T16:41:48.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78dceecb80 1 -- 172.21.15.111:0/4054785679 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5605f7553630 con 0x5605f77858b0 2023-10-11T16:41:48.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.474+0000 7f78aa7fc700 1 -- 172.21.15.111:0/4054785679 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78b400cbd0 con 0x5605f77858b0 2023-10-11T16:41:48.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.475+0000 7f78aa7fc700 1 -- 172.21.15.111:0/4054785679 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 38) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f78bc004610 con 0x5605f77858b0 2023-10-11T16:41:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.475+0000 7f78dceecb80 10 client.17028 fetch_fsmap finished waiting for FSMap version 38 2023-10-11T16:41:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.475+0000 7f78dceecb80 10 client.17028 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:41:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.475+0000 7f78dceecb80 20 client.17028 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:48.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.475+0000 7f78dceecb80 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5605f77ef700 0x5605f77f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.475+0000 7f78dceecb80 4 client.17028 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:41:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.475+0000 7f78dceecb80 1 -- 172.21.15.111:0/4054785679 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5605f76a0660 con 0x5605f77ef700 2023-10-11T16:41:48.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.475+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5605f77ef700 0x5605f77f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78aa7fc700 10 client.17028 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:41:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78c2ffd700 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5605f77ef700 0x5605f77f1b10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e03b35411d6745e server_cookie=349e301ea85a3d96 in_seq=0 out_seq=0 2023-10-11T16:41:48.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78aa7fc700 1 -- 172.21.15.111:0/4054785679 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5605f76a0660 con 0x5605f77ef700 2023-10-11T16:41:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78aa7fc700 10 client.17028 handle_command_reply: tid=0 2023-10-11T16:41:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 client.17028 shutdown 2023-10-11T16:41:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 -- 172.21.15.111:0/4054785679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f77858b0 msgr2=0x5605f77e7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f77858b0 0x5605f77e7690 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x5605f778b6a0 tx=0x7f78bc004b40 comp rx=0 tx=0).stop 2023-10-11T16:41:48.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 -- 172.21.15.111:0/4054785679 shutdown_connections 2023-10-11T16:41:48.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5605f77ef700 0x5605f77f1b10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5605f77858b0 0x5605f77e7690 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5605f7780b60 0x5605f778db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 --2- 172.21.15.111:0/4054785679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5605f7781470 0x5605f778e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 -- 172.21.15.111:0/4054785679 >> 172.21.15.111:0/4054785679 conn(0x5605f7776360 msgr2=0x5605f7783970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78aa7fc700 0 client.17028 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:41:48.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 -- 172.21.15.111:0/4054785679 shutdown_connections 2023-10-11T16:41:48.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 1 -- 172.21.15.111:0/4054785679 wait complete. 2023-10-11T16:41:48.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.476+0000 7f78dceecb80 20 client.17028 trim_cache size 0 max 16384 2023-10-11T16:41:48.529 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:41:48.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 -- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f78b00a1340 msgr2=0x7f78b00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f78b00a1340 0x7f78b00a37f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009ec0 tx=0x7f78bc00fc40 comp rx=0 tx=0).stop 2023-10-11T16:41:48.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 -- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4100cc0 msgr2=0x7f78c41986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4100cc0 0x7f78c41986b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f78b400eb30 tx=0x7f78b400ee40 comp rx=0 tx=0).stop 2023-10-11T16:41:48.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 -- 172.21.15.111:0/1741909791 shutdown_connections 2023-10-11T16:41:48.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f78b00a1340 0x7f78b00a37f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78c4105620 0x7f78c419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78c4100cc0 0x7f78c41986b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 --2- 172.21.15.111:0/1741909791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4101660 0x7f78c4198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 -- 172.21.15.111:0/1741909791 >> 172.21.15.111:0/1741909791 conn(0x7f78c4077560 msgr2=0x7f78c41001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 -- 172.21.15.111:0/1741909791 shutdown_connections 2023-10-11T16:41:48.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.477+0000 7f78c95ff700 1 -- 172.21.15.111:0/1741909791 wait complete. 2023-10-11T16:41:48.534 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:41:48.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 -- 172.21.15.111:0/3474592775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc102ed0 msgr2=0x7f75dc10f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 --2- 172.21.15.111:0/3474592775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc102ed0 0x7f75dc10f310 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f75d400b0a0 tx=0x7f75d400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 -- 172.21.15.111:0/3474592775 shutdown_connections 2023-10-11T16:41:48.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 --2- 172.21.15.111:0/3474592775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc102ed0 0x7f75dc10f310 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 --2- 172.21.15.111:0/3474592775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75dc108550 0x7f75dc108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 --2- 172.21.15.111:0/3474592775 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75dc102520 0x7f75dc102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 -- 172.21.15.111:0/3474592775 >> 172.21.15.111:0/3474592775 conn(0x7f75dc0fdfc0 msgr2=0x7f75dc1003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 -- 172.21.15.111:0/3474592775 shutdown_connections 2023-10-11T16:41:48.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.732+0000 7f75e47ff700 1 -- 172.21.15.111:0/3474592775 wait complete. 2023-10-11T16:41:48.738 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.733+0000 7f75e47ff700 1 Processor -- start 2023-10-11T16:41:48.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.733+0000 7f75e47ff700 1 -- start start 2023-10-11T16:41:48.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.733+0000 7f75e47ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75dc102520 0x7f75dc19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.733+0000 7f75e47ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75dc102ed0 0x7f75dc19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.739 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e47ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc108550 0x7f75dc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e47ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc198c10 con 0x7f75dc108550 2023-10-11T16:41:48.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e47ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc198d80 con 0x7f75dc102ed0 2023-10-11T16:41:48.740 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e47ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc198f40 con 0x7f75dc102520 2023-10-11T16:41:48.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e259b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75dc102520 0x7f75dc19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e2d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc108550 0x7f75dc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e259b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75dc102520 0x7f75dc19e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56290/0 (socket says 172.21.15.111:56290) 2023-10-11T16:41:48.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e259b700 1 -- 172.21.15.111:0/1303297038 learned_addr learned my addr 172.21.15.111:0/1303297038 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75dc102ed0 0x7f75dc19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e2d9c700 1 -- 172.21.15.111:0/1303297038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75dc102520 msgr2=0x7f75dc19e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75dc102520 0x7f75dc19e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e2d9c700 1 -- 172.21.15.111:0/1303297038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75dc102ed0 msgr2=0x7f75dc19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75dc102ed0 0x7f75dc19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e2d9c700 1 -- 172.21.15.111:0/1303297038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d4009d00 con 0x7f75dc108550 2023-10-11T16:41:48.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75dc102ed0 0x7f75dc19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:48.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e259b700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75dc102520 0x7f75dc19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:48.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.734+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc108550 0x7f75dc1985a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009fd0 tx=0x7f75d400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.735+0000 7f75d37fe700 1 -- 172.21.15.111:0/1303297038 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d400e040 con 0x7f75dc108550 2023-10-11T16:41:48.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.735+0000 7f75d37fe700 1 -- 172.21.15.111:0/1303297038 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75d40094c0 con 0x7f75dc108550 2023-10-11T16:41:48.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.735+0000 7f75d37fe700 1 -- 172.21.15.111:0/1303297038 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d40052a0 con 0x7f75dc108550 2023-10-11T16:41:48.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.735+0000 7f75e47ff700 1 -- 172.21.15.111:0/1303297038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc112570 con 0x7f75dc108550 2023-10-11T16:41:48.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.735+0000 7f75e47ff700 1 -- 172.21.15.111:0/1303297038 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc1123e0 con 0x7f75dc108550 2023-10-11T16:41:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.738+0000 7f75d37fe700 1 -- 172.21.15.111:0/1303297038 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f75d401d470 con 0x7f75dc108550 2023-10-11T16:41:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75f80dcb80 1 Processor -- start 2023-10-11T16:41:48.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75f80dcb80 1 -- start start 2023-10-11T16:41:48.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75d37fe700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f75c80a1340 0x7f75c80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75d37fe700 1 -- 172.21.15.111:0/1303297038 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f75d40ce7c0 con 0x7f75dc108550 2023-10-11T16:41:48.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f451370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f451d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f352060 con 0x55ef6f4518b0 2023-10-11T16:41:48.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f3a48a0 con 0x55ef6f44d470 2023-10-11T16:41:48.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.740+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f39dd30 con 0x55ef6f44cb60 2023-10-11T16:41:48.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f75c80a1340 0x7f75c80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e259b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e259b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56294/0 (socket says 172.21.15.111:56294) 2023-10-11T16:41:48.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e259b700 1 -- 172.21.15.111:0/1449665251 learned_addr learned my addr 172.21.15.111:0/1449665251 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f451d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f451370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e259b700 1 -- 172.21.15.111:0/1449665251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 msgr2=0x55ef6f451370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e259b700 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f451370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e259b700 1 -- 172.21.15.111:0/1449665251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 msgr2=0x55ef6f451d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e259b700 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f451d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e259b700 1 -- 172.21.15.111:0/1449665251 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef6f21f630 con 0x55ef6f44cb60 2023-10-11T16:41:48.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.741+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f75c80a1340 0x7f75c80a37f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f75cc005e90 tx=0x7f75cc013980 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.742+0000 7f75e259b700 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44cf30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f75d8011650 tx=0x7f75d8011960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5a1d607315130b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.742+0000 7f75d0ff9700 1 -- 172.21.15.111:0/1449665251 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8019630 con 0x55ef6f44cb60 2023-10-11T16:41:48.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.742+0000 7f75d0ff9700 1 -- 172.21.15.111:0/1449665251 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75d8019c50 con 0x55ef6f44cb60 2023-10-11T16:41:48.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.742+0000 7f75d0ff9700 1 -- 172.21.15.111:0/1449665251 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d800ec90 con 0x55ef6f44cb60 2023-10-11T16:41:48.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.742+0000 7f75f80dcb80 1 -- 172.21.15.111:0/1449665251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 msgr2=0x55ef6f44cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.742+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44cf30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f75d8011650 tx=0x7f75d8011960 comp rx=0 tx=0).stop 2023-10-11T16:41:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 -- 172.21.15.111:0/1449665251 shutdown_connections 2023-10-11T16:41:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f451d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44cf30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/1449665251 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f451370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 -- 172.21.15.111:0/1449665251 >> 172.21.15.111:0/1449665251 conn(0x55ef6f442360 msgr2=0x55ef6f447b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 -- 172.21.15.111:0/1449665251 shutdown_connections 2023-10-11T16:41:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 -- 172.21.15.111:0/1449665251 wait complete. 2023-10-11T16:41:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 -- start start 2023-10-11T16:41:48.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.743+0000 7f75f80dcb80 1 Processor -- start 2023-10-11T16:41:48.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f459050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f456340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f33f8e0 con 0x55ef6f4518b0 2023-10-11T16:41:48.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f33f580 con 0x55ef6f44d470 2023-10-11T16:41:48.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f221920 con 0x55ef6f44cb60 2023-10-11T16:41:48.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75f80dcb80 10 client.0 _ll_register_callbacks cb 0x55ef6f35d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:48.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75e2d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75e2d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39972/0 (socket says 172.21.15.111:39972) 2023-10-11T16:41:48.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75e1d9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f456340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75e2d9c700 1 -- 172.21.15.111:0/2697992835 learned_addr learned my addr 172.21.15.111:0/2697992835 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.744+0000 7f75e259b700 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f459050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75e2d9c700 1 -- 172.21.15.111:0/2697992835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 msgr2=0x55ef6f459050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f459050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75e2d9c700 1 -- 172.21.15.111:0/2697992835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 msgr2=0x55ef6f456340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f456340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75e2d9c700 1 -- 172.21.15.111:0/2697992835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d8009e10 con 0x55ef6f4518b0 2023-10-11T16:41:48.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4569c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f75d409b1b0 tx=0x7f75d400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19a6e14dcfb5470 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75c5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:48.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2697992835 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d402e350 con 0x55ef6f4518b0 2023-10-11T16:41:48.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2697992835 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f75d409b4b0 con 0x55ef6f4518b0 2023-10-11T16:41:48.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2697992835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ef6f43fb20 con 0x55ef6f4518b0 2023-10-11T16:41:48.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2697992835 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75d409f4e0 con 0x55ef6f4518b0 2023-10-11T16:41:48.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75f80dcb80 10 client.17040 fetch_fsmap learned FSMap version 39 2023-10-11T16:41:48.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.745+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2697992835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55ef6f364190 con 0x55ef6f4518b0 2023-10-11T16:41:48.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2697992835 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ef6f43fb20 con 0x55ef6f4518b0 2023-10-11T16:41:48.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2697992835 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f75d409a9a0 con 0x55ef6f4518b0 2023-10-11T16:41:48.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75f80dcb80 10 client.17040 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:41:48.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75f80dcb80 10 client.17040 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:41:48.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75f80dcb80 20 client.17040 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:48.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef6f466700 0x55ef6f468b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75f80dcb80 4 client.17040 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:48.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2697992835 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef6f3600a0 con 0x55ef6f466700 2023-10-11T16:41:48.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.746+0000 7f75e259b700 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef6f466700 0x55ef6f468b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.747+0000 7f75c5ffb700 10 client.17040 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:41:48.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.747+0000 7f75e259b700 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef6f466700 0x55ef6f468b10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b4891a8e584e9ac server_cookie=17a69861162208f3 in_seq=0 out_seq=0 2023-10-11T16:41:48.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2697992835 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ef6f3600a0 con 0x55ef6f466700 2023-10-11T16:41:48.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75c5ffb700 10 client.17040 handle_command_reply: tid=0 2023-10-11T16:41:48.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 client.17040 shutdown 2023-10-11T16:41:48.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2697992835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 msgr2=0x55ef6f4569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4569c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f75d409b1b0 tx=0x7f75d400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2697992835 shutdown_connections 2023-10-11T16:41:48.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef6f466700 0x55ef6f468b10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4569c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f459050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75c5ffb700 0 client.17040 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:41:48.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2697992835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f456340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2697992835 >> 172.21.15.111:0/2697992835 conn(0x55ef6f442360 msgr2=0x55ef6f452d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2697992835 shutdown_connections 2023-10-11T16:41:48.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.748+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2697992835 wait complete. 2023-10-11T16:41:48.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.749+0000 7f75f80dcb80 20 client.17040 trim_cache size 0 max 16384 2023-10-11T16:41:48.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.762+0000 7f75f80dcb80 1 Processor -- start 2023-10-11T16:41:48.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.762+0000 7f75f80dcb80 1 -- start start 2023-10-11T16:41:48.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.762+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.762+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f44d470 0x55ef6f45d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.762+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f4518b0 0x55ef6f44fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.762+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f44dbc0 con 0x55ef6f44d470 2023-10-11T16:41:48.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.762+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f4546a0 con 0x55ef6f4518b0 2023-10-11T16:41:48.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.762+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f442170 con 0x55ef6f44cb60 2023-10-11T16:41:48.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f44d470 0x55ef6f45d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f44d470 0x55ef6f45d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39974/0 (socket says 172.21.15.111:39974) 2023-10-11T16:41:48.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 -- 172.21.15.111:0/3943712395 learned_addr learned my addr 172.21.15.111:0/3943712395 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e259b700 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f4518b0 0x55ef6f44fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 -- 172.21.15.111:0/3943712395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 msgr2=0x55ef6f44f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 -- 172.21.15.111:0/3943712395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f4518b0 msgr2=0x55ef6f44fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f4518b0 0x55ef6f44fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 -- 172.21.15.111:0/3943712395 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef6f451f50 con 0x55ef6f44d470 2023-10-11T16:41:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e259b700 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f44d470 0x55ef6f45d9e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0207c0 tx=0x7f75cc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958c0670e1c8149b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75d17fa700 1 -- 172.21.15.111:0/3943712395 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc025070 con 0x55ef6f44d470 2023-10-11T16:41:48.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.763+0000 7f75d17fa700 1 -- 172.21.15.111:0/3943712395 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75cc019070 con 0x55ef6f44d470 2023-10-11T16:41:48.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75d17fa700 1 -- 172.21.15.111:0/3943712395 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc02a920 con 0x55ef6f44d470 2023-10-11T16:41:48.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 -- 172.21.15.111:0/3943712395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f44d470 msgr2=0x55ef6f45d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f44d470 0x55ef6f45d9e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0207c0 tx=0x7f75cc01c000 comp rx=0 tx=0).stop 2023-10-11T16:41:48.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 -- 172.21.15.111:0/3943712395 shutdown_connections 2023-10-11T16:41:48.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f4518b0 0x55ef6f44fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f44f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/3943712395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f44d470 0x55ef6f45d9e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 -- 172.21.15.111:0/3943712395 >> 172.21.15.111:0/3943712395 conn(0x55ef6f442360 msgr2=0x55ef6f462b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 -- 172.21.15.111:0/3943712395 shutdown_connections 2023-10-11T16:41:48.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.764+0000 7f75f80dcb80 1 -- 172.21.15.111:0/3943712395 wait complete. 2023-10-11T16:41:48.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 1 -- start start 2023-10-11T16:41:48.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 1 Processor -- start 2023-10-11T16:41:48.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f4ae550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f4aea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f4af240 con 0x55ef6f4518b0 2023-10-11T16:41:48.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f4af3b0 con 0x55ef6f44d470 2023-10-11T16:41:48.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef6f4b9b40 con 0x55ef6f44cb60 2023-10-11T16:41:48.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.765+0000 7f75f80dcb80 10 client.0 _ll_register_callbacks cb 0x7f75d40977f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:48.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e1d9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f4aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e259b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f4ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39976/0 (socket says 172.21.15.111:39976) 2023-10-11T16:41:48.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 -- 172.21.15.111:0/2087010233 learned_addr learned my addr 172.21.15.111:0/2087010233 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:48.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 -- 172.21.15.111:0/2087010233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 msgr2=0x55ef6f4ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f4ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 -- 172.21.15.111:0/2087010233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 msgr2=0x55ef6f4aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:48.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f4aea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 -- 172.21.15.111:0/2087010233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc020500 con 0x55ef6f4518b0 2023-10-11T16:41:48.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e1d9a700 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f4aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:48.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75e2d9c700 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4b95d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f75d409b440 tx=0x7f75d4004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36019643ec5af65a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:48.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75c5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:48.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2087010233 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d409c050 con 0x55ef6f4518b0 2023-10-11T16:41:48.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2087010233 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f75d409bbb0 con 0x55ef6f4518b0 2023-10-11T16:41:48.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.766+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2087010233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75d8019c50 con 0x55ef6f4518b0 2023-10-11T16:41:48.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2087010233 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75d402e350 con 0x55ef6f4518b0 2023-10-11T16:41:48.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75f80dcb80 10 client.17046 fetch_fsmap learned FSMap version 39 2023-10-11T16:41:48.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2087010233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55ef6f1cbde0 con 0x55ef6f4518b0 2023-10-11T16:41:48.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2087010233 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75d8019c50 con 0x55ef6f4518b0 2023-10-11T16:41:48.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2087010233 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f75d409b710 con 0x55ef6f4518b0 2023-10-11T16:41:48.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75f80dcb80 10 client.17046 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:41:48.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75f80dcb80 10 client.17046 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:41:48.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75f80dcb80 20 client.17046 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:48.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef6f4bad70 0x55ef6f4bd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:48.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75f80dcb80 4 client.17046 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:41:48.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2087010233 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ef6f36c660 con 0x55ef6f4bad70 2023-10-11T16:41:48.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.767+0000 7f75e259b700 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef6f4bad70 0x55ef6f4bd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:48.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75c5ffb700 10 client.17046 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:41:48.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75e259b700 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef6f4bad70 0x55ef6f4bd150 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b3b53871ae58dca server_cookie=76ca44a1d7987eec in_seq=0 out_seq=0 2023-10-11T16:41:48.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75c5ffb700 1 -- 172.21.15.111:0/2087010233 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ef6f36c660 con 0x55ef6f4bad70 2023-10-11T16:41:48.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75c5ffb700 10 client.17046 handle_command_reply: tid=0 2023-10-11T16:41:48.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 client.17046 shutdown 2023-10-11T16:41:48.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2087010233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 msgr2=0x55ef6f4b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4b95d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f75d409b440 tx=0x7f75d4004f60 comp rx=0 tx=0).stop 2023-10-11T16:41:48.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2087010233 shutdown_connections 2023-10-11T16:41:48.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55ef6f4bad70 0x55ef6f4bd150 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55ef6f4518b0 0x55ef6f4b95d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55ef6f44cb60 0x55ef6f4ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 --2- 172.21.15.111:0/2087010233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef6f44d470 0x55ef6f4aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75c5ffb700 0 client.17046 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:41:48.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.768+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2087010233 >> 172.21.15.111:0/2087010233 conn(0x55ef6f442360 msgr2=0x55ef6f44fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2087010233 shutdown_connections 2023-10-11T16:41:48.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75f80dcb80 1 -- 172.21.15.111:0/2087010233 wait complete. 2023-10-11T16:41:48.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75f80dcb80 20 client.17046 trim_cache size 0 max 16384 2023-10-11T16:41:48.807 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:41:48.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 -- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f75c80a1340 msgr2=0x7f75c80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f75c80a1340 0x7f75c80a37f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f75cc005e90 tx=0x7f75cc013980 comp rx=0 tx=0).stop 2023-10-11T16:41:48.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 -- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc108550 msgr2=0x7f75dc1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:48.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc108550 0x7f75dc1985a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009fd0 tx=0x7f75d400bce0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 -- 172.21.15.111:0/1303297038 shutdown_connections 2023-10-11T16:41:48.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f75c80a1340 0x7f75c80a37f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75dc108550 0x7f75dc1985a0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75dc102520 0x7f75dc19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 --2- 172.21.15.111:0/1303297038 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75dc102ed0 0x7f75dc19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:48.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.769+0000 7f75e47ff700 1 -- 172.21.15.111:0/1303297038 >> 172.21.15.111:0/1303297038 conn(0x7f75dc0fdfc0 msgr2=0x7f75dc0ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:48.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.770+0000 7f75e47ff700 1 -- 172.21.15.111:0/1303297038 shutdown_connections 2023-10-11T16:41:48.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:48.770+0000 7f75e47ff700 1 -- 172.21.15.111:0/1303297038 wait complete. 2023-10-11T16:41:48.812 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:48 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1034679683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:41:48.813 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:41:48.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:48 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1034679683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:41:48.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:48 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1034679683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:41:49.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 -- 172.21.15.111:0/657594349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 msgr2=0x7fdcec101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/657594349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 0x7fdcec101090 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fdce80099c0 tx=0x7fdce8009cd0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 -- 172.21.15.111:0/657594349 shutdown_connections 2023-10-11T16:41:49.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/657594349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcec105620 0x7fdcec10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/657594349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 0x7fdcec101090 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/657594349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcec101660 0x7fdcec104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 -- 172.21.15.111:0/657594349 >> 172.21.15.111:0/657594349 conn(0x7fdcec077560 msgr2=0x7fdcec077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:49.017 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 -- 172.21.15.111:0/657594349 shutdown_connections 2023-10-11T16:41:49.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.013+0000 7fdcf43ff700 1 -- 172.21.15.111:0/657594349 wait complete. 2023-10-11T16:41:49.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.014+0000 7fdcf43ff700 1 Processor -- start 2023-10-11T16:41:49.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.014+0000 7fdcf43ff700 1 -- start start 2023-10-11T16:41:49.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.014+0000 7fdcf43ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 0x7fdcec198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf43ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcec101660 0x7fdcec198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf43ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcec105620 0x7fdcec19c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf43ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec19c610 con 0x7fdcec100cc0 2023-10-11T16:41:49.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf43ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec19c780 con 0x7fdcec101660 2023-10-11T16:41:49.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf43ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec19c940 con 0x7fdcec105620 2023-10-11T16:41:49.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 0x7fdcec198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 0x7fdcec198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39996/0 (socket says 172.21.15.111:39996) 2023-10-11T16:41:49.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 -- 172.21.15.111:0/941816496 learned_addr learned my addr 172.21.15.111:0/941816496 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:49.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf299c700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcec105620 0x7fdcec19c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf199a700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcec101660 0x7fdcec198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 -- 172.21.15.111:0/941816496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcec105620 msgr2=0x7fdcec19c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcec105620 0x7fdcec19c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 -- 172.21.15.111:0/941816496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcec101660 msgr2=0x7fdcec198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcec101660 0x7fdcec198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 -- 172.21.15.111:0/941816496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce80095f0 con 0x7fdcec100cc0 2023-10-11T16:41:49.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.015+0000 7fdcf219b700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 0x7fdcec198720 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdce800b580 tx=0x7fdce800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:49.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.016+0000 7fdce37fe700 1 -- 172.21.15.111:0/941816496 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce801d070 con 0x7fdcec100cc0 2023-10-11T16:41:49.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.016+0000 7fdce37fe700 1 -- 172.21.15.111:0/941816496 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdce8005050 con 0x7fdcec100cc0 2023-10-11T16:41:49.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.016+0000 7fdcf43ff700 1 -- 172.21.15.111:0/941816496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcec1103d0 con 0x7fdcec100cc0 2023-10-11T16:41:49.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.016+0000 7fdce37fe700 1 -- 172.21.15.111:0/941816496 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce8004a40 con 0x7fdcec100cc0 2023-10-11T16:41:49.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.016+0000 7fdcf43ff700 1 -- 172.21.15.111:0/941816496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec110240 con 0x7fdcec100cc0 2023-10-11T16:41:49.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.019+0000 7fdce37fe700 1 -- 172.21.15.111:0/941816496 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fdce80051c0 con 0x7fdcec100cc0 2023-10-11T16:41:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdd07dc5b80 1 Processor -- start 2023-10-11T16:41:49.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdd07dc5b80 1 -- start start 2023-10-11T16:41:49.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdce37fe700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdcd80a12c0 0x7fdcd80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b680ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6813370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6813d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdce37fe700 1 -- 172.21.15.111:0/941816496 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fdce80cdef0 con 0x7fdcec100cc0 2023-10-11T16:41:49.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b6714060 con 0x55b3b68138b0 2023-10-11T16:41:49.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b67668a0 con 0x55b3b680f470 2023-10-11T16:41:49.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.020+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b675fd30 con 0x55b3b680eb60 2023-10-11T16:41:49.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdcd80a12c0 0x7fdcd80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6813d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.036 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6813d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40010/0 (socket says 172.21.15.111:40010) 2023-10-11T16:41:49.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 -- 172.21.15.111:0/1845705604 learned_addr learned my addr 172.21.15.111:0/1845705604 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:49.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf199a700 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6813370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.037 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf219b700 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b680ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 -- 172.21.15.111:0/1845705604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 msgr2=0x55b3b680ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:49.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b680ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.038 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 -- 172.21.15.111:0/1845705604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 msgr2=0x55b3b6813370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6813370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 -- 172.21.15.111:0/1845705604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3b65e1630 con 0x55b3b68138b0 2023-10-11T16:41:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf219b700 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b680ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.021+0000 7fdcf299c700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdcd80a12c0 0x7fdcd80a3770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdce4009550 tx=0x7fdce4009380 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.022+0000 7fdcf299c700 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6813d00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fdce401aa50 tx=0x7fdce401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525383b7888c5496 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.022+0000 7fdce0bfe700 1 -- 172.21.15.111:0/1845705604 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400e070 con 0x55b3b68138b0 2023-10-11T16:41:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.022+0000 7fdce0bfe700 1 -- 172.21.15.111:0/1845705604 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdce400f040 con 0x55b3b68138b0 2023-10-11T16:41:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.022+0000 7fdce0bfe700 1 -- 172.21.15.111:0/1845705604 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce401b9c0 con 0x55b3b68138b0 2023-10-11T16:41:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.022+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/1845705604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 msgr2=0x55b3b6813d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.041 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.022+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6813d00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fdce401aa50 tx=0x7fdce401d000 comp rx=0 tx=0).stop 2023-10-11T16:41:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/1845705604 shutdown_connections 2023-10-11T16:41:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6813d00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b680ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/1845705604 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6813370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/1845705604 >> 172.21.15.111:0/1845705604 conn(0x55b3b6804360 msgr2=0x55b3b6809b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/1845705604 shutdown_connections 2023-10-11T16:41:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/1845705604 wait complete. 2023-10-11T16:41:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 -- start start 2023-10-11T16:41:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.023+0000 7fdd07dc5b80 1 Processor -- start 2023-10-11T16:41:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b681b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b68182f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6818990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b67018e0 con 0x55b3b68138b0 2023-10-11T16:41:49.045 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b6701580 con 0x55b3b680f470 2023-10-11T16:41:49.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b65e3920 con 0x55b3b680eb60 2023-10-11T16:41:49.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdd07dc5b80 10 client.0 _ll_register_callbacks cb 0x55b3b671f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:49.046 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdcf299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6818990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdcf299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6818990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40016/0 (socket says 172.21.15.111:40016) 2023-10-11T16:41:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdcf299c700 1 -- 172.21.15.111:0/2691910648 learned_addr learned my addr 172.21.15.111:0/2691910648 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:49.047 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdcf199a700 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b68182f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.024+0000 7fdcf219b700 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b681b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcf299c700 1 -- 172.21.15.111:0/2691910648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 msgr2=0x55b3b681b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:49.048 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcf299c700 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b681b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcf299c700 1 -- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 msgr2=0x55b3b68182f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcf299c700 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b68182f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcf299c700 1 -- 172.21.15.111:0/2691910648 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce4008dd0 con 0x55b3b68138b0 2023-10-11T16:41:49.049 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcf199a700 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b68182f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcf299c700 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6818990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fdce4009e00 tx=0x7fdce401af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ca8d56eae3d83e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcd5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:49.050 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/2691910648 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400e070 con 0x55b3b68138b0 2023-10-11T16:41:49.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/2691910648 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fdce4021070 con 0x55b3b68138b0 2023-10-11T16:41:49.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2691910648 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3b6801b20 con 0x55b3b68138b0 2023-10-11T16:41:49.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/2691910648 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdce4004e20 con 0x55b3b68138b0 2023-10-11T16:41:49.051 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdd07dc5b80 10 client.17058 fetch_fsmap learned FSMap version 39 2023-10-11T16:41:49.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.025+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2691910648 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55b3b6726190 con 0x55b3b68138b0 2023-10-11T16:41:49.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/2691910648 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b3b6801b20 con 0x55b3b68138b0 2023-10-11T16:41:49.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/2691910648 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdce4007950 con 0x55b3b68138b0 2023-10-11T16:41:49.052 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdd07dc5b80 10 client.17058 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:41:49.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdd07dc5b80 10 client.17058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:49.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdd07dc5b80 20 client.17058 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:49.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b3b68287f0 0x55b3b682ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdd07dc5b80 4 client.17058 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2691910648 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3b67220a0 con 0x55b3b68287f0 2023-10-11T16:41:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.026+0000 7fdcf219b700 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b3b68287f0 0x55b3b682ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.027+0000 7fdcd5ffb700 10 client.17058 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.027+0000 7fdcf219b700 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b3b68287f0 0x55b3b682ac00 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b663e99c596f6504 server_cookie=cd16e783011a57ba in_seq=0 out_seq=0 2023-10-11T16:41:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/2691910648 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b3b67220a0 con 0x55b3b68287f0 2023-10-11T16:41:49.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdcd5ffb700 10 client.17058 handle_command_reply: tid=0 2023-10-11T16:41:49.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 client.17058 shutdown 2023-10-11T16:41:49.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2691910648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 msgr2=0x55b3b6818990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6818990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fdce4009e00 tx=0x7fdce401af30 comp rx=0 tx=0).stop 2023-10-11T16:41:49.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2691910648 shutdown_connections 2023-10-11T16:41:49.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b3b68287f0 0x55b3b682ac00 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6818990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b681b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2691910648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b68182f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdcd5ffb700 0 client.17058 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:49.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2691910648 >> 172.21.15.111:0/2691910648 conn(0x55b3b6804360 msgr2=0x55b3b680d960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:49.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2691910648 shutdown_connections 2023-10-11T16:41:49.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2691910648 wait complete. 2023-10-11T16:41:49.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.028+0000 7fdd07dc5b80 20 client.17058 trim_cache size 0 max 16384 2023-10-11T16:41:49.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.041+0000 7fdd07dc5b80 1 Processor -- start 2023-10-11T16:41:49.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdd07dc5b80 1 -- start start 2023-10-11T16:41:49.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b681f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6812f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6814c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b680a9d0 con 0x55b3b68138b0 2023-10-11T16:41:49.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b6803ee0 con 0x55b3b680f470 2023-10-11T16:41:49.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b680f840 con 0x55b3b680eb60 2023-10-11T16:41:49.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6814c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6814c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40030/0 (socket says 172.21.15.111:40030) 2023-10-11T16:41:49.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf219b700 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b681f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf299c700 1 -- 172.21.15.111:0/2080157340 learned_addr learned my addr 172.21.15.111:0/2080157340 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:49.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf199a700 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6812f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf299c700 1 -- 172.21.15.111:0/2080157340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 msgr2=0x55b3b681f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:49.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf299c700 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b681f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf299c700 1 -- 172.21.15.111:0/2080157340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 msgr2=0x55b3b6812f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:49.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf299c700 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6812f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.042+0000 7fdcf299c700 1 -- 172.21.15.111:0/2080157340 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3b6725ee0 con 0x55b3b68138b0 2023-10-11T16:41:49.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.043+0000 7fdcf199a700 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6812f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:49.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.043+0000 7fdcf299c700 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6814c90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fdce4007cb0 tx=0x7fdce40099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73a170f709b063d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:49.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.043+0000 7fdce13ff700 1 -- 172.21.15.111:0/2080157340 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400e070 con 0x55b3b68138b0 2023-10-11T16:41:49.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.043+0000 7fdce13ff700 1 -- 172.21.15.111:0/2080157340 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdce4004a70 con 0x55b3b68138b0 2023-10-11T16:41:49.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.043+0000 7fdce13ff700 1 -- 172.21.15.111:0/2080157340 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce401b730 con 0x55b3b68138b0 2023-10-11T16:41:49.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.043+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2080157340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 msgr2=0x55b3b6814c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.043+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6814c90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fdce4007cb0 tx=0x7fdce40099e0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2080157340 shutdown_connections 2023-10-11T16:41:49.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6814c90 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680eb60 0x55b3b681f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/2080157340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680f470 0x55b3b6812f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2080157340 >> 172.21.15.111:0/2080157340 conn(0x55b3b6804360 msgr2=0x55b3b6812870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:49.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2080157340 shutdown_connections 2023-10-11T16:41:49.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/2080157340 wait complete. 2023-10-11T16:41:49.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 -- start start 2023-10-11T16:41:49.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.044+0000 7fdd07dc5b80 1 Processor -- start 2023-10-11T16:41:49.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680eb60 0x55b3b6817b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680f470 0x55b3b6818080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdd07dc5b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6816ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b681ff20 con 0x55b3b68138b0 2023-10-11T16:41:49.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b6817000 con 0x55b3b680eb60 2023-10-11T16:41:49.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdd07dc5b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3b68171c0 con 0x55b3b680f470 2023-10-11T16:41:49.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdd07dc5b80 10 client.0 _ll_register_callbacks cb 0x55b3b671f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:49.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6816ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf219b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680eb60 0x55b3b6817b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf199a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680f470 0x55b3b6818080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf219b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680eb60 0x55b3b6817b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:41082/0 (socket says 172.21.15.111:41082) 2023-10-11T16:41:49.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf219b700 1 -- 172.21.15.111:0/3951901025 learned_addr learned my addr 172.21.15.111:0/3951901025 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf299c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6816ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40046/0 (socket says 172.21.15.111:40046) 2023-10-11T16:41:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf219b700 1 -- 172.21.15.111:0/3951901025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680f470 msgr2=0x55b3b6818080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf219b700 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680f470 0x55b3b6818080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf219b700 1 -- 172.21.15.111:0/3951901025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 msgr2=0x55b3b6816ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf219b700 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6816ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf219b700 1 -- 172.21.15.111:0/3951901025 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b3b6726190 con 0x55b3b680eb60 2023-10-11T16:41:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.045+0000 7fdcf299c700 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6816ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdcf219b700 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680eb60 0x55b3b6817b40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdce8000c00 tx=0x7fdce809a850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be49c27a3fd9b74a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:49.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdcd5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:41:49.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/3951901025 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce80056b0 con 0x55b3b680eb60 2023-10-11T16:41:49.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/3951901025 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fdce809ae40 con 0x55b3b680eb60 2023-10-11T16:41:49.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/3951901025 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdce400f040 con 0x55b3b680eb60 2023-10-11T16:41:49.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/3951901025 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdce809b000 con 0x55b3b680eb60 2023-10-11T16:41:49.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdd07dc5b80 10 client.27203 fetch_fsmap learned FSMap version 39 2023-10-11T16:41:49.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/3951901025 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55b3b6576c60 con 0x55b3b680eb60 2023-10-11T16:41:49.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/3951901025 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdce400f040 con 0x55b3b680eb60 2023-10-11T16:41:49.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/3951901025 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdce8004ba0 con 0x55b3b680eb60 2023-10-11T16:41:49.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdd07dc5b80 10 client.27203 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:41:49.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.046+0000 7fdd07dc5b80 10 client.27203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:49.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.047+0000 7fdd07dc5b80 20 client.27203 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:49.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.047+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b3b687d0c0 0x55b3b687f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:49.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.047+0000 7fdd07dc5b80 4 client.27203 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:41:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.047+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/3951901025 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b3b672e660 con 0x55b3b687d0c0 2023-10-11T16:41:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.047+0000 7fdcf199a700 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b3b687d0c0 0x55b3b687f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.047+0000 7fdcd5ffb700 10 client.27203 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:49.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.047+0000 7fdcf199a700 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b3b687d0c0 0x55b3b687f4a0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f2dadc9e324029e server_cookie=d9291b99cf0537c2 in_seq=0 out_seq=0 2023-10-11T16:41:49.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdcd5ffb700 1 -- 172.21.15.111:0/3951901025 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b3b672e660 con 0x55b3b687d0c0 2023-10-11T16:41:49.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdcd5ffb700 10 client.27203 handle_command_reply: tid=0 2023-10-11T16:41:49.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 client.27203 shutdown 2023-10-11T16:41:49.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/3951901025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680eb60 msgr2=0x55b3b6817b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680eb60 0x55b3b6817b40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdce8000c00 tx=0x7fdce809a850 comp rx=0 tx=0).stop 2023-10-11T16:41:49.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/3951901025 shutdown_connections 2023-10-11T16:41:49.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b3b687d0c0 0x55b3b687f4a0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b3b68138b0 0x55b3b6816ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdcd5ffb700 0 client.27203 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b3b680eb60 0x55b3b6817b40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 --2- 172.21.15.111:0/3951901025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b3b680f470 0x55b3b6818080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/3951901025 >> 172.21.15.111:0/3951901025 conn(0x55b3b6804360 msgr2=0x55b3b680e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/3951901025 shutdown_connections 2023-10-11T16:41:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 1 -- 172.21.15.111:0/3951901025 wait complete. 2023-10-11T16:41:49.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.048+0000 7fdd07dc5b80 20 client.27203 trim_cache size 0 max 16384 2023-10-11T16:41:49.089 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:41:49.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 -- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdcd80a12c0 msgr2=0x7fdcd80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdcd80a12c0 0x7fdcd80a3770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdce4009550 tx=0x7fdce4009380 comp rx=0 tx=0).stop 2023-10-11T16:41:49.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 -- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 msgr2=0x7fdcec198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:49.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 0x7fdcec198720 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdce800b580 tx=0x7fdce800bfd0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 -- 172.21.15.111:0/941816496 shutdown_connections 2023-10-11T16:41:49.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdcd80a12c0 0x7fdcd80a3770 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcec105620 0x7fdcec19c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcec100cc0 0x7fdcec198720 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 --2- 172.21.15.111:0/941816496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcec101660 0x7fdcec198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:49.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 -- 172.21.15.111:0/941816496 >> 172.21.15.111:0/941816496 conn(0x7fdcec077560 msgr2=0x7fdcec0fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:49.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 -- 172.21.15.111:0/941816496 shutdown_connections 2023-10-11T16:41:49.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:49.049+0000 7fdcf43ff700 1 -- 172.21.15.111:0/941816496 wait complete. 2023-10-11T16:41:49.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:49 smithi130 ceph-mon[131859]: pgmap v284: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:49.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:49 smithi130 ceph-mon[131859]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:49.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:49 smithi111 ceph-mon[98527]: pgmap v284: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:49.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:49 smithi111 ceph-mon[98527]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:49.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:49 smithi150 ceph-mon[132217]: pgmap v284: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:49.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:49 smithi150 ceph-mon[132217]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-11T16:41:50.095 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:41:50.096 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:41:50.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 -- 172.21.15.111:0/3243947475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 msgr2=0x7fa27c10d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3243947475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 0x7fa27c10d0a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa27400b0a0 tx=0x7fa27400b3b0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.309 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 -- 172.21.15.111:0/3243947475 shutdown_connections 2023-10-11T16:41:50.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3243947475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 0x7fa27c10d0a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.310 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3243947475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa27c1063f0 0x7fa27c1067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3243947475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa27c100a50 0x7fa27c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.311 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 -- 172.21.15.111:0/3243947475 >> 172.21.15.111:0/3243947475 conn(0x7fa27c077590 msgr2=0x7fa27c077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 -- 172.21.15.111:0/3243947475 shutdown_connections 2023-10-11T16:41:50.312 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.307+0000 7fa2845ff700 1 -- 172.21.15.111:0/3243947475 wait complete. 2023-10-11T16:41:50.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.308+0000 7fa2845ff700 1 Processor -- start 2023-10-11T16:41:50.314 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.308+0000 7fa2845ff700 1 -- start start 2023-10-11T16:41:50.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa2845ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa27c100a50 0x7fa27c19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa2845ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 0x7fa27c19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.315 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa2845ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa27c1063f0 0x7fa27c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa2845ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c198c20 con 0x7fa27c101400 2023-10-11T16:41:50.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa2845ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c198d90 con 0x7fa27c100a50 2023-10-11T16:41:50.316 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa2845ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c198f50 con 0x7fa27c1063f0 2023-10-11T16:41:50.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa281b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 0x7fa27c19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa282b9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa27c1063f0 0x7fa27c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa281b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 0x7fa27c19ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40068/0 (socket says 172.21.15.111:40068) 2023-10-11T16:41:50.317 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa28239b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa27c100a50 0x7fa27c19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa281b9a700 1 -- 172.21.15.111:0/3740706588 learned_addr learned my addr 172.21.15.111:0/3740706588 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa281b9a700 1 -- 172.21.15.111:0/3740706588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa27c1063f0 msgr2=0x7fa27c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.318 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa281b9a700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa27c1063f0 0x7fa27c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa281b9a700 1 -- 172.21.15.111:0/3740706588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa27c100a50 msgr2=0x7fa27c19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa281b9a700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa27c100a50 0x7fa27c19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.319 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa281b9a700 1 -- 172.21.15.111:0/3740706588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa278009520 con 0x7fa27c101400 2023-10-11T16:41:50.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.309+0000 7fa282b9c700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa27c1063f0 0x7fa27c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:50.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.310+0000 7fa281b9a700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 0x7fa27c19ea80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00e9a0 tx=0x7fa26c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.310+0000 7fa2737fe700 1 -- 172.21.15.111:0/3740706588 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c009790 con 0x7fa27c101400 2023-10-11T16:41:50.320 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.310+0000 7fa2737fe700 1 -- 172.21.15.111:0/3740706588 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa26c010430 con 0x7fa27c101400 2023-10-11T16:41:50.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.310+0000 7fa2845ff700 1 -- 172.21.15.111:0/3740706588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa274009d00 con 0x7fa27c101400 2023-10-11T16:41:50.321 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.310+0000 7fa2737fe700 1 -- 172.21.15.111:0/3740706588 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c018680 con 0x7fa27c101400 2023-10-11T16:41:50.322 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.310+0000 7fa2845ff700 1 -- 172.21.15.111:0/3740706588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c110410 con 0x7fa27c101400 2023-10-11T16:41:50.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.313+0000 7fa2737fe700 1 -- 172.21.15.111:0/3740706588 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fa26c0105a0 con 0x7fa27c101400 2023-10-11T16:41:50.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.314+0000 7fa297f19b80 1 Processor -- start 2023-10-11T16:41:50.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.314+0000 7fa2737fe700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2680a1340 0x7fa2680a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa297f19b80 1 -- start start 2023-10-11T16:41:50.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa2737fe700 1 -- 172.21.15.111:0/3740706588 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fa26c014070 con 0x7fa27c101400 2023-10-11T16:41:50.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa28239b700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2680a1340 0x7fa2680a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.330 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffbfffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc004f0 0x55b9ffc043f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc04930 0x55b9ffc04d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.331 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa297f19b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffb050e0 con 0x55b9ffc04930 2023-10-11T16:41:50.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa297f19b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffb57920 con 0x55b9ffc004f0 2023-10-11T16:41:50.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa297f19b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffb50db0 con 0x55b9ffbffbe0 2023-10-11T16:41:50.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa282b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc04930 0x55b9ffc04d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.332 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa281b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc004f0 0x55b9ffc043f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa282b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc04930 0x55b9ffc04d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40072/0 (socket says 172.21.15.111:40072) 2023-10-11T16:41:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa281b9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc004f0 0x55b9ffc043f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:41100/0 (socket says 172.21.15.111:41100) 2023-10-11T16:41:50.333 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.315+0000 7fa282b9c700 1 -- 172.21.15.111:0/200819925 learned_addr learned my addr 172.21.15.111:0/200819925 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa28239b700 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffbfffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa281b9a700 1 -- 172.21.15.111:0/200819925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 msgr2=0x55b9ffbfffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa281b9a700 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffbfffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.334 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa281b9a700 1 -- 172.21.15.111:0/200819925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc04930 msgr2=0x55b9ffc04d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa281b9a700 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc04930 0x55b9ffc04d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa281b9a700 1 -- 172.21.15.111:0/200819925 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9ff9d26b0 con 0x55b9ffc004f0 2023-10-11T16:41:50.335 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa28239b700 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffbfffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:41:50.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa28239b700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2680a1340 0x7fa2680a37f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa278009d80 tx=0x7fa278011720 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.336 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa281b9a700 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc004f0 0x55b9ffc043f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0ca5e0 tx=0x7fa26c099990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3006fe287614e5c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa270bfe700 1 -- 172.21.15.111:0/200819925 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0a23b0 con 0x55b9ffc004f0 2023-10-11T16:41:50.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa270bfe700 1 -- 172.21.15.111:0/200819925 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa26c010880 con 0x55b9ffc004f0 2023-10-11T16:41:50.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.316+0000 7fa270bfe700 1 -- 172.21.15.111:0/200819925 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c099720 con 0x55b9ffc004f0 2023-10-11T16:41:50.337 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 -- 172.21.15.111:0/200819925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc004f0 msgr2=0x55b9ffc043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc004f0 0x55b9ffc043f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0ca5e0 tx=0x7fa26c099990 comp rx=0 tx=0).stop 2023-10-11T16:41:50.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 -- 172.21.15.111:0/200819925 shutdown_connections 2023-10-11T16:41:50.338 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc04930 0x55b9ffc04d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffbfffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 --2- 172.21.15.111:0/200819925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc004f0 0x55b9ffc043f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.339 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 -- 172.21.15.111:0/200819925 >> 172.21.15.111:0/200819925 conn(0x55b9ffbf53e0 msgr2=0x55b9ffbfabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 -- 172.21.15.111:0/200819925 shutdown_connections 2023-10-11T16:41:50.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.317+0000 7fa297f19b80 1 -- 172.21.15.111:0/200819925 wait complete. 2023-10-11T16:41:50.340 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 1 -- start start 2023-10-11T16:41:50.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 1 Processor -- start 2023-10-11T16:41:50.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.341 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc09e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffaf2960 con 0x55b9ffbffbe0 2023-10-11T16:41:50.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffaf2600 con 0x55b9ffc04930 2023-10-11T16:41:50.342 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ff9d49a0 con 0x55b9ffc004f0 2023-10-11T16:41:50.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.318+0000 7fa297f19b80 10 client.0 _ll_register_callbacks cb 0x55b9ffb10800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:50.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40084/0 (socket says 172.21.15.111:40084) 2023-10-11T16:41:50.343 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 -- 172.21.15.111:0/3827580608 learned_addr learned my addr 172.21.15.111:0/3827580608 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa282b9c700 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc09e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa281b9a700 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.344 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 -- 172.21.15.111:0/3827580608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc004f0 msgr2=0x55b9ffc09910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 -- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 msgr2=0x55b9ffc09e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc09e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.345 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 -- 172.21.15.111:0/3827580608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c110410 con 0x55b9ffbffbe0 2023-10-11T16:41:50.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa282b9c700 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc09e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:41:50.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa28239b700 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc093d0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa27800fa70 tx=0x7fa2780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d576bef1780cf6c server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.346 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.319+0000 7fa265ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:50.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa265ffb700 1 -- 172.21.15.111:0/3827580608 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27801eae0 con 0x55b9ffbffbe0 2023-10-11T16:41:50.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa265ffb700 1 -- 172.21.15.111:0/3827580608 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fa27801ec40 con 0x55b9ffbffbe0 2023-10-11T16:41:50.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa297f19b80 1 -- 172.21.15.111:0/3827580608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9ffbf2ba0 con 0x55b9ffbffbe0 2023-10-11T16:41:50.347 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa265ffb700 1 -- 172.21.15.111:0/3827580608 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa27801ee00 con 0x55b9ffbffbe0 2023-10-11T16:41:50.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa297f19b80 10 client.17076 fetch_fsmap learned FSMap version 39 2023-10-11T16:41:50.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa297f19b80 1 -- 172.21.15.111:0/3827580608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55b9ffb17210 con 0x55b9ffbffbe0 2023-10-11T16:41:50.348 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa265ffb700 1 -- 172.21.15.111:0/3827580608 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b9ffbf2ba0 con 0x55b9ffbffbe0 2023-10-11T16:41:50.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa265ffb700 1 -- 172.21.15.111:0/3827580608 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa27800e440 con 0x55b9ffbffbe0 2023-10-11T16:41:50.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa297f19b80 10 client.17076 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:41:50.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.320+0000 7fa297f19b80 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:50.349 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.321+0000 7fa297f19b80 20 client.17076 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:50.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.321+0000 7fa297f19b80 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b9ffc19a90 0x55b9ffc1bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.321+0000 7fa297f19b80 4 client.17076 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:50.350 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.321+0000 7fa297f19b80 1 -- 172.21.15.111:0/3827580608 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9ffb13120 con 0x55b9ffc19a90 2023-10-11T16:41:50.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.321+0000 7fa281b9a700 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b9ffc19a90 0x55b9ffc1bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.321+0000 7fa281b9a700 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b9ffc19a90 0x55b9ffc1bea0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3630169a45abde7 server_cookie=b526b07f2982c8b7 in_seq=0 out_seq=0 2023-10-11T16:41:50.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.321+0000 7fa265ffb700 10 client.17076 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:50.351 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.322+0000 7fa265ffb700 1 -- 172.21.15.111:0/3827580608 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b9ffb13120 con 0x55b9ffc19a90 2023-10-11T16:41:50.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.322+0000 7fa265ffb700 10 client.17076 handle_command_reply: tid=0 2023-10-11T16:41:50.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.322+0000 7fa297f19b80 1 client.17076 shutdown 2023-10-11T16:41:50.352 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.322+0000 7fa297f19b80 1 -- 172.21.15.111:0/3827580608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffbffbe0 msgr2=0x55b9ffc093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.322+0000 7fa297f19b80 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc093d0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa27800fa70 tx=0x7fa2780047c0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.322+0000 7fa297f19b80 1 -- 172.21.15.111:0/3827580608 shutdown_connections 2023-10-11T16:41:50.353 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.322+0000 7fa297f19b80 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b9ffc19a90 0x55b9ffc1bea0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.322+0000 7fa297f19b80 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc09e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.323+0000 7fa265ffb700 0 client.17076 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:50.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.323+0000 7fa297f19b80 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc093d0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.354 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.323+0000 7fa297f19b80 1 --2- 172.21.15.111:0/3827580608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.355 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.323+0000 7fa297f19b80 1 -- 172.21.15.111:0/3827580608 >> 172.21.15.111:0/3827580608 conn(0x55b9ffbf53e0 msgr2=0x55b9ffc05de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.323+0000 7fa297f19b80 1 -- 172.21.15.111:0/3827580608 shutdown_connections 2023-10-11T16:41:50.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.323+0000 7fa297f19b80 1 -- 172.21.15.111:0/3827580608 wait complete. 2023-10-11T16:41:50.358 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.323+0000 7fa297f19b80 20 client.17076 trim_cache size 0 max 16384 2023-10-11T16:41:50.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.340+0000 7fa297f19b80 1 Processor -- start 2023-10-11T16:41:50.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.340+0000 7fa297f19b80 1 -- start start 2023-10-11T16:41:50.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.340+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc041a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc10a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc04930 0x55b9ffbff0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa297f19b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffbf4f60 con 0x55b9ffc004f0 2023-10-11T16:41:50.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa297f19b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffc009c0 con 0x55b9ffbffbe0 2023-10-11T16:41:50.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa297f19b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffc07980 con 0x55b9ffc04930 2023-10-11T16:41:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc10a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc10a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40086/0 (socket says 172.21.15.111:40086) 2023-10-11T16:41:50.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 -- 172.21.15.111:0/4250402626 learned_addr learned my addr 172.21.15.111:0/4250402626 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa282b9c700 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc04930 0x55b9ffbff0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa28239b700 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 -- 172.21.15.111:0/4250402626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc04930 msgr2=0x55b9ffbff0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc04930 0x55b9ffbff0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 -- 172.21.15.111:0/4250402626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffbffbe0 msgr2=0x55b9ffc041a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc041a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 -- 172.21.15.111:0/4250402626 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9ffb16f60 con 0x55b9ffc004f0 2023-10-11T16:41:50.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.341+0000 7fa281b9a700 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc10a20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0cc010 tx=0x7fa26c09a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df0563ce39ae777 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa2713ff700 1 -- 172.21.15.111:0/4250402626 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0b6070 con 0x55b9ffc004f0 2023-10-11T16:41:50.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa2713ff700 1 -- 172.21.15.111:0/4250402626 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa26c0b1070 con 0x55b9ffc004f0 2023-10-11T16:41:50.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa2713ff700 1 -- 172.21.15.111:0/4250402626 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c099450 con 0x55b9ffc004f0 2023-10-11T16:41:50.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 -- 172.21.15.111:0/4250402626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 msgr2=0x55b9ffc10a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc10a20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0cc010 tx=0x7fa26c09a0e0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 -- 172.21.15.111:0/4250402626 shutdown_connections 2023-10-11T16:41:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffc04930 0x55b9ffbff0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 --2- 172.21.15.111:0/4250402626 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc10a20 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 -- 172.21.15.111:0/4250402626 >> 172.21.15.111:0/4250402626 conn(0x55b9ffbf53e0 msgr2=0x55b9ffbfe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 -- 172.21.15.111:0/4250402626 shutdown_connections 2023-10-11T16:41:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.342+0000 7fa297f19b80 1 -- 172.21.15.111:0/4250402626 wait complete. 2023-10-11T16:41:50.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 1 -- start start 2023-10-11T16:41:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 1 Processor -- start 2023-10-11T16:41:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc08c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc096d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffc10fc0 con 0x55b9ffc004f0 2023-10-11T16:41:50.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffc6d930 con 0x55b9ffc04930 2023-10-11T16:41:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9ffc6da70 con 0x55b9ffbffbe0 2023-10-11T16:41:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.343+0000 7fa297f19b80 10 client.0 _ll_register_callbacks cb 0x7fa27800f860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40100/0 (socket says 172.21.15.111:40100) 2023-10-11T16:41:50.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 -- 172.21.15.111:0/737157006 learned_addr learned my addr 172.21.15.111:0/737157006 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa282b9c700 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa28239b700 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc08c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 -- 172.21.15.111:0/737157006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 msgr2=0x55b9ffc08c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc08c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 -- 172.21.15.111:0/737157006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 msgr2=0x55b9ffc096d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc096d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 -- 172.21.15.111:0/737157006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9ff9d26b0 con 0x55b9ffc004f0 2023-10-11T16:41:50.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa281b9a700 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09190 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0982b0 tx=0x7fa26c010b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a838106ad09df806 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.344+0000 7fa265ffb700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:41:50.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa265ffb700 1 -- 172.21.15.111:0/737157006 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0b6070 con 0x55b9ffc004f0 2023-10-11T16:41:50.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa265ffb700 1 -- 172.21.15.111:0/737157006 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fa26c099660 con 0x55b9ffc004f0 2023-10-11T16:41:50.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 1 -- 172.21.15.111:0/737157006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa26c010880 con 0x55b9ffc004f0 2023-10-11T16:41:50.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa265ffb700 1 -- 172.21.15.111:0/737157006 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa26c0a2430 con 0x55b9ffc004f0 2023-10-11T16:41:50.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 10 client.17082 fetch_fsmap learned FSMap version 39 2023-10-11T16:41:50.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 1 -- 172.21.15.111:0/737157006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55b9ff967ce0 con 0x55b9ffc004f0 2023-10-11T16:41:50.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa265ffb700 1 -- 172.21.15.111:0/737157006 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa26c07c080 con 0x55b9ffc004f0 2023-10-11T16:41:50.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa265ffb700 1 -- 172.21.15.111:0/737157006 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa26c09a250 con 0x55b9ffc004f0 2023-10-11T16:41:50.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 10 client.17082 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:41:50.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 10 client.17082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:50.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 20 client.17082 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:50.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b9ffc6ec90 0x55b9ffc71070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 4 client.17082 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:41:50.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.345+0000 7fa297f19b80 1 -- 172.21.15.111:0/737157006 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b9ffb1f6e0 con 0x55b9ffc6ec90 2023-10-11T16:41:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.346+0000 7fa28239b700 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b9ffc6ec90 0x55b9ffc71070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.346+0000 7fa265ffb700 10 client.17082 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.346+0000 7fa28239b700 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b9ffc6ec90 0x55b9ffc71070 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd18cd91da5ff15c server_cookie=a2e99958a1015161 in_seq=0 out_seq=0 2023-10-11T16:41:50.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa265ffb700 1 -- 172.21.15.111:0/737157006 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b9ffb1f6e0 con 0x55b9ffc6ec90 2023-10-11T16:41:50.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa265ffb700 10 client.17082 handle_command_reply: tid=0 2023-10-11T16:41:50.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 client.17082 shutdown 2023-10-11T16:41:50.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 -- 172.21.15.111:0/737157006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 msgr2=0x55b9ffc09190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09190 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0982b0 tx=0x7fa26c010b80 comp rx=0 tx=0).stop 2023-10-11T16:41:50.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 -- 172.21.15.111:0/737157006 shutdown_connections 2023-10-11T16:41:50.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55b9ffc6ec90 0x55b9ffc71070 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9ffc04930 0x55b9ffc096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55b9ffbffbe0 0x55b9ffc08c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa265ffb700 0 client.17082 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 --2- 172.21.15.111:0/737157006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55b9ffc004f0 0x55b9ffc09190 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.387 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 -- 172.21.15.111:0/737157006 >> 172.21.15.111:0/737157006 conn(0x55b9ffbf53e0 msgr2=0x55b9ffbff270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 -- 172.21.15.111:0/737157006 shutdown_connections 2023-10-11T16:41:50.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.347+0000 7fa297f19b80 1 -- 172.21.15.111:0/737157006 wait complete. 2023-10-11T16:41:50.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa297f19b80 20 client.17082 trim_cache size 0 max 16384 2023-10-11T16:41:50.389 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:41:50.389 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:41:50.389 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "5a3a0797-2ff6-48e9-aeef-c4ee92b420b8", 2023-10-11T16:41:50.390 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:41:50.390 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:41:50.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 -- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2680a1340 msgr2=0x7fa2680a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2680a1340 0x7fa2680a37f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa278009d80 tx=0x7fa278011720 comp rx=0 tx=0).stop 2023-10-11T16:41:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 -- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 msgr2=0x7fa27c19ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 0x7fa27c19ea80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00e9a0 tx=0x7fa26c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 -- 172.21.15.111:0/3740706588 shutdown_connections 2023-10-11T16:41:50.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa2680a1340 0x7fa2680a37f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa27c1063f0 0x7fa27c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa27c100a50 0x7fa27c19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 --2- 172.21.15.111:0/3740706588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c101400 0x7fa27c19ea80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 -- 172.21.15.111:0/3740706588 >> 172.21.15.111:0/3740706588 conn(0x7fa27c077590 msgr2=0x7fa27c10b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.348+0000 7fa2845ff700 1 -- 172.21.15.111:0/3740706588 shutdown_connections 2023-10-11T16:41:50.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.349+0000 7fa2845ff700 1 -- 172.21.15.111:0/3740706588 wait complete. 2023-10-11T16:41:50.395 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:41:50.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 -- 172.21.15.111:0/739496985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 msgr2=0x7f5a400747d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.596 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 --2- 172.21.15.111:0/739496985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 0x7f5a400747d0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009990 tx=0x7f5a30009ca0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 -- 172.21.15.111:0/739496985 shutdown_connections 2023-10-11T16:41:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 --2- 172.21.15.111:0/739496985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a400736d0 0x7f5a401136a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 --2- 172.21.15.111:0/739496985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 0x7f5a400747d0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 --2- 172.21.15.111:0/739496985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a40072d20 0x7f5a40073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 -- 172.21.15.111:0/739496985 >> 172.21.15.111:0/739496985 conn(0x7f5a40077590 msgr2=0x7f5a40077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 -- 172.21.15.111:0/739496985 shutdown_connections 2023-10-11T16:41:50.599 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.594+0000 7f5a469ff700 1 -- 172.21.15.111:0/739496985 wait complete. 2023-10-11T16:41:50.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.595+0000 7f5a469ff700 1 Processor -- start 2023-10-11T16:41:50.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.595+0000 7f5a469ff700 1 -- start start 2023-10-11T16:41:50.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.595+0000 7f5a469ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a40072d20 0x7f5a401a2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.595+0000 7f5a469ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a400736d0 0x7f5a401a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a469ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 0x7f5a4019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a469ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019cfb0 con 0x7f5a40074400 2023-10-11T16:41:50.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a469ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019d120 con 0x7f5a40072d20 2023-10-11T16:41:50.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a469ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4019d2e0 con 0x7f5a400736d0 2023-10-11T16:41:50.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 0x7f5a4019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.605 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 0x7f5a4019c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40114/0 (socket says 172.21.15.111:40114) 2023-10-11T16:41:50.606 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 -- 172.21.15.111:0/1793479656 learned_addr learned my addr 172.21.15.111:0/1793479656 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.607 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a3ffff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a40072d20 0x7f5a401a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a37fff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a400736d0 0x7f5a401a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 -- 172.21.15.111:0/1793479656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a400736d0 msgr2=0x7f5a401a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.608 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a400736d0 0x7f5a401a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 -- 172.21.15.111:0/1793479656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a40072d20 msgr2=0x7f5a401a2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a40072d20 0x7f5a401a2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.609 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 -- 172.21.15.111:0/1793479656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300095f0 con 0x7f5a40074400 2023-10-11T16:41:50.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.596+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 0x7f5a4019c7e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5a38007ee0 tx=0x7f5a3800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.597+0000 7f5a3dffb700 1 -- 172.21.15.111:0/1793479656 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38017070 con 0x7f5a40074400 2023-10-11T16:41:50.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.597+0000 7f5a3dffb700 1 -- 172.21.15.111:0/1793479656 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a3800f040 con 0x7f5a40074400 2023-10-11T16:41:50.610 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.597+0000 7f5a469ff700 1 -- 172.21.15.111:0/1793479656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a401169a0 con 0x7f5a40074400 2023-10-11T16:41:50.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.597+0000 7f5a3dffb700 1 -- 172.21.15.111:0/1793479656 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38005580 con 0x7f5a40074400 2023-10-11T16:41:50.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.597+0000 7f5a469ff700 1 -- 172.21.15.111:0/1793479656 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a40116810 con 0x7f5a40074400 2023-10-11T16:41:50.611 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.600+0000 7f5a3dffb700 1 -- 172.21.15.111:0/1793479656 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f5a3800db20 con 0x7f5a40074400 2023-10-11T16:41:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.601+0000 7f5a5a259b80 1 Processor -- start 2023-10-11T16:41:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.601+0000 7f5a3dffb700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5a2c0a12c0 0x7f5a2c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.601+0000 7f5a5a259b80 1 -- start start 2023-10-11T16:41:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.601+0000 7f5a3dffb700 1 -- 172.21.15.111:0/1793479656 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f5a380cc880 con 0x7f5a40074400 2023-10-11T16:41:50.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a37fff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5a2c0a12c0 0x7f5a2c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 0x55e2a0412f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.618 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 0x55e2a0417370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a04178b0 0x55e2a0417d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a0318060 con 0x55e2a04178b0 2023-10-11T16:41:50.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a036a8a0 con 0x55e2a0413470 2023-10-11T16:41:50.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a0363d30 con 0x55e2a0412b60 2023-10-11T16:41:50.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a44f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a04178b0 0x55e2a0417d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a3ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 0x55e2a0412f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.620 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a44f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a04178b0 0x55e2a0417d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40116/0 (socket says 172.21.15.111:40116) 2023-10-11T16:41:50.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a44f9c700 1 -- 172.21.15.111:0/2639853289 learned_addr learned my addr 172.21.15.111:0/2639853289 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a37fff700 1 --2- 172.21.15.111:0/2639853289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 0x55e2a0417370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a44f9c700 1 -- 172.21.15.111:0/2639853289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 msgr2=0x55e2a0412f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.621 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/2639853289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 0x55e2a0412f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a37fff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5a2c0a12c0 0x7f5a2c0a3770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5a4019e010 tx=0x7f5a28007f80 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a44f9c700 1 -- 172.21.15.111:0/2639853289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 msgr2=0x55e2a0417370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.622 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/2639853289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 0x55e2a0417370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.602+0000 7f5a44f9c700 1 -- 172.21.15.111:0/2639853289 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2a01e5630 con 0x55e2a04178b0 2023-10-11T16:41:50.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.603+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/2639853289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a04178b0 0x55e2a0417d00 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5a380a1400 tx=0x7f5a38000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54a08dcc9f26e08 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.623 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.603+0000 7f5a367fc700 1 -- 172.21.15.111:0/2639853289 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380a1930 con 0x55e2a04178b0 2023-10-11T16:41:50.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.603+0000 7f5a367fc700 1 -- 172.21.15.111:0/2639853289 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a38097880 con 0x55e2a04178b0 2023-10-11T16:41:50.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.603+0000 7f5a367fc700 1 -- 172.21.15.111:0/2639853289 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38095dc0 con 0x55e2a04178b0 2023-10-11T16:41:50.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2639853289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a04178b0 msgr2=0x55e2a0417d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.624 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/2639853289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a04178b0 0x55e2a0417d00 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f5a380a1400 tx=0x7f5a38000f40 comp rx=0 tx=0).stop 2023-10-11T16:41:50.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2639853289 shutdown_connections 2023-10-11T16:41:50.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/2639853289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a04178b0 0x55e2a0417d00 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/2639853289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 0x55e2a0412f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/2639853289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 0x55e2a0417370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2639853289 >> 172.21.15.111:0/2639853289 conn(0x55e2a0408360 msgr2=0x55e2a040db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2639853289 shutdown_connections 2023-10-11T16:41:50.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2639853289 wait complete. 2023-10-11T16:41:50.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 -- start start 2023-10-11T16:41:50.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.604+0000 7f5a5a259b80 1 Processor -- start 2023-10-11T16:41:50.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0412b60 0x55e2a041f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a041c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a03058e0 con 0x55e2a0413470 2023-10-11T16:41:50.628 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a0305580 con 0x55e2a0412b60 2023-10-11T16:41:50.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a01e7920 con 0x55e2a04178b0 2023-10-11T16:41:50.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a5a259b80 10 client.0 _ll_register_callbacks cb 0x55e2a0323780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:50.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a041c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a3ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0412b60 0x55e2a041f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a3ffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0412b60 0x55e2a041f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:41174/0 (socket says 172.21.15.111:41174) 2023-10-11T16:41:50.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a3ffff700 1 -- 172.21.15.111:0/1012007090 learned_addr learned my addr 172.21.15.111:0/1012007090 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.605+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a3ffff700 1 -- 172.21.15.111:0/1012007090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 msgr2=0x55e2a041c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a3ffff700 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a3ffff700 1 -- 172.21.15.111:0/1012007090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 msgr2=0x55e2a041c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a3ffff700 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a041c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a3ffff700 1 -- 172.21.15.111:0/1012007090 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a40116810 con 0x55e2a0412b60 2023-10-11T16:41:50.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a3ffff700 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0412b60 0x55e2a041f020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5a30005f20 tx=0x7f5a30005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87dc331f73d23c82 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a1f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:41:50.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/1012007090 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3001d040 con 0x55e2a0412b60 2023-10-11T16:41:50.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a5a259b80 1 -- 172.21.15.111:0/1012007090 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e2a0405b20 con 0x55e2a0412b60 2023-10-11T16:41:50.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/1012007090 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f5a30003d60 con 0x55e2a0412b60 2023-10-11T16:41:50.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/1012007090 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a30005070 con 0x55e2a0412b60 2023-10-11T16:41:50.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a5a259b80 10 client.27233 fetch_fsmap learned FSMap version 39 2023-10-11T16:41:50.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.606+0000 7f5a5a259b80 1 -- 172.21.15.111:0/1012007090 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55e2a032a190 con 0x55e2a0412b60 2023-10-11T16:41:50.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.607+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/1012007090 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e2a0405b20 con 0x55e2a0412b60 2023-10-11T16:41:50.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.607+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/1012007090 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5a3000fa00 con 0x55e2a0412b60 2023-10-11T16:41:50.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.607+0000 7f5a5a259b80 10 client.27233 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:41:50.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.607+0000 7f5a5a259b80 10 client.27233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:50.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.607+0000 7f5a5a259b80 20 client.27233 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:50.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.607+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e2a042c8f0 0x55e2a042ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.607+0000 7f5a5a259b80 4 client.27233 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:41:50.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.607+0000 7f5a5a259b80 1 -- 172.21.15.111:0/1012007090 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e2a03260a0 con 0x55e2a042c8f0 2023-10-11T16:41:50.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.608+0000 7f5a37fff700 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e2a042c8f0 0x55e2a042ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.608+0000 7f5a1f7fe700 10 client.27233 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:50.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.608+0000 7f5a37fff700 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e2a042c8f0 0x55e2a042ed00 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1b1bf6f525b2ec9 server_cookie=97459e615a2e1365 in_seq=0 out_seq=0 2023-10-11T16:41:50.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/1012007090 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e2a03260a0 con 0x55e2a042c8f0 2023-10-11T16:41:50.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a1f7fe700 10 client.27233 handle_command_reply: tid=0 2023-10-11T16:41:50.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 client.27233 shutdown 2023-10-11T16:41:50.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 -- 172.21.15.111:0/1012007090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0412b60 msgr2=0x55e2a041f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0412b60 0x55e2a041f020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5a30005f20 tx=0x7f5a30005c90 comp rx=0 tx=0).stop 2023-10-11T16:41:50.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 -- 172.21.15.111:0/1012007090 shutdown_connections 2023-10-11T16:41:50.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e2a042c8f0 0x55e2a042ed00 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0412b60 0x55e2a041f020 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a1f7fe700 0 client.27233 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:50.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/1012007090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a041c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 -- 172.21.15.111:0/1012007090 >> 172.21.15.111:0/1012007090 conn(0x55e2a0408360 msgr2=0x55e2a040cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 -- 172.21.15.111:0/1012007090 shutdown_connections 2023-10-11T16:41:50.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 1 -- 172.21.15.111:0/1012007090 wait complete. 2023-10-11T16:41:50.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.609+0000 7f5a5a259b80 20 client.27233 trim_cache size 0 max 16384 2023-10-11T16:41:50.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a5a259b80 1 Processor -- start 2023-10-11T16:41:50.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a5a259b80 1 -- start start 2023-10-11T16:41:50.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 0x55e2a04239a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a0419c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a04178b0 0x55e2a041ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.647 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a041d8b0 con 0x55e2a0413470 2023-10-11T16:41:50.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a0407ee0 con 0x55e2a04178b0 2023-10-11T16:41:50.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a041a800 con 0x55e2a0412b60 2023-10-11T16:41:50.648 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a0419c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a37fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a0419c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40126/0 (socket says 172.21.15.111:40126) 2023-10-11T16:41:50.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a44f9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a04178b0 0x55e2a041ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a37fff700 1 -- 172.21.15.111:0/2691132302 learned_addr learned my addr 172.21.15.111:0/2691132302 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.649 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a3ffff700 1 --2- 172.21.15.111:0/2691132302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 0x55e2a04239a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a37fff700 1 -- 172.21.15.111:0/2691132302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 msgr2=0x55e2a04239a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a37fff700 1 --2- 172.21.15.111:0/2691132302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 0x55e2a04239a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.650 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a37fff700 1 -- 172.21.15.111:0/2691132302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a04178b0 msgr2=0x55e2a041ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a37fff700 1 --2- 172.21.15.111:0/2691132302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a04178b0 0x55e2a041ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.627+0000 7f5a37fff700 1 -- 172.21.15.111:0/2691132302 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2a0417f50 con 0x55e2a0413470 2023-10-11T16:41:50.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a37fff700 1 --2- 172.21.15.111:0/2691132302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a0419c60 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5a4019e0a0 tx=0x7f5a28015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9108b9e36465ce server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.651 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a36ffd700 1 -- 172.21.15.111:0/2691132302 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28024040 con 0x55e2a0413470 2023-10-11T16:41:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a36ffd700 1 -- 172.21.15.111:0/2691132302 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a28028530 con 0x55e2a0413470 2023-10-11T16:41:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a36ffd700 1 -- 172.21.15.111:0/2691132302 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28028780 con 0x55e2a0413470 2023-10-11T16:41:50.652 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2691132302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 msgr2=0x55e2a0419c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/2691132302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a0419c60 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5a4019e0a0 tx=0x7f5a28015000 comp rx=0 tx=0).stop 2023-10-11T16:41:50.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2691132302 shutdown_connections 2023-10-11T16:41:50.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/2691132302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a04178b0 0x55e2a041ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.653 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/2691132302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a0412b60 0x55e2a04239a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/2691132302 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0413470 0x55e2a0419c60 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.628+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2691132302 >> 172.21.15.111:0/2691132302 conn(0x55e2a0408360 msgr2=0x55e2a0428be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.654 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.629+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2691132302 shutdown_connections 2023-10-11T16:41:50.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.629+0000 7f5a5a259b80 1 -- 172.21.15.111:0/2691132302 wait complete. 2023-10-11T16:41:50.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.629+0000 7f5a5a259b80 1 -- start start 2023-10-11T16:41:50.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.629+0000 7f5a5a259b80 1 Processor -- start 2023-10-11T16:41:50.655 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0412b60 0x55e2a041bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 0x55e2a041aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a5a259b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.656 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a041b4d0 con 0x55e2a0412b60 2023-10-11T16:41:50.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a041b640 con 0x55e2a0413470 2023-10-11T16:41:50.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a5a259b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a041b800 con 0x55e2a04178b0 2023-10-11T16:41:50.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a5a259b80 10 client.0 _ll_register_callbacks cb 0x55e2a0323780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:41:50.657 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0412b60 0x55e2a041bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a37fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 0x55e2a041aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.658 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:56492/0 (socket says 172.21.15.111:56492) 2023-10-11T16:41:50.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 -- 172.21.15.111:0/3936196245 learned_addr learned my addr 172.21.15.111:0/3936196245 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:41:50.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 -- 172.21.15.111:0/3936196245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 msgr2=0x55e2a041aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 0x55e2a041aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.659 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 -- 172.21.15.111:0/3936196245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0412b60 msgr2=0x55e2a041bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:41:50.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0412b60 0x55e2a041bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 -- 172.21.15.111:0/3936196245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a280045c0 con 0x55e2a04178b0 2023-10-11T16:41:50.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a44f9c700 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041af90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5a380991e0 tx=0x7f5a38099640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a24cfca04fa2383 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:41:50.660 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.630+0000 7f5a1f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:41:50.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/3936196245 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380992b0 con 0x55e2a04178b0 2023-10-11T16:41:50.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/3936196245 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f5a380b6040 con 0x55e2a04178b0 2023-10-11T16:41:50.661 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 1 -- 172.21.15.111:0/3936196245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a38097880 con 0x55e2a04178b0 2023-10-11T16:41:50.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/3936196245 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a380a1930 con 0x55e2a04178b0 2023-10-11T16:41:50.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 10 client.27247 fetch_fsmap learned FSMap version 39 2023-10-11T16:41:50.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 1 -- 172.21.15.111:0/3936196245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55e2a0191de0 con 0x55e2a04178b0 2023-10-11T16:41:50.662 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/3936196245 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a38097880 con 0x55e2a04178b0 2023-10-11T16:41:50.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/3936196245 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5a380c5d20 con 0x55e2a04178b0 2023-10-11T16:41:50.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 10 client.27247 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:41:50.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 10 client.27247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:41:50.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 20 client.27247 populate_metadata read hostname 'smithi111' 2023-10-11T16:41:50.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e2a0481710 0x55e2a0483b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:41:50.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 4 client.27247 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:41:50.664 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.631+0000 7f5a5a259b80 1 -- 172.21.15.111:0/3936196245 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e2a0332660 con 0x55e2a0481710 2023-10-11T16:41:50.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.632+0000 7f5a3ffff700 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e2a0481710 0x55e2a0483b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:41:50.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.632+0000 7f5a3ffff700 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e2a0481710 0x55e2a0483b20 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e487dfb116c1f8b7 server_cookie=bdb08027a69c5354 in_seq=0 out_seq=0 2023-10-11T16:41:50.665 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.632+0000 7f5a1f7fe700 10 client.27247 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:50.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.632+0000 7f5a1f7fe700 1 -- 172.21.15.111:0/3936196245 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e2a0332660 con 0x55e2a0481710 2023-10-11T16:41:50.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.632+0000 7f5a1f7fe700 10 client.27247 handle_command_reply: tid=0 2023-10-11T16:41:50.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.632+0000 7f5a5a259b80 1 client.27247 shutdown 2023-10-11T16:41:50.666 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 -- 172.21.15.111:0/3936196245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 msgr2=0x55e2a041af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041af90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5a380991e0 tx=0x7f5a38099640 comp rx=0 tx=0).stop 2023-10-11T16:41:50.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 -- 172.21.15.111:0/3936196245 shutdown_connections 2023-10-11T16:41:50.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55e2a0481710 0x55e2a0483b20 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.667 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55e2a04178b0 0x55e2a041af90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e2a0412b60 0x55e2a041bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a1f7fe700 0 client.27247 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:41:50.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 --2- 172.21.15.111:0/3936196245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2a0413470 0x55e2a041aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 -- 172.21.15.111:0/3936196245 >> 172.21.15.111:0/3936196245 conn(0x55e2a0408360 msgr2=0x55e2a04158f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 -- 172.21.15.111:0/3936196245 shutdown_connections 2023-10-11T16:41:50.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 1 -- 172.21.15.111:0/3936196245 wait complete. 2023-10-11T16:41:50.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a5a259b80 20 client.27247 trim_cache size 0 max 16384 2023-10-11T16:41:50.670 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:41:50.670 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:41:50.671 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:41:50.671 INFO:teuthology.orchestra.run.smithi111.stdout: "5a3a0797-2ff6-48e9-aeef-c4ee92b420b8": { 2023-10-11T16:41:50.671 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:41:50.672 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "5a3a0797-2ff6-48e9-aeef-c4ee92b420b8", 2023-10-11T16:41:50.672 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:41:50.672 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:41:50.672 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:41:50.673 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:41:50.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a469ff700 1 -- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5a2c0a12c0 msgr2=0x7f5a2c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a469ff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5a2c0a12c0 0x7f5a2c0a3770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5a4019e010 tx=0x7f5a28007f80 comp rx=0 tx=0).stop 2023-10-11T16:41:50.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a469ff700 1 -- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 msgr2=0x7f5a4019c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:41:50.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a469ff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 0x7f5a4019c7e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5a38007ee0 tx=0x7f5a3800d3b0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.675 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.633+0000 7f5a469ff700 1 -- 172.21.15.111:0/1793479656 shutdown_connections 2023-10-11T16:41:50.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.634+0000 7f5a469ff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f5a2c0a12c0 0x7f5a2c0a3770 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.634+0000 7f5a469ff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a40074400 0x7f5a4019c7e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.634+0000 7f5a469ff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a40072d20 0x7f5a401a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.676 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.634+0000 7f5a469ff700 1 --2- 172.21.15.111:0/1793479656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a400736d0 0x7f5a401a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:41:50.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.634+0000 7f5a469ff700 1 -- 172.21.15.111:0/1793479656 >> 172.21.15.111:0/1793479656 conn(0x7f5a40077590 msgr2=0x7f5a40112750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:41:50.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.634+0000 7f5a469ff700 1 -- 172.21.15.111:0/1793479656 shutdown_connections 2023-10-11T16:41:50.677 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:41:50.634+0000 7f5a469ff700 1 -- 172.21.15.111:0/1793479656 wait complete. 2023-10-11T16:41:50.678 INFO:tasks.cephfs.filesystem:scrub status for tag:5a3a0797-2ff6-48e9-aeef-c4ee92b420b8 - {'path': '/', 'tag': '5a3a0797-2ff6-48e9-aeef-c4ee92b420b8', 'options': 'recursive,force'} 2023-10-11T16:41:51.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:51 smithi130 ceph-mon[131859]: pgmap v285: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:51.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:51 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:41:51.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:51 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:51.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:51 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:41:51.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:51 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:41:51.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:51 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:51.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:51 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:51.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:51 smithi111 ceph-mon[98527]: pgmap v285: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:51.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:51 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:41:51.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:51 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:51.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:51 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:41:51.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:51 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:41:51.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:51 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:51.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:51 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:51 smithi150 ceph-mon[132217]: pgmap v285: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:41:51.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:51 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:41:51.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:51 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:51.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:51 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:41:51.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:51 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:41:51.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:51 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:51.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:51 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:41:53.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:53 smithi130 ceph-mon[131859]: pgmap v286: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:53.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:53 smithi111 ceph-mon[98527]: pgmap v286: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:53.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:53 smithi150 ceph-mon[132217]: pgmap v286: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:41:55.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:55 smithi130 ceph-mon[131859]: pgmap v287: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2023-10-11T16:41:55.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:55 smithi111 ceph-mon[98527]: pgmap v287: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2023-10-11T16:41:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:55 smithi150 ceph-mon[132217]: pgmap v287: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2023-10-11T16:41:56.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:56 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:41:56.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:56 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:41:56.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:56 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:41:56.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:56 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:41:56.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:56 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:41:56.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:56 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:41:57.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:57 smithi130 ceph-mon[131859]: pgmap v288: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:41:57.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:57 smithi111 ceph-mon[98527]: pgmap v288: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:41:57.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:57 smithi150 ceph-mon[132217]: pgmap v288: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:41:59.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:41:59 smithi130 ceph-mon[131859]: pgmap v289: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:41:59.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:41:59 smithi111 ceph-mon[98527]: pgmap v289: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:41:59.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:41:59 smithi150 ceph-mon[132217]: pgmap v289: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:42:01.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:01 smithi130 ceph-mon[131859]: pgmap v290: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:42:01.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:01 smithi111 ceph-mon[98527]: pgmap v290: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:42:01.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:01 smithi150 ceph-mon[132217]: pgmap v290: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2023-10-11T16:42:03.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:03 smithi130 ceph-mon[131859]: pgmap v291: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:42:03.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:03 smithi111 ceph-mon[98527]: pgmap v291: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:42:03.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:03 smithi150 ceph-mon[132217]: pgmap v291: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:42:05.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:05 smithi130 ceph-mon[131859]: pgmap v292: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-11T16:42:05.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:05 smithi111 ceph-mon[98527]: pgmap v292: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-11T16:42:05.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:05 smithi150 ceph-mon[132217]: pgmap v292: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-11T16:42:07.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:07 smithi130 ceph-mon[131859]: pgmap v293: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-11T16:42:07.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:07 smithi111 ceph-mon[98527]: pgmap v293: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-11T16:42:07.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:07 smithi150 ceph-mon[132217]: pgmap v293: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-11T16:42:09.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:09 smithi130 ceph-mon[131859]: pgmap v294: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:42:09.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:09 smithi111 ceph-mon[98527]: pgmap v294: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:42:09.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:09 smithi150 ceph-mon[132217]: pgmap v294: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-11T16:42:11.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:11 smithi130 ceph-mon[131859]: pgmap v295: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:42:11.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:11 smithi111 ceph-mon[98527]: pgmap v295: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:42:11.956 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:11 smithi150 ceph-mon[132217]: pgmap v295: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-10-11T16:42:12.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:12 smithi130 ceph-mon[131859]: osdmap e99: 12 total, 12 up, 12 in 2023-10-11T16:42:12.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:12 smithi111 ceph-mon[98527]: osdmap e99: 12 total, 12 up, 12 in 2023-10-11T16:42:12.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:12 smithi150 ceph-mon[132217]: osdmap e99: 12 total, 12 up, 12 in 2023-10-11T16:42:13.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:13 smithi130 ceph-mon[131859]: pgmap v297: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:42:13.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:13 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:42:13.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:13 smithi111 ceph-mon[98527]: pgmap v297: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:42:13.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:13 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:42:13.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:13 smithi150 ceph-mon[132217]: pgmap v297: 129 pgs: 129 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-11T16:42:13.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:13 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:42:15.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:15 smithi111 ceph-mon[98527]: pgmap v298: 129 pgs: 35 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 82 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:42:15.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:15 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:42:15.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:15 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:42:15.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:15 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:42:15.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:15 smithi150 ceph-mon[132217]: pgmap v298: 129 pgs: 35 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 82 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:42:15.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:15 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:42:15.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:15 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:42:15.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:15 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:42:16.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:15 smithi130 ceph-mon[131859]: pgmap v298: 129 pgs: 35 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 82 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:42:16.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:15 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:42:16.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:15 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:42:16.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:15 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:42:16.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:16 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:42:16.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:16 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:42:16.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:16 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:42:16.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:16 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:42:16.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:16 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:42:16.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:16 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:42:16.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:16 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:42:16.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:16 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:42:17.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:16 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:42:17.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:16 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:42:17.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:16 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:42:17.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:16 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:42:17.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:17 smithi111 ceph-mon[98527]: pgmap v299: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:17.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:17 smithi150 ceph-mon[132217]: pgmap v299: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:18.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:17 smithi130 ceph-mon[131859]: pgmap v299: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:19.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:19 smithi111 ceph-mon[98527]: pgmap v300: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:19 smithi150 ceph-mon[132217]: pgmap v300: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:20.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:19 smithi130 ceph-mon[131859]: pgmap v300: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:20.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:42:21.053 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.051+0000 7fe5473ff700 1 -- 172.21.15.111:0/2987569561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 msgr2=0x7fe54010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.051+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2987569561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 0x7fe54010ccd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe53800b0a0 tx=0x7fe53800b3b0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.054 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.052+0000 7fe5473ff700 1 -- 172.21.15.111:0/2987569561 shutdown_connections 2023-10-11T16:42:21.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.052+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2987569561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 0x7fe54010ccd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.052+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2987569561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe540101390 0x7fe540101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.055 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.052+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2987569561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe540101ca0 0x7fe54010a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.052+0000 7fe5473ff700 1 -- 172.21.15.111:0/2987569561 >> 172.21.15.111:0/2987569561 conn(0x7fe5400770b0 msgr2=0x7fe5400774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:21.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.052+0000 7fe5473ff700 1 -- 172.21.15.111:0/2987569561 shutdown_connections 2023-10-11T16:42:21.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.052+0000 7fe5473ff700 1 -- 172.21.15.111:0/2987569561 wait complete. 2023-10-11T16:42:21.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.053+0000 7fe5473ff700 1 Processor -- start 2023-10-11T16:42:21.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.054+0000 7fe5473ff700 1 -- start start 2023-10-11T16:42:21.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.054+0000 7fe5473ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe540101390 0x7fe5400ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.054+0000 7fe5473ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe540101ca0 0x7fe5400ff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.054+0000 7fe5473ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 0x7fe5401a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe5473ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400fff30 con 0x7fe540101ca0 2023-10-11T16:42:21.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe5473ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5401a4c50 con 0x7fe54010a8b0 2023-10-11T16:42:21.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe5473ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5401a4d90 con 0x7fe540101390 2023-10-11T16:42:21.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe54599c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 0x7fe5401a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe54599c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 0x7fe5401a4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54804/0 (socket says 172.21.15.111:54804) 2023-10-11T16:42:21.060 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe54599c700 1 -- 172.21.15.111:0/2712239219 learned_addr learned my addr 172.21.15.111:0/2712239219 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:21.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe54499a700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe540101ca0 0x7fe5400ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe54599c700 1 -- 172.21.15.111:0/2712239219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe540101390 msgr2=0x7fe5400ff1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.055+0000 7fe54599c700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe540101390 0x7fe5400ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.056+0000 7fe54599c700 1 -- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe540101ca0 msgr2=0x7fe5400ff700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.056+0000 7fe54599c700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe540101ca0 0x7fe5400ff700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.056+0000 7fe54599c700 1 -- 172.21.15.111:0/2712239219 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe538009d00 con 0x7fe54010a8b0 2023-10-11T16:42:21.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.056+0000 7fe54599c700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 0x7fe5401a4710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe53800b070 tx=0x7fe5380095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:21.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.058+0000 7fe5367fc700 1 -- 172.21.15.111:0/2712239219 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53800e040 con 0x7fe54010a8b0 2023-10-11T16:42:21.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.058+0000 7fe5367fc700 1 -- 172.21.15.111:0/2712239219 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe538004d60 con 0x7fe54010a8b0 2023-10-11T16:42:21.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.058+0000 7fe5367fc700 1 -- 172.21.15.111:0/2712239219 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53801b450 con 0x7fe54010a8b0 2023-10-11T16:42:21.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.058+0000 7fe5473ff700 1 -- 172.21.15.111:0/2712239219 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5401103d0 con 0x7fe54010a8b0 2023-10-11T16:42:21.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.059+0000 7fe5473ff700 1 -- 172.21.15.111:0/2712239219 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe540110240 con 0x7fe54010a8b0 2023-10-11T16:42:21.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.069+0000 7fe5367fc700 1 -- 172.21.15.111:0/2712239219 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fe538003a00 con 0x7fe54010a8b0 2023-10-11T16:42:21.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.071+0000 7fe5367fc700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe52c0a1410 0x7fe52c0a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.071+0000 7fe55ad18b80 1 Processor -- start 2023-10-11T16:42:21.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.072+0000 7fe55ad18b80 1 -- start start 2023-10-11T16:42:21.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.072+0000 7fe5367fc700 1 -- 172.21.15.111:0/2712239219 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fe5380ce860 con 0x7fe54010a8b0 2023-10-11T16:42:21.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.072+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9930d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.072+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b9943280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.072+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9945ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.072+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9836060 con 0x5654b9930970 2023-10-11T16:42:21.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.072+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b98888a0 con 0x5654b9931280 2023-10-11T16:42:21.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.072+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9881d30 con 0x5654b99437c0 2023-10-11T16:42:21.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54599c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9945ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54599c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9945ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43318/0 (socket says 172.21.15.111:43318) 2023-10-11T16:42:21.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54499a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b9943280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54599c700 1 -- 172.21.15.111:0/2670027840 learned_addr learned my addr 172.21.15.111:0/2670027840 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:21.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54519b700 1 --2- 172.21.15.111:0/2670027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9930d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54599c700 1 -- 172.21.15.111:0/2670027840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 msgr2=0x5654b9943280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:21.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54599c700 1 --2- 172.21.15.111:0/2670027840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b9943280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54599c700 1 -- 172.21.15.111:0/2670027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 msgr2=0x5654b9930d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:21.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54599c700 1 --2- 172.21.15.111:0/2670027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9930d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.073+0000 7fe54599c700 1 -- 172.21.15.111:0/2670027840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654b9703630 con 0x5654b99437c0 2023-10-11T16:42:21.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.074+0000 7fe54519b700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe52c0a1410 0x7fe52c0a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.074+0000 7fe54599c700 1 --2- 172.21.15.111:0/2670027840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9945ba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe538097fd0 tx=0x7fe5380978a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a593fa9696042a09 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:21.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.074+0000 7fe52b7fe700 1 -- 172.21.15.111:0/2670027840 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53809aec0 con 0x5654b99437c0 2023-10-11T16:42:21.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.075+0000 7fe52b7fe700 1 -- 172.21.15.111:0/2670027840 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe538019350 con 0x5654b99437c0 2023-10-11T16:42:21.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.075+0000 7fe52b7fe700 1 -- 172.21.15.111:0/2670027840 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5380cbdd0 con 0x5654b99437c0 2023-10-11T16:42:21.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.076+0000 7fe55ad18b80 1 -- 172.21.15.111:0/2670027840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 msgr2=0x5654b9945ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.076+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/2670027840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9945ba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe538097fd0 tx=0x7fe5380978a0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.076+0000 7fe55ad18b80 1 -- 172.21.15.111:0/2670027840 shutdown_connections 2023-10-11T16:42:21.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.076+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/2670027840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9945ba0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.076+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/2670027840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9930d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.076+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/2670027840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b9943280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.076+0000 7fe55ad18b80 1 -- 172.21.15.111:0/2670027840 >> 172.21.15.111:0/2670027840 conn(0x5654b9926360 msgr2=0x5654b992ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:21.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.076+0000 7fe54519b700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe52c0a1410 0x7fe52c0a38c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe53c005950 tx=0x7fe53c00d320 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:21.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.077+0000 7fe55ad18b80 1 -- 172.21.15.111:0/2670027840 shutdown_connections 2023-10-11T16:42:21.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.077+0000 7fe55ad18b80 1 -- 172.21.15.111:0/2670027840 wait complete. 2023-10-11T16:42:21.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.077+0000 7fe55ad18b80 1 -- start start 2023-10-11T16:42:21.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.078+0000 7fe55ad18b80 1 Processor -- start 2023-10-11T16:42:21.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.079+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9943b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.079+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b99440d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.080+0000 7fe54519b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9943b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.080+0000 7fe54519b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9943b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37592/0 (socket says 172.21.15.111:37592) 2023-10-11T16:42:21.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.080+0000 7fe54519b700 1 -- 172.21.15.111:0/3589819682 learned_addr learned my addr 172.21.15.111:0/3589819682 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:21.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.080+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9944650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.080+0000 7fe54499a700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b99440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.080+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b98238e0 con 0x5654b9930970 2023-10-11T16:42:21.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.080+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9823580 con 0x5654b9931280 2023-10-11T16:42:21.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.080+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9705920 con 0x5654b99437c0 2023-10-11T16:42:21.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.081+0000 7fe55ad18b80 10 client.0 _ll_register_callbacks cb 0x5654b9841780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:21.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.081+0000 7fe54599c700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9944650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.081+0000 7fe54499a700 1 -- 172.21.15.111:0/3589819682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 msgr2=0x5654b9944650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.082+0000 7fe54499a700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9944650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.082+0000 7fe54499a700 1 -- 172.21.15.111:0/3589819682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 msgr2=0x5654b9943b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:21.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.082+0000 7fe54499a700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9943b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.082+0000 7fe54499a700 1 -- 172.21.15.111:0/3589819682 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe540110240 con 0x5654b9931280 2023-10-11T16:42:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.082+0000 7fe54599c700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9944650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:42:21.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.082+0000 7fe54519b700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9943b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:42:21.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.082+0000 7fe54499a700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b99440d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5401009b0 tx=0x7fe530009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c578052d7f7ec7e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:21.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.082+0000 7fe528ff9700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:42:21.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.083+0000 7fe528ff9700 1 -- 172.21.15.111:0/3589819682 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530014440 con 0x5654b9931280 2023-10-11T16:42:21.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.083+0000 7fe528ff9700 1 -- 172.21.15.111:0/3589819682 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fe5300145a0 con 0x5654b9931280 2023-10-11T16:42:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.083+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654b9923b20 con 0x5654b9931280 2023-10-11T16:42:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.084+0000 7fe528ff9700 1 -- 172.21.15.111:0/3589819682 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe530014760 con 0x5654b9931280 2023-10-11T16:42:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.084+0000 7fe55ad18b80 10 client.27251 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:21.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.084+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5654b9848190 con 0x5654b9931280 2023-10-11T16:42:21.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.084+0000 7fe528ff9700 1 -- 172.21.15.111:0/3589819682 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5654b9923b20 con 0x5654b9931280 2023-10-11T16:42:21.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.084+0000 7fe528ff9700 1 -- 172.21.15.111:0/3589819682 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe53001b020 con 0x5654b9931280 2023-10-11T16:42:21.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.084+0000 7fe55ad18b80 10 client.27251 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:21.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.084+0000 7fe55ad18b80 10 client.27251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:21.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.085+0000 7fe55ad18b80 20 client.27251 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:21.100 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.085+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5654b9952ea0 0x5654b99552b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.085+0000 7fe55ad18b80 4 client.27251 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:21.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.085+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654b98440a0 con 0x5654b9952ea0 2023-10-11T16:42:21.101 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.085+0000 7fe54519b700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5654b9952ea0 0x5654b99552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.086+0000 7fe528ff9700 10 client.27251 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.086+0000 7fe54519b700 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5654b9952ea0 0x5654b99552b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d27bded1bad74719 server_cookie=4f82c7e9681b9e27 in_seq=0 out_seq=0 2023-10-11T16:42:21.102 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.086+0000 7fe528ff9700 1 -- 172.21.15.111:0/3589819682 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5654b98440a0 con 0x5654b9952ea0 2023-10-11T16:42:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.087+0000 7fe528ff9700 10 client.27251 handle_command_reply: tid=0 2023-10-11T16:42:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.087+0000 7fe55ad18b80 1 client.27251 shutdown 2023-10-11T16:42:21.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 msgr2=0x5654b99440d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b99440d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5401009b0 tx=0x7fe530009f90 comp rx=0 tx=0).stop 2023-10-11T16:42:21.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 shutdown_connections 2023-10-11T16:42:21.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5654b9952ea0 0x5654b99552b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b9944650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b9943b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/3589819682 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9931280 0x5654b99440d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 >> 172.21.15.111:0/3589819682 conn(0x5654b9926360 msgr2=0x5654b992ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:21.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe528ff9700 0 client.27251 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:21.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 shutdown_connections 2023-10-11T16:42:21.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.088+0000 7fe55ad18b80 1 -- 172.21.15.111:0/3589819682 wait complete. 2023-10-11T16:42:21.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.089+0000 7fe55ad18b80 20 client.27251 trim_cache size 0 max 16384 2023-10-11T16:42:21.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.126+0000 7fe55ad18b80 1 Processor -- start 2023-10-11T16:42:21.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.127+0000 7fe55ad18b80 1 -- start start 2023-10-11T16:42:21.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.127+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9930970 0x5654b9948670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.127+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9931280 0x5654b9943b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.127+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b99440d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.127+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9932470 con 0x5654b9931280 2023-10-11T16:42:21.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.127+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9931b90 con 0x5654b9930970 2023-10-11T16:42:21.130 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.127+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b99505e0 con 0x5654b99437c0 2023-10-11T16:42:21.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.127+0000 7fe54519b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9930970 0x5654b9948670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54519b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9930970 0x5654b9948670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54832/0 (socket says 172.21.15.111:54832) 2023-10-11T16:42:21.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54519b700 1 -- 172.21.15.111:0/274927686 learned_addr learned my addr 172.21.15.111:0/274927686 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:21.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54599c700 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b99440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54519b700 1 -- 172.21.15.111:0/274927686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 msgr2=0x5654b99440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:21.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54519b700 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b99440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54519b700 1 -- 172.21.15.111:0/274927686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9931280 msgr2=0x5654b9943b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:21.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54519b700 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9931280 0x5654b9943b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54519b700 1 -- 172.21.15.111:0/274927686 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654b99423d0 con 0x5654b9930970 2023-10-11T16:42:21.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.128+0000 7fe54599c700 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b99440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:21.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.129+0000 7fe54519b700 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9930970 0x5654b9948670 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00da50 tx=0x7fe53c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97a98c4519cc420e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:21.135 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.129+0000 7fe52bfff700 1 -- 172.21.15.111:0/274927686 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c024070 con 0x5654b9930970 2023-10-11T16:42:21.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.129+0000 7fe52bfff700 1 -- 172.21.15.111:0/274927686 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe53c015070 con 0x5654b9930970 2023-10-11T16:42:21.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.129+0000 7fe52bfff700 1 -- 172.21.15.111:0/274927686 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c0119d0 con 0x5654b9930970 2023-10-11T16:42:21.136 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 -- 172.21.15.111:0/274927686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9930970 msgr2=0x5654b9948670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9930970 0x5654b9948670 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00da50 tx=0x7fe53c013000 comp rx=0 tx=0).stop 2023-10-11T16:42:21.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 -- 172.21.15.111:0/274927686 shutdown_connections 2023-10-11T16:42:21.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b99437c0 0x5654b99440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.137 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b9930970 0x5654b9948670 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/274927686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9931280 0x5654b9943b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 -- 172.21.15.111:0/274927686 >> 172.21.15.111:0/274927686 conn(0x5654b9926360 msgr2=0x5654b993d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:21.138 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 -- 172.21.15.111:0/274927686 shutdown_connections 2023-10-11T16:42:21.139 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.131+0000 7fe55ad18b80 1 -- 172.21.15.111:0/274927686 wait complete. 2023-10-11T16:42:21.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.132+0000 7fe55ad18b80 1 -- start start 2023-10-11T16:42:21.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.132+0000 7fe55ad18b80 1 Processor -- start 2023-10-11T16:42:21.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.134+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b999f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.141 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.134+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b9931280 0x5654b999fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.135+0000 7fe55ad18b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b99437c0 0x5654b9947bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.135+0000 7fe54519b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b999f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.135+0000 7fe54519b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b999f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37608/0 (socket says 172.21.15.111:37608) 2023-10-11T16:42:21.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.135+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9948100 con 0x5654b9930970 2023-10-11T16:42:21.143 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.135+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9948270 con 0x5654b99437c0 2023-10-11T16:42:21.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.135+0000 7fe55ad18b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b9948430 con 0x5654b9931280 2023-10-11T16:42:21.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.135+0000 7fe54519b700 1 -- 172.21.15.111:0/413044985 learned_addr learned my addr 172.21.15.111:0/413044985 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:21.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.135+0000 7fe55ad18b80 10 client.0 _ll_register_callbacks cb 0x7fe530009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:21.144 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.136+0000 7fe54519b700 1 -- 172.21.15.111:0/413044985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b9931280 msgr2=0x5654b999fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:21.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.136+0000 7fe54519b700 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b9931280 0x5654b999fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.136+0000 7fe54519b700 1 -- 172.21.15.111:0/413044985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b99437c0 msgr2=0x5654b9947bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:21.145 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.136+0000 7fe54519b700 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b99437c0 0x5654b9947bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.136+0000 7fe54519b700 1 -- 172.21.15.111:0/413044985 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c010cd0 con 0x5654b9930970 2023-10-11T16:42:21.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.136+0000 7fe54519b700 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b999f500 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe53c000c00 tx=0x7fe53c025e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d38b7b19890b47 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:21.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.136+0000 7fe528ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:21.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.137+0000 7fe528ff9700 1 -- 172.21.15.111:0/413044985 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c024050 con 0x5654b9930970 2023-10-11T16:42:21.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.137+0000 7fe528ff9700 1 -- 172.21.15.111:0/413044985 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fe53c017070 con 0x5654b9930970 2023-10-11T16:42:21.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.137+0000 7fe55ad18b80 1 -- 172.21.15.111:0/413044985 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe538019350 con 0x5654b9930970 2023-10-11T16:42:21.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.138+0000 7fe528ff9700 1 -- 172.21.15.111:0/413044985 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe53c004450 con 0x5654b9930970 2023-10-11T16:42:21.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.138+0000 7fe55ad18b80 10 client.17112 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:21.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.138+0000 7fe55ad18b80 1 -- 172.21.15.111:0/413044985 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5654b96afde0 con 0x5654b9930970 2023-10-11T16:42:21.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.139+0000 7fe528ff9700 1 -- 172.21.15.111:0/413044985 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe538019350 con 0x5654b9930970 2023-10-11T16:42:21.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.139+0000 7fe528ff9700 1 -- 172.21.15.111:0/413044985 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe53c037070 con 0x5654b9930970 2023-10-11T16:42:21.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.139+0000 7fe55ad18b80 10 client.17112 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:21.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.139+0000 7fe55ad18b80 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:21.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.139+0000 7fe55ad18b80 20 client.17112 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:21.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.139+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5654b99b8820 0x5654b99bac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.140+0000 7fe55ad18b80 4 client.17112 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:42:21.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.140+0000 7fe55ad18b80 1 -- 172.21.15.111:0/413044985 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5654b9850660 con 0x5654b99b8820 2023-10-11T16:42:21.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.141+0000 7fe54499a700 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5654b99b8820 0x5654b99bac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.142+0000 7fe54499a700 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5654b99b8820 0x5654b99bac30 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4be7644ad936b323 server_cookie=b99a5ee32530a3d6 in_seq=0 out_seq=0 2023-10-11T16:42:21.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.142+0000 7fe528ff9700 10 client.17112 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:21.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.142+0000 7fe528ff9700 1 -- 172.21.15.111:0/413044985 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5654b9850660 con 0x5654b99b8820 2023-10-11T16:42:21.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.142+0000 7fe528ff9700 10 client.17112 handle_command_reply: tid=0 2023-10-11T16:42:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.142+0000 7fe55ad18b80 1 client.17112 shutdown 2023-10-11T16:42:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.143+0000 7fe55ad18b80 1 -- 172.21.15.111:0/413044985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 msgr2=0x5654b999f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.143+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b999f500 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe53c000c00 tx=0x7fe53c025e90 comp rx=0 tx=0).stop 2023-10-11T16:42:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe55ad18b80 1 -- 172.21.15.111:0/413044985 shutdown_connections 2023-10-11T16:42:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x5654b99b8820 0x5654b99bac30 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654b99437c0 0x5654b9947bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5654b9930970 0x5654b999f500 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe55ad18b80 1 --2- 172.21.15.111:0/413044985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5654b9931280 0x5654b999fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe55ad18b80 1 -- 172.21.15.111:0/413044985 >> 172.21.15.111:0/413044985 conn(0x5654b9926360 msgr2=0x5654b993dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:21.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe528ff9700 0 client.17112 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:21.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe55ad18b80 1 -- 172.21.15.111:0/413044985 shutdown_connections 2023-10-11T16:42:21.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.144+0000 7fe55ad18b80 1 -- 172.21.15.111:0/413044985 wait complete. 2023-10-11T16:42:21.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.145+0000 7fe55ad18b80 20 client.17112 trim_cache size 0 max 16384 2023-10-11T16:42:21.157 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:42:21.157 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:42:21.157 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:42:21.158 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:42:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.146+0000 7fe5473ff700 1 -- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe52c0a1410 msgr2=0x7fe52c0a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.146+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe52c0a1410 0x7fe52c0a38c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe53c005950 tx=0x7fe53c00d320 comp rx=0 tx=0).stop 2023-10-11T16:42:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.146+0000 7fe5473ff700 1 -- 172.21.15.111:0/2712239219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 msgr2=0x7fe5401a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.146+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 0x7fe5401a4710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe53800b070 tx=0x7fe5380095c0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.147+0000 7fe5473ff700 1 -- 172.21.15.111:0/2712239219 shutdown_connections 2023-10-11T16:42:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.147+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe52c0a1410 0x7fe52c0a38c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.147+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe54010a8b0 0x7fe5401a4710 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.147+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe540101390 0x7fe5400ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.147+0000 7fe5473ff700 1 --2- 172.21.15.111:0/2712239219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe540101ca0 0x7fe5400ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.147+0000 7fe5473ff700 1 -- 172.21.15.111:0/2712239219 >> 172.21.15.111:0/2712239219 conn(0x7fe5400770b0 msgr2=0x7fe540102290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.148+0000 7fe5473ff700 1 -- 172.21.15.111:0/2712239219 shutdown_connections 2023-10-11T16:42:21.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.148+0000 7fe5473ff700 1 -- 172.21.15.111:0/2712239219 wait complete. 2023-10-11T16:42:21.183 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:42:21.183 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:42:21.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.537+0000 7f73dabff700 1 -- 172.21.15.111:0/4091745867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 msgr2=0x7f73d4106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.537+0000 7f73dabff700 1 --2- 172.21.15.111:0/4091745867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 0x7f73d4106960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0098f0 tx=0x7f73bc009c00 comp rx=0 tx=0).stop 2023-10-11T16:42:21.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.537+0000 7f73dabff700 1 -- 172.21.15.111:0/4091745867 shutdown_connections 2023-10-11T16:42:21.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.537+0000 7f73dabff700 1 --2- 172.21.15.111:0/4091745867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d4107090 0x7f73d41136e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.537+0000 7f73dabff700 1 --2- 172.21.15.111:0/4091745867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73d410c920 0x7f73d410ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.537+0000 7f73dabff700 1 --2- 172.21.15.111:0/4091745867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 0x7f73d4106960 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.537+0000 7f73dabff700 1 -- 172.21.15.111:0/4091745867 >> 172.21.15.111:0/4091745867 conn(0x7f73d40738c0 msgr2=0x7f73d4075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:21.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.538+0000 7f73dabff700 1 -- 172.21.15.111:0/4091745867 shutdown_connections 2023-10-11T16:42:21.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.538+0000 7f73dabff700 1 -- 172.21.15.111:0/4091745867 wait complete. 2023-10-11T16:42:21.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.539+0000 7f73dabff700 1 Processor -- start 2023-10-11T16:42:21.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.540+0000 7f73dabff700 1 -- start start 2023-10-11T16:42:21.543 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.540+0000 7f73dabff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73d4107090 0x7f73d4079e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.540+0000 7f73dabff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d410c920 0x7f73d407a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.540+0000 7f73dabff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 0x7f73d41b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.541+0000 7f73dabff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d41b14f0 con 0x7f73d410c920 2023-10-11T16:42:21.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.541+0000 7f73d899b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73d4107090 0x7f73d4079e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.541+0000 7f73d899b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73d4107090 0x7f73d4079e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43358/0 (socket says 172.21.15.111:43358) 2023-10-11T16:42:21.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.541+0000 7f73d919c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 0x7f73d41b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.541+0000 7f73d899b700 1 -- 172.21.15.111:0/1853433079 learned_addr learned my addr 172.21.15.111:0/1853433079 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:21.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.541+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d41b1660 con 0x7f73d410d230 2023-10-11T16:42:21.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.541+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d41b1820 con 0x7f73d4107090 2023-10-11T16:42:21.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.542+0000 7f73d3fff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d410c920 0x7f73d407a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.547 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.543+0000 7f73d3fff700 1 -- 172.21.15.111:0/1853433079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73d4107090 msgr2=0x7f73d4079e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.543+0000 7f73d3fff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73d4107090 0x7f73d4079e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.543+0000 7f73d3fff700 1 -- 172.21.15.111:0/1853433079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 msgr2=0x7f73d41b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.543+0000 7f73d3fff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 0x7f73d41b0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.543+0000 7f73d3fff700 1 -- 172.21.15.111:0/1853433079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc009520 con 0x7f73d410c920 2023-10-11T16:42:21.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.543+0000 7f73d3fff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d410c920 0x7f73d407a390 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00b2e0 tx=0x7f73bc00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:21.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.544+0000 7f73d919c700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 0x7f73d41b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:21.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.544+0000 7f73d1ffb700 1 -- 172.21.15.111:0/1853433079 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc01c070 con 0x7f73d410c920 2023-10-11T16:42:21.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.544+0000 7f73d1ffb700 1 -- 172.21.15.111:0/1853433079 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73bc022420 con 0x7f73d410c920 2023-10-11T16:42:21.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.545+0000 7f73d899b700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73d4107090 0x7f73d4079e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:21.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.545+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d4116a00 con 0x7f73d410c920 2023-10-11T16:42:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.545+0000 7f73d1ffb700 1 -- 172.21.15.111:0/1853433079 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc017660 con 0x7f73d410c920 2023-10-11T16:42:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.545+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d4116870 con 0x7f73d410c920 2023-10-11T16:42:21.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.548+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d4116b90 con 0x7f73d410c920 2023-10-11T16:42:21.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.555+0000 7f73d1ffb700 1 -- 172.21.15.111:0/1853433079 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f73bc020070 con 0x7f73d410c920 2023-10-11T16:42:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.562+0000 7f73d1ffb700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f73c00a1410 0x7f73c00a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:21.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.562+0000 7f73d1ffb700 1 -- 172.21.15.111:0/1853433079 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f73bc0ce2d0 con 0x7f73d410c920 2023-10-11T16:42:21.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.563+0000 7f73d899b700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f73c00a1410 0x7f73c00a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:21.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.566+0000 7f73d1ffb700 1 -- 172.21.15.111:0/1853433079 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f73bc05c730 con 0x7f73d410c920 2023-10-11T16:42:21.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.566+0000 7f73d899b700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f73c00a1410 0x7f73c00a38c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009cd0 tx=0x7f73cc0092a0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:21.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:21 smithi111 ceph-mon[98527]: pgmap v301: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:21.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:21 smithi150 ceph-mon[132217]: pgmap v301: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:21.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.958+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73d41b19e0 con 0x7f73d410c920 2023-10-11T16:42:21.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.962+0000 7f73d1ffb700 1 -- 172.21.15.111:0/1853433079 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+10820 (secure 0 0 0) 0x7f73bc027090 con 0x7f73d410c920 2023-10-11T16:42:21.966 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:42:21.967 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:41:48.484142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:42:21.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f73c00a1410 msgr2=0x7f73c00a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f73c00a1410 0x7f73c00a38c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009cd0 tx=0x7f73cc0092a0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d410c920 msgr2=0x7f73d407a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:21.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d410c920 0x7f73d407a390 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f73bc00b2e0 tx=0x7f73bc00b6c0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 shutdown_connections 2023-10-11T16:42:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f73c00a1410 0x7f73c00a38c0 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009cd0 tx=0x7f73cc0092a0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73d410d230 0x7f73d41b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73d4107090 0x7f73d4079e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 --2- 172.21.15.111:0/1853433079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73d410c920 0x7f73d407a390 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 >> 172.21.15.111:0/1853433079 conn(0x7f73d40738c0 msgr2=0x7f73d4074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:21.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 shutdown_connections 2023-10-11T16:42:21.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:21.966+0000 7f73dabff700 1 -- 172.21.15.111:0/1853433079 wait complete. 2023-10-11T16:42:21.973 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 39 2023-10-11T16:42:22.002 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:42:22.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:21 smithi130 ceph-mon[131859]: pgmap v301: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:42:22.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.370+0000 7fbb2759e700 1 -- 172.21.15.111:0/1976471347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb2009a6f0 msgr2=0x7fbb200a22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.370+0000 7fbb2759e700 1 --2- 172.21.15.111:0/1976471347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb2009a6f0 0x7fbb200a22c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbb280669a0 tx=0x7fbb28066fd0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.370+0000 7fbb2759e700 1 -- 172.21.15.111:0/1976471347 shutdown_connections 2023-10-11T16:42:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.370+0000 7fbb2759e700 1 --2- 172.21.15.111:0/1976471347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb2009a6f0 0x7fbb200a22c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.370+0000 7fbb2759e700 1 --2- 172.21.15.111:0/1976471347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb200956b0 0x7fbb20095a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.370+0000 7fbb2759e700 1 --2- 172.21.15.111:0/1976471347 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb20096050 0x7fbb2009a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.370+0000 7fbb2759e700 1 -- 172.21.15.111:0/1976471347 >> 172.21.15.111:0/1976471347 conn(0x7fbb2000c230 msgr2=0x7fbb2000c630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.370+0000 7fbb2759e700 1 -- 172.21.15.111:0/1976471347 shutdown_connections 2023-10-11T16:42:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.371+0000 7fbb2759e700 1 -- 172.21.15.111:0/1976471347 wait complete. 2023-10-11T16:42:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.372+0000 7fbb2759e700 1 Processor -- start 2023-10-11T16:42:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.373+0000 7fbb2759e700 1 -- start start 2023-10-11T16:42:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.373+0000 7fbb2759e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb200956b0 0x7fbb2013c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.373+0000 7fbb2759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb20096050 0x7fbb2013ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.373+0000 7fbb2759e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb2009a6f0 0x7fbb20140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.373+0000 7fbb2759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20141410 con 0x7fbb20096050 2023-10-11T16:42:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.373+0000 7fbb2759e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20141580 con 0x7fbb2009a6f0 2023-10-11T16:42:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.373+0000 7fbb2759e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20141740 con 0x7fbb200956b0 2023-10-11T16:42:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.374+0000 7fbb25d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb20096050 0x7fbb2013ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.375+0000 7fbb2659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb200956b0 0x7fbb2013c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.375+0000 7fbb2659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb200956b0 0x7fbb2013c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43370/0 (socket says 172.21.15.111:43370) 2023-10-11T16:42:22.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.375+0000 7fbb2659c700 1 -- 172.21.15.111:0/3489889607 learned_addr learned my addr 172.21.15.111:0/3489889607 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.375+0000 7fbb25d9b700 1 -- 172.21.15.111:0/3489889607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb200956b0 msgr2=0x7fbb2013c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.375+0000 7fbb25d9b700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb200956b0 0x7fbb2013c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.375+0000 7fbb25d9b700 1 -- 172.21.15.111:0/3489889607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb2009a6f0 msgr2=0x7fbb20140ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.375+0000 7fbb25d9b700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb2009a6f0 0x7fbb20140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.375+0000 7fbb25d9b700 1 -- 172.21.15.111:0/3489889607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2804f760 con 0x7fbb20096050 2023-10-11T16:42:22.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.376+0000 7fbb25d9b700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb20096050 0x7fbb2013ca70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbb20096420 tx=0x7fbb1c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.377+0000 7fbb177fe700 1 -- 172.21.15.111:0/3489889607 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c00cbd0 con 0x7fbb20096050 2023-10-11T16:42:22.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.377+0000 7fbb177fe700 1 -- 172.21.15.111:0/3489889607 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb1c00ee80 con 0x7fbb20096050 2023-10-11T16:42:22.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.377+0000 7fbb177fe700 1 -- 172.21.15.111:0/3489889607 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c018680 con 0x7fbb20096050 2023-10-11T16:42:22.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.377+0000 7fbb2759e700 1 -- 172.21.15.111:0/3489889607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb200a54e0 con 0x7fbb20096050 2023-10-11T16:42:22.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.377+0000 7fbb2759e700 1 -- 172.21.15.111:0/3489889607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb200a5350 con 0x7fbb20096050 2023-10-11T16:42:22.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.385+0000 7fbb408a5b80 1 Processor -- start 2023-10-11T16:42:22.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.386+0000 7fbb408a5b80 1 -- start start 2023-10-11T16:42:22.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.386+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e3dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.386+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e37bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.386+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e38530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.386+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2d3d060 con 0x5610e2e38100 2023-10-11T16:42:22.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.386+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2d8f8a0 con 0x5610e2e3e1e0 2023-10-11T16:42:22.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.386+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2d88d30 con 0x5610e2e3d8d0 2023-10-11T16:42:22.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb26d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e3dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb26d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e3dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43376/0 (socket says 172.21.15.111:43376) 2023-10-11T16:42:22.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb26d9d700 1 -- 172.21.15.111:0/973386297 learned_addr learned my addr 172.21.15.111:0/973386297 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb2659c700 1 --2- 172.21.15.111:0/973386297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e37bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb26d9d700 1 -- 172.21.15.111:0/973386297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 msgr2=0x5610e2e37bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/973386297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e37bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb26d9d700 1 -- 172.21.15.111:0/973386297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 msgr2=0x5610e2e38530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:42:22.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/973386297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e38530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.387+0000 7fbb26d9d700 1 -- 172.21.15.111:0/973386297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610e2c0a630 con 0x5610e2e3d8d0 2023-10-11T16:42:22.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.388+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/973386297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e3dca0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbb28000eb0 tx=0x7fbb28066b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0e22e23df018ba1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.388+0000 7fbb177fe700 1 -- 172.21.15.111:0/3489889607 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fbb1c00cd30 con 0x7fbb20096050 2023-10-11T16:42:22.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.389+0000 7fbb14ff9700 1 -- 172.21.15.111:0/973386297 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2807a040 con 0x5610e2e3d8d0 2023-10-11T16:42:22.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.390+0000 7fbb14ff9700 1 -- 172.21.15.111:0/973386297 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb2807e530 con 0x5610e2e3d8d0 2023-10-11T16:42:22.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.390+0000 7fbb14ff9700 1 -- 172.21.15.111:0/973386297 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28072560 con 0x5610e2e3d8d0 2023-10-11T16:42:22.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.391+0000 7fbb408a5b80 1 -- 172.21.15.111:0/973386297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 msgr2=0x5610e2e3dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.391+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/973386297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e3dca0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbb28000eb0 tx=0x7fbb28066b10 comp rx=0 tx=0).stop 2023-10-11T16:42:22.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.391+0000 7fbb177fe700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fbb100a14d0 0x7fbb100a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.392+0000 7fbb177fe700 1 -- 172.21.15.111:0/3489889607 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fbb1c014070 con 0x7fbb20096050 2023-10-11T16:42:22.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.392+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fbb100a14d0 0x7fbb100a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.393+0000 7fbb408a5b80 1 -- 172.21.15.111:0/973386297 shutdown_connections 2023-10-11T16:42:22.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.393+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/973386297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e38530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.393+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/973386297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e3dca0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.393+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/973386297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e37bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.393+0000 7fbb408a5b80 1 -- 172.21.15.111:0/973386297 >> 172.21.15.111:0/973386297 conn(0x5610e2e2d5b0 msgr2=0x5610e2e32ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.394+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fbb100a14d0 0x7fbb100a3980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbb2806a010 tx=0x7fbb28072740 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.394+0000 7fbb408a5b80 1 -- 172.21.15.111:0/973386297 shutdown_connections 2023-10-11T16:42:22.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.394+0000 7fbb408a5b80 1 -- 172.21.15.111:0/973386297 wait complete. 2023-10-11T16:42:22.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.395+0000 7fbb408a5b80 1 -- start start 2023-10-11T16:42:22.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.395+0000 7fbb408a5b80 1 Processor -- start 2023-10-11T16:42:22.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.398+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e44150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.399+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e413d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.399+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e41a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.399+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2d2a8e0 con 0x5610e2e38100 2023-10-11T16:42:22.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.399+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2d2a580 con 0x5610e2e3e1e0 2023-10-11T16:42:22.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.399+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2c0c920 con 0x5610e2e3d8d0 2023-10-11T16:42:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.399+0000 7fbb26d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e41a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.399+0000 7fbb26d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e41a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54886/0 (socket says 172.21.15.111:54886) 2023-10-11T16:42:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.399+0000 7fbb26d9d700 1 -- 172.21.15.111:0/640163463 learned_addr learned my addr 172.21.15.111:0/640163463 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb25d9b700 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb26d9d700 1 -- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 msgr2=0x5610e2e413d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e413d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb26d9d700 1 -- 172.21.15.111:0/640163463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 msgr2=0x5610e2e44150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e44150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb26d9d700 1 -- 172.21.15.111:0/640163463 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb200a2b00 con 0x5610e2e3e1e0 2023-10-11T16:42:22.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb25d9b700 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:22.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb408a5b80 10 client.0 _ll_register_callbacks cb 0x5610e2d48780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:22.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.400+0000 7fbb408a5b80 1 -- 172.21.15.111:0/640163463 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5610e2e2ad70 con 0x5610e2e3e1e0 2023-10-11T16:42:22.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.402+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e41a50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbb280641e0 tx=0x7fbb2804eb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fbac7d44041f050 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.403+0000 7fbb0dffb700 10 client.27266 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:42:22.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.403+0000 7fbb0dffb700 1 -- 172.21.15.111:0/640163463 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2807a030 con 0x5610e2e3e1e0 2023-10-11T16:42:22.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.403+0000 7fbb0dffb700 1 -- 172.21.15.111:0/640163463 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fbb2806b690 con 0x5610e2e3e1e0 2023-10-11T16:42:22.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.403+0000 7fbb0dffb700 1 -- 172.21.15.111:0/640163463 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb2806b890 con 0x5610e2e3e1e0 2023-10-11T16:42:22.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.403+0000 7fbb408a5b80 10 client.27266 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:22.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.403+0000 7fbb408a5b80 1 -- 172.21.15.111:0/640163463 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=39}) v3 -- 0x5610e2d4f190 con 0x5610e2e3e1e0 2023-10-11T16:42:22.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.404+0000 7fbb0dffb700 1 -- 172.21.15.111:0/640163463 <== mon.2 v2:172.21.15.130:3300/0 4 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbb2806b9f0 con 0x5610e2e3e1e0 2023-10-11T16:42:22.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.404+0000 7fbb408a5b80 10 client.27266 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:22.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.404+0000 7fbb408a5b80 10 client.27266 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:42:22.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.404+0000 7fbb408a5b80 20 client.27266 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.404+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5610e2e512e0 0x5610e2e536c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.404+0000 7fbb408a5b80 4 client.27266 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.404+0000 7fbb408a5b80 1 -- 172.21.15.111:0/640163463 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610e2d4b0a0 con 0x5610e2e512e0 2023-10-11T16:42:22.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.404+0000 7fbb2659c700 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5610e2e512e0 0x5610e2e536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.405+0000 7fbb2659c700 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5610e2e512e0 0x5610e2e536c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d84609365af8fb96 server_cookie=40b299ff986f8b51 in_seq=0 out_seq=0 2023-10-11T16:42:22.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.407+0000 7fbb0dffb700 10 client.27266 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:42:22.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.407+0000 7fbb0dffb700 1 -- 172.21.15.111:0/640163463 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5610e2d4b0a0 con 0x5610e2e512e0 2023-10-11T16:42:22.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.407+0000 7fbb0dffb700 10 client.27266 handle_command_reply: tid=0 2023-10-11T16:42:22.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.408+0000 7fbb408a5b80 1 client.27266 shutdown 2023-10-11T16:42:22.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.408+0000 7fbb408a5b80 1 -- 172.21.15.111:0/640163463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 msgr2=0x5610e2e41a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.408+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e41a50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbb280641e0 tx=0x7fbb2804eb70 comp rx=0 tx=0).stop 2023-10-11T16:42:22.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.409+0000 7fbb408a5b80 1 -- 172.21.15.111:0/640163463 shutdown_connections 2023-10-11T16:42:22.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.409+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5610e2e512e0 0x5610e2e536c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.409+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e41a50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.409+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e44150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.409+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/640163463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.409+0000 7fbb408a5b80 1 -- 172.21.15.111:0/640163463 >> 172.21.15.111:0/640163463 conn(0x5610e2e2d5b0 msgr2=0x5610e2e32030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.409+0000 7fbb0dffb700 0 client.27266 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:42:22.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.410+0000 7fbb408a5b80 1 -- 172.21.15.111:0/640163463 shutdown_connections 2023-10-11T16:42:22.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.410+0000 7fbb408a5b80 1 -- 172.21.15.111:0/640163463 wait complete. 2023-10-11T16:42:22.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.410+0000 7fbb408a5b80 20 client.27266 trim_cache size 0 max 16384 2023-10-11T16:42:22.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.437+0000 7fbb408a5b80 1 Processor -- start 2023-10-11T16:42:22.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.437+0000 7fbb408a5b80 1 -- start start 2023-10-11T16:42:22.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.438+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.438+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e43cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.438+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e327b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.438+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2e4e300 con 0x5610e2e38100 2023-10-11T16:42:22.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.439+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2e2d0e0 con 0x5610e2e3e1e0 2023-10-11T16:42:22.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.439+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2e336b0 con 0x5610e2e3d8d0 2023-10-11T16:42:22.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.439+0000 7fbb2659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.439+0000 7fbb26d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.439+0000 7fbb2659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37666/0 (socket says 172.21.15.111:37666) 2023-10-11T16:42:22.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.439+0000 7fbb2659c700 1 -- 172.21.15.111:0/4036866116 learned_addr learned my addr 172.21.15.111:0/4036866116 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.440+0000 7fbb26d9d700 1 -- 172.21.15.111:0/4036866116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 msgr2=0x5610e2e43cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.440+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/4036866116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e43cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.440+0000 7fbb26d9d700 1 -- 172.21.15.111:0/4036866116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 msgr2=0x5610e2e48a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.440+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/4036866116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.440+0000 7fbb26d9d700 1 -- 172.21.15.111:0/4036866116 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610e2e2ed50 con 0x5610e2e3e1e0 2023-10-11T16:42:22.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.443+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/4036866116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e327b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbb28066770 tx=0x7fbb28094000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a237a3a15ce5569 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.443+0000 7fbb157fa700 1 -- 172.21.15.111:0/4036866116 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2807a070 con 0x5610e2e3e1e0 2023-10-11T16:42:22.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.443+0000 7fbb157fa700 1 -- 172.21.15.111:0/4036866116 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb280a2030 con 0x5610e2e3e1e0 2023-10-11T16:42:22.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.443+0000 7fbb157fa700 1 -- 172.21.15.111:0/4036866116 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2806b870 con 0x5610e2e3e1e0 2023-10-11T16:42:22.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.444+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4036866116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 msgr2=0x5610e2e327b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.444+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4036866116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e327b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbb28066770 tx=0x7fbb28094000 comp rx=0 tx=0).stop 2023-10-11T16:42:22.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.445+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4036866116 shutdown_connections 2023-10-11T16:42:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.445+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4036866116 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e327b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.445+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4036866116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.445+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4036866116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e43cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.445+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4036866116 >> 172.21.15.111:0/4036866116 conn(0x5610e2e2d5b0 msgr2=0x5610e2e3b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.445+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4036866116 shutdown_connections 2023-10-11T16:42:22.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.445+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4036866116 wait complete. 2023-10-11T16:42:22.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.445+0000 7fbb408a5b80 1 -- start start 2023-10-11T16:42:22.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.446+0000 7fbb408a5b80 1 Processor -- start 2023-10-11T16:42:22.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.447+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.447+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e39310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.447+0000 7fbb408a5b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e39850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.447+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2e39d90 con 0x5610e2e38100 2023-10-11T16:42:22.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.447+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2e39f00 con 0x5610e2e3e1e0 2023-10-11T16:42:22.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.447+0000 7fbb408a5b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610e2e3a0c0 con 0x5610e2e3d8d0 2023-10-11T16:42:22.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.447+0000 7fbb408a5b80 10 client.0 _ll_register_callbacks cb 0x7fbb28069e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:22.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.448+0000 7fbb26d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e39850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.448+0000 7fbb2659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.448+0000 7fbb26d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e39850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54892/0 (socket says 172.21.15.111:54892) 2023-10-11T16:42:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.448+0000 7fbb26d9d700 1 -- 172.21.15.111:0/4185778764 learned_addr learned my addr 172.21.15.111:0/4185778764 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.448+0000 7fbb26d9d700 1 -- 172.21.15.111:0/4185778764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 msgr2=0x5610e2e39310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.448+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e39310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.448+0000 7fbb26d9d700 1 -- 172.21.15.111:0/4185778764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 msgr2=0x5610e2e48a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.449+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.449+0000 7fbb26d9d700 1 -- 172.21.15.111:0/4185778764 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610e2d4f190 con 0x5610e2e3e1e0 2023-10-11T16:42:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.450+0000 7fbb26d9d700 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e39850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbb28072cf0 tx=0x7fbb2807eac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fe2ea9615d973 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.450+0000 7fbb0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:42:22.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.450+0000 7fbb0dffb700 1 -- 172.21.15.111:0/4185778764 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2807a040 con 0x5610e2e3e1e0 2023-10-11T16:42:22.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.450+0000 7fbb0dffb700 1 -- 172.21.15.111:0/4185778764 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fbb2807f630 con 0x5610e2e3e1e0 2023-10-11T16:42:22.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.451+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4185778764 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb2807e530 con 0x5610e2e3e1e0 2023-10-11T16:42:22.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.451+0000 7fbb0dffb700 1 -- 172.21.15.111:0/4185778764 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb2807f7f0 con 0x5610e2e3e1e0 2023-10-11T16:42:22.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.451+0000 7fbb408a5b80 10 client.27272 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:22.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.451+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4185778764 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5610e2bb6de0 con 0x5610e2e3e1e0 2023-10-11T16:42:22.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.452+0000 7fbb0dffb700 1 -- 172.21.15.111:0/4185778764 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb2807e530 con 0x5610e2e3e1e0 2023-10-11T16:42:22.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.452+0000 7fbb0dffb700 1 -- 172.21.15.111:0/4185778764 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbb280afa40 con 0x5610e2e3e1e0 2023-10-11T16:42:22.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.452+0000 7fbb408a5b80 10 client.27272 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:22.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.452+0000 7fbb408a5b80 10 client.27272 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:42:22.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.452+0000 7fbb408a5b80 20 client.27272 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:22.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.452+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5610e2ea5180 0x5610e2ea7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.452+0000 7fbb408a5b80 4 client.27272 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:42:22.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.452+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4185778764 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5610e2d57660 con 0x5610e2ea5180 2023-10-11T16:42:22.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.453+0000 7fbb2659c700 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5610e2ea5180 0x5610e2ea7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.456+0000 7fbb0dffb700 10 client.27272 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:42:22.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.456+0000 7fbb2659c700 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5610e2ea5180 0x5610e2ea7560 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=896aca135e48bca3 server_cookie=498aca31c79718d7 in_seq=0 out_seq=0 2023-10-11T16:42:22.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.456+0000 7fbb0dffb700 1 -- 172.21.15.111:0/4185778764 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5610e2d57660 con 0x5610e2ea5180 2023-10-11T16:42:22.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.456+0000 7fbb0dffb700 10 client.27272 handle_command_reply: tid=0 2023-10-11T16:42:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.456+0000 7fbb408a5b80 1 client.27272 shutdown 2023-10-11T16:42:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4185778764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 msgr2=0x5610e2e39850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e39850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbb28072cf0 tx=0x7fbb2807eac0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4185778764 shutdown_connections 2023-10-11T16:42:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x5610e2ea5180 0x5610e2ea7560 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610e2e3e1e0 0x5610e2e39850 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x5610e2e38100 0x5610e2e48a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 --2- 172.21.15.111:0/4185778764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5610e2e3d8d0 0x5610e2e39310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4185778764 >> 172.21.15.111:0/4185778764 conn(0x5610e2e2d5b0 msgr2=0x5610e2e3c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb0dffb700 0 client.27272 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:42:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4185778764 shutdown_connections 2023-10-11T16:42:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.458+0000 7fbb408a5b80 1 -- 172.21.15.111:0/4185778764 wait complete. 2023-10-11T16:42:22.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.459+0000 7fbb408a5b80 20 client.27272 trim_cache size 0 max 16384 2023-10-11T16:42:22.468 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:42:22.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 -- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fbb100a14d0 msgr2=0x7fbb100a3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fbb100a14d0 0x7fbb100a3980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbb2806a010 tx=0x7fbb28072740 comp rx=0 tx=0).stop 2023-10-11T16:42:22.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 -- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb20096050 msgr2=0x7fbb2013ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb20096050 0x7fbb2013ca70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbb20096420 tx=0x7fbb1c00eca0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 -- 172.21.15.111:0/3489889607 shutdown_connections 2023-10-11T16:42:22.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fbb100a14d0 0x7fbb100a3980 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb2009a6f0 0x7fbb20140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb200956b0 0x7fbb2013c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 --2- 172.21.15.111:0/3489889607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb20096050 0x7fbb2013ca70 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.460+0000 7fbb2759e700 1 -- 172.21.15.111:0/3489889607 >> 172.21.15.111:0/3489889607 conn(0x7fbb2000c230 msgr2=0x7fbb20092f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.461+0000 7fbb2759e700 1 -- 172.21.15.111:0/3489889607 shutdown_connections 2023-10-11T16:42:22.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.461+0000 7fbb2759e700 1 -- 172.21.15.111:0/3489889607 wait complete. 2023-10-11T16:42:22.498 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:42:22.863 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.861+0000 7f9a64dff700 1 -- 172.21.15.111:0/1054254290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 msgr2=0x7f9a60107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.861+0000 7f9a64dff700 1 --2- 172.21.15.111:0/1054254290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 0x7f9a60107af0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009990 tx=0x7f9a50009ca0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.862+0000 7f9a64dff700 1 -- 172.21.15.111:0/1054254290 shutdown_connections 2023-10-11T16:42:22.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.863+0000 7f9a64dff700 1 --2- 172.21.15.111:0/1054254290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60115300 0x7f9a601176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.863+0000 7f9a64dff700 1 --2- 172.21.15.111:0/1054254290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 0x7f9a60107af0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.863+0000 7f9a64dff700 1 --2- 172.21.15.111:0/1054254290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a601080c0 0x7f9a60108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.863+0000 7f9a64dff700 1 -- 172.21.15.111:0/1054254290 >> 172.21.15.111:0/1054254290 conn(0x7f9a6006ca60 msgr2=0x7f9a60071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.863+0000 7f9a64dff700 1 -- 172.21.15.111:0/1054254290 shutdown_connections 2023-10-11T16:42:22.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.863+0000 7f9a64dff700 1 -- 172.21.15.111:0/1054254290 wait complete. 2023-10-11T16:42:22.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.865+0000 7f9a64dff700 1 Processor -- start 2023-10-11T16:42:22.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.865+0000 7f9a64dff700 1 -- start start 2023-10-11T16:42:22.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.866+0000 7f9a64dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 0x7f9a60120840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.866+0000 7f9a64dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a601080c0 0x7f9a60120d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.866+0000 7f9a64dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60115300 0x7f9a60114930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.867+0000 7f9a5e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 0x7f9a60120840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.867+0000 7f9a5e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 0x7f9a60120840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43424/0 (socket says 172.21.15.111:43424) 2023-10-11T16:42:22.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.867+0000 7f9a64dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a60114f30 con 0x7f9a601080c0 2023-10-11T16:42:22.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.867+0000 7f9a64dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a601150a0 con 0x7f9a60115300 2023-10-11T16:42:22.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.867+0000 7f9a64dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a601ba180 con 0x7f9a60072900 2023-10-11T16:42:22.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.867+0000 7f9a5e59c700 1 -- 172.21.15.111:0/293917857 learned_addr learned my addr 172.21.15.111:0/293917857 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.867+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60115300 0x7f9a60114930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.868+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a601080c0 0x7f9a60120d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.868+0000 7f9a5e59c700 1 -- 172.21.15.111:0/293917857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60115300 msgr2=0x7f9a60114930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.868+0000 7f9a5e59c700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60115300 0x7f9a60114930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.868+0000 7f9a5e59c700 1 -- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a601080c0 msgr2=0x7f9a60120d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.868+0000 7f9a5e59c700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a601080c0 0x7f9a60120d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.868+0000 7f9a5e59c700 1 -- 172.21.15.111:0/293917857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a500095f0 con 0x7f9a60072900 2023-10-11T16:42:22.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.869+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60115300 0x7f9a60114930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:42:22.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.869+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a601080c0 0x7f9a60120d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:22.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.869+0000 7f9a5e59c700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 0x7f9a60120840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9a50005b40 tx=0x7f9a50003c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.870+0000 7f9a4f7fe700 1 -- 172.21.15.111:0/293917857 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5001d070 con 0x7f9a60072900 2023-10-11T16:42:22.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.870+0000 7f9a4f7fe700 1 -- 172.21.15.111:0/293917857 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a50003e60 con 0x7f9a60072900 2023-10-11T16:42:22.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.870+0000 7f9a4f7fe700 1 -- 172.21.15.111:0/293917857 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50022420 con 0x7f9a60072900 2023-10-11T16:42:22.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.870+0000 7f9a64dff700 1 -- 172.21.15.111:0/293917857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6010c020 con 0x7f9a60072900 2023-10-11T16:42:22.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.870+0000 7f9a64dff700 1 -- 172.21.15.111:0/293917857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6010be90 con 0x7f9a60072900 2023-10-11T16:42:22.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.880+0000 7f9a4f7fe700 1 -- 172.21.15.111:0/293917857 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f9a5000f460 con 0x7f9a60072900 2023-10-11T16:42:22.882 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.881+0000 7f9a78736b80 1 Processor -- start 2023-10-11T16:42:22.883 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.882+0000 7f9a78736b80 1 -- start start 2023-10-11T16:42:22.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.882+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a91727bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.882+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a917280370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.882+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917280d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.882+0000 7f9a78736b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a917181060 con 0x55a91727c470 2023-10-11T16:42:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.882+0000 7f9a78736b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9171d38a0 con 0x55a91727bb60 2023-10-11T16:42:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.882+0000 7f9a78736b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9171ccd30 con 0x55a9172808b0 2023-10-11T16:42:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.883+0000 7f9a5ed9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a91727bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.883+0000 7f9a5ed9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a91727bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54912/0 (socket says 172.21.15.111:54912) 2023-10-11T16:42:22.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.883+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/4144815679 learned_addr learned my addr 172.21.15.111:0/4144815679 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.883+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917280d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:22 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1853433079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:42:22.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.884+0000 7f9a5e59c700 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a917280370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.884+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/4144815679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 msgr2=0x55a917280d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.884+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917280d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.884+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/4144815679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 msgr2=0x55a917280370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.884+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a917280370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.884+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/4144815679 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a91704e630 con 0x55a91727bb60 2023-10-11T16:42:22.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.886+0000 7f9a4f7fe700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9a480a1340 0x7f9a480a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.886+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a91727bf30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c550 tx=0x7f9a5800c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da6b9b6533a34c60 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.886+0000 7f9a5e59c700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9a480a1340 0x7f9a480a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.886+0000 7f9a4cff9700 1 -- 172.21.15.111:0/4144815679 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58011070 con 0x55a91727bb60 2023-10-11T16:42:22.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.886+0000 7f9a4f7fe700 1 -- 172.21.15.111:0/293917857 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f9a500ce280 con 0x7f9a60072900 2023-10-11T16:42:22.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.886+0000 7f9a4cff9700 1 -- 172.21.15.111:0/4144815679 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a58009bc0 con 0x55a91727bb60 2023-10-11T16:42:22.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.886+0000 7f9a4cff9700 1 -- 172.21.15.111:0/4144815679 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58015da0 con 0x55a91727bb60 2023-10-11T16:42:22.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.889+0000 7f9a78736b80 1 -- 172.21.15.111:0/4144815679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 msgr2=0x55a91727bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.889+0000 7f9a5e59c700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9a480a1340 0x7f9a480a37f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9a5002a000 tx=0x7f9a50009ca0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.889+0000 7f9a78736b80 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a91727bf30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c550 tx=0x7f9a5800c580 comp rx=0 tx=0).stop 2023-10-11T16:42:22.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.889+0000 7f9a78736b80 1 -- 172.21.15.111:0/4144815679 shutdown_connections 2023-10-11T16:42:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.889+0000 7f9a78736b80 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917280d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.889+0000 7f9a78736b80 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a91727bf30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.889+0000 7f9a78736b80 1 --2- 172.21.15.111:0/4144815679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a917280370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.889+0000 7f9a78736b80 1 -- 172.21.15.111:0/4144815679 >> 172.21.15.111:0/4144815679 conn(0x55a917271360 msgr2=0x55a917276b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.890+0000 7f9a78736b80 1 -- 172.21.15.111:0/4144815679 shutdown_connections 2023-10-11T16:42:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.890+0000 7f9a78736b80 1 -- 172.21.15.111:0/4144815679 wait complete. 2023-10-11T16:42:22.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.891+0000 7f9a78736b80 1 -- start start 2023-10-11T16:42:22.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.891+0000 7f9a78736b80 1 Processor -- start 2023-10-11T16:42:22.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.893+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727bb60 0x55a917288090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.894+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727c470 0x55a917285380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.894+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917285ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.894+0000 7f9a5ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727bb60 0x55a917288090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.894+0000 7f9a5ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727bb60 0x55a917288090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37716/0 (socket says 172.21.15.111:37716) 2023-10-11T16:42:22.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.894+0000 7f9a5dd9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727c470 0x55a917285380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.894+0000 7f9a5dd9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727c470 0x55a917285380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54926/0 (socket says 172.21.15.111:54926) 2023-10-11T16:42:22.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.894+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/2653732705 learned_addr learned my addr 172.21.15.111:0/2653732705 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.895+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a91716e8e0 con 0x55a91727bb60 2023-10-11T16:42:22.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.895+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a91716e580 con 0x55a91727c470 2023-10-11T16:42:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.895+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a917050920 con 0x55a9172808b0 2023-10-11T16:42:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.895+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917285ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a78736b80 10 client.0 _ll_register_callbacks cb 0x55a91718c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:22.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/2653732705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727c470 msgr2=0x55a917285380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727c470 0x55a917285380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/2653732705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727bb60 msgr2=0x55a917288090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.905 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727bb60 0x55a917288090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/2653732705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a58009d40 con 0x55a9172808b0 2023-10-11T16:42:22.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727c470 0x55a917285380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:42:22.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917285ac0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9a58000f80 tx=0x7f9a58009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d28ffb6c8b6aa97 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a45ffb700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:42:22.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.896+0000 7f9a45ffb700 1 -- 172.21.15.111:0/2653732705 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58011040 con 0x55a9172808b0 2023-10-11T16:42:22.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.897+0000 7f9a45ffb700 1 -- 172.21.15.111:0/2653732705 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f9a58015d20 con 0x55a9172808b0 2023-10-11T16:42:22.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.897+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a91718f0a0 con 0x55a9172808b0 2023-10-11T16:42:22.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.897+0000 7f9a45ffb700 1 -- 172.21.15.111:0/2653732705 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a58003aa0 con 0x55a9172808b0 2023-10-11T16:42:22.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.898+0000 7f9a78736b80 10 client.27280 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:22.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.898+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55a917193190 con 0x55a9172808b0 2023-10-11T16:42:22.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a45ffb700 1 -- 172.21.15.111:0/2653732705 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a91718f0a0 con 0x55a9172808b0 2023-10-11T16:42:22.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a45ffb700 1 -- 172.21.15.111:0/2653732705 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9a5801c030 con 0x55a9172808b0 2023-10-11T16:42:22.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a78736b80 10 client.27280 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:22.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a78736b80 10 client.27280 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:42:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a78736b80 20 client.27280 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a78736b80 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a917295790 0x55a917297ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a78736b80 4 client.27280 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:22.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a91719b660 con 0x55a917295790 2023-10-11T16:42:22.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.899+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a917295790 0x55a917297ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.900+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a917295790 0x55a917297ba0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9bd07e7fbb027c5f server_cookie=b2ec5e7eacb0d0c9 in_seq=0 out_seq=0 2023-10-11T16:42:22.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.900+0000 7f9a45ffb700 10 client.27280 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:42:22.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.901+0000 7f9a45ffb700 1 -- 172.21.15.111:0/2653732705 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a91719b660 con 0x55a917295790 2023-10-11T16:42:22.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.901+0000 7f9a45ffb700 10 client.27280 handle_command_reply: tid=0 2023-10-11T16:42:22.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.901+0000 7f9a78736b80 1 client.27280 shutdown 2023-10-11T16:42:22.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.902+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 msgr2=0x55a917285ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.902+0000 7f9a78736b80 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917285ac0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9a58000f80 tx=0x7f9a58009650 comp rx=0 tx=0).stop 2023-10-11T16:42:22.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.902+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 shutdown_connections 2023-10-11T16:42:22.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.902+0000 7f9a78736b80 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a917295790 0x55a917297ba0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.902+0000 7f9a78736b80 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a9172808b0 0x55a917285ac0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.902+0000 7f9a78736b80 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727bb60 0x55a917288090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.917 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.902+0000 7f9a78736b80 1 --2- 172.21.15.111:0/2653732705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727c470 0x55a917285380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.902+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 >> 172.21.15.111:0/2653732705 conn(0x55a917271360 msgr2=0x55a917275de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.903+0000 7f9a45ffb700 0 client.27280 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:42:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.903+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 shutdown_connections 2023-10-11T16:42:22.918 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.904+0000 7f9a78736b80 1 -- 172.21.15.111:0/2653732705 wait complete. 2023-10-11T16:42:22.919 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.904+0000 7f9a78736b80 20 client.27280 trim_cache size 0 max 16384 2023-10-11T16:42:22.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.930+0000 7f9a78736b80 1 Processor -- start 2023-10-11T16:42:22.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.931+0000 7f9a78736b80 1 -- start start 2023-10-11T16:42:22.933 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.931+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a917271360 0x55a91727f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.931+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a9172977a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.931+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a91727c470 0x55a91727fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a78736b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a917270ee0 con 0x55a917271360 2023-10-11T16:42:22.934 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a78736b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a917280c80 con 0x55a91727bb60 2023-10-11T16:42:22.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a78736b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a917280e40 con 0x55a91727c470 2023-10-11T16:42:22.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5dd9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a9172977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.935 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5dd9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a9172977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54930/0 (socket says 172.21.15.111:54930) 2023-10-11T16:42:22.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5dd9b700 1 -- 172.21.15.111:0/578453767 learned_addr learned my addr 172.21.15.111:0/578453767 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.936 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/578453767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a91727c470 0x55a91727fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5dd9b700 1 -- 172.21.15.111:0/578453767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a91727c470 msgr2=0x55a91727fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.937 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/578453767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a91727c470 0x55a91727fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5dd9b700 1 -- 172.21.15.111:0/578453767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a917271360 msgr2=0x55a91727f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/578453767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a917271360 0x55a91727f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.938 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.932+0000 7f9a5dd9b700 1 -- 172.21.15.111:0/578453767 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a917192ee0 con 0x55a91727bb60 2023-10-11T16:42:22.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.933+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/578453767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a9172977a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9a60121e50 tx=0x7f9a54004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb88a1f3a0991fd server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.933+0000 7f9a4d7fa700 1 -- 172.21.15.111:0/578453767 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5401a3b0 con 0x55a91727bb60 2023-10-11T16:42:22.939 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.933+0000 7f9a4d7fa700 1 -- 172.21.15.111:0/578453767 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a5401a040 con 0x55a91727bb60 2023-10-11T16:42:22.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.934+0000 7f9a4d7fa700 1 -- 172.21.15.111:0/578453767 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54023460 con 0x55a91727bb60 2023-10-11T16:42:22.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.935+0000 7f9a78736b80 1 -- 172.21.15.111:0/578453767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 msgr2=0x55a9172977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.941 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.935+0000 7f9a78736b80 1 --2- 172.21.15.111:0/578453767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a9172977a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9a60121e50 tx=0x7f9a54004750 comp rx=0 tx=0).stop 2023-10-11T16:42:22.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.935+0000 7f9a78736b80 1 -- 172.21.15.111:0/578453767 shutdown_connections 2023-10-11T16:42:22.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.935+0000 7f9a78736b80 1 --2- 172.21.15.111:0/578453767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a91727c470 0x55a91727fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.942 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.935+0000 7f9a78736b80 1 --2- 172.21.15.111:0/578453767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a917271360 0x55a91727f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.935+0000 7f9a78736b80 1 --2- 172.21.15.111:0/578453767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a9172977a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.935+0000 7f9a78736b80 1 -- 172.21.15.111:0/578453767 >> 172.21.15.111:0/578453767 conn(0x55a917285c30 msgr2=0x55a917291c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.935+0000 7f9a78736b80 1 -- 172.21.15.111:0/578453767 shutdown_connections 2023-10-11T16:42:22.943 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.936+0000 7f9a78736b80 1 -- 172.21.15.111:0/578453767 wait complete. 2023-10-11T16:42:22.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.937+0000 7f9a78736b80 1 -- start start 2023-10-11T16:42:22.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.937+0000 7f9a78736b80 1 Processor -- start 2023-10-11T16:42:22.946 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.938+0000 7f9a78736b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a917271360 0x55a917288980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.939+0000 7f9a5ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a917271360 0x55a917288980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.939+0000 7f9a5ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a917271360 0x55a917288980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43458/0 (socket says 172.21.15.111:43458) 2023-10-11T16:42:22.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.939+0000 7f9a5ed9d700 1 -- 172.21.15.111:0/3029006726 learned_addr learned my addr 172.21.15.111:0/3029006726 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:22.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.939+0000 7f9a78736b80 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a917288ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.939+0000 7f9a78736b80 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a9172e8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.939+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a917288ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.939+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9172897d0 con 0x55a91727c470 2023-10-11T16:42:22.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.939+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9172e8b40 con 0x55a91727bb60 2023-10-11T16:42:22.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.940+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9172e8c80 con 0x55a917271360 2023-10-11T16:42:22.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.940+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a9172e8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.940+0000 7f9a5dd9b700 1 -- 172.21.15.111:0/3029006726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a917271360 msgr2=0x55a917288980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.940+0000 7f9a78736b80 10 client.0 _ll_register_callbacks cb 0x7f9a58018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.941+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a917271360 0x55a917288980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.941+0000 7f9a5dd9b700 1 -- 172.21.15.111:0/3029006726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 msgr2=0x55a9172e8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.941+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a9172e8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.941+0000 7f9a5dd9b700 1 -- 172.21.15.111:0/3029006726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a54014410 con 0x55a91727bb60 2023-10-11T16:42:22.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.941+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a9172e8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:22.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.941+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a91726eb20 con 0x55a91727bb60 2023-10-11T16:42:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.941+0000 7f9a5dd9b700 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a917288ec0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9a60121e50 tx=0x7f9a5401ae50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d27361efe07e184 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.941+0000 7f9a45ffb700 10 client.27287 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:42:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.942+0000 7f9a45ffb700 1 -- 172.21.15.111:0/3029006726 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5401a1b0 con 0x55a91727bb60 2023-10-11T16:42:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.942+0000 7f9a45ffb700 1 -- 172.21.15.111:0/3029006726 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f9a5401a310 con 0x55a91727bb60 2023-10-11T16:42:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.942+0000 7f9a45ffb700 1 -- 172.21.15.111:0/3029006726 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a5401a510 con 0x55a91727bb60 2023-10-11T16:42:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.942+0000 7f9a78736b80 10 client.27287 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.943+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({fsmap=39}) v3 -- 0x55a916fe3c60 con 0x55a91727bb60 2023-10-11T16:42:22.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.943+0000 7f9a45ffb700 1 -- 172.21.15.111:0/3029006726 <== mon.2 v2:172.21.15.130:3300/0 4 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9a5401a670 con 0x55a91727bb60 2023-10-11T16:42:22.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.944+0000 7f9a78736b80 10 client.27287 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:22.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.944+0000 7f9a78736b80 10 client.27287 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:42:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.944+0000 7f9a78736b80 20 client.27287 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.944+0000 7f9a78736b80 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a9172808b0 0x55a9172ec060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.944+0000 7f9a78736b80 4 client.27287 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:42:22.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.944+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a91701bf70 con 0x55a9172808b0 2023-10-11T16:42:22.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.945+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a9172808b0 0x55a9172ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:22.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.946+0000 7f9a45ffb700 10 client.27287 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:42:22.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.946+0000 7f9a5ed9d700 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a9172808b0 0x55a9172ec060 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=921fcf6f845a6629 server_cookie=b3e4d8f59b2f1d2e in_seq=0 out_seq=0 2023-10-11T16:42:22.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.947+0000 7f9a45ffb700 1 -- 172.21.15.111:0/3029006726 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a91701bf70 con 0x55a9172808b0 2023-10-11T16:42:22.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.947+0000 7f9a45ffb700 10 client.27287 handle_command_reply: tid=0 2023-10-11T16:42:22.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.948+0000 7f9a78736b80 1 client.27287 shutdown 2023-10-11T16:42:22.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.948+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 msgr2=0x55a917288ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.948+0000 7f9a78736b80 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a917288ec0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9a60121e50 tx=0x7f9a5401ae50 comp rx=0 tx=0).stop 2023-10-11T16:42:22.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.948+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 shutdown_connections 2023-10-11T16:42:22.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.948+0000 7f9a78736b80 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a9172808b0 0x55a9172ec060 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.948+0000 7f9a78736b80 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a91727c470 0x55a9172e8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.948+0000 7f9a45ffb700 0 client.27287 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:42:22.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.949+0000 7f9a78736b80 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a917271360 0x55a917288980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.949+0000 7f9a78736b80 1 --2- 172.21.15.111:0/3029006726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a91727bb60 0x55a917288ec0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.949+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 >> 172.21.15.111:0/3029006726 conn(0x55a917285c30 msgr2=0x55a91727a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.949+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 shutdown_connections 2023-10-11T16:42:22.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.949+0000 7f9a78736b80 1 -- 172.21.15.111:0/3029006726 wait complete. 2023-10-11T16:42:22.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.950+0000 7f9a78736b80 20 client.27287 trim_cache size 0 max 16384 2023-10-11T16:42:22.965 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:42:22.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 -- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9a480a1340 msgr2=0x7f9a480a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9a480a1340 0x7f9a480a37f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9a5002a000 tx=0x7f9a50009ca0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 -- 172.21.15.111:0/293917857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 msgr2=0x7f9a60120840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:22.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 0x7f9a60120840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9a50005b40 tx=0x7f9a50003c40 comp rx=0 tx=0).stop 2023-10-11T16:42:22.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 -- 172.21.15.111:0/293917857 shutdown_connections 2023-10-11T16:42:22.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f9a480a1340 0x7f9a480a37f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60115300 0x7f9a60114930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a60072900 0x7f9a60120840 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 --2- 172.21.15.111:0/293917857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a601080c0 0x7f9a60120d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:22.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 -- 172.21.15.111:0/293917857 >> 172.21.15.111:0/293917857 conn(0x7f9a6006ca60 msgr2=0x7f9a60071750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:22.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 -- 172.21.15.111:0/293917857 shutdown_connections 2023-10-11T16:42:22.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:22.951+0000 7f9a64dff700 1 -- 172.21.15.111:0/293917857 wait complete. 2023-10-11T16:42:22.971 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:22 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1853433079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:42:22.989 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:42:23.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:22 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1853433079' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:42:23.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.358+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/2432084993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb8104680 msgr2=0x7f2bb8104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.358+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/2432084993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb8104680 0x7f2bb8104af0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000b0a0 tx=0x7f2bb000b3b0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.359+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/2432084993 shutdown_connections 2023-10-11T16:42:23.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.359+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/2432084993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb8104680 0x7f2bb8104af0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.359+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/2432084993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb80ff180 0x7f2bb80ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.363 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.359+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/2432084993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb80ffa90 0x7f2bb8104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.359+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/2432084993 >> 172.21.15.111:0/2432084993 conn(0x7f2bb80770b0 msgr2=0x7f2bb80774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:23.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.359+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/2432084993 shutdown_connections 2023-10-11T16:42:23.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.359+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/2432084993 wait complete. 2023-10-11T16:42:23.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.361+0000 7f2bbe7ff700 1 Processor -- start 2023-10-11T16:42:23.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.361+0000 7f2bbe7ff700 1 -- start start 2023-10-11T16:42:23.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bbe7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb80ff180 0x7f2bb81a2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bbe7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb80ffa90 0x7f2bb81a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bbe7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb8104680 0x7f2bb819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bbe7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb819d060 con 0x7f2bb80ff180 2023-10-11T16:42:23.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bbe7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb819d1d0 con 0x7f2bb80ffa90 2023-10-11T16:42:23.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bbe7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb819d390 con 0x7f2bb8104680 2023-10-11T16:42:23.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bb77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb80ffa90 0x7f2bb81a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bb77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb80ffa90 0x7f2bb81a2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54964/0 (socket says 172.21.15.111:54964) 2023-10-11T16:42:23.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bb77fe700 1 -- 172.21.15.111:0/3166795715 learned_addr learned my addr 172.21.15.111:0/3166795715 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:23.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb80ff180 0x7f2bb81a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.362+0000 7f2bbcd9c700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb8104680 0x7f2bb819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.363+0000 7f2bb77fe700 1 -- 172.21.15.111:0/3166795715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb8104680 msgr2=0x7f2bb819c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.363+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb8104680 0x7f2bb819c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.363+0000 7f2bb77fe700 1 -- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb80ff180 msgr2=0x7f2bb81a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.363+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb80ff180 0x7f2bb81a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.363+0000 7f2bb77fe700 1 -- 172.21.15.111:0/3166795715 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb0009d00 con 0x7f2bb80ffa90 2023-10-11T16:42:23.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.363+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb80ff180 0x7f2bb81a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:42:23.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.364+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb80ffa90 0x7f2bb81a2ec0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00b530 tx=0x7f2bac00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:23.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.365+0000 7f2bb57fa700 1 -- 172.21.15.111:0/3166795715 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac011830 con 0x7f2bb80ffa90 2023-10-11T16:42:23.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.365+0000 7f2bb57fa700 1 -- 172.21.15.111:0/3166795715 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2bac011e50 con 0x7f2bb80ffa90 2023-10-11T16:42:23.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.365+0000 7f2bb57fa700 1 -- 172.21.15.111:0/3166795715 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac00f2e0 con 0x7f2bb80ffa90 2023-10-11T16:42:23.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.365+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/3166795715 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb8116a40 con 0x7f2bb80ffa90 2023-10-11T16:42:23.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.365+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/3166795715 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb81168b0 con 0x7f2bb80ffa90 2023-10-11T16:42:23.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.372+0000 7f2bb57fa700 1 -- 172.21.15.111:0/3166795715 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2bac011990 con 0x7f2bb80ffa90 2023-10-11T16:42:23.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.374+0000 7f2bb57fa700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ba00a13c0 0x7f2ba00a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.374+0000 7f2bb57fa700 1 -- 172.21.15.111:0/3166795715 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f2bac0cca00 con 0x7f2bb80ffa90 2023-10-11T16:42:23.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.374+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ba00a13c0 0x7f2ba00a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.375+0000 7f2bd2088b80 1 Processor -- start 2023-10-11T16:42:23.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.375+0000 7f2bd2088b80 1 -- start start 2023-10-11T16:42:23.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.376+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a02b60 0x563ec8a02f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.376+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a07370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.376+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a07d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.376+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec8908060 con 0x563ec8a02b60 2023-10-11T16:42:23.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.376+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec895a8a0 con 0x563ec8a078b0 2023-10-11T16:42:23.383 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.376+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec8953d30 con 0x563ec8a03470 2023-10-11T16:42:23.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bb77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a07370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bbcd9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a07d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bb77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a07370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:43488/0 (socket says 172.21.15.111:43488) 2023-10-11T16:42:23.384 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bb77fe700 1 -- 172.21.15.111:0/753514466 learned_addr learned my addr 172.21.15.111:0/753514466 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:23.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bb77fe700 1 -- 172.21.15.111:0/753514466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 msgr2=0x563ec8a07d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:23.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/753514466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a07d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.385 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bb77fe700 1 -- 172.21.15.111:0/753514466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a02b60 msgr2=0x563ec8a02f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:23.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/753514466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a02b60 0x563ec8a02f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.386 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.377+0000 7f2bb77fe700 1 -- 172.21.15.111:0/753514466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ec87d5630 con 0x563ec8a03470 2023-10-11T16:42:23.388 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.379+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/753514466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a07370 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0996b0 tx=0x7f2bac095b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=827766004fb7cd5e server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.379+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ba00a13c0 0x7f2ba00a3870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009e30 tx=0x7f2ba800e4b0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.380+0000 7f2b9e7fc700 1 -- 172.21.15.111:0/753514466 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac099720 con 0x563ec8a03470 2023-10-11T16:42:23.389 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.380+0000 7f2b9e7fc700 1 -- 172.21.15.111:0/753514466 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2bac09d750 con 0x563ec8a03470 2023-10-11T16:42:23.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.380+0000 7f2b9e7fc700 1 -- 172.21.15.111:0/753514466 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac0b90b0 con 0x563ec8a03470 2023-10-11T16:42:23.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.382+0000 7f2bd2088b80 1 -- 172.21.15.111:0/753514466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 msgr2=0x563ec8a07370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.383+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/753514466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a07370 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0996b0 tx=0x7f2bac095b70 comp rx=0 tx=0).stop 2023-10-11T16:42:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.384+0000 7f2bd2088b80 1 -- 172.21.15.111:0/753514466 shutdown_connections 2023-10-11T16:42:23.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.384+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/753514466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a07d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.384+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/753514466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a02b60 0x563ec8a02f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.384+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/753514466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a07370 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.384+0000 7f2bd2088b80 1 -- 172.21.15.111:0/753514466 >> 172.21.15.111:0/753514466 conn(0x563ec89f8360 msgr2=0x563ec89fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:23.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.384+0000 7f2bd2088b80 1 -- 172.21.15.111:0/753514466 shutdown_connections 2023-10-11T16:42:23.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.384+0000 7f2bd2088b80 1 -- 172.21.15.111:0/753514466 wait complete. 2023-10-11T16:42:23.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.385+0000 7f2bd2088b80 1 -- start start 2023-10-11T16:42:23.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.386+0000 7f2bd2088b80 1 Processor -- start 2023-10-11T16:42:23.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.386+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a0f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.386+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 0x563ec8a0c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.387+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a078b0 0x563ec8a0c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.387+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec88f58e0 con 0x563ec8a03470 2023-10-11T16:42:23.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.387+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec88f5580 con 0x563ec8a02b60 2023-10-11T16:42:23.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.387+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec87d7920 con 0x563ec8a078b0 2023-10-11T16:42:23.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.387+0000 7f2bd2088b80 10 client.0 _ll_register_callbacks cb 0x563ec8913780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:23.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.388+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a0f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.388+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a0f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54970/0 (socket says 172.21.15.111:54970) 2023-10-11T16:42:23.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.388+0000 7f2bb7fff700 1 -- 172.21.15.111:0/2801382094 learned_addr learned my addr 172.21.15.111:0/2801382094 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:23.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.389+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 0x563ec8a0c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.389+0000 7f2bbcd9c700 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a078b0 0x563ec8a0c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.389+0000 7f2bb7fff700 1 -- 172.21.15.111:0/2801382094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a078b0 msgr2=0x563ec8a0c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:23.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.389+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a078b0 0x563ec8a0c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.389+0000 7f2bb7fff700 1 -- 172.21.15.111:0/2801382094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 msgr2=0x563ec8a0c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:23.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.389+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 0x563ec8a0c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.389+0000 7f2bb7fff700 1 -- 172.21.15.111:0/2801382094 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb81168b0 con 0x563ec8a02b60 2023-10-11T16:42:23.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.390+0000 7f2bbcd9c700 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a078b0 0x563ec8a0c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:23.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.390+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a0f090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80156f0 tx=0x7f2ba8004860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be1f35b5ed8478d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:23.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.390+0000 7f2b977fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:42:23.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.391+0000 7f2b977fe700 1 -- 172.21.15.111:0/2801382094 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80164c0 con 0x563ec8a02b60 2023-10-11T16:42:23.405 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.391+0000 7f2b977fe700 1 -- 172.21.15.111:0/2801382094 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f2ba8016620 con 0x563ec8a02b60 2023-10-11T16:42:23.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.391+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2801382094 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ec89f5b20 con 0x563ec8a02b60 2023-10-11T16:42:23.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.392+0000 7f2b977fe700 1 -- 172.21.15.111:0/2801382094 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ba80298c0 con 0x563ec8a02b60 2023-10-11T16:42:23.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.392+0000 7f2bd2088b80 10 client.27299 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:23.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.392+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2801382094 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x563ec891a190 con 0x563ec8a02b60 2023-10-11T16:42:23.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.392+0000 7f2b977fe700 1 -- 172.21.15.111:0/2801382094 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563ec89f5b20 con 0x563ec8a02b60 2023-10-11T16:42:23.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.393+0000 7f2b977fe700 1 -- 172.21.15.111:0/2801382094 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ba8016a40 con 0x563ec8a02b60 2023-10-11T16:42:23.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.393+0000 7f2bd2088b80 10 client.27299 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:23.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.393+0000 7f2bd2088b80 10 client.27299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.393+0000 7f2bd2088b80 20 client.27299 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.393+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563ec8a1c950 0x563ec8a1ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.393+0000 7f2bd2088b80 4 client.27299 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.393+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2801382094 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ec89160a0 con 0x563ec8a1c950 2023-10-11T16:42:23.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.394+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563ec8a1c950 0x563ec8a1ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.400+0000 7f2b977fe700 10 client.27299 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:23.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.400+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563ec8a1c950 0x563ec8a1ed60 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eed516551409bab0 server_cookie=bf5898ba25c666e0 in_seq=0 out_seq=0 2023-10-11T16:42:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.404+0000 7f2b977fe700 1 -- 172.21.15.111:0/2801382094 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563ec89160a0 con 0x563ec8a1c950 2023-10-11T16:42:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.404+0000 7f2b977fe700 10 client.27299 handle_command_reply: tid=0 2023-10-11T16:42:23.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.404+0000 7f2bd2088b80 1 client.27299 shutdown 2023-10-11T16:42:23.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2801382094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 msgr2=0x563ec8a0f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a0f090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80156f0 tx=0x7f2ba8004860 comp rx=0 tx=0).stop 2023-10-11T16:42:23.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2801382094 shutdown_connections 2023-10-11T16:42:23.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563ec8a1c950 0x563ec8a1ed60 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a078b0 0x563ec8a0c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a0f090 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2801382094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 0x563ec8a0c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2801382094 >> 172.21.15.111:0/2801382094 conn(0x563ec89f8360 msgr2=0x563ec8a08d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.405+0000 7f2b977fe700 0 client.27299 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.406+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2801382094 shutdown_connections 2023-10-11T16:42:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.406+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2801382094 wait complete. 2023-10-11T16:42:23.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.406+0000 7f2bd2088b80 20 client.27299 trim_cache size 0 max 16384 2023-10-11T16:42:23.435 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.434+0000 7f2bd2088b80 1 Processor -- start 2023-10-11T16:42:23.438 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.437+0000 7f2bd2088b80 1 -- start start 2023-10-11T16:42:23.439 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.437+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a02b60 0x563ec8a139a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.437+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 0x563ec89fc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.437+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a1e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.440 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.437+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec8a0cf90 con 0x563ec8a03470 2023-10-11T16:42:23.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.437+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec89ff1c0 con 0x563ec8a078b0 2023-10-11T16:42:23.441 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.437+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec8a063f0 con 0x563ec8a02b60 2023-10-11T16:42:23.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.439+0000 7f2bbcd9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a1e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.439+0000 7f2bbcd9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a1e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54978/0 (socket says 172.21.15.111:54978) 2023-10-11T16:42:23.442 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.439+0000 7f2bbcd9c700 1 -- 172.21.15.111:0/2517309622 learned_addr learned my addr 172.21.15.111:0/2517309622 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.439+0000 7f2bbcd9c700 1 -- 172.21.15.111:0/2517309622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a02b60 msgr2=0x563ec8a139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.439+0000 7f2bbcd9c700 1 --2- 172.21.15.111:0/2517309622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a02b60 0x563ec8a139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.439+0000 7f2bbcd9c700 1 -- 172.21.15.111:0/2517309622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 msgr2=0x563ec89fc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:23.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.441+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/2517309622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 0x563ec89fc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.441+0000 7f2bbcd9c700 1 --2- 172.21.15.111:0/2517309622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 0x563ec89fc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.441+0000 7f2bbcd9c700 1 -- 172.21.15.111:0/2517309622 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ec8a07f50 con 0x563ec8a078b0 2023-10-11T16:42:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.441+0000 7f2bbcd9c700 1 --2- 172.21.15.111:0/2517309622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a1e960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000b0a0 tx=0x7f2bb0009420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bc67ea09a2e6247 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.442+0000 7f2b9effd700 1 -- 172.21.15.111:0/2517309622 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb000e040 con 0x563ec8a078b0 2023-10-11T16:42:23.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.442+0000 7f2b9effd700 1 -- 172.21.15.111:0/2517309622 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2bb0007960 con 0x563ec8a078b0 2023-10-11T16:42:23.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.442+0000 7f2b9effd700 1 -- 172.21.15.111:0/2517309622 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0005780 con 0x563ec8a078b0 2023-10-11T16:42:23.446 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2517309622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 msgr2=0x563ec8a1e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2517309622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a1e960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000b0a0 tx=0x7f2bb0009420 comp rx=0 tx=0).stop 2023-10-11T16:42:23.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2517309622 shutdown_connections 2023-10-11T16:42:23.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2517309622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a078b0 0x563ec8a1e960 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2517309622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a02b60 0x563ec8a139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/2517309622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a03470 0x563ec89fc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2517309622 >> 172.21.15.111:0/2517309622 conn(0x563ec89f8360 msgr2=0x563ec89fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2517309622 shutdown_connections 2023-10-11T16:42:23.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.443+0000 7f2bd2088b80 1 -- 172.21.15.111:0/2517309622 wait complete. 2023-10-11T16:42:23.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.444+0000 7f2bd2088b80 1 -- start start 2023-10-11T16:42:23.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.444+0000 7f2bd2088b80 1 Processor -- start 2023-10-11T16:42:23.452 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.445+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a01a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.445+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a01f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.445+0000 7f2bd2088b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a078b0 0x563ec8a0dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.445+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec8a02800 con 0x563ec8a078b0 2023-10-11T16:42:23.453 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.445+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec8a13c30 con 0x563ec8a02b60 2023-10-11T16:42:23.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.445+0000 7f2bd2088b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ec8a13df0 con 0x563ec8a03470 2023-10-11T16:42:23.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.445+0000 7f2bd2088b80 10 client.0 _ll_register_callbacks cb 0x7f2ba800ebf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:23.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.446+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a01a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.446+0000 7f2bb7fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a01a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:54994/0 (socket says 172.21.15.111:54994) 2023-10-11T16:42:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.446+0000 7f2bbcd9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a078b0 0x563ec8a0dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.446+0000 7f2bb7fff700 1 -- 172.21.15.111:0/1517691227 learned_addr learned my addr 172.21.15.111:0/1517691227 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:23.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.447+0000 7f2bb7fff700 1 -- 172.21.15.111:0/1517691227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 msgr2=0x563ec8a01f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:42:23.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.447+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a01f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.447+0000 7f2bb7fff700 1 -- 172.21.15.111:0/1517691227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a078b0 msgr2=0x563ec8a0dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:23.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.447+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a078b0 0x563ec8a0dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.447+0000 7f2bb7fff700 1 -- 172.21.15.111:0/1517691227 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb8114090 con 0x563ec8a02b60 2023-10-11T16:42:23.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.447+0000 7f2bb7fff700 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a01a00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800e930 tx=0x7f2ba8015ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01f5f76d6eef3c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:23.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.447+0000 7f2b977fe700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-10-11T16:42:23.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.448+0000 7f2b977fe700 1 -- 172.21.15.111:0/1517691227 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8020b80 con 0x563ec8a02b60 2023-10-11T16:42:23.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.448+0000 7f2b977fe700 1 -- 172.21.15.111:0/1517691227 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f2ba8009a30 con 0x563ec8a02b60 2023-10-11T16:42:23.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.448+0000 7f2bd2088b80 1 -- 172.21.15.111:0/1517691227 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2bac09d750 con 0x563ec8a02b60 2023-10-11T16:42:23.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.448+0000 7f2b977fe700 1 -- 172.21.15.111:0/1517691227 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ba8020420 con 0x563ec8a02b60 2023-10-11T16:42:23.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.449+0000 7f2bd2088b80 10 client.27305 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:23.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.449+0000 7f2bd2088b80 1 -- 172.21.15.111:0/1517691227 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x563ec8781de0 con 0x563ec8a02b60 2023-10-11T16:42:23.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.449+0000 7f2b977fe700 1 -- 172.21.15.111:0/1517691227 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ba8041030 con 0x563ec8a02b60 2023-10-11T16:42:23.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.450+0000 7f2b977fe700 1 -- 172.21.15.111:0/1517691227 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ba8038020 con 0x563ec8a02b60 2023-10-11T16:42:23.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.451+0000 7f2bd2088b80 10 client.27305 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:23.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.451+0000 7f2bd2088b80 10 client.27305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:23.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.451+0000 7f2bd2088b80 20 client.27305 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.451+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563ec8a70ff0 0x563ec8a73400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.451+0000 7f2bd2088b80 4 client.27305 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:42:23.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.451+0000 7f2bd2088b80 1 -- 172.21.15.111:0/1517691227 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563ec8922660 con 0x563ec8a70ff0 2023-10-11T16:42:23.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.451+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563ec8a70ff0 0x563ec8a73400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:23.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.452+0000 7f2b977fe700 10 client.27305 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:23.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.452+0000 7f2bb77fe700 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563ec8a70ff0 0x563ec8a73400 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cccc8a15dde6b71 server_cookie=a97bfd12f05a9b88 in_seq=0 out_seq=0 2023-10-11T16:42:23.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.453+0000 7f2b977fe700 1 -- 172.21.15.111:0/1517691227 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563ec8922660 con 0x563ec8a70ff0 2023-10-11T16:42:23.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.453+0000 7f2b977fe700 10 client.27305 handle_command_reply: tid=0 2023-10-11T16:42:23.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.453+0000 7f2bd2088b80 1 client.27305 shutdown 2023-10-11T16:42:23.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 -- 172.21.15.111:0/1517691227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 msgr2=0x563ec8a01a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a01a00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800e930 tx=0x7f2ba8015ef0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 -- 172.21.15.111:0/1517691227 shutdown_connections 2023-10-11T16:42:23.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x563ec8a70ff0 0x563ec8a73400 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x563ec8a078b0 0x563ec8a0dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ec8a02b60 0x563ec8a01a00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 --2- 172.21.15.111:0/1517691227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x563ec8a03470 0x563ec8a01f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 -- 172.21.15.111:0/1517691227 >> 172.21.15.111:0/1517691227 conn(0x563ec89f8360 msgr2=0x563ec89fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:23.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2b977fe700 0 client.27305 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:23.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 -- 172.21.15.111:0/1517691227 shutdown_connections 2023-10-11T16:42:23.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.454+0000 7f2bd2088b80 1 -- 172.21.15.111:0/1517691227 wait complete. 2023-10-11T16:42:23.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.455+0000 7f2bd2088b80 20 client.27305 trim_cache size 0 max 16384 2023-10-11T16:42:23.471 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:42:23.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.455+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ba00a13c0 msgr2=0x7f2ba00a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.455+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ba00a13c0 0x7f2ba00a3870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009e30 tx=0x7f2ba800e4b0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/3166795715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb80ffa90 msgr2=0x7f2bb81a2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb80ffa90 0x7f2bb81a2ec0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00b530 tx=0x7f2bac00b840 comp rx=0 tx=0).stop 2023-10-11T16:42:23.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/3166795715 shutdown_connections 2023-10-11T16:42:23.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2ba00a13c0 0x7f2ba00a3870 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bb8104680 0x7f2bb819c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bb80ff180 0x7f2bb81a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 --2- 172.21.15.111:0/3166795715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bb80ffa90 0x7f2bb81a2ec0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:23.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/3166795715 >> 172.21.15.111:0/3166795715 conn(0x7f2bb80770b0 msgr2=0x7f2bb8102090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:23.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/3166795715 shutdown_connections 2023-10-11T16:42:23.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:23.456+0000 7f2bbe7ff700 1 -- 172.21.15.111:0/3166795715 wait complete. 2023-10-11T16:42:23.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:23 smithi111 ceph-mon[98527]: pgmap v302: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 197 B/s wr, 6 op/s 2023-10-11T16:42:23.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:23 smithi150 ceph-mon[132217]: pgmap v302: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 197 B/s wr, 6 op/s 2023-10-11T16:42:24.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:23 smithi130 ceph-mon[131859]: pgmap v302: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 197 B/s wr, 6 op/s 2023-10-11T16:42:24.495 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-11T16:42:24.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-11T16:42:24.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.862+0000 7fac5999d700 1 -- 172.21.15.111:0/720530485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c0a62c0 msgr2=0x7fac4c0a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.862+0000 7fac5999d700 1 --2- 172.21.15.111:0/720530485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c0a62c0 0x7fac4c0a4d00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fac48009990 tx=0x7fac48009ca0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.864+0000 7fac5999d700 1 -- 172.21.15.111:0/720530485 shutdown_connections 2023-10-11T16:42:24.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.864+0000 7fac5999d700 1 --2- 172.21.15.111:0/720530485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac4c0b09b0 0x7fac4c0b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.864+0000 7fac5999d700 1 --2- 172.21.15.111:0/720530485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c0a62c0 0x7fac4c0a4d00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.864+0000 7fac5999d700 1 --2- 172.21.15.111:0/720530485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac4c0a52d0 0x7fac4c0a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.864+0000 7fac5999d700 1 -- 172.21.15.111:0/720530485 >> 172.21.15.111:0/720530485 conn(0x7fac4c01a2f0 msgr2=0x7fac4c01a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:24.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.865+0000 7fac5999d700 1 -- 172.21.15.111:0/720530485 shutdown_connections 2023-10-11T16:42:24.868 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.866+0000 7fac5999d700 1 -- 172.21.15.111:0/720530485 wait complete. 2023-10-11T16:42:24.869 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.868+0000 7fac5999d700 1 Processor -- start 2023-10-11T16:42:24.870 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.868+0000 7fac5999d700 1 -- start start 2023-10-11T16:42:24.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5999d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac4c0a52d0 0x7fac4c143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5999d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c0b09b0 0x7fac4c143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.871 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5999d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac4c13f0f0 0x7fac4c13f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5999d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c13fbd0 con 0x7fac4c13f0f0 2023-10-11T16:42:24.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5999d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c13fd40 con 0x7fac4c0a52d0 2023-10-11T16:42:24.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5999d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c13ff00 con 0x7fac4c0b09b0 2023-10-11T16:42:24.872 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5899b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac4c0a52d0 0x7fac4c143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac53fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c0b09b0 0x7fac4c143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5899b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac4c0a52d0 0x7fac4c143430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52174/0 (socket says 172.21.15.111:52174) 2023-10-11T16:42:24.873 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.869+0000 7fac5899b700 1 -- 172.21.15.111:0/1649475769 learned_addr learned my addr 172.21.15.111:0/1649475769 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:24.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.870+0000 7fac5899b700 1 -- 172.21.15.111:0/1649475769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c0b09b0 msgr2=0x7fac4c143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.874 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.870+0000 7fac5899b700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c0b09b0 0x7fac4c143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.870+0000 7fac5899b700 1 -- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac4c13f0f0 msgr2=0x7fac4c13f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.870+0000 7fac5919c700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac4c13f0f0 0x7fac4c13f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.870+0000 7fac5899b700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac4c13f0f0 0x7fac4c13f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.875 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.870+0000 7fac5899b700 1 -- 172.21.15.111:0/1649475769 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480095f0 con 0x7fac4c0a52d0 2023-10-11T16:42:24.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.870+0000 7fac5899b700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac4c0a52d0 0x7fac4c143430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fac48000c00 tx=0x7fac48005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:24.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.871+0000 7fac51ffb700 1 -- 172.21.15.111:0/1649475769 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4801d070 con 0x7fac4c0a52d0 2023-10-11T16:42:24.876 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.871+0000 7fac51ffb700 1 -- 172.21.15.111:0/1649475769 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac48022470 con 0x7fac4c0a52d0 2023-10-11T16:42:24.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.871+0000 7fac51ffb700 1 -- 172.21.15.111:0/1649475769 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48017650 con 0x7fac4c0a52d0 2023-10-11T16:42:24.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.871+0000 7fac5999d700 1 -- 172.21.15.111:0/1649475769 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4c0a6b80 con 0x7fac4c0a52d0 2023-10-11T16:42:24.877 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.871+0000 7fac5999d700 1 -- 172.21.15.111:0/1649475769 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4c0a69f0 con 0x7fac4c0a52d0 2023-10-11T16:42:24.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.881+0000 7fac51ffb700 1 -- 172.21.15.111:0/1649475769 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fac4800fad0 con 0x7fac4c0a52d0 2023-10-11T16:42:24.885 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.884+0000 7fac6e462b80 1 Processor -- start 2023-10-11T16:42:24.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.884+0000 7fac6e462b80 1 -- start start 2023-10-11T16:42:24.886 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.885+0000 7fac51ffb700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fac440a1340 0x7fac440a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.885+0000 7fac51ffb700 1 -- 172.21.15.111:0/1649475769 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fac480ce630 con 0x7fac4c0a52d0 2023-10-11T16:42:24.887 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.885+0000 7fac5919c700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fac440a1340 0x7fac440a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 0x55efb6902d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb68fcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb68fd180 0x55efb68fd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac6e462b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb68020e0 con 0x55efb68fd180 2023-10-11T16:42:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac6e462b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb6854920 con 0x55efb6902950 2023-10-11T16:42:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac6e462b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb684ddb0 con 0x55efb6903260 2023-10-11T16:42:24.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac53fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb68fcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac53fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb68fcc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:42010/0 (socket says 172.21.15.111:42010) 2023-10-11T16:42:24.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.886+0000 7fac53fff700 1 -- 172.21.15.111:0/4095467196 learned_addr learned my addr 172.21.15.111:0/4095467196 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:24.890 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.887+0000 7fac5919c700 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb68fd180 0x55efb68fd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.891 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.887+0000 7fac53fff700 1 -- 172.21.15.111:0/4095467196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 msgr2=0x55efb6902d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:24.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.887+0000 7fac53fff700 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 0x55efb6902d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.887+0000 7fac53fff700 1 -- 172.21.15.111:0/4095467196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb68fd180 msgr2=0x55efb68fd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:24.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.887+0000 7fac53fff700 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb68fd180 0x55efb68fd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.887+0000 7fac53fff700 1 -- 172.21.15.111:0/4095467196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efb66cf6b0 con 0x55efb6903260 2023-10-11T16:42:24.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.887+0000 7fac53fff700 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb68fcc40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fac4000bd30 tx=0x7fac400106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc24225ee00b5aa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:24.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.888+0000 7fac3effd700 1 -- 172.21.15.111:0/4095467196 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40010ec0 con 0x55efb6903260 2023-10-11T16:42:24.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.888+0000 7fac5919c700 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb68fd180 0x55efb68fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:24.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.888+0000 7fac5919c700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fac440a1340 0x7fac440a37f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac54066400 tx=0x7fac5404ef20 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:24.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.888+0000 7fac3effd700 1 -- 172.21.15.111:0/4095467196 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac40015070 con 0x55efb6903260 2023-10-11T16:42:24.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.888+0000 7fac3effd700 1 -- 172.21.15.111:0/4095467196 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac400117c0 con 0x55efb6903260 2023-10-11T16:42:24.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.889+0000 7fac6e462b80 1 -- 172.21.15.111:0/4095467196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 msgr2=0x55efb68fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.889+0000 7fac6e462b80 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb68fcc40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fac4000bd30 tx=0x7fac400106b0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.889+0000 7fac6e462b80 1 -- 172.21.15.111:0/4095467196 shutdown_connections 2023-10-11T16:42:24.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.889+0000 7fac6e462b80 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb68fd180 0x55efb68fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.889+0000 7fac6e462b80 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 0x55efb6902d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.889+0000 7fac6e462b80 1 --2- 172.21.15.111:0/4095467196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb68fcc40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.896 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.889+0000 7fac6e462b80 1 -- 172.21.15.111:0/4095467196 >> 172.21.15.111:0/4095467196 conn(0x55efb68f2630 msgr2=0x55efb68f7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:24.897 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.890+0000 7fac6e462b80 1 -- 172.21.15.111:0/4095467196 shutdown_connections 2023-10-11T16:42:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.891+0000 7fac6e462b80 1 -- 172.21.15.111:0/4095467196 wait complete. 2023-10-11T16:42:24.898 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.892+0000 7fac6e462b80 1 -- start start 2023-10-11T16:42:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.892+0000 7fac6e462b80 1 Processor -- start 2023-10-11T16:42:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.893+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb69092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.899 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.893+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 0x55efb6906530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.893+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6903260 0x55efb6906c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.893+0000 7fac6e462b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb67ef960 con 0x55efb6903260 2023-10-11T16:42:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.893+0000 7fac6e462b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb67ef600 con 0x55efb6902950 2023-10-11T16:42:24.900 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.893+0000 7fac6e462b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb66d19a0 con 0x55efb68fd180 2023-10-11T16:42:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.893+0000 7fac6e462b80 10 client.0 _ll_register_callbacks cb 0x55efb680d800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.894+0000 7fac5919c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6903260 0x55efb6906c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.901 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.895+0000 7fac53fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 0x55efb6906530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.895+0000 7fac5919c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6903260 0x55efb6906c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33864/0 (socket says 172.21.15.111:33864) 2023-10-11T16:42:24.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.895+0000 7fac5919c700 1 -- 172.21.15.111:0/2422204193 learned_addr learned my addr 172.21.15.111:0/2422204193 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:24.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.895+0000 7fac5899b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb69092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.902 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.896+0000 7fac5919c700 1 -- 172.21.15.111:0/2422204193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 msgr2=0x55efb69092b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:24.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.896+0000 7fac5919c700 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb69092b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.896+0000 7fac5919c700 1 -- 172.21.15.111:0/2422204193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 msgr2=0x55efb6906530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:24.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.896+0000 7fac5919c700 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 0x55efb6906530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.903 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.896+0000 7fac5919c700 1 -- 172.21.15.111:0/2422204193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4000b960 con 0x55efb6903260 2023-10-11T16:42:24.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.896+0000 7fac5899b700 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb69092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:24.904 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.896+0000 7fac53fff700 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 0x55efb6906530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:24.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.897+0000 7fac5919c700 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6903260 0x55efb6906c70 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fac54066770 tx=0x7fac5404eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661012f7ba5bd2fa server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:24.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.897+0000 7fac33fff700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:24.906 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.899+0000 7fac33fff700 1 -- 172.21.15.111:0/2422204193 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54083070 con 0x55efb6903260 2023-10-11T16:42:24.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.899+0000 7fac33fff700 1 -- 172.21.15.111:0/2422204193 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fac5406f7f0 con 0x55efb6903260 2023-10-11T16:42:24.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.899+0000 7fac6e462b80 1 -- 172.21.15.111:0/2422204193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55efb68efdf0 con 0x55efb6903260 2023-10-11T16:42:24.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.900+0000 7fac33fff700 1 -- 172.21.15.111:0/2422204193 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac54076070 con 0x55efb6903260 2023-10-11T16:42:24.907 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.900+0000 7fac6e462b80 10 client.17145 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:24.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.900+0000 7fac6e462b80 1 -- 172.21.15.111:0/2422204193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55efb6814210 con 0x55efb6903260 2023-10-11T16:42:24.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.900+0000 7fac33fff700 1 -- 172.21.15.111:0/2422204193 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55efb68efdf0 con 0x55efb6903260 2023-10-11T16:42:24.908 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.901+0000 7fac33fff700 1 -- 172.21.15.111:0/2422204193 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fac54092a20 con 0x55efb6903260 2023-10-11T16:42:24.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.901+0000 7fac6e462b80 10 client.17145 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:24.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.901+0000 7fac6e462b80 10 client.17145 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:24.909 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.901+0000 7fac6e462b80 20 client.17145 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:24.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.901+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55efb691f470 0x55efb6921880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.901+0000 7fac6e462b80 4 client.17145 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:24.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.902+0000 7fac6e462b80 1 -- 172.21.15.111:0/2422204193 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efb6810120 con 0x55efb691f470 2023-10-11T16:42:24.910 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.902+0000 7fac5899b700 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55efb691f470 0x55efb6921880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.903+0000 7fac5899b700 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55efb691f470 0x55efb6921880 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7089e46497bd2e3b server_cookie=5a94a99241c05f5c in_seq=0 out_seq=0 2023-10-11T16:42:24.911 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.903+0000 7fac33fff700 10 client.17145 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:24.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.905+0000 7fac33fff700 1 -- 172.21.15.111:0/2422204193 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55efb6810120 con 0x55efb691f470 2023-10-11T16:42:24.912 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.905+0000 7fac33fff700 10 client.17145 handle_command_reply: tid=0 2023-10-11T16:42:24.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.905+0000 7fac6e462b80 1 client.17145 shutdown 2023-10-11T16:42:24.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.907+0000 7fac6e462b80 1 -- 172.21.15.111:0/2422204193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6903260 msgr2=0x55efb6906c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.907+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6903260 0x55efb6906c70 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fac54066770 tx=0x7fac5404eb70 comp rx=0 tx=0).stop 2023-10-11T16:42:24.913 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac6e462b80 1 -- 172.21.15.111:0/2422204193 shutdown_connections 2023-10-11T16:42:24.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55efb691f470 0x55efb6921880 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6903260 0x55efb6906c70 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.914 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb69092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2422204193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6902950 0x55efb6906530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac6e462b80 1 -- 172.21.15.111:0/2422204193 >> 172.21.15.111:0/2422204193 conn(0x55efb68f2630 msgr2=0x55efb68f70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac33fff700 0 client.17145 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:24.915 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac6e462b80 1 -- 172.21.15.111:0/2422204193 shutdown_connections 2023-10-11T16:42:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.908+0000 7fac6e462b80 1 -- 172.21.15.111:0/2422204193 wait complete. 2023-10-11T16:42:24.916 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.909+0000 7fac6e462b80 20 client.17145 trim_cache size 0 max 16384 2023-10-11T16:42:24.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.945+0000 7fac6e462b80 1 Processor -- start 2023-10-11T16:42:24.947 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.946+0000 7fac6e462b80 1 -- start start 2023-10-11T16:42:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.946+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb68fd180 0x55efb68fcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.946+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 0x55efb68f7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.948 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.946+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb6905d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.946+0000 7fac6e462b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb6902560 con 0x55efb6902950 2023-10-11T16:42:24.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.946+0000 7fac6e462b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb6904af0 con 0x55efb68fd180 2023-10-11T16:42:24.949 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.946+0000 7fac6e462b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb6901240 con 0x55efb6903260 2023-10-11T16:42:24.950 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.948+0000 7fac53fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 0x55efb68f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.948+0000 7fac53fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 0x55efb68f7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33872/0 (socket says 172.21.15.111:33872) 2023-10-11T16:42:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.948+0000 7fac53fff700 1 -- 172.21.15.111:0/2400612325 learned_addr learned my addr 172.21.15.111:0/2400612325 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:24.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.948+0000 7fac5919c700 1 --2- 172.21.15.111:0/2400612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb6905d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.949+0000 7fac5919c700 1 -- 172.21.15.111:0/2400612325 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb68fd180 msgr2=0x55efb68fcc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:24.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.949+0000 7fac5919c700 1 --2- 172.21.15.111:0/2400612325 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb68fd180 0x55efb68fcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.949+0000 7fac5919c700 1 -- 172.21.15.111:0/2400612325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 msgr2=0x55efb68f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:24.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.949+0000 7fac5919c700 1 --2- 172.21.15.111:0/2400612325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 0x55efb68f7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.949+0000 7fac5919c700 1 -- 172.21.15.111:0/2400612325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efb6813f60 con 0x55efb6903260 2023-10-11T16:42:24.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.949+0000 7fac5919c700 1 --2- 172.21.15.111:0/2400612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb6905d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fac540640e0 tx=0x7fac5407e320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cecb2a7dbd8f211 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:24.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.949+0000 7fac3f7fe700 1 -- 172.21.15.111:0/2400612325 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54083040 con 0x55efb6903260 2023-10-11T16:42:24.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.950+0000 7fac3f7fe700 1 -- 172.21.15.111:0/2400612325 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac5407c450 con 0x55efb6903260 2023-10-11T16:42:24.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.950+0000 7fac3f7fe700 1 -- 172.21.15.111:0/2400612325 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54076040 con 0x55efb6903260 2023-10-11T16:42:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.951+0000 7fac6e462b80 1 -- 172.21.15.111:0/2400612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 msgr2=0x55efb6905d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.951+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2400612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb6905d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fac540640e0 tx=0x7fac5407e320 comp rx=0 tx=0).stop 2023-10-11T16:42:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.951+0000 7fac6e462b80 1 -- 172.21.15.111:0/2400612325 shutdown_connections 2023-10-11T16:42:24.955 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.951+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2400612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb6903260 0x55efb6905d50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.951+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2400612325 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb68fd180 0x55efb68fcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.951+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2400612325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 0x55efb68f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.951+0000 7fac6e462b80 1 -- 172.21.15.111:0/2400612325 >> 172.21.15.111:0/2400612325 conn(0x55efb68f2630 msgr2=0x55efb69086d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:24.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.952+0000 7fac6e462b80 1 -- 172.21.15.111:0/2400612325 shutdown_connections 2023-10-11T16:42:24.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.952+0000 7fac6e462b80 1 -- 172.21.15.111:0/2400612325 wait complete. 2023-10-11T16:42:24.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.952+0000 7fac6e462b80 1 -- start start 2023-10-11T16:42:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.953+0000 7fac6e462b80 1 Processor -- start 2023-10-11T16:42:24.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.955+0000 7fac6e462b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb6925750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.957+0000 7fac5899b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb6925750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.958+0000 7fac5899b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb6925750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:42042/0 (socket says 172.21.15.111:42042) 2023-10-11T16:42:24.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.958+0000 7fac5899b700 1 -- 172.21.15.111:0/2212287951 learned_addr learned my addr 172.21.15.111:0/2212287951 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:24.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.958+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 0x55efb6925c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.958+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6903260 0x55efb69261d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.958+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb696c910 con 0x55efb6902950 2023-10-11T16:42:24.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.958+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb696ca80 con 0x55efb6903260 2023-10-11T16:42:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.958+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efb696cc40 con 0x55efb68fd180 2023-10-11T16:42:24.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.959+0000 7fac6e462b80 10 client.0 _ll_register_callbacks cb 0x7fac54071f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:24.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.960+0000 7fac5899b700 1 -- 172.21.15.111:0/2212287951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6903260 msgr2=0x55efb69261d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:24.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.960+0000 7fac5899b700 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6903260 0x55efb69261d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.961+0000 7fac5899b700 1 -- 172.21.15.111:0/2212287951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 msgr2=0x55efb6925c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:24.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.961+0000 7fac5899b700 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 0x55efb6925c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.961+0000 7fac5899b700 1 -- 172.21.15.111:0/2212287951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efb6814210 con 0x55efb68fd180 2023-10-11T16:42:24.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.961+0000 7fac5899b700 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb6925750 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fac48097da0 tx=0x7fac48086120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f48573f6f8f082f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:24.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.961+0000 7fac33fff700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:42:24.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.962+0000 7fac33fff700 1 -- 172.21.15.111:0/2212287951 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4809f3c0 con 0x55efb68fd180 2023-10-11T16:42:24.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.962+0000 7fac33fff700 1 -- 172.21.15.111:0/2212287951 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fac480a36f0 con 0x55efb68fd180 2023-10-11T16:42:24.966 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.962+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac40015070 con 0x55efb68fd180 2023-10-11T16:42:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.962+0000 7fac33fff700 1 -- 172.21.15.111:0/2212287951 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac4809f170 con 0x55efb68fd180 2023-10-11T16:42:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.962+0000 7fac6e462b80 10 client.27307 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.962+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55efb6664ce0 con 0x55efb68fd180 2023-10-11T16:42:24.968 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.963+0000 7fac33fff700 1 -- 172.21.15.111:0/2212287951 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac40015070 con 0x55efb68fd180 2023-10-11T16:42:24.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.963+0000 7fac33fff700 1 -- 172.21.15.111:0/2212287951 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fac4809b590 con 0x55efb68fd180 2023-10-11T16:42:24.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.963+0000 7fac6e462b80 10 client.27307 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:24.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.963+0000 7fac6e462b80 10 client.27307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:24.969 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.963+0000 7fac6e462b80 20 client.27307 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:24.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.963+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55efb6910090 0x55efb6910490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:24.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.963+0000 7fac6e462b80 4 client.27307 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-11T16:42:24.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.963+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55efb681c6e0 con 0x55efb6910090 2023-10-11T16:42:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.964+0000 7fac53fff700 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55efb6910090 0x55efb6910490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:24.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.966+0000 7fac53fff700 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55efb6910090 0x55efb6910490 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44a9bd6cb50093e4 server_cookie=178c7eeb338f8c62 in_seq=0 out_seq=0 2023-10-11T16:42:24.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.966+0000 7fac33fff700 10 client.27307 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:24.972 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac33fff700 1 -- 172.21.15.111:0/2212287951 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55efb681c6e0 con 0x55efb6910090 2023-10-11T16:42:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac33fff700 10 client.27307 handle_command_reply: tid=0 2023-10-11T16:42:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 client.27307 shutdown 2023-10-11T16:42:24.973 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 msgr2=0x55efb6925750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb6925750 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fac48097da0 tx=0x7fac48086120 comp rx=0 tx=0).stop 2023-10-11T16:42:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 shutdown_connections 2023-10-11T16:42:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x55efb6910090 0x55efb6910490 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.974 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55efb6903260 0x55efb69261d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55efb68fd180 0x55efb6925750 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 --2- 172.21.15.111:0/2212287951 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55efb6902950 0x55efb6925c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.967+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 >> 172.21.15.111:0/2212287951 conn(0x55efb68f2630 msgr2=0x55efb691a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:24.975 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.968+0000 7fac33fff700 0 client.27307 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:24.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.968+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 shutdown_connections 2023-10-11T16:42:24.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.968+0000 7fac6e462b80 1 -- 172.21.15.111:0/2212287951 wait complete. 2023-10-11T16:42:24.976 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.969+0000 7fac6e462b80 20 client.27307 trim_cache size 0 max 16384 2023-10-11T16:42:24.977 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:42:24.977 INFO:teuthology.orchestra.run.smithi111.stdout: "return_code": 0, 2023-10-11T16:42:24.977 INFO:teuthology.orchestra.run.smithi111.stdout: "scrub_tag": "bbbf7543-d1f8-4005-8f63-58f58295ae81", 2023-10-11T16:42:24.978 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "asynchronous" 2023-10-11T16:42:24.978 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:42:24.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 -- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fac440a1340 msgr2=0x7fac440a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fac440a1340 0x7fac440a37f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac54066400 tx=0x7fac5404ef20 comp rx=0 tx=0).stop 2023-10-11T16:42:24.979 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 -- 172.21.15.111:0/1649475769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac4c0a52d0 msgr2=0x7fac4c143430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:24.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac4c0a52d0 0x7fac4c143430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fac48000c00 tx=0x7fac48005740 comp rx=0 tx=0).stop 2023-10-11T16:42:24.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 -- 172.21.15.111:0/1649475769 shutdown_connections 2023-10-11T16:42:24.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fac440a1340 0x7fac440a37f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.980 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac4c13f0f0 0x7fac4c13f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fac4c0a52d0 0x7fac4c143430 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 --2- 172.21.15.111:0/1649475769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac4c0b09b0 0x7fac4c143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:24.981 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 -- 172.21.15.111:0/1649475769 >> 172.21.15.111:0/1649475769 conn(0x7fac4c01a2f0 msgr2=0x7fac4c0a7360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 -- 172.21.15.111:0/1649475769 shutdown_connections 2023-10-11T16:42:24.982 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:24.970+0000 7fac5999d700 1 -- 172.21.15.111:0/1649475769 wait complete. 2023-10-11T16:42:25.005 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:42:25.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.369+0000 7f5208d9d700 1 -- 172.21.15.111:0/4218586979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 msgr2=0x7f51fc0a56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:25.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.369+0000 7f5208d9d700 1 --2- 172.21.15.111:0/4218586979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 0x7f51fc0a56b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009990 tx=0x7f51f0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.370+0000 7f5208d9d700 1 -- 172.21.15.111:0/4218586979 shutdown_connections 2023-10-11T16:42:25.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.370+0000 7f5208d9d700 1 --2- 172.21.15.111:0/4218586979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51fc0b2d00 0x7f51fc0b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.370+0000 7f5208d9d700 1 --2- 172.21.15.111:0/4218586979 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51fc0a8350 0x7f51fc0a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.370+0000 7f5208d9d700 1 --2- 172.21.15.111:0/4218586979 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 0x7f51fc0a56b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.370+0000 7f5208d9d700 1 -- 172.21.15.111:0/4218586979 >> 172.21.15.111:0/4218586979 conn(0x7f51fc01a2f0 msgr2=0x7f51fc01a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:25.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.371+0000 7f5208d9d700 1 -- 172.21.15.111:0/4218586979 shutdown_connections 2023-10-11T16:42:25.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.371+0000 7f5208d9d700 1 -- 172.21.15.111:0/4218586979 wait complete. 2023-10-11T16:42:25.375 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.372+0000 7f5208d9d700 1 Processor -- start 2023-10-11T16:42:25.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.373+0000 7f5208d9d700 1 -- start start 2023-10-11T16:42:25.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.373+0000 7f5208d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 0x7f51fc0b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.376 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.373+0000 7f5208d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51fc0a8350 0x7f51fc0af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.374+0000 7f5208d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51fc0b2d00 0x7f51fc0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.374+0000 7f5208d9d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0afd20 con 0x7f51fc0a5240 2023-10-11T16:42:25.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.374+0000 7f5208d9d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0afe90 con 0x7f51fc0b2d00 2023-10-11T16:42:25.377 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.374+0000 7f5208d9d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0b0050 con 0x7f51fc0a8350 2023-10-11T16:42:25.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.374+0000 7f52037fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 0x7f51fc0b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.378 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.374+0000 7f52037fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 0x7f51fc0b2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33902/0 (socket says 172.21.15.111:33902) 2023-10-11T16:42:25.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.374+0000 7f52037fe700 1 -- 172.21.15.111:0/447215424 learned_addr learned my addr 172.21.15.111:0/447215424 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:25.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.375+0000 7f52037fe700 1 -- 172.21.15.111:0/447215424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51fc0a8350 msgr2=0x7f51fc0af2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-11T16:42:25.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.375+0000 7f52037fe700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51fc0a8350 0x7f51fc0af2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.379 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.375+0000 7f52037fe700 1 -- 172.21.15.111:0/447215424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51fc0b2d00 msgr2=0x7f51fc0af7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-11T16:42:25.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.375+0000 7f52037fe700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51fc0b2d00 0x7f51fc0af7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.375+0000 7f52037fe700 1 -- 172.21.15.111:0/447215424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520404f760 con 0x7f51fc0a5240 2023-10-11T16:42:25.380 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.376+0000 7f52037fe700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 0x7f51fc0b2290 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f51f800b530 tx=0x7f51f800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:25.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.378+0000 7f5200ff9700 1 -- 172.21.15.111:0/447215424 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8011830 con 0x7f51fc0a5240 2023-10-11T16:42:25.381 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.379+0000 7f5200ff9700 1 -- 172.21.15.111:0/447215424 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51f8011e50 con 0x7f51fc0a5240 2023-10-11T16:42:25.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.379+0000 7f5200ff9700 1 -- 172.21.15.111:0/447215424 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f800f2e0 con 0x7f51fc0a5240 2023-10-11T16:42:25.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.379+0000 7f5208d9d700 1 -- 172.21.15.111:0/447215424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f00095f0 con 0x7f51fc0a5240 2023-10-11T16:42:25.382 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.379+0000 7f5208d9d700 1 -- 172.21.15.111:0/447215424 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc0b86d0 con 0x7f51fc0a5240 2023-10-11T16:42:25.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.389+0000 7f5200ff9700 1 -- 172.21.15.111:0/447215424 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f51f80105a0 con 0x7f51fc0a5240 2023-10-11T16:42:25.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.392+0000 7f521da21b80 1 Processor -- start 2023-10-11T16:42:25.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.392+0000 7f521da21b80 1 -- start start 2023-10-11T16:42:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.393+0000 7f5200ff9700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51f40a12c0 0x7f51f40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.393+0000 7f5200ff9700 1 -- 172.21.15.111:0/447215424 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f51f80ccc20 con 0x7f51fc0a5240 2023-10-11T16:42:25.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.393+0000 7f521da21b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.394+0000 7f521da21b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e95bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.394+0000 7f52037fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.394+0000 7f52037fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33906/0 (socket says 172.21.15.111:33906) 2023-10-11T16:42:25.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.394+0000 7f52037fe700 1 -- 172.21.15.111:0/1153630609 learned_addr learned my addr 172.21.15.111:0/1153630609 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:25.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.394+0000 7f5203fff700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51f40a12c0 0x7f51f40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.394+0000 7f521da21b80 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04e96530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.395+0000 7f521da21b80 1 -- 172.21.15.111:0/1153630609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04d9b060 con 0x559a04e9b8d0 2023-10-11T16:42:25.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.395+0000 7f521da21b80 1 -- 172.21.15.111:0/1153630609 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04ded8a0 con 0x559a04e96100 2023-10-11T16:42:25.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.395+0000 7f521da21b80 1 -- 172.21.15.111:0/1153630609 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04de6d30 con 0x559a04e9c1e0 2023-10-11T16:42:25.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.395+0000 7f5203fff700 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04e96530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.396+0000 7f5202ffd700 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e95bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.396+0000 7f52037fe700 1 -- 172.21.15.111:0/1153630609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 msgr2=0x559a04e95bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:25.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.396+0000 7f52037fe700 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e95bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.396+0000 7f52037fe700 1 -- 172.21.15.111:0/1153630609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 msgr2=0x559a04e96530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.396+0000 7f52037fe700 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04e96530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.396+0000 7f52037fe700 1 -- 172.21.15.111:0/1153630609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a04c68630 con 0x559a04e9b8d0 2023-10-11T16:42:25.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.396+0000 7f5203fff700 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04e96530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:42:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.396+0000 7f5203fff700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51f40a12c0 0x7f51f40a3770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5204066400 tx=0x7f5204070770 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.397+0000 7f52037fe700 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9bca0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f51f8095ea0 tx=0x7f51f8099610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cd0ea209c6e9f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:25.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.397+0000 7f51edffb700 1 -- 172.21.15.111:0/1153630609 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f80a1df0 con 0x559a04e9b8d0 2023-10-11T16:42:25.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.398+0000 7f51edffb700 1 -- 172.21.15.111:0/1153630609 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51f8010380 con 0x559a04e9b8d0 2023-10-11T16:42:25.406 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.399+0000 7f521da21b80 1 -- 172.21.15.111:0/1153630609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 msgr2=0x559a04e9bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:25.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.399+0000 7f521da21b80 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9bca0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f51f8095ea0 tx=0x7f51f8099610 comp rx=0 tx=0).stop 2023-10-11T16:42:25.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.400+0000 7f521da21b80 1 -- 172.21.15.111:0/1153630609 shutdown_connections 2023-10-11T16:42:25.407 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.400+0000 7f521da21b80 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04e96530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.400+0000 7f521da21b80 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9bca0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.400+0000 7f521da21b80 1 --2- 172.21.15.111:0/1153630609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e95bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.400+0000 7f521da21b80 1 -- 172.21.15.111:0/1153630609 >> 172.21.15.111:0/1153630609 conn(0x559a04e8b5b0 msgr2=0x559a04e90ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:25.408 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.400+0000 7f521da21b80 1 -- 172.21.15.111:0/1153630609 shutdown_connections 2023-10-11T16:42:25.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.401+0000 7f521da21b80 1 -- 172.21.15.111:0/1153630609 wait complete. 2023-10-11T16:42:25.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.401+0000 7f521da21b80 1 -- start start 2023-10-11T16:42:25.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.402+0000 7f521da21b80 1 Processor -- start 2023-10-11T16:42:25.409 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.403+0000 7f521da21b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04ea21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.403+0000 7f521da21b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.404+0000 7f52037fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04ea21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.410 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.404+0000 7f52037fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04ea21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52252/0 (socket says 172.21.15.111:52252) 2023-10-11T16:42:25.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.404+0000 7f52037fe700 1 -- 172.21.15.111:0/1065767703 learned_addr learned my addr 172.21.15.111:0/1065767703 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:25.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.404+0000 7f521da21b80 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e9fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.411 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.404+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04d888e0 con 0x559a04e9b8d0 2023-10-11T16:42:25.412 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.405+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04d88580 con 0x559a04e96100 2023-10-11T16:42:25.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.405+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04c6a920 con 0x559a04e9c1e0 2023-10-11T16:42:25.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.405+0000 7f5203fff700 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e9fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.405+0000 7f5203fff700 1 -- 172.21.15.111:0/1065767703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 msgr2=0x559a04ea21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:25.413 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.405+0000 7f5203fff700 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04ea21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.405+0000 7f5203fff700 1 -- 172.21.15.111:0/1065767703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 msgr2=0x559a04e9f460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-11T16:42:25.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.405+0000 7f521da21b80 10 client.0 _ll_register_callbacks cb 0x559a04da6780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:25.414 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.406+0000 7f5202ffd700 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.406+0000 7f5203fff700 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.406+0000 7f5203fff700 1 -- 172.21.15.111:0/1065767703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc0b86d0 con 0x559a04e9c1e0 2023-10-11T16:42:25.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.406+0000 7f5203fff700 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e9fb50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f52040641e0 tx=0x7f520404eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db328cc6baa9f7db server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:25.415 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.406+0000 7f51e2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:42:25.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.407+0000 7f5202ffd700 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:25.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.407+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1065767703 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520408a640 con 0x559a04e9c1e0 2023-10-11T16:42:25.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.407+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1065767703 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f520407f400 con 0x559a04e9c1e0 2023-10-11T16:42:25.416 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.407+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a04e88d70 con 0x559a04e9c1e0 2023-10-11T16:42:25.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.410+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1065767703 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f520408a7a0 con 0x559a04e9c1e0 2023-10-11T16:42:25.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.410+0000 7f521da21b80 10 client.27316 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:25.417 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.410+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({fsmap=39}) v3 -- 0x559a04dad190 con 0x559a04e9c1e0 2023-10-11T16:42:25.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.411+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1065767703 <== mon.1 v2:172.21.15.150:3300/0 4 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f520408a900 con 0x559a04e9c1e0 2023-10-11T16:42:25.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.411+0000 7f521da21b80 10 client.27316 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:25.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.411+0000 7f521da21b80 10 client.27316 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:25.418 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.411+0000 7f521da21b80 20 client.27316 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:25.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.411+0000 7f521da21b80 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x559a04eaf8b0 0x559a04eb1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.419 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.411+0000 7f521da21b80 4 client.27316 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:25.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.411+0000 7f52037fe700 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x559a04eaf8b0 0x559a04eb1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.411+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559a04da90a0 con 0x559a04eaf8b0 2023-10-11T16:42:25.420 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.417+0000 7f51e2ffd700 10 client.27316 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:25.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.417+0000 7f52037fe700 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x559a04eaf8b0 0x559a04eb1ce0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e546952c64c06008 server_cookie=a7dd4fe245dc76b4 in_seq=0 out_seq=0 2023-10-11T16:42:25.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.419+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1065767703 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559a04da90a0 con 0x559a04eaf8b0 2023-10-11T16:42:25.421 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f51e2ffd700 10 client.27316 handle_command_reply: tid=0 2023-10-11T16:42:25.422 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 client.27316 shutdown 2023-10-11T16:42:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 msgr2=0x559a04e9fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e9fb50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f52040641e0 tx=0x7f520404eb70 comp rx=0 tx=0).stop 2023-10-11T16:42:25.423 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 shutdown_connections 2023-10-11T16:42:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x559a04eaf8b0 0x559a04eb1ce0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f51e2ffd700 0 client.27316 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9c1e0 0x559a04e9fb50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.424 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e96100 0x559a04ea21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 --2- 172.21.15.111:0/1065767703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9b8d0 0x559a04e9f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.420+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 >> 172.21.15.111:0/1065767703 conn(0x559a04e8b5b0 msgr2=0x559a04e90030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:25.425 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.421+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 shutdown_connections 2023-10-11T16:42:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.421+0000 7f521da21b80 1 -- 172.21.15.111:0/1065767703 wait complete. 2023-10-11T16:42:25.426 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.421+0000 7f521da21b80 20 client.27316 trim_cache size 0 max 16384 2023-10-11T16:42:25.454 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.453+0000 7f521da21b80 1 Processor -- start 2023-10-11T16:42:25.455 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.453+0000 7f521da21b80 1 -- start start 2023-10-11T16:42:25.456 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.453+0000 7f521da21b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e96100 0x559a04e9aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.454+0000 7f521da21b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9b8d0 0x559a04e907b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.454+0000 7f521da21b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9c1e0 0x559a04ea1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.457 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.454+0000 7f521da21b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04e91730 con 0x559a04e96100 2023-10-11T16:42:25.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.454+0000 7f521da21b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04e8b0e0 con 0x559a04e9c1e0 2023-10-11T16:42:25.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.454+0000 7f521da21b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04e9b500 con 0x559a04e9b8d0 2023-10-11T16:42:25.458 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.454+0000 7f5203fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9c1e0 0x559a04ea1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.454+0000 7f5203fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9c1e0 0x559a04ea1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52258/0 (socket says 172.21.15.111:52258) 2023-10-11T16:42:25.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.454+0000 7f5203fff700 1 -- 172.21.15.111:0/68506821 learned_addr learned my addr 172.21.15.111:0/68506821 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:25.459 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.455+0000 7f5202ffd700 1 --2- 172.21.15.111:0/68506821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9b8d0 0x559a04e907b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.455+0000 7f5202ffd700 1 -- 172.21.15.111:0/68506821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9c1e0 msgr2=0x559a04ea1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:25.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.455+0000 7f5202ffd700 1 --2- 172.21.15.111:0/68506821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9c1e0 0x559a04ea1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.460 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.455+0000 7f5202ffd700 1 -- 172.21.15.111:0/68506821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e96100 msgr2=0x559a04e9aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:25.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.455+0000 7f5202ffd700 1 --2- 172.21.15.111:0/68506821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e96100 0x559a04e9aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.461 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.455+0000 7f5202ffd700 1 -- 172.21.15.111:0/68506821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a04dacee0 con 0x559a04e9b8d0 2023-10-11T16:42:25.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.455+0000 7f5202ffd700 1 --2- 172.21.15.111:0/68506821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9b8d0 0x559a04e907b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f51f0000c00 tx=0x7f51f0005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca5251bf1ba3be3b server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:25.463 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.456+0000 7f51ee7fc700 1 -- 172.21.15.111:0/68506821 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f000be30 con 0x559a04e9b8d0 2023-10-11T16:42:25.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.456+0000 7f51ee7fc700 1 -- 172.21.15.111:0/68506821 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51f001c070 con 0x559a04e9b8d0 2023-10-11T16:42:25.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.456+0000 7f51ee7fc700 1 -- 172.21.15.111:0/68506821 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0004c90 con 0x559a04e9b8d0 2023-10-11T16:42:25.464 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.457+0000 7f521da21b80 1 -- 172.21.15.111:0/68506821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9b8d0 msgr2=0x559a04e907b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:25.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.457+0000 7f521da21b80 1 --2- 172.21.15.111:0/68506821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9b8d0 0x559a04e907b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f51f0000c00 tx=0x7f51f0005bc0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.457+0000 7f521da21b80 1 -- 172.21.15.111:0/68506821 shutdown_connections 2023-10-11T16:42:25.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.457+0000 7f521da21b80 1 --2- 172.21.15.111:0/68506821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9c1e0 0x559a04ea1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.465 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.457+0000 7f521da21b80 1 --2- 172.21.15.111:0/68506821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e96100 0x559a04e9aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.457+0000 7f521da21b80 1 --2- 172.21.15.111:0/68506821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e9b8d0 0x559a04e907b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.457+0000 7f521da21b80 1 -- 172.21.15.111:0/68506821 >> 172.21.15.111:0/68506821 conn(0x559a04e8b5b0 msgr2=0x559a04e9a590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:25.466 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.458+0000 7f521da21b80 1 -- 172.21.15.111:0/68506821 shutdown_connections 2023-10-11T16:42:25.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.458+0000 7f521da21b80 1 -- 172.21.15.111:0/68506821 wait complete. 2023-10-11T16:42:25.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.459+0000 7f521da21b80 1 -- start start 2023-10-11T16:42:25.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.459+0000 7f521da21b80 1 Processor -- start 2023-10-11T16:42:25.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.460+0000 7f521da21b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e96100 0x559a04ea5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.460+0000 7f521da21b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9b8d0 0x559a04ea57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.461+0000 7f521da21b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9c1e0 0x559a04ea5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.468 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.461+0000 7f521da21b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04ea7220 con 0x559a04e9c1e0 2023-10-11T16:42:25.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.461+0000 7f521da21b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04f037e0 con 0x559a04e9b8d0 2023-10-11T16:42:25.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.461+0000 7f521da21b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a04f03920 con 0x559a04e96100 2023-10-11T16:42:25.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.461+0000 7f521da21b80 10 client.0 _ll_register_callbacks cb 0x7f5204061fa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:25.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.461+0000 7f52037fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e96100 0x559a04ea5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.462+0000 7f52037fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e96100 0x559a04ea5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:42112/0 (socket says 172.21.15.111:42112) 2023-10-11T16:42:25.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.462+0000 7f52037fe700 1 -- 172.21.15.111:0/1356293415 learned_addr learned my addr 172.21.15.111:0/1356293415 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:25.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.462+0000 7f5202ffd700 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9b8d0 0x559a04ea57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.463+0000 7f5203fff700 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9c1e0 0x559a04ea5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.465+0000 7f52037fe700 1 -- 172.21.15.111:0/1356293415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9b8d0 msgr2=0x559a04ea57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:25.474 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.465+0000 7f52037fe700 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9b8d0 0x559a04ea57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.465+0000 7f52037fe700 1 -- 172.21.15.111:0/1356293415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9c1e0 msgr2=0x559a04ea5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:25.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.465+0000 7f52037fe700 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9c1e0 0x559a04ea5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.465+0000 7f52037fe700 1 -- 172.21.15.111:0/1356293415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc0b5cd0 con 0x559a04e96100 2023-10-11T16:42:25.475 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.466+0000 7f52037fe700 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e96100 0x559a04ea5260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f51f800bcf0 tx=0x7f51f80109a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=558d6d7e19ea7ec6 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:25.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.466+0000 7f51e2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2023-10-11T16:42:25.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.466+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1356293415 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8009300 con 0x559a04e96100 2023-10-11T16:42:25.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.466+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1356293415 <== mon.1 v2:172.21.15.150:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f51f80b2040 con 0x559a04e96100 2023-10-11T16:42:25.476 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f521da21b80 1 -- 172.21.15.111:0/1356293415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f51f8010380 con 0x559a04e96100 2023-10-11T16:42:25.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1356293415 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51f8097dc0 con 0x559a04e96100 2023-10-11T16:42:25.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f521da21b80 10 client.27322 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:25.477 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f521da21b80 1 -- 172.21.15.111:0/1356293415 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x559a04bfdc60 con 0x559a04e96100 2023-10-11T16:42:25.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1356293415 <== mon.1 v2:172.21.15.150:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51f8010380 con 0x559a04e96100 2023-10-11T16:42:25.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1356293415 <== mon.1 v2:172.21.15.150:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f51f80a1960 con 0x559a04e96100 2023-10-11T16:42:25.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f521da21b80 10 client.27322 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:25.478 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f521da21b80 10 client.27322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:25.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f521da21b80 20 client.27322 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:25.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.467+0000 7f521da21b80 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x559a04f04cc0 0x559a04f070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:25.479 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.468+0000 7f521da21b80 4 client.27322 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:42:25.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.468+0000 7f521da21b80 1 -- 172.21.15.111:0/1356293415 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559a04db5660 con 0x559a04f04cc0 2023-10-11T16:42:25.480 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.468+0000 7f5202ffd700 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x559a04f04cc0 0x559a04f070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:25.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.469+0000 7f5202ffd700 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x559a04f04cc0 0x559a04f070a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=683a5a76fd2abe8e server_cookie=d8881ef2bf32b9d7 in_seq=0 out_seq=0 2023-10-11T16:42:25.481 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.469+0000 7f51e2ffd700 10 client.27322 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:25.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.469+0000 7f51e2ffd700 1 -- 172.21.15.111:0/1356293415 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559a04db5660 con 0x559a04f04cc0 2023-10-11T16:42:25.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.469+0000 7f51e2ffd700 10 client.27322 handle_command_reply: tid=0 2023-10-11T16:42:25.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.469+0000 7f521da21b80 1 client.27322 shutdown 2023-10-11T16:42:25.482 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f521da21b80 1 -- 172.21.15.111:0/1356293415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e96100 msgr2=0x559a04ea5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:25.483 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f521da21b80 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e96100 0x559a04ea5260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f51f800bcf0 tx=0x7f51f80109a0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f521da21b80 1 -- 172.21.15.111:0/1356293415 shutdown_connections 2023-10-11T16:42:25.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f521da21b80 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x559a04f04cc0 0x559a04f070a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f521da21b80 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x559a04e9c1e0 0x559a04ea5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.484 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f521da21b80 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x559a04e96100 0x559a04ea5260 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f521da21b80 1 --2- 172.21.15.111:0/1356293415 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a04e9b8d0 0x559a04ea57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f521da21b80 1 -- 172.21.15.111:0/1356293415 >> 172.21.15.111:0/1356293415 conn(0x559a04e8b5b0 msgr2=0x559a04e94bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:25.485 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.472+0000 7f51e2ffd700 0 client.27322 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:25.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.473+0000 7f521da21b80 1 -- 172.21.15.111:0/1356293415 shutdown_connections 2023-10-11T16:42:25.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.473+0000 7f521da21b80 1 -- 172.21.15.111:0/1356293415 wait complete. 2023-10-11T16:42:25.486 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.475+0000 7f521da21b80 20 client.27322 trim_cache size 0 max 16384 2023-10-11T16:42:25.487 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:42:25.487 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-11T16:42:25.487 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": { 2023-10-11T16:42:25.488 INFO:teuthology.orchestra.run.smithi111.stdout: "bbbf7543-d1f8-4005-8f63-58f58295ae81": { 2023-10-11T16:42:25.488 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2023-10-11T16:42:25.488 INFO:teuthology.orchestra.run.smithi111.stdout: "tag": "bbbf7543-d1f8-4005-8f63-58f58295ae81", 2023-10-11T16:42:25.488 INFO:teuthology.orchestra.run.smithi111.stdout: "options": "recursive,force" 2023-10-11T16:42:25.489 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:42:25.489 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:42:25.489 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:42:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 -- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51f40a12c0 msgr2=0x7f51f40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:25.490 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51f40a12c0 0x7f51f40a3770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5204066400 tx=0x7f5204070770 comp rx=0 tx=0).stop 2023-10-11T16:42:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 -- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 msgr2=0x7f51fc0b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 0x7f51fc0b2290 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f51f800b530 tx=0x7f51f800b8f0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 -- 172.21.15.111:0/447215424 shutdown_connections 2023-10-11T16:42:25.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51f40a12c0 0x7f51f40a3770 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51fc0b2d00 0x7f51fc0af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0a5240 0x7f51fc0b2290 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 --2- 172.21.15.111:0/447215424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51fc0a8350 0x7f51fc0af2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:25.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 -- 172.21.15.111:0/447215424 >> 172.21.15.111:0/447215424 conn(0x7f51fc01a2f0 msgr2=0x7f51fc0b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:25.493 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.476+0000 7f5208d9d700 1 -- 172.21.15.111:0/447215424 shutdown_connections 2023-10-11T16:42:25.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:25.477+0000 7f5208d9d700 1 -- 172.21.15.111:0/447215424 wait complete. 2023-10-11T16:42:25.513 INFO:tasks.cephfs.filesystem:scrub status for tag:bbbf7543-d1f8-4005-8f63-58f58295ae81 - {'path': '/', 'tag': 'bbbf7543-d1f8-4005-8f63-58f58295ae81', 'options': 'recursive,force'} 2023-10-11T16:42:25.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:25 smithi111 ceph-mon[98527]: pgmap v303: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:42:25.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:25 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:42:25.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:25 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:25 smithi150 ceph-mon[132217]: pgmap v303: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:42:25.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:25 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:42:25.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:25 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:26.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:25 smithi130 ceph-mon[131859]: pgmap v303: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:42:26.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:25 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:42:26.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:25 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:26.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:26 smithi150 ceph-mon[132217]: scrub queued for path: / 2023-10-11T16:42:26.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:26 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:26.954 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:26 smithi150 ceph-mon[132217]: scrub summary: active paths [/] 2023-10-11T16:42:26.955 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:26 smithi150 ceph-mon[132217]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:27.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:26 smithi130 ceph-mon[131859]: scrub queued for path: / 2023-10-11T16:42:27.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:26 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:27.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:26 smithi130 ceph-mon[131859]: scrub summary: active paths [/] 2023-10-11T16:42:27.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:26 smithi130 ceph-mon[131859]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:27.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:26 smithi111 ceph-mon[98527]: scrub queued for path: / 2023-10-11T16:42:27.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:26 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:27.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:26 smithi111 ceph-mon[98527]: scrub summary: active paths [/] 2023-10-11T16:42:27.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:26 smithi111 ceph-mon[98527]: scrub summary: idle+waiting paths [/] 2023-10-11T16:42:27.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:27 smithi150 ceph-mon[132217]: pgmap v304: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:42:28.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:27 smithi130 ceph-mon[131859]: pgmap v304: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:42:28.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:27 smithi111 ceph-mon[98527]: pgmap v304: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:42:29.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:29 smithi150 ceph-mon[132217]: pgmap v305: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:42:30.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:29 smithi130 ceph-mon[131859]: pgmap v305: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:42:30.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:29 smithi111 ceph-mon[98527]: pgmap v305: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:42:31.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:31 smithi130 ceph-mon[131859]: pgmap v306: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 8 op/s 2023-10-11T16:42:31.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:31 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:42:31.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:31 smithi130 ceph-mon[131859]: scrub summary: idle 2023-10-11T16:42:31.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:31 smithi111 ceph-mon[98527]: pgmap v306: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 8 op/s 2023-10-11T16:42:31.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:31 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:42:31.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:31 smithi111 ceph-mon[98527]: scrub summary: idle 2023-10-11T16:42:31.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:31 smithi150 ceph-mon[132217]: pgmap v306: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 8 op/s 2023-10-11T16:42:31.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:31 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:42:31.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:31 smithi150 ceph-mon[132217]: scrub summary: idle 2023-10-11T16:42:33.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:33 smithi130 ceph-mon[131859]: pgmap v307: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:42:33.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:33 smithi111 ceph-mon[98527]: pgmap v307: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:42:33.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:33 smithi150 ceph-mon[132217]: pgmap v307: 129 pgs: 66 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 42 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-11T16:42:35.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:35 smithi130 ceph-mon[131859]: pgmap v308: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 6 op/s 2023-10-11T16:42:35.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:35 smithi111 ceph-mon[98527]: pgmap v308: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 6 op/s 2023-10-11T16:42:35.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:35 smithi150 ceph-mon[132217]: pgmap v308: 129 pgs: 54 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 57 active+clean; 663 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 6 op/s 2023-10-11T16:42:37.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:37 smithi130 ceph-mon[131859]: pgmap v309: 129 pgs: 51 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 61 active+clean; 663 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:42:37.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:37 smithi111 ceph-mon[98527]: pgmap v309: 129 pgs: 51 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 61 active+clean; 663 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:42:37.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:37 smithi150 ceph-mon[132217]: pgmap v309: 129 pgs: 51 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 61 active+clean; 663 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-11T16:42:39.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:39 smithi130 ceph-mon[131859]: pgmap v310: 129 pgs: 51 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 61 active+clean; 663 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5 op/s 2023-10-11T16:42:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:39 smithi111 ceph-mon[98527]: pgmap v310: 129 pgs: 51 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 61 active+clean; 663 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5 op/s 2023-10-11T16:42:39.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:39 smithi150 ceph-mon[132217]: pgmap v310: 129 pgs: 51 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 61 active+clean; 663 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5 op/s 2023-10-11T16:42:40.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-10-11T16:42:41.150 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.148+0000 7f0797bff700 1 -- 172.21.15.111:0/3106156226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0790111020 msgr2=0x7f0790113400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:41.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.148+0000 7f0797bff700 1 --2- 172.21.15.111:0/3106156226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0790111020 0x7f0790113400 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f07800098f0 tx=0x7f0780009c00 comp rx=0 tx=0).stop 2023-10-11T16:42:41.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.148+0000 7f0797bff700 1 -- 172.21.15.111:0/3106156226 shutdown_connections 2023-10-11T16:42:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.148+0000 7f0797bff700 1 --2- 172.21.15.111:0/3106156226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0790111020 0x7f0790113400 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.148+0000 7f0797bff700 1 --2- 172.21.15.111:0/3106156226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900fe300 0x7f07900fe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.148+0000 7f0797bff700 1 --2- 172.21.15.111:0/3106156226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fec10 0x7f0790110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.148+0000 7f0797bff700 1 -- 172.21.15.111:0/3106156226 >> 172.21.15.111:0/3106156226 conn(0x7f07900762a0 msgr2=0x7f07900766a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:41.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.149+0000 7f0797bff700 1 -- 172.21.15.111:0/3106156226 shutdown_connections 2023-10-11T16:42:41.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.149+0000 7f0797bff700 1 -- 172.21.15.111:0/3106156226 wait complete. 2023-10-11T16:42:41.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.150+0000 7f0797bff700 1 Processor -- start 2023-10-11T16:42:41.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.150+0000 7f0797bff700 1 -- start start 2023-10-11T16:42:41.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f0797bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fe300 0x7f07901a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:41.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f0797bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900fec10 0x7f07901a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:41.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f0797bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0790111020 0x7f079019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:41.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f0797bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079019ce30 con 0x7f07900fec10 2023-10-11T16:42:41.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f0797bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079019cfa0 con 0x7f0790111020 2023-10-11T16:42:41.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f079599b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fe300 0x7f07901a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:41.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f0797bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079019d160 con 0x7f07900fe300 2023-10-11T16:42:41.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f079599b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fe300 0x7f07901a2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:53290/0 (socket says 172.21.15.111:53290) 2023-10-11T16:42:41.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.151+0000 7f079599b700 1 -- 172.21.15.111:0/2630870025 learned_addr learned my addr 172.21.15.111:0/2630870025 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:41.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.152+0000 7f079619c700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0790111020 0x7f079019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:41.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.152+0000 7f079519a700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900fec10 0x7f07901a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:41.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.152+0000 7f079599b700 1 -- 172.21.15.111:0/2630870025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0790111020 msgr2=0x7f079019c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:41.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.152+0000 7f079599b700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0790111020 0x7f079019c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.152+0000 7f079599b700 1 -- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900fec10 msgr2=0x7f07901a2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:41.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.152+0000 7f079599b700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900fec10 0x7f07901a2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.153+0000 7f079599b700 1 -- 172.21.15.111:0/2630870025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0780009520 con 0x7f07900fe300 2023-10-11T16:42:41.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.155+0000 7f079599b700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fe300 0x7f07901a2700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0788007f10 tx=0x7f078800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:41.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.155+0000 7f0786ffd700 1 -- 172.21.15.111:0/2630870025 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788017070 con 0x7f07900fe300 2023-10-11T16:42:41.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.156+0000 7f0786ffd700 1 -- 172.21.15.111:0/2630870025 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f078800f040 con 0x7f07900fe300 2023-10-11T16:42:41.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.156+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0790116920 con 0x7f07900fe300 2023-10-11T16:42:41.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.156+0000 7f0786ffd700 1 -- 172.21.15.111:0/2630870025 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788004630 con 0x7f07900fe300 2023-10-11T16:42:41.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.156+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0790116790 con 0x7f07900fe300 2023-10-11T16:42:41.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.158+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0790116ab0 con 0x7f07900fe300 2023-10-11T16:42:41.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.160+0000 7f0786ffd700 1 -- 172.21.15.111:0/2630870025 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f0788004790 con 0x7f07900fe300 2023-10-11T16:42:41.165 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.162+0000 7f0786ffd700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f077c0a1410 0x7f077c0a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:41.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.163+0000 7f0786ffd700 1 -- 172.21.15.111:0/2630870025 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7f07880cc7a0 con 0x7f07900fe300 2023-10-11T16:42:41.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.164+0000 7f079519a700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f077c0a1410 0x7f077c0a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:41.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.165+0000 7f079519a700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f077c0a1410 0x7f077c0a38c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f079019dfc0 tx=0x7f078c009310 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:41.170 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.169+0000 7f0786ffd700 1 -- 172.21.15.111:0/2630870025 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0788095750 con 0x7f07900fe300 2023-10-11T16:42:41.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:41 smithi130 ceph-mon[131859]: pgmap v311: 129 pgs: 30 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 87 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2023-10-11T16:42:41.619 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.618+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 --> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f079010f1e0 con 0x7f077c0a1410 2023-10-11T16:42:41.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:41 smithi111 ceph-mon[98527]: pgmap v311: 129 pgs: 30 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 87 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2023-10-11T16:42:41.663 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.662+0000 7f0786ffd700 1 -- 172.21.15.111:0/2630870025 <== mgr.25544 v2:172.21.15.111:6800/1233945144 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+335 (secure 0 0 0) 0x7f079010f1e0 con 0x7f077c0a1410 2023-10-11T16:42:41.664 INFO:teuthology.orchestra.run.smithi111.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-10-11T00:00:00", "created": "2023-10-11T16:32:38", "first": "2023-10-11T16:33:00", "last": "2023-10-11T16:42:00", "last_pruned": "2023-10-11T16:42:00", "created_count": 10, "pruned_count": 4, "active": true} 2023-10-11T16:42:41.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f077c0a1410 msgr2=0x7f077c0a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:41.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f077c0a1410 0x7f077c0a38c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f079019dfc0 tx=0x7f078c009310 comp rx=0 tx=0).stop 2023-10-11T16:42:41.668 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fe300 msgr2=0x7f07901a2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:41.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fe300 0x7f07901a2700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0788007f10 tx=0x7f078800d3b0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 shutdown_connections 2023-10-11T16:42:41.669 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f077c0a1410 0x7f077c0a38c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0790111020 0x7f079019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07900fe300 0x7f07901a2700 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 --2- 172.21.15.111:0/2630870025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900fec10 0x7f07901a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:41.670 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 >> 172.21.15.111:0/2630870025 conn(0x7f07900762a0 msgr2=0x7f079010ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:41.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.666+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 shutdown_connections 2023-10-11T16:42:41.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:41.667+0000 7f0797bff700 1 -- 172.21.15.111:0/2630870025 wait complete. 2023-10-11T16:42:41.701 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-10-11T16:42:41.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:41 smithi150 ceph-mon[132217]: pgmap v311: 129 pgs: 30 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 87 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2023-10-11T16:42:42.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.065+0000 7fb9c5bff700 1 -- 172.21.15.111:0/2098676625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0119a40 msgr2=0x7fb9c011be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:42.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.065+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/2098676625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0119a40 0x7fb9c011be20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb9b801c0d0 tx=0x7fb9b801c3e0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.066+0000 7fb9c5bff700 1 -- 172.21.15.111:0/2098676625 shutdown_connections 2023-10-11T16:42:42.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.066+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/2098676625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0119a40 0x7fb9c011be20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.066+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/2098676625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9c0107af0 0x7fb9c010eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.066+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/2098676625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c0107ec0 0x7fb9c0108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.069 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.066+0000 7fb9c5bff700 1 -- 172.21.15.111:0/2098676625 >> 172.21.15.111:0/2098676625 conn(0x7fb9c006c510 msgr2=0x7fb9c00716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:42.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.066+0000 7fb9c5bff700 1 -- 172.21.15.111:0/2098676625 shutdown_connections 2023-10-11T16:42:42.070 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.066+0000 7fb9c5bff700 1 -- 172.21.15.111:0/2098676625 wait complete. 2023-10-11T16:42:42.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.069+0000 7fb9c5bff700 1 Processor -- start 2023-10-11T16:42:42.071 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.069+0000 7fb9c5bff700 1 -- start start 2023-10-11T16:42:42.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9c5bff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c0107af0 0x7fb9c0114e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:42.072 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9c5bff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0107ec0 0x7fb9c01153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:42.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9c5bff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9c0119a40 0x7fb9c01a2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:42.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9c5bff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c01a2c90 con 0x7fb9c0107af0 2023-10-11T16:42:42.073 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9c5bff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c01a2e00 con 0x7fb9c0119a40 2023-10-11T16:42:42.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9c5bff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c01a2fc0 con 0x7fb9c0107ec0 2023-10-11T16:42:42.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9beffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0107ec0 0x7fb9c01153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:42.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9beffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0107ec0 0x7fb9c01153b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:53298/0 (socket says 172.21.15.111:53298) 2023-10-11T16:42:42.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9beffd700 1 -- 172.21.15.111:0/4043020212 learned_addr learned my addr 172.21.15.111:0/4043020212 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:42.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9bffff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9c0119a40 0x7fb9c01a2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:42.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.070+0000 7fb9bf7fe700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c0107af0 0x7fb9c0114e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:42.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.071+0000 7fb9bffff700 1 -- 172.21.15.111:0/4043020212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0107ec0 msgr2=0x7fb9c01153b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:42.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.071+0000 7fb9bffff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0107ec0 0x7fb9c01153b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.071+0000 7fb9bffff700 1 -- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c0107af0 msgr2=0x7fb9c0114e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:42.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.071+0000 7fb9bffff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c0107af0 0x7fb9c0114e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.071+0000 7fb9bffff700 1 -- 172.21.15.111:0/4043020212 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b801ad10 con 0x7fb9c0119a40 2023-10-11T16:42:42.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.071+0000 7fb9bffff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9c0119a40 0x7fb9c01a2650 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb9b801c7b0 tx=0x7fb9b8018940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:42.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.073+0000 7fb9bcff9700 1 -- 172.21.15.111:0/4043020212 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b801f050 con 0x7fb9c0119a40 2023-10-11T16:42:42.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.073+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c0111a30 con 0x7fb9c0119a40 2023-10-11T16:42:42.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.073+0000 7fb9bcff9700 1 -- 172.21.15.111:0/4043020212 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9b80050b0 con 0x7fb9c0119a40 2023-10-11T16:42:42.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.073+0000 7fb9bcff9700 1 -- 172.21.15.111:0/4043020212 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b802c6b0 con 0x7fb9c0119a40 2023-10-11T16:42:42.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.073+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c01118a0 con 0x7fb9c0119a40 2023-10-11T16:42:42.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.075+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c0111bc0 con 0x7fb9c0119a40 2023-10-11T16:42:42.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.078+0000 7fb9bcff9700 1 -- 172.21.15.111:0/4043020212 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fb9b8003a00 con 0x7fb9c0119a40 2023-10-11T16:42:42.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.083+0000 7fb9bcff9700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb9a80a12c0 0x7fb9a80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:42.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.083+0000 7fb9bcff9700 1 -- 172.21.15.111:0/4043020212 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9644+0+0 (secure 0 0 0) 0x7fb9b80df5a0 con 0x7fb9c0119a40 2023-10-11T16:42:42.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.083+0000 7fb9bf7fe700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb9a80a12c0 0x7fb9a80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:42.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.084+0000 7fb9bf7fe700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb9a80a12c0 0x7fb9a80a3770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009c60 tx=0x7fb9b4009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:42.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.086+0000 7fb9bcff9700 1 -- 172.21.15.111:0/4043020212 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb9b80a85e0 con 0x7fb9c0119a40 2023-10-11T16:42:42.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.523+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 --> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb9c0067c80 con 0x7fb9a80a12c0 2023-10-11T16:42:42.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.543+0000 7fb9bcff9700 1 -- 172.21.15.111:0/4043020212 <== mgr.25544 v2:172.21.15.111:6800/1233945144 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fb9c0067c80 con 0x7fb9a80a12c0 2023-10-11T16:42:42.545 INFO:teuthology.orchestra.run.smithi111.stdout:/ 1M 6M3h 2023-10-11T16:42:42.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb9a80a12c0 msgr2=0x7fb9a80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:42.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb9a80a12c0 0x7fb9a80a3770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009c60 tx=0x7fb9b4009260 comp rx=0 tx=0).stop 2023-10-11T16:42:42.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9c0119a40 msgr2=0x7fb9c01a2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:42.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9c0119a40 0x7fb9c01a2650 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb9b801c7b0 tx=0x7fb9b8018940 comp rx=0 tx=0).stop 2023-10-11T16:42:42.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 shutdown_connections 2023-10-11T16:42:42.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb9a80a12c0 0x7fb9a80a3770 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb9c0119a40 0x7fb9c01a2650 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c0107af0 0x7fb9c0114e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 --2- 172.21.15.111:0/4043020212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c0107ec0 0x7fb9c01153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:42.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 >> 172.21.15.111:0/4043020212 conn(0x7fb9c006c510 msgr2=0x7fb9c0108ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:42.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 shutdown_connections 2023-10-11T16:42:42.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:42.547+0000 7fb9c5bff700 1 -- 172.21.15.111:0/4043020212 wait complete. 2023-10-11T16:42:42.582 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2023-10-11T16:42:42.595 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2023-10-11T16:42:42.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:42 smithi111 ceph-mon[98527]: from='client.27328 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:42:42.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:42 smithi150 ceph-mon[132217]: from='client.27328 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:42:42.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:42 smithi130 ceph-mon[131859]: from='client.27328 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:42:43.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:43 smithi111 ceph-mon[98527]: pgmap v312: 129 pgs: 30 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 87 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-11T16:42:43.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:43 smithi111 ceph-mon[98527]: from='client.27335 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:42:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:43 smithi150 ceph-mon[132217]: pgmap v312: 129 pgs: 30 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 87 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-11T16:42:43.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:43 smithi150 ceph-mon[132217]: from='client.27335 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:42:43.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:43 smithi130 ceph-mon[131859]: pgmap v312: 129 pgs: 30 active+clean+snaptrim_wait, 12 active+clean+snaptrim, 87 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-11T16:42:43.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:43 smithi130 ceph-mon[131859]: from='client.27335 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-11T16:42:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:45 smithi111 ceph-mon[98527]: pgmap v313: 129 pgs: 22 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 97 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2023-10-11T16:42:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:45 smithi150 ceph-mon[132217]: pgmap v313: 129 pgs: 22 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 97 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2023-10-11T16:42:45.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:45 smithi130 ceph-mon[131859]: pgmap v313: 129 pgs: 22 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 97 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2023-10-11T16:42:47.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:47 smithi111 ceph-mon[98527]: pgmap v314: 129 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 117 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-11T16:42:47.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:47 smithi150 ceph-mon[132217]: pgmap v314: 129 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 117 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-11T16:42:47.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:47 smithi130 ceph-mon[131859]: pgmap v314: 129 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 117 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-11T16:42:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:49 smithi111 ceph-mon[98527]: pgmap v315: 129 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 117 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-11T16:42:49.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:49 smithi150 ceph-mon[132217]: pgmap v315: 129 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 117 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-11T16:42:49.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:49 smithi130 ceph-mon[131859]: pgmap v315: 129 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 117 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-11T16:42:51.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:51 smithi130 ceph-mon[131859]: pgmap v316: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-11T16:42:51.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:51 smithi130 ceph-mon[131859]: osdmap e100: 12 total, 12 up, 12 in 2023-10-11T16:42:51.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:51 smithi111 ceph-mon[98527]: pgmap v316: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-11T16:42:51.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:51 smithi111 ceph-mon[98527]: osdmap e100: 12 total, 12 up, 12 in 2023-10-11T16:42:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:51 smithi150 ceph-mon[132217]: pgmap v316: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-11T16:42:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:51 smithi150 ceph-mon[132217]: osdmap e100: 12 total, 12 up, 12 in 2023-10-11T16:42:53.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:53 smithi130 ceph-mon[131859]: pgmap v318: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-11T16:42:53.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:53 smithi111 ceph-mon[98527]: pgmap v318: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-11T16:42:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:53 smithi150 ceph-mon[132217]: pgmap v318: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-11T16:42:55.515 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-11T16:42:55.581 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:55 smithi111 ceph-mon[98527]: pgmap v319: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-11T16:42:55.581 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:55 smithi111 ceph-mon[98527]: osdmap e101: 12 total, 12 up, 12 in 2023-10-11T16:42:55.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:55 smithi130 ceph-mon[131859]: pgmap v319: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-11T16:42:55.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:55 smithi130 ceph-mon[131859]: osdmap e101: 12 total, 12 up, 12 in 2023-10-11T16:42:55.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:55 smithi150 ceph-mon[132217]: pgmap v319: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-11T16:42:55.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:55 smithi150 ceph-mon[132217]: osdmap e101: 12 total, 12 up, 12 in 2023-10-11T16:42:55.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 -- 172.21.15.111:0/2292659308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc105620 msgr2=0x7f8ccc10d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/2292659308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc105620 0x7f8ccc10d1a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400b0d0 tx=0x7f8cc400b3e0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 -- 172.21.15.111:0/2292659308 shutdown_connections 2023-10-11T16:42:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/2292659308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc105620 0x7f8ccc10d1a0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/2292659308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ccc100cc0 0x7f8ccc101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/2292659308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ccc101660 0x7f8ccc104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 -- 172.21.15.111:0/2292659308 >> 172.21.15.111:0/2292659308 conn(0x7f8ccc077560 msgr2=0x7f8ccc077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:55.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 -- 172.21.15.111:0/2292659308 shutdown_connections 2023-10-11T16:42:55.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.724+0000 7f8cd39ff700 1 -- 172.21.15.111:0/2292659308 wait complete. 2023-10-11T16:42:55.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.726+0000 7f8cd39ff700 1 Processor -- start 2023-10-11T16:42:55.730 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.726+0000 7f8cd39ff700 1 -- start start 2023-10-11T16:42:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.726+0000 7f8cd39ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc100cc0 0x7f8ccc072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.726+0000 7f8cd39ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ccc101660 0x7f8ccc073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.731 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.726+0000 7f8cd39ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ccc105620 0x7f8ccc1a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.726+0000 7f8cd39ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc073b60 con 0x7f8ccc100cc0 2023-10-11T16:42:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.726+0000 7f8cd39ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc1a4d50 con 0x7f8ccc105620 2023-10-11T16:42:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.726+0000 7f8cd39ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc1a4e90 con 0x7f8ccc101660 2023-10-11T16:42:55.732 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc100cc0 0x7f8ccc072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc100cc0 0x7f8ccc072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44980/0 (socket says 172.21.15.111:44980) 2023-10-11T16:42:55.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 -- 172.21.15.111:0/1274684609 learned_addr learned my addr 172.21.15.111:0/1274684609 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:55.733 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ccc101660 0x7f8ccc073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd1f9c700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ccc105620 0x7f8ccc1a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 -- 172.21.15.111:0/1274684609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ccc101660 msgr2=0x7f8ccc073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ccc101660 0x7f8ccc073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.734 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 -- 172.21.15.111:0/1274684609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ccc105620 msgr2=0x7f8ccc1a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ccc105620 0x7f8ccc1a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 -- 172.21.15.111:0/1274684609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc4009d00 con 0x7f8ccc100cc0 2023-10-11T16:42:55.735 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd179b700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc100cc0 0x7f8ccc072df0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8cc800e9a0 tx=0x7f8cc800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:55.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cc27fc700 1 -- 172.21.15.111:0/1274684609 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8009790 con 0x7f8ccc100cc0 2023-10-11T16:42:55.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cc27fc700 1 -- 172.21.15.111:0/1274684609 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8cc800cbd0 con 0x7f8ccc100cc0 2023-10-11T16:42:55.736 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.727+0000 7f8cd39ff700 1 -- 172.21.15.111:0/1274684609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ccc1103e0 con 0x7f8ccc100cc0 2023-10-11T16:42:55.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.728+0000 7f8cc27fc700 1 -- 172.21.15.111:0/1274684609 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8018680 con 0x7f8ccc100cc0 2023-10-11T16:42:55.737 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.728+0000 7f8cd39ff700 1 -- 172.21.15.111:0/1274684609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ccc110250 con 0x7f8ccc100cc0 2023-10-11T16:42:55.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.731+0000 7f8cc27fc700 1 -- 172.21.15.111:0/1274684609 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f8cc800cd40 con 0x7f8ccc100cc0 2023-10-11T16:42:55.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.732+0000 7f8ce733cb80 1 Processor -- start 2023-10-11T16:42:55.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8cc27fc700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8cb80a12c0 0x7f8cb80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8ce733cb80 1 -- start start 2023-10-11T16:42:55.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8cc27fc700 1 -- 172.21.15.111:0/1274684609 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f8cc8014070 con 0x7f8ccc100cc0 2023-10-11T16:42:55.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8cd1f9c700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8cb80a12c0 0x7f8cb80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.745 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b5470 0x557ef69b9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69b9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef68ba060 con 0x557ef69b4b60 2023-10-11T16:42:55.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef690c8a0 con 0x557ef69b5470 2023-10-11T16:42:55.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef6905d30 con 0x557ef69b98b0 2023-10-11T16:42:55.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8cd179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8cd0f9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b5470 0x557ef69b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8cd0f9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b5470 0x557ef69b9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:52602/0 (socket says 172.21.15.111:52602) 2023-10-11T16:42:55.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.733+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/999007647 learned_addr learned my addr 172.21.15.111:0/999007647 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:55.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd1f9c700 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69b9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/999007647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 msgr2=0x557ef69b9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:55.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69b9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/999007647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 msgr2=0x557ef69b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:55.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/999007647 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ef6787630 con 0x557ef69b5470 2023-10-11T16:42:55.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd1f9c700 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69b9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:55.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd1f9c700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8cb80a12c0 0x7f8cb80a3770 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc074f00 tx=0x7f8cc4009520 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:55.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b5470 0x557ef69b9370 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009cb0 tx=0x7f8cbc0058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c2f046674d87a29 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:55.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cb77fe700 1 -- 172.21.15.111:0/999007647 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc011ce0 con 0x557ef69b5470 2023-10-11T16:42:55.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cb77fe700 1 -- 172.21.15.111:0/999007647 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8cbc011e40 con 0x557ef69b5470 2023-10-11T16:42:55.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.734+0000 7f8cb77fe700 1 -- 172.21.15.111:0/999007647 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc005220 con 0x557ef69b5470 2023-10-11T16:42:55.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 -- 172.21.15.111:0/999007647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b5470 msgr2=0x557ef69b9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b5470 0x557ef69b9370 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009cb0 tx=0x7f8cbc0058e0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 -- 172.21.15.111:0/999007647 shutdown_connections 2023-10-11T16:42:55.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69b9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/999007647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b5470 0x557ef69b9370 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 -- 172.21.15.111:0/999007647 >> 172.21.15.111:0/999007647 conn(0x557ef69aa360 msgr2=0x557ef69afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:55.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 -- 172.21.15.111:0/999007647 shutdown_connections 2023-10-11T16:42:55.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.735+0000 7f8ce733cb80 1 -- 172.21.15.111:0/999007647 wait complete. 2023-10-11T16:42:55.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.736+0000 7f8ce733cb80 1 -- start start 2023-10-11T16:42:55.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.736+0000 7f8ce733cb80 1 Processor -- start 2023-10-11T16:42:55.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.736+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b4b60 0x557ef69c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.736+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69be2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.736+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69bea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.736+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef68a78e0 con 0x557ef69b5470 2023-10-11T16:42:55.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.736+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef68a7580 con 0x557ef69b4b60 2023-10-11T16:42:55.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.736+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef6789920 con 0x557ef69b98b0 2023-10-11T16:42:55.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8ce733cb80 10 client.0 _ll_register_callbacks cb 0x557ef68c5780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:55.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69be2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69be2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45002/0 (socket says 172.21.15.111:45002) 2023-10-11T16:42:55.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/351903492 learned_addr learned my addr 172.21.15.111:0/351903492 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:55.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd1f9c700 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69bea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd179b700 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b4b60 0x557ef69c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/351903492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 msgr2=0x557ef69bea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:55.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69bea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/351903492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b4b60 msgr2=0x557ef69c10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:55.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b4b60 0x557ef69c10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/351903492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc0098e0 con 0x557ef69b5470 2023-10-11T16:42:55.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69be2f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc004d80 tx=0x7f8cbc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1096085f91e421d server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:55.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.737+0000 7f8cb4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/351903492 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc004260 con 0x557ef69b5470 2023-10-11T16:42:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/351903492 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f8cbc0043c0 con 0x557ef69b5470 2023-10-11T16:42:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8ce733cb80 1 -- 172.21.15.111:0/351903492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ef69a7b20 con 0x557ef69b5470 2023-10-11T16:42:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/351903492 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cbc004580 con 0x557ef69b5470 2023-10-11T16:42:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8ce733cb80 10 client.17184 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8ce733cb80 1 -- 172.21.15.111:0/351903492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x557ef68cc190 con 0x557ef69b5470 2023-10-11T16:42:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/351903492 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557ef69a7b20 con 0x557ef69b5470 2023-10-11T16:42:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/351903492 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8cbc005230 con 0x557ef69b5470 2023-10-11T16:42:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8ce733cb80 10 client.17184 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.738+0000 7f8ce733cb80 10 client.17184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.739+0000 7f8ce733cb80 20 client.17184 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.739+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557ef69ce720 0x557ef69d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.739+0000 7f8ce733cb80 4 client.17184 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.739+0000 7f8ce733cb80 1 -- 172.21.15.111:0/351903492 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ef68c80a0 con 0x557ef69ce720 2023-10-11T16:42:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.739+0000 7f8cd179b700 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557ef69ce720 0x557ef69d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.739+0000 7f8cb4ff9700 10 client.17184 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.739+0000 7f8cd179b700 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557ef69ce720 0x557ef69d0b30 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4fc49cecab5be35 server_cookie=c715ae6f248a95c9 in_seq=0 out_seq=0 2023-10-11T16:42:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.740+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/351903492 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557ef68c80a0 con 0x557ef69ce720 2023-10-11T16:42:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.740+0000 7f8cb4ff9700 10 client.17184 handle_command_reply: tid=0 2023-10-11T16:42:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.740+0000 7f8ce733cb80 1 client.17184 shutdown 2023-10-11T16:42:55.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 -- 172.21.15.111:0/351903492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 msgr2=0x557ef69be2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.767 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69be2f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc004d80 tx=0x7f8cbc011bf0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 -- 172.21.15.111:0/351903492 shutdown_connections 2023-10-11T16:42:55.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557ef69ce720 0x557ef69d0b30 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b98b0 0x557ef69bea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b4b60 0x557ef69c10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8cb4ff9700 0 client.17184 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:55.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/351903492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69be2f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 -- 172.21.15.111:0/351903492 >> 172.21.15.111:0/351903492 conn(0x557ef69aa360 msgr2=0x557ef69bad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:55.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 -- 172.21.15.111:0/351903492 shutdown_connections 2023-10-11T16:42:55.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 1 -- 172.21.15.111:0/351903492 wait complete. 2023-10-11T16:42:55.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.741+0000 7f8ce733cb80 20 client.17184 trim_cache size 0 max 16384 2023-10-11T16:42:55.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8ce733cb80 1 Processor -- start 2023-10-11T16:42:55.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8ce733cb80 1 -- start start 2023-10-11T16:42:55.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b4b60 0x557ef69c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 0x557ef69bdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef69b5bc0 con 0x557ef69b5470 2023-10-11T16:42:55.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef69b5900 con 0x557ef69b98b0 2023-10-11T16:42:55.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef69a7960 con 0x557ef69b4b60 2023-10-11T16:42:55.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.760+0000 7f8cd0f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd0f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69c0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45010/0 (socket says 172.21.15.111:45010) 2023-10-11T16:42:55.779 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd1f9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 0x557ef69bdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/1452108452 learned_addr learned my addr 172.21.15.111:0/1452108452 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:55.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd179b700 1 --2- 172.21.15.111:0/1452108452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b4b60 0x557ef69c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.780 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/1452108452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b4b60 msgr2=0x557ef69c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:55.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/1452108452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b4b60 0x557ef69c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/1452108452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 msgr2=0x557ef69bdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:55.781 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/1452108452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 0x557ef69bdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd0f9a700 1 -- 172.21.15.111:0/1452108452 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ef69abb00 con 0x557ef69b5470 2023-10-11T16:42:55.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/1452108452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69c0c10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc005a80 tx=0x7f8cbc0096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85962d8376b5c715 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:55.782 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cb7fff700 1 -- 172.21.15.111:0/1452108452 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc0117a0 con 0x557ef69b5470 2023-10-11T16:42:55.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cb7fff700 1 -- 172.21.15.111:0/1452108452 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8cbc011c60 con 0x557ef69b5470 2023-10-11T16:42:55.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.761+0000 7f8cb7fff700 1 -- 172.21.15.111:0/1452108452 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc022690 con 0x557ef69b5470 2023-10-11T16:42:55.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 -- 172.21.15.111:0/1452108452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 msgr2=0x557ef69c0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/1452108452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69c0c10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc005a80 tx=0x7f8cbc0096d0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 -- 172.21.15.111:0/1452108452 shutdown_connections 2023-10-11T16:42:55.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/1452108452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 0x557ef69bdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/1452108452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b4b60 0x557ef69c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/1452108452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b5470 0x557ef69c0c10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 -- 172.21.15.111:0/1452108452 >> 172.21.15.111:0/1452108452 conn(0x557ef69aa360 msgr2=0x557ef69b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:55.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 -- 172.21.15.111:0/1452108452 shutdown_connections 2023-10-11T16:42:55.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.762+0000 7f8ce733cb80 1 -- 172.21.15.111:0/1452108452 wait complete. 2023-10-11T16:42:55.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 1 -- start start 2023-10-11T16:42:55.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 1 Processor -- start 2023-10-11T16:42:55.787 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69c3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b5470 0x557ef69c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 0x557ef69c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef6a215c0 con 0x557ef69b4b60 2023-10-11T16:42:55.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef6a21700 con 0x557ef69b98b0 2023-10-11T16:42:55.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef6a21840 con 0x557ef69b5470 2023-10-11T16:42:55.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.763+0000 7f8ce733cb80 10 client.0 _ll_register_callbacks cb 0x7f8cc8010aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:55.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69c3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45020/0 (socket says 172.21.15.111:45020) 2023-10-11T16:42:55.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 -- 172.21.15.111:0/2533037975 learned_addr learned my addr 172.21.15.111:0/2533037975 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:55.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd1f9c700 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 0x557ef69c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b5470 0x557ef69c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 -- 172.21.15.111:0/2533037975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b5470 msgr2=0x557ef69c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b5470 0x557ef69c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 -- 172.21.15.111:0/2533037975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 msgr2=0x557ef69c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 0x557ef69c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 -- 172.21.15.111:0/2533037975 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ef68cc190 con 0x557ef69b4b60 2023-10-11T16:42:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b5470 0x557ef69c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:42:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cd179b700 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69c3990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8099fd0 tx=0x7f8cc8096380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75dcb91604bca5f server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cb4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:55.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/2533037975 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc809a4a0 con 0x557ef69b4b60 2023-10-11T16:42:55.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/2533037975 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f8cc8099530 con 0x557ef69b4b60 2023-10-11T16:42:55.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.764+0000 7f8ce733cb80 1 -- 172.21.15.111:0/2533037975 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cbc011e40 con 0x557ef69b4b60 2023-10-11T16:42:55.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/2533037975 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cc80cd1e0 con 0x557ef69b4b60 2023-10-11T16:42:55.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8ce733cb80 10 client.17190 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:55.798 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8ce733cb80 1 -- 172.21.15.111:0/2533037975 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x557ef6733de0 con 0x557ef69b4b60 2023-10-11T16:42:55.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/2533037975 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8cc80ca080 con 0x557ef69b4b60 2023-10-11T16:42:55.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/2533037975 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8cc80a2000 con 0x557ef69b4b60 2023-10-11T16:42:55.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8ce733cb80 10 client.17190 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8ce733cb80 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8ce733cb80 20 client.17190 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557ef6a22ed0 0x557ef6a252e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:55.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8ce733cb80 4 client.17190 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2023-10-11T16:42:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8ce733cb80 1 -- 172.21.15.111:0/2533037975 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557ef68d4660 con 0x557ef6a22ed0 2023-10-11T16:42:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.765+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557ef6a22ed0 0x557ef6a252e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:55.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8cd0f9a700 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557ef6a22ed0 0x557ef6a252e0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73549eab23a91cf9 server_cookie=1df0c1015a5bef4c in_seq=0 out_seq=0 2023-10-11T16:42:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8cb4ff9700 10 client.17190 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8cb4ff9700 1 -- 172.21.15.111:0/2533037975 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557ef68d4660 con 0x557ef6a22ed0 2023-10-11T16:42:55.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8cb4ff9700 10 client.17190 handle_command_reply: tid=0 2023-10-11T16:42:55.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 client.17190 shutdown 2023-10-11T16:42:55.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 -- 172.21.15.111:0/2533037975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 msgr2=0x557ef69c3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69c3990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8099fd0 tx=0x7f8cc8096380 comp rx=0 tx=0).stop 2023-10-11T16:42:55.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 -- 172.21.15.111:0/2533037975 shutdown_connections 2023-10-11T16:42:55.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x557ef6a22ed0 0x557ef6a252e0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557ef69b98b0 0x557ef69c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x557ef69b4b60 0x557ef69c3990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 --2- 172.21.15.111:0/2533037975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557ef69b5470 0x557ef69c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8cb4ff9700 0 client.17190 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:55.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 -- 172.21.15.111:0/2533037975 >> 172.21.15.111:0/2533037975 conn(0x557ef69aa360 msgr2=0x557ef69b8830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.766+0000 7f8ce733cb80 1 -- 172.21.15.111:0/2533037975 shutdown_connections 2023-10-11T16:42:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8ce733cb80 1 -- 172.21.15.111:0/2533037975 wait complete. 2023-10-11T16:42:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8ce733cb80 20 client.17190 trim_cache size 0 max 16384 2023-10-11T16:42:55.807 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:42:55.807 INFO:teuthology.orchestra.run.smithi111.stdout: "status": "no active scrubs running", 2023-10-11T16:42:55.808 INFO:teuthology.orchestra.run.smithi111.stdout: "scrubs": {} 2023-10-11T16:42:55.808 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:42:55.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 -- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8cb80a12c0 msgr2=0x7f8cb80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8cb80a12c0 0x7f8cb80a3770 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc074f00 tx=0x7f8cc4009520 comp rx=0 tx=0).stop 2023-10-11T16:42:55.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 -- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc100cc0 msgr2=0x7f8ccc072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:55.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc100cc0 0x7f8ccc072df0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8cc800e9a0 tx=0x7f8cc800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 -- 172.21.15.111:0/1274684609 shutdown_connections 2023-10-11T16:42:55.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8cb80a12c0 0x7f8cb80a3770 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ccc105620 0x7f8ccc1a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ccc100cc0 0x7f8ccc072df0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 --2- 172.21.15.111:0/1274684609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ccc101660 0x7f8ccc073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:55.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 -- 172.21.15.111:0/1274684609 >> 172.21.15.111:0/1274684609 conn(0x7f8ccc077560 msgr2=0x7f8ccc103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:55.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 -- 172.21.15.111:0/1274684609 shutdown_connections 2023-10-11T16:42:55.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:55.767+0000 7f8cd39ff700 1 -- 172.21.15.111:0/1274684609 wait complete. 2023-10-11T16:42:55.814 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-11T16:42:55.814 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:42:56.013 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 -- 172.21.15.111:0/3584708440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac0737a0 msgr2=0x7f2dac10c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 --2- 172.21.15.111:0/3584708440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac0737a0 0x7f2dac10c3c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009990 tx=0x7f2da0009ca0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 -- 172.21.15.111:0/3584708440 shutdown_connections 2023-10-11T16:42:56.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 --2- 172.21.15.111:0/3584708440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dac10c900 0x7f2dac10ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 --2- 172.21.15.111:0/3584708440 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dac072e90 0x7f2dac073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 --2- 172.21.15.111:0/3584708440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac0737a0 0x7f2dac10c3c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 -- 172.21.15.111:0/3584708440 >> 172.21.15.111:0/3584708440 conn(0x7f2dac0fbbc0 msgr2=0x7f2dac0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 -- 172.21.15.111:0/3584708440 shutdown_connections 2023-10-11T16:42:56.016 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.012+0000 7f2dabfff700 1 -- 172.21.15.111:0/3584708440 wait complete. 2023-10-11T16:42:56.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.013+0000 7f2dabfff700 1 Processor -- start 2023-10-11T16:42:56.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2dabfff700 1 -- start start 2023-10-11T16:42:56.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2dabfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac072e90 0x7f2dac1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2dabfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dac0737a0 0x7f2dac198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2dabfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dac10c900 0x7f2dac19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2dabfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac19d3b0 con 0x7f2dac072e90 2023-10-11T16:42:56.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2dabfff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac19d520 con 0x7f2dac0737a0 2023-10-11T16:42:56.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2dabfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac19d6e0 con 0x7f2dac10c900 2023-10-11T16:42:56.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2daaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac072e90 0x7f2dac1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2daaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac072e90 0x7f2dac1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45036/0 (socket says 172.21.15.111:45036) 2023-10-11T16:42:56.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2daaffd700 1 -- 172.21.15.111:0/1315228085 learned_addr learned my addr 172.21.15.111:0/1315228085 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2daa7fc700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dac0737a0 0x7f2dac198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2dab7fe700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dac10c900 0x7f2dac19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2daaffd700 1 -- 172.21.15.111:0/1315228085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dac10c900 msgr2=0x7f2dac19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.014+0000 7f2daaffd700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dac10c900 0x7f2dac19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2daaffd700 1 -- 172.21.15.111:0/1315228085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dac0737a0 msgr2=0x7f2dac198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2daaffd700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dac0737a0 0x7f2dac198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2daaffd700 1 -- 172.21.15.111:0/1315228085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da00095f0 con 0x7f2dac072e90 2023-10-11T16:42:56.023 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2daaffd700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac072e90 0x7f2dac1983d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00e9a0 tx=0x7f2d9c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2d93fff700 1 -- 172.21.15.111:0/1315228085 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c009790 con 0x7f2dac072e90 2023-10-11T16:42:56.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2d93fff700 1 -- 172.21.15.111:0/1315228085 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d9c00cbd0 con 0x7f2dac072e90 2023-10-11T16:42:56.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2d93fff700 1 -- 172.21.15.111:0/1315228085 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c018680 con 0x7f2dac072e90 2023-10-11T16:42:56.024 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac112340 con 0x7f2dac072e90 2023-10-11T16:42:56.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.015+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac1121b0 con 0x7f2dac072e90 2023-10-11T16:42:56.025 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.016+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac1124d0 con 0x7f2dac072e90 2023-10-11T16:42:56.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.018+0000 7f2d93fff700 1 -- 172.21.15.111:0/1315228085 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2d9c00cd40 con 0x7f2dac072e90 2023-10-11T16:42:56.026 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.020+0000 7f2d93fff700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2d940a1340 0x7f2d940a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.020+0000 7f2d93fff700 1 -- 172.21.15.111:0/1315228085 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f2d9c014070 con 0x7f2dac072e90 2023-10-11T16:42:56.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.020+0000 7f2daa7fc700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2d940a1340 0x7f2d940a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.021+0000 7f2daa7fc700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2d940a1340 0x7f2d940a37f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2dac1999e0 tx=0x7f2da0005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.027 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.024+0000 7f2d93fff700 1 -- 172.21.15.111:0/1315228085 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2d9c096310 con 0x7f2dac072e90 2023-10-11T16:42:56.272 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.272+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dac107220 con 0x7f2dac072e90 2023-10-11T16:42:56.273 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.273+0000 7f2d93fff700 1 -- 172.21.15.111:0/1315228085 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+10820 (secure 0 0 0) 0x7f2d9c099930 con 0x7f2dac072e90 2023-10-11T16:42:56.274 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:42:56.274 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:41:48.484142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:42:56.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2d940a1340 msgr2=0x7f2d940a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2d940a1340 0x7f2d940a37f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2dac1999e0 tx=0x7f2da0005fb0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac072e90 msgr2=0x7f2dac1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.276 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac072e90 0x7f2dac1983d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00e9a0 tx=0x7f2d9c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 shutdown_connections 2023-10-11T16:42:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2d940a1340 0x7f2d940a37f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.277 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dac10c900 0x7f2dac19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2dac072e90 0x7f2dac1983d0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 --2- 172.21.15.111:0/1315228085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2dac0737a0 0x7f2dac198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.278 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 >> 172.21.15.111:0/1315228085 conn(0x7f2dac0fbbc0 msgr2=0x7f2dac104360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 shutdown_connections 2023-10-11T16:42:56.279 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.275+0000 7f2dabfff700 1 -- 172.21.15.111:0/1315228085 wait complete. 2023-10-11T16:42:56.279 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 39 2023-10-11T16:42:56.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-11T16:42:56.498 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 -- 172.21.15.111:0/3521985126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c105620 msgr2=0x7f613c10d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 --2- 172.21.15.111:0/3521985126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c105620 0x7f613c10d1a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f613400b0d0 tx=0x7f613400b3e0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 -- 172.21.15.111:0/3521985126 shutdown_connections 2023-10-11T16:42:56.499 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 --2- 172.21.15.111:0/3521985126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c105620 0x7f613c10d1a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 --2- 172.21.15.111:0/3521985126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f613c100cc0 0x7f613c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.500 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 --2- 172.21.15.111:0/3521985126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c101660 0x7f613c104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 -- 172.21.15.111:0/3521985126 >> 172.21.15.111:0/3521985126 conn(0x7f613c077560 msgr2=0x7f613c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.501 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 -- 172.21.15.111:0/3521985126 shutdown_connections 2023-10-11T16:42:56.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.496+0000 7f61425ff700 1 -- 172.21.15.111:0/3521985126 wait complete. 2023-10-11T16:42:56.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.497+0000 7f61425ff700 1 Processor -- start 2023-10-11T16:42:56.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f61425ff700 1 -- start start 2023-10-11T16:42:56.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f61425ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c100cc0 0x7f613c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f61425ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f613c101660 0x7f613c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f61425ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105620 0x7f613c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f61425ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c198d20 con 0x7f613c100cc0 2023-10-11T16:42:56.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f61425ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c198e90 con 0x7f613c101660 2023-10-11T16:42:56.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f61425ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c199050 con 0x7f613c105620 2023-10-11T16:42:56.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f613bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c100cc0 0x7f613c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f613bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c100cc0 0x7f613c19e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45040/0 (socket says 172.21.15.111:45040) 2023-10-11T16:42:56.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f613bfff700 1 -- 172.21.15.111:0/2832252164 learned_addr learned my addr 172.21.15.111:0/2832252164 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f613b7fe700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f613c101660 0x7f613c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.498+0000 7f6140b9c700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105620 0x7f613c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f613bfff700 1 -- 172.21.15.111:0/2832252164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105620 msgr2=0x7f613c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f613bfff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105620 0x7f613c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f613bfff700 1 -- 172.21.15.111:0/2832252164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f613c101660 msgr2=0x7f613c19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f613bfff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f613c101660 0x7f613c19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f613bfff700 1 -- 172.21.15.111:0/2832252164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6134009d00 con 0x7f613c100cc0 2023-10-11T16:42:56.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f613bfff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c100cc0 0x7f613c19e640 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f612c00e9a0 tx=0x7f612c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f61397fa700 1 -- 172.21.15.111:0/2832252164 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c009790 con 0x7f613c100cc0 2023-10-11T16:42:56.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f61397fa700 1 -- 172.21.15.111:0/2832252164 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f612c00cbd0 con 0x7f613c100cc0 2023-10-11T16:42:56.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f61425ff700 1 -- 172.21.15.111:0/2832252164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c1103e0 con 0x7f613c100cc0 2023-10-11T16:42:56.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f61397fa700 1 -- 172.21.15.111:0/2832252164 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c018680 con 0x7f613c100cc0 2023-10-11T16:42:56.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.499+0000 7f61425ff700 1 -- 172.21.15.111:0/2832252164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c110250 con 0x7f613c100cc0 2023-10-11T16:42:56.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.502+0000 7f61397fa700 1 -- 172.21.15.111:0/2832252164 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f612c00cd40 con 0x7f613c100cc0 2023-10-11T16:42:56.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.504+0000 7f6155eccb80 1 Processor -- start 2023-10-11T16:42:56.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.504+0000 7f6155eccb80 1 -- start start 2023-10-11T16:42:56.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f61397fa700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f61240a1340 0x7f61240a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f997882f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f997887370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997887d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f6155eccb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f997788060 con 0x55f997882b60 2023-10-11T16:42:56.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f6155eccb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9977da8a0 con 0x55f9978878b0 2023-10-11T16:42:56.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f6155eccb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9977d3d30 con 0x55f997883470 2023-10-11T16:42:56.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f61397fa700 1 -- 172.21.15.111:0/2832252164 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f612c014070 con 0x7f613c100cc0 2023-10-11T16:42:56.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f997887370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f6140b9c700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f61240a1340 0x7f61240a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f997882f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f997887370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:52164/0 (socket says 172.21.15.111:52164) 2023-10-11T16:42:56.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613b7fe700 1 -- 172.21.15.111:0/1355246689 learned_addr learned my addr 172.21.15.111:0/1355246689 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f6140b9c700 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997887d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613bfff700 1 -- 172.21.15.111:0/1355246689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 msgr2=0x55f997887370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613bfff700 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f997887370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613bfff700 1 -- 172.21.15.111:0/1355246689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 msgr2=0x55f997887d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613bfff700 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997887d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.526 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.505+0000 7f613bfff700 1 -- 172.21.15.111:0/1355246689 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f997655630 con 0x55f997882b60 2023-10-11T16:42:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.506+0000 7f6140b9c700 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997887d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:42:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.506+0000 7f613bfff700 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f997882f30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f612c0ca640 tx=0x7f612c099920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5c177ef2a785b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.527 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.506+0000 7f6140b9c700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f61240a1340 0x7f61240a37f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6134009850 tx=0x7f6134009780 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.506+0000 7f61227fc700 1 -- 172.21.15.111:0/1355246689 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c0cd170 con 0x55f997882b60 2023-10-11T16:42:56.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.506+0000 7f61227fc700 1 -- 172.21.15.111:0/1355246689 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f612c0a2410 con 0x55f997882b60 2023-10-11T16:42:56.528 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.506+0000 7f61227fc700 1 -- 172.21.15.111:0/1355246689 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c0c4a10 con 0x55f997882b60 2023-10-11T16:42:56.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 -- 172.21.15.111:0/1355246689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 msgr2=0x55f997882f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f997882f30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f612c0ca640 tx=0x7f612c099920 comp rx=0 tx=0).stop 2023-10-11T16:42:56.529 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 -- 172.21.15.111:0/1355246689 shutdown_connections 2023-10-11T16:42:56.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997887d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f997882f30 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 --2- 172.21.15.111:0/1355246689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f997887370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.530 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 -- 172.21.15.111:0/1355246689 >> 172.21.15.111:0/1355246689 conn(0x55f997878360 msgr2=0x55f99787db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 -- 172.21.15.111:0/1355246689 shutdown_connections 2023-10-11T16:42:56.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 -- 172.21.15.111:0/1355246689 wait complete. 2023-10-11T16:42:56.531 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.507+0000 7f6155eccb80 1 -- start start 2023-10-11T16:42:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.508+0000 7f6155eccb80 1 Processor -- start 2023-10-11T16:42:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.508+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f99788f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.508+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997883470 0x55f99788c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.532 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f9978878b0 0x55f99788c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f6155eccb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9977758e0 con 0x55f997882b60 2023-10-11T16:42:56.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f6155eccb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f997775580 con 0x55f997883470 2023-10-11T16:42:56.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f6155eccb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f997657920 con 0x55f9978878b0 2023-10-11T16:42:56.533 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f6155eccb80 10 client.0 _ll_register_callbacks cb 0x55f997793780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f99788f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f99788f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45060/0 (socket says 172.21.15.111:45060) 2023-10-11T16:42:56.534 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613bfff700 1 -- 172.21.15.111:0/3306145089 learned_addr learned my addr 172.21.15.111:0/3306145089 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613b7fe700 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997883470 0x55f99788c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f6140b9c700 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f9978878b0 0x55f99788c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613bfff700 1 -- 172.21.15.111:0/3306145089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f9978878b0 msgr2=0x55f99788c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.535 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613bfff700 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f9978878b0 0x55f99788c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613bfff700 1 -- 172.21.15.111:0/3306145089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997883470 msgr2=0x55f99788c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613bfff700 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997883470 0x55f99788c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.536 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.509+0000 7f613bfff700 1 -- 172.21.15.111:0/3306145089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c110250 con 0x55f997882b60 2023-10-11T16:42:56.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f613bfff700 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f99788f000 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f612c0a1ee0 tx=0x7f612c09e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac74c6739b5698a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f611b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:56.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f611b7fe700 1 -- 172.21.15.111:0/3306145089 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612c0caac0 con 0x55f997882b60 2023-10-11T16:42:56.537 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f611b7fe700 1 -- 172.21.15.111:0/3306145089 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f612c0b2610 con 0x55f997882b60 2023-10-11T16:42:56.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f6155eccb80 1 -- 172.21.15.111:0/3306145089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f997875b20 con 0x55f997882b60 2023-10-11T16:42:56.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f611b7fe700 1 -- 172.21.15.111:0/3306145089 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f612c0b27d0 con 0x55f997882b60 2023-10-11T16:42:56.538 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f6155eccb80 10 client.17208 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:56.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f6155eccb80 1 -- 172.21.15.111:0/3306145089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55f99779a190 con 0x55f997882b60 2023-10-11T16:42:56.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f611b7fe700 1 -- 172.21.15.111:0/3306145089 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f997875b20 con 0x55f997882b60 2023-10-11T16:42:56.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f611b7fe700 1 -- 172.21.15.111:0/3306145089 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f612c09a2b0 con 0x55f997882b60 2023-10-11T16:42:56.539 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f6155eccb80 10 client.17208 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:56.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f6155eccb80 10 client.17208 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:42:56.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f6155eccb80 20 client.17208 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:56.540 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.510+0000 7f6155eccb80 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f99789c320 0x55f99789e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.511+0000 7f6155eccb80 4 client.17208 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:56.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.511+0000 7f6155eccb80 1 -- 172.21.15.111:0/3306145089 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9977960a0 con 0x55f99789c320 2023-10-11T16:42:56.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.511+0000 7f613b7fe700 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f99789c320 0x55f99789e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.541 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.511+0000 7f611b7fe700 10 client.17208 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:42:56.542 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.511+0000 7f613b7fe700 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f99789c320 0x55f99789e730 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e7e5c5319c5dbfb server_cookie=580f4aa0479d14d2 in_seq=0 out_seq=0 2023-10-11T16:42:56.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f611b7fe700 1 -- 172.21.15.111:0/3306145089 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f9977960a0 con 0x55f99789c320 2023-10-11T16:42:56.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f611b7fe700 10 client.17208 handle_command_reply: tid=0 2023-10-11T16:42:56.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 client.17208 shutdown 2023-10-11T16:42:56.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 -- 172.21.15.111:0/3306145089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 msgr2=0x55f99788f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f99788f000 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f612c0a1ee0 tx=0x7f612c09e0b0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 -- 172.21.15.111:0/3306145089 shutdown_connections 2023-10-11T16:42:56.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f99789c320 0x55f99789e730 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f9978878b0 0x55f99788c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f611b7fe700 0 client.17208 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:42:56.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997882b60 0x55f99788f000 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 --2- 172.21.15.111:0/3306145089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997883470 0x55f99788c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 -- 172.21.15.111:0/3306145089 >> 172.21.15.111:0/3306145089 conn(0x55f997878360 msgr2=0x55f997888d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 -- 172.21.15.111:0/3306145089 shutdown_connections 2023-10-11T16:42:56.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.512+0000 7f6155eccb80 1 -- 172.21.15.111:0/3306145089 wait complete. 2023-10-11T16:42:56.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.513+0000 7f6155eccb80 20 client.17208 trim_cache size 0 max 16384 2023-10-11T16:42:56.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.526+0000 7f6155eccb80 1 Processor -- start 2023-10-11T16:42:56.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.526+0000 7f6155eccb80 1 -- start start 2023-10-11T16:42:56.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997882b60 0x55f9978851a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f9978939a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9978878b0 0x55f99788eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6155eccb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99788a8a0 con 0x55f9978878b0 2023-10-11T16:42:56.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6155eccb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f997886480 con 0x55f997882b60 2023-10-11T16:42:56.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6155eccb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f997875960 con 0x55f997883470 2023-10-11T16:42:56.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6140b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9978878b0 0x55f99788eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f613b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f9978939a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6140b9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9978878b0 0x55f99788eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45076/0 (socket says 172.21.15.111:45076) 2023-10-11T16:42:56.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f6140b9c700 1 -- 172.21.15.111:0/2378738972 learned_addr learned my addr 172.21.15.111:0/2378738972 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f613bfff700 1 --2- 172.21.15.111:0/2378738972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997882b60 0x55f9978851a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f613b7fe700 1 -- 172.21.15.111:0/2378738972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997882b60 msgr2=0x55f9978851a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f613b7fe700 1 --2- 172.21.15.111:0/2378738972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997882b60 0x55f9978851a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f613b7fe700 1 -- 172.21.15.111:0/2378738972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9978878b0 msgr2=0x55f99788eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f613b7fe700 1 --2- 172.21.15.111:0/2378738972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9978878b0 0x55f99788eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f613b7fe700 1 -- 172.21.15.111:0/2378738972 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f997879b00 con 0x55f997883470 2023-10-11T16:42:56.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.527+0000 7f613b7fe700 1 --2- 172.21.15.111:0/2378738972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f9978939a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f612c098310 tx=0x7f6130009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=117dcb35ed59b334 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6122ffd700 1 -- 172.21.15.111:0/2378738972 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130017030 con 0x55f997883470 2023-10-11T16:42:56.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6122ffd700 1 -- 172.21.15.111:0/2378738972 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6130018040 con 0x55f997883470 2023-10-11T16:42:56.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6122ffd700 1 -- 172.21.15.111:0/2378738972 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613001e680 con 0x55f997883470 2023-10-11T16:42:56.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6155eccb80 1 -- 172.21.15.111:0/2378738972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 msgr2=0x55f9978939a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6155eccb80 1 --2- 172.21.15.111:0/2378738972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f9978939a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f612c098310 tx=0x7f6130009260 comp rx=0 tx=0).stop 2023-10-11T16:42:56.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6155eccb80 1 -- 172.21.15.111:0/2378738972 shutdown_connections 2023-10-11T16:42:56.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6155eccb80 1 --2- 172.21.15.111:0/2378738972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f9978878b0 0x55f99788eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.559 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6155eccb80 1 --2- 172.21.15.111:0/2378738972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f997882b60 0x55f9978851a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6155eccb80 1 --2- 172.21.15.111:0/2378738972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997883470 0x55f9978939a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.528+0000 7f6155eccb80 1 -- 172.21.15.111:0/2378738972 >> 172.21.15.111:0/2378738972 conn(0x55f997878360 msgr2=0x55f997889c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.560 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.529+0000 7f6155eccb80 1 -- 172.21.15.111:0/2378738972 shutdown_connections 2023-10-11T16:42:56.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.529+0000 7f6155eccb80 1 -- 172.21.15.111:0/2378738972 wait complete. 2023-10-11T16:42:56.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.529+0000 7f6155eccb80 1 -- start start 2023-10-11T16:42:56.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.529+0000 7f6155eccb80 1 Processor -- start 2023-10-11T16:42:56.561 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.529+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997882b60 0x55f997891990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.529+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997883470 0x55f997891ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f6155eccb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997892410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.562 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f6155eccb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9978ef280 con 0x55f997883470 2023-10-11T16:42:56.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f6155eccb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9978ef3f0 con 0x55f9978878b0 2023-10-11T16:42:56.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f6155eccb80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9978ef5b0 con 0x55f997882b60 2023-10-11T16:42:56.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f6155eccb80 10 client.0 _ll_register_callbacks cb 0x7f6130009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:56.563 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997883470 0x55f997891ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997882b60 0x55f997891990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997883470 0x55f997891ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45078/0 (socket says 172.21.15.111:45078) 2023-10-11T16:42:56.564 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 -- 172.21.15.111:0/4096824767 learned_addr learned my addr 172.21.15.111:0/4096824767 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f6140b9c700 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997892410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 -- 172.21.15.111:0/4096824767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997882b60 msgr2=0x55f997891990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997882b60 0x55f997891990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 -- 172.21.15.111:0/4096824767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 msgr2=0x55f997892410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997892410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 -- 172.21.15.111:0/4096824767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6130009b90 con 0x55f997883470 2023-10-11T16:42:56.566 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613bfff700 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997882b60 0x55f997891990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:42:56.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f613b7fe700 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997883470 0x55f997891ed0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x55f997875af0 tx=0x7f613001f6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2994db85b15d730a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.530+0000 7f611b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:56.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f611b7fe700 1 -- 172.21.15.111:0/4096824767 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130017030 con 0x55f997883470 2023-10-11T16:42:56.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f611b7fe700 1 -- 172.21.15.111:0/4096824767 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f613001f250 con 0x55f997883470 2023-10-11T16:42:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 1 -- 172.21.15.111:0/4096824767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f612c0a2410 con 0x55f997883470 2023-10-11T16:42:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f611b7fe700 1 -- 172.21.15.111:0/4096824767 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f613001fa40 con 0x55f997883470 2023-10-11T16:42:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 10 client.17214 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 1 -- 172.21.15.111:0/4096824767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55f997601de0 con 0x55f997883470 2023-10-11T16:42:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f611b7fe700 1 -- 172.21.15.111:0/4096824767 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f612c0a2410 con 0x55f997883470 2023-10-11T16:42:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f611b7fe700 1 -- 172.21.15.111:0/4096824767 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6130004950 con 0x55f997883470 2023-10-11T16:42:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 10 client.17214 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:56.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 10 client.17214 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-11T16:42:56.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 20 client.17214 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:56.570 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f9978f0bf0 0x55f9978f3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 4 client.17214 mds_command: new command op to 24532 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:42:56.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.531+0000 7f6155eccb80 1 -- 172.21.15.111:0/4096824767 --> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f9977a2660 con 0x55f9978f0bf0 2023-10-11T16:42:56.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.532+0000 7f613bfff700 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f9978f0bf0 0x55f9978f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.571 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.532+0000 7f613bfff700 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f9978f0bf0 0x55f9978f3000 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ee82329f91a728f server_cookie=c822e3bf34a3effd in_seq=0 out_seq=0 2023-10-11T16:42:56.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.532+0000 7f611b7fe700 10 client.17214 ms_handle_connect on v2:172.21.15.150:6836/612139379 2023-10-11T16:42:56.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.532+0000 7f611b7fe700 1 -- 172.21.15.111:0/4096824767 <== mds.2 v2:172.21.15.150:6836/612139379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f9977a2660 con 0x55f9978f0bf0 2023-10-11T16:42:56.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.532+0000 7f611b7fe700 10 client.17214 handle_command_reply: tid=0 2023-10-11T16:42:56.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.532+0000 7f6155eccb80 1 client.17214 shutdown 2023-10-11T16:42:56.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 -- 172.21.15.111:0/4096824767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997883470 msgr2=0x55f997891ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997883470 0x55f997891ed0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x55f997875af0 tx=0x7f613001f6e0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 -- 172.21.15.111:0/4096824767 shutdown_connections 2023-10-11T16:42:56.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.150:6836/612139379,v1:172.21.15.150:6837/612139379] conn(0x55f9978f0bf0 0x55f9978f3000 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f9978878b0 0x55f997892410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55f997882b60 0x55f997891990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f611b7fe700 0 client.17214 ms_handle_reset on v2:172.21.15.150:6836/612139379 2023-10-11T16:42:56.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 --2- 172.21.15.111:0/4096824767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55f997883470 0x55f997891ed0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 -- 172.21.15.111:0/4096824767 >> 172.21.15.111:0/4096824767 conn(0x55f997878360 msgr2=0x55f997881960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 -- 172.21.15.111:0/4096824767 shutdown_connections 2023-10-11T16:42:56.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 1 -- 172.21.15.111:0/4096824767 wait complete. 2023-10-11T16:42:56.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.533+0000 7f6155eccb80 20 client.17214 trim_cache size 0 max 16384 2023-10-11T16:42:56.576 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:42:56.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 -- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f61240a1340 msgr2=0x7f61240a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f61240a1340 0x7f61240a37f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6134009850 tx=0x7f6134009780 comp rx=0 tx=0).stop 2023-10-11T16:42:56.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 -- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c100cc0 msgr2=0x7f613c19e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.578 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c100cc0 0x7f613c19e640 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f612c00e9a0 tx=0x7f612c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 -- 172.21.15.111:0/2832252164 shutdown_connections 2023-10-11T16:42:56.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f61240a1340 0x7f61240a37f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.579 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c105620 0x7f613c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f613c100cc0 0x7f613c19e640 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 --2- 172.21.15.111:0/2832252164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f613c101660 0x7f613c19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.580 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 -- 172.21.15.111:0/2832252164 >> 172.21.15.111:0/2832252164 conn(0x7f613c077560 msgr2=0x7f613c0fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 -- 172.21.15.111:0/2832252164 shutdown_connections 2023-10-11T16:42:56.581 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.534+0000 7f61425ff700 1 -- 172.21.15.111:0/2832252164 wait complete. 2023-10-11T16:42:56.581 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:56 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1315228085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:42:56.583 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-11T16:42:56.583 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:56 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1315228085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:42:56.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:56 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1315228085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:42:56.783 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/2420666358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc410a8b0 msgr2=0x7f6cc410ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/2420666358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc410a8b0 0x7f6cc410ccd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00b0a0 tx=0x7f6cbc00b3b0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/2420666358 shutdown_connections 2023-10-11T16:42:56.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/2420666358 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc410a8b0 0x7f6cc410ccd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.784 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/2420666358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc4101390 0x7f6cc4101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/2420666358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cc4101ca0 0x7f6cc410a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/2420666358 >> 172.21.15.111:0/2420666358 conn(0x7f6cc40770b0 msgr2=0x7f6cc40774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.785 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/2420666358 shutdown_connections 2023-10-11T16:42:56.786 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.782+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/2420666358 wait complete. 2023-10-11T16:42:56.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.784+0000 7f6ccb3ff700 1 Processor -- start 2023-10-11T16:42:56.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.784+0000 7f6ccb3ff700 1 -- start start 2023-10-11T16:42:56.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6ccb3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc4101390 0x7f6cc419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.788 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6ccb3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc4101ca0 0x7f6cc419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6ccb3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cc410a8b0 0x7f6cc41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6ccb3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc4198c40 con 0x7f6cc4101390 2023-10-11T16:42:56.789 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6ccb3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc4198db0 con 0x7f6cc410a8b0 2023-10-11T16:42:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6ccb3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc4198f70 con 0x7f6cc4101ca0 2023-10-11T16:42:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc4101390 0x7f6cc419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc4101390 0x7f6cc419e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45100/0 (socket says 172.21.15.111:45100) 2023-10-11T16:42:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc919b700 1 -- 172.21.15.111:0/1685330442 learned_addr learned my addr 172.21.15.111:0/1685330442 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc899a700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc4101ca0 0x7f6cc419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc999c700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cc410a8b0 0x7f6cc41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc919b700 1 -- 172.21.15.111:0/1685330442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc4101ca0 msgr2=0x7f6cc419eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc919b700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc4101ca0 0x7f6cc419eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc919b700 1 -- 172.21.15.111:0/1685330442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cc410a8b0 msgr2=0x7f6cc41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc919b700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cc410a8b0 0x7f6cc41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.785+0000 7f6cc919b700 1 -- 172.21.15.111:0/1685330442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc009d00 con 0x7f6cc4101390 2023-10-11T16:42:56.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.786+0000 7f6cc919b700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc4101390 0x7f6cc419e560 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000e9a0 tx=0x7f6cc000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.786+0000 7f6cba7fc700 1 -- 172.21.15.111:0/1685330442 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0009790 con 0x7f6cc4101390 2023-10-11T16:42:56.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.786+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/1685330442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc4110350 con 0x7f6cc4101390 2023-10-11T16:42:56.793 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.786+0000 7f6cba7fc700 1 -- 172.21.15.111:0/1685330442 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cc0010430 con 0x7f6cc4101390 2023-10-11T16:42:56.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.786+0000 7f6cba7fc700 1 -- 172.21.15.111:0/1685330442 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0018680 con 0x7f6cc4101390 2023-10-11T16:42:56.794 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.786+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/1685330442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc41101c0 con 0x7f6cc4101390 2023-10-11T16:42:56.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.790+0000 7f6cba7fc700 1 -- 172.21.15.111:0/1685330442 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f6cc00105a0 con 0x7f6cc4101390 2023-10-11T16:42:56.799 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.791+0000 7f6cdecd9b80 1 Processor -- start 2023-10-11T16:42:56.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.791+0000 7f6cdecd9b80 1 -- start start 2023-10-11T16:42:56.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.791+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818ac2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac78b0 0x55a818ac7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8189c8060 con 0x55a818ac78b0 2023-10-11T16:42:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818a1a8a0 con 0x55a818ac3470 2023-10-11T16:42:56.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818a13d30 con 0x55a818ac2b60 2023-10-11T16:42:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc899a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac78b0 0x55a818ac7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc999c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818ac2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc919b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc899a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac78b0 0x55a818ac7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45106/0 (socket says 172.21.15.111:45106) 2023-10-11T16:42:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc899a700 1 -- 172.21.15.111:0/4259709151 learned_addr learned my addr 172.21.15.111:0/4259709151 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cba7fc700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6cb00a12c0 0x7f6cb00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc919b700 1 -- 172.21.15.111:0/4259709151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 msgr2=0x55a818ac2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc919b700 1 --2- 172.21.15.111:0/4259709151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818ac2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc919b700 1 -- 172.21.15.111:0/4259709151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac78b0 msgr2=0x55a818ac7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc919b700 1 --2- 172.21.15.111:0/4259709151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac78b0 0x55a818ac7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.792+0000 7f6cc919b700 1 -- 172.21.15.111:0/4259709151 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a818895630 con 0x55a818ac3470 2023-10-11T16:42:56.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.793+0000 7f6cba7fc700 1 -- 172.21.15.111:0/1685330442 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f6cc0014070 con 0x7f6cc4101390 2023-10-11T16:42:56.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.793+0000 7f6cc999c700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6cb00a12c0 0x7f6cb00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.793+0000 7f6cc919b700 1 --2- 172.21.15.111:0/4259709151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac7370 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00ca630 tx=0x7f6cc0010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=886dd34ea2e297ad server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.793+0000 7f6cc999c700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6cb00a12c0 0x7f6cb00a3770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6cc419f260 tx=0x7f6cbc0099e0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.793+0000 7f6caf7fe700 1 -- 172.21.15.111:0/4259709151 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00d0d00 con 0x55a818ac3470 2023-10-11T16:42:56.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.793+0000 7f6caf7fe700 1 -- 172.21.15.111:0/4259709151 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cc0099af0 con 0x55a818ac3470 2023-10-11T16:42:56.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.793+0000 7f6caf7fe700 1 -- 172.21.15.111:0/4259709151 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0099fb0 con 0x55a818ac3470 2023-10-11T16:42:56.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/4259709151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 msgr2=0x55a818ac7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/4259709151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac7370 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00ca630 tx=0x7f6cc0010c80 comp rx=0 tx=0).stop 2023-10-11T16:42:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/4259709151 shutdown_connections 2023-10-11T16:42:56.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/4259709151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac78b0 0x55a818ac7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/4259709151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818ac2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/4259709151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac7370 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/4259709151 >> 172.21.15.111:0/4259709151 conn(0x55a818ab8360 msgr2=0x55a818abdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/4259709151 shutdown_connections 2023-10-11T16:42:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.794+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/4259709151 wait complete. 2023-10-11T16:42:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.795+0000 7f6cdecd9b80 1 -- start start 2023-10-11T16:42:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.795+0000 7f6cdecd9b80 1 Processor -- start 2023-10-11T16:42:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818acefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac3470 0x55a818acc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 0x55a818acc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8189b58e0 con 0x55a818ac2b60 2023-10-11T16:42:56.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8189b5580 con 0x55a818ac78b0 2023-10-11T16:42:56.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818897920 con 0x55a818ac3470 2023-10-11T16:42:56.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cdecd9b80 10 client.0 _ll_register_callbacks cb 0x55a8189d3780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:56.811 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818acefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818acefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45118/0 (socket says 172.21.15.111:45118) 2023-10-11T16:42:56.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc899a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac3470 0x55a818acc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc919b700 1 -- 172.21.15.111:0/2644992226 learned_addr learned my addr 172.21.15.111:0/2644992226 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.812 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc999c700 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 0x55a818acc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc919b700 1 -- 172.21.15.111:0/2644992226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac3470 msgr2=0x55a818acc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc919b700 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac3470 0x55a818acc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc919b700 1 -- 172.21.15.111:0/2644992226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 msgr2=0x55a818acc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc919b700 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 0x55a818acc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.796+0000 7f6cc919b700 1 -- 172.21.15.111:0/2644992226 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc41101c0 con 0x55a818ac2b60 2023-10-11T16:42:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cc919b700 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818acefe0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009230 tx=0x7f6cc00d0df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658c409d1f9cb4ca server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cacff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cacff9700 1 -- 172.21.15.111:0/2644992226 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00cd120 con 0x55a818ac2b60 2023-10-11T16:42:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cacff9700 1 -- 172.21.15.111:0/2644992226 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f6cc00d08e0 con 0x55a818ac2b60 2023-10-11T16:42:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/2644992226 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a818ab5b20 con 0x55a818ac2b60 2023-10-11T16:42:56.815 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cacff9700 1 -- 172.21.15.111:0/2644992226 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cc009dfe0 con 0x55a818ac2b60 2023-10-11T16:42:56.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 10 client.17226 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:56.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/2644992226 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55a8189da190 con 0x55a818ac2b60 2023-10-11T16:42:56.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cacff9700 1 -- 172.21.15.111:0/2644992226 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a818ab5b20 con 0x55a818ac2b60 2023-10-11T16:42:56.816 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cacff9700 1 -- 172.21.15.111:0/2644992226 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6cc00a1f70 con 0x55a818ac2b60 2023-10-11T16:42:56.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 10 client.17226 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:56.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 10 client.17226 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:42:56.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 20 client.17226 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:56.817 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a818adc4c0 0x55a818ade8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 4 client.17226 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:56.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.797+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/2644992226 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8189d60a0 con 0x55a818adc4c0 2023-10-11T16:42:56.818 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.798+0000 7f6cc899a700 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a818adc4c0 0x55a818ade8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.799+0000 7f6cacff9700 10 client.17226 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:42:56.819 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.799+0000 7f6cc899a700 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a818adc4c0 0x55a818ade8a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=876ad3bf4f3e48a8 server_cookie=310bd4533b128e99 in_seq=0 out_seq=0 2023-10-11T16:42:56.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cacff9700 1 -- 172.21.15.111:0/2644992226 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a8189d60a0 con 0x55a818adc4c0 2023-10-11T16:42:56.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cacff9700 10 client.17226 handle_command_reply: tid=0 2023-10-11T16:42:56.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 client.17226 shutdown 2023-10-11T16:42:56.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/2644992226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 msgr2=0x55a818acefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818acefe0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009230 tx=0x7f6cc00d0df0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/2644992226 shutdown_connections 2023-10-11T16:42:56.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a818adc4c0 0x55a818ade8a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 0x55a818acc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818acefe0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/2644992226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac3470 0x55a818acc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/2644992226 >> 172.21.15.111:0/2644992226 conn(0x55a818ab8360 msgr2=0x55a818abc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cacff9700 0 client.17226 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:42:56.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/2644992226 shutdown_connections 2023-10-11T16:42:56.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/2644992226 wait complete. 2023-10-11T16:42:56.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.800+0000 7f6cdecd9b80 20 client.17226 trim_cache size 0 max 16384 2023-10-11T16:42:56.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cdecd9b80 1 Processor -- start 2023-10-11T16:42:56.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cdecd9b80 1 -- start start 2023-10-11T16:42:56.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818aceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac78b0 0x55a818ac5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818abe950 con 0x55a818ac2b60 2023-10-11T16:42:56.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818ac63e0 con 0x55a818ac3470 2023-10-11T16:42:56.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818ac3910 con 0x55a818ac78b0 2023-10-11T16:42:56.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cc919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818aceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cc919b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818aceb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45132/0 (socket says 172.21.15.111:45132) 2023-10-11T16:42:56.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cc919b700 1 -- 172.21.15.111:0/3647303738 learned_addr learned my addr 172.21.15.111:0/3647303738 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.814+0000 7f6cc899a700 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6cc999c700 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac78b0 0x55a818ac5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6cc919b700 1 -- 172.21.15.111:0/3647303738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac78b0 msgr2=0x55a818ac5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6cc919b700 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac78b0 0x55a818ac5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6cc919b700 1 -- 172.21.15.111:0/3647303738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 msgr2=0x55a818ac58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.833 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6cc919b700 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6cc919b700 1 -- 172.21.15.111:0/3647303738 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a818ab9b00 con 0x55a818ac2b60 2023-10-11T16:42:56.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6cc919b700 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818aceb80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0099f40 tx=0x7f6cc0010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10936de33bd93f45 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6caffff700 1 -- 172.21.15.111:0/3647303738 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00cd120 con 0x55a818ac2b60 2023-10-11T16:42:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6caffff700 1 -- 172.21.15.111:0/3647303738 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cc009dfe0 con 0x55a818ac2b60 2023-10-11T16:42:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.815+0000 7f6caffff700 1 -- 172.21.15.111:0/3647303738 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0099700 con 0x55a818ac2b60 2023-10-11T16:42:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3647303738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 msgr2=0x55a818aceb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818aceb80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0099f40 tx=0x7f6cc0010c80 comp rx=0 tx=0).stop 2023-10-11T16:42:56.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3647303738 shutdown_connections 2023-10-11T16:42:56.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac78b0 0x55a818ac5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac2b60 0x55a818aceb80 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3647303738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac3470 0x55a818ac58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3647303738 >> 172.21.15.111:0/3647303738 conn(0x55a818ab8360 msgr2=0x55a818ac9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3647303738 shutdown_connections 2023-10-11T16:42:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3647303738 wait complete. 2023-10-11T16:42:56.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 -- start start 2023-10-11T16:42:56.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.816+0000 7f6cdecd9b80 1 Processor -- start 2023-10-11T16:42:56.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818b28bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.838 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac3470 0x55a818b29100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cdecd9b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 0x55a818ad2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818ad32f0 con 0x55a818ac3470 2023-10-11T16:42:56.839 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818ad3460 con 0x55a818ac78b0 2023-10-11T16:42:56.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cdecd9b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a818ad3620 con 0x55a818ac2b60 2023-10-11T16:42:56.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cdecd9b80 10 client.0 _ll_register_callbacks cb 0x7f6cb4009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:56.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cc899a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac3470 0x55a818b29100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.840 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cc899a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac3470 0x55a818b29100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45136/0 (socket says 172.21.15.111:45136) 2023-10-11T16:42:56.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cc899a700 1 -- 172.21.15.111:0/3233744624 learned_addr learned my addr 172.21.15.111:0/3233744624 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:56.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.817+0000 7f6cc919b700 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818b28bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cc999c700 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 0x55a818ad2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.841 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cc899a700 1 -- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 msgr2=0x55a818b28bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cc899a700 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818b28bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cc899a700 1 -- 172.21.15.111:0/3233744624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 msgr2=0x55a818ad2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:56.842 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cc899a700 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 0x55a818ad2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cc899a700 1 -- 172.21.15.111:0/3233744624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8189da190 con 0x55a818ac3470 2023-10-11T16:42:56.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cc919b700 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818b28bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:42:56.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cc899a700 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac3470 0x55a818b29100 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x55a818acd810 tx=0x7f6cb4004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a5fb5fb19e2834 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:56.843 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cacff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:56.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cacff9700 1 -- 172.21.15.111:0/3233744624 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4016040 con 0x55a818ac3470 2023-10-11T16:42:56.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3233744624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6cc0099af0 con 0x55a818ac3470 2023-10-11T16:42:56.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.818+0000 7f6cacff9700 1 -- 172.21.15.111:0/3233744624 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f6cb4003cc0 con 0x55a818ac3470 2023-10-11T16:42:56.844 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cacff9700 1 -- 172.21.15.111:0/3233744624 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cb401d9d0 con 0x55a818ac3470 2023-10-11T16:42:56.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cdecd9b80 10 client.17232 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:56.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3233744624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55a818841de0 con 0x55a818ac3470 2023-10-11T16:42:56.845 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cacff9700 1 -- 172.21.15.111:0/3233744624 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cc0099af0 con 0x55a818ac3470 2023-10-11T16:42:56.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cacff9700 1 -- 172.21.15.111:0/3233744624 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6cb4010ae0 con 0x55a818ac3470 2023-10-11T16:42:56.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cdecd9b80 10 client.17232 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:56.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cdecd9b80 10 client.17232 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-11T16:42:56.846 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cdecd9b80 20 client.17232 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:56.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a818b30a90 0x55a818b32ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:56.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cdecd9b80 4 client.17232 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:42:56.847 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.819+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3233744624 --> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a8189e2660 con 0x55a818b30a90 2023-10-11T16:42:56.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.820+0000 7f6cc919b700 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a818b30a90 0x55a818b32ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:56.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.820+0000 7f6cacff9700 10 client.17232 ms_handle_connect on v2:172.21.15.150:6838/3698951370 2023-10-11T16:42:56.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.820+0000 7f6cc919b700 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a818b30a90 0x55a818b32ea0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=77b7b8009e4534b0 server_cookie=b2caab926464546f in_seq=0 out_seq=0 2023-10-11T16:42:56.848 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.820+0000 7f6cacff9700 1 -- 172.21.15.111:0/3233744624 <== mds.1 v2:172.21.15.150:6838/3698951370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a8189e2660 con 0x55a818b30a90 2023-10-11T16:42:56.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.820+0000 7f6cacff9700 10 client.17232 handle_command_reply: tid=0 2023-10-11T16:42:56.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.820+0000 7f6cdecd9b80 1 client.17232 shutdown 2023-10-11T16:42:56.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3233744624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac3470 msgr2=0x55a818b29100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.849 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac3470 0x55a818b29100 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x55a818acd810 tx=0x7f6cb4004320 comp rx=0 tx=0).stop 2023-10-11T16:42:56.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3233744624 shutdown_connections 2023-10-11T16:42:56.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:6838/3698951370,v1:172.21.15.150:6839/3698951370] conn(0x55a818b30a90 0x55a818b32ea0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.850 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a818ac78b0 0x55a818ad2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a818ac2b60 0x55a818b28bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 --2- 172.21.15.111:0/3233744624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55a818ac3470 0x55a818b29100 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cacff9700 0 client.17232 ms_handle_reset on v2:172.21.15.150:6838/3698951370 2023-10-11T16:42:56.851 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3233744624 >> 172.21.15.111:0/3233744624 conn(0x55a818ab8360 msgr2=0x55a818ac58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3233744624 shutdown_connections 2023-10-11T16:42:56.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 1 -- 172.21.15.111:0/3233744624 wait complete. 2023-10-11T16:42:56.852 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6cdecd9b80 20 client.17232 trim_cache size 0 max 16384 2023-10-11T16:42:56.853 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:42:56.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6cb00a12c0 msgr2=0x7f6cb00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.854 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6cb00a12c0 0x7f6cb00a3770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6cc419f260 tx=0x7f6cbc0099e0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc4101390 msgr2=0x7f6cc419e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:56.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.821+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc4101390 0x7f6cc419e560 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000e9a0 tx=0x7f6cc000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.822+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/1685330442 shutdown_connections 2023-10-11T16:42:56.855 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.822+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6cb00a12c0 0x7f6cb00a3770 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.822+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6cc410a8b0 0x7f6cc41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.822+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc4101390 0x7f6cc419e560 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.856 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.822+0000 7f6ccb3ff700 1 --2- 172.21.15.111:0/1685330442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc4101ca0 0x7f6cc419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:56.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.822+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/1685330442 >> 172.21.15.111:0/1685330442 conn(0x7f6cc40770b0 msgr2=0x7f6cc40fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:56.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.822+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/1685330442 shutdown_connections 2023-10-11T16:42:56.857 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:56.822+0000 7f6ccb3ff700 1 -- 172.21.15.111:0/1685330442 wait complete. 2023-10-11T16:42:56.858 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-11T16:42:57.056 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.055+0000 7ff3879ff700 1 -- 172.21.15.111:0/414038677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380105620 msgr2=0x7ff38010d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.055+0000 7ff3879ff700 1 --2- 172.21.15.111:0/414038677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380105620 0x7ff38010d1a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff37800b0d0 tx=0x7ff37800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.057 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.056+0000 7ff3879ff700 1 -- 172.21.15.111:0/414038677 shutdown_connections 2023-10-11T16:42:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.056+0000 7ff3879ff700 1 --2- 172.21.15.111:0/414038677 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380105620 0x7ff38010d1a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.056+0000 7ff3879ff700 1 --2- 172.21.15.111:0/414038677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff380100cc0 0x7ff380101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.056+0000 7ff3879ff700 1 --2- 172.21.15.111:0/414038677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff380101660 0x7ff380104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.056+0000 7ff3879ff700 1 -- 172.21.15.111:0/414038677 >> 172.21.15.111:0/414038677 conn(0x7ff380077560 msgr2=0x7ff380077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:57.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.056+0000 7ff3879ff700 1 -- 172.21.15.111:0/414038677 shutdown_connections 2023-10-11T16:42:57.059 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.056+0000 7ff3879ff700 1 -- 172.21.15.111:0/414038677 wait complete. 2023-10-11T16:42:57.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff3879ff700 1 Processor -- start 2023-10-11T16:42:57.061 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff3879ff700 1 -- start start 2023-10-11T16:42:57.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff3879ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff380100cc0 0x7ff380196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff3879ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380101660 0x7ff380196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.062 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff3879ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff380105620 0x7ff38019afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff3879ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38019b620 con 0x7ff380101660 2023-10-11T16:42:57.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff3879ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38019b790 con 0x7ff380105620 2023-10-11T16:42:57.063 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff3879ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38019b950 con 0x7ff380100cc0 2023-10-11T16:42:57.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff384f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380101660 0x7ff380196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff384f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380101660 0x7ff380196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45148/0 (socket says 172.21.15.111:45148) 2023-10-11T16:42:57.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.057+0000 7ff384f9a700 1 -- 172.21.15.111:0/1998182051 learned_addr learned my addr 172.21.15.111:0/1998182051 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:57.064 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff38579b700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff380100cc0 0x7ff380196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff385f9c700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff380105620 0x7ff38019afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff384f9a700 1 -- 172.21.15.111:0/1998182051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff380100cc0 msgr2=0x7ff380196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff384f9a700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff380100cc0 0x7ff380196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.065 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff384f9a700 1 -- 172.21.15.111:0/1998182051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff380105620 msgr2=0x7ff38019afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff384f9a700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff380105620 0x7ff38019afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff384f9a700 1 -- 172.21.15.111:0/1998182051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378009d00 con 0x7ff380101660 2023-10-11T16:42:57.066 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff385f9c700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff380105620 0x7ff38019afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:42:57.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff384f9a700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380101660 0x7ff380196b80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff37000e910 tx=0x7ff37000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:57.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff3767fc700 1 -- 172.21.15.111:0/1998182051 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37000cbd0 con 0x7ff380101660 2023-10-11T16:42:57.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff3767fc700 1 -- 172.21.15.111:0/1998182051 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37000ee60 con 0x7ff380101660 2023-10-11T16:42:57.067 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff3879ff700 1 -- 172.21.15.111:0/1998182051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3801103e0 con 0x7ff380101660 2023-10-11T16:42:57.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff3767fc700 1 -- 172.21.15.111:0/1998182051 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700055c0 con 0x7ff380101660 2023-10-11T16:42:57.068 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.058+0000 7ff3879ff700 1 -- 172.21.15.111:0/1998182051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff380110250 con 0x7ff380101660 2023-10-11T16:42:57.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.061+0000 7ff3767fc700 1 -- 172.21.15.111:0/1998182051 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7ff370004750 con 0x7ff380101660 2023-10-11T16:42:57.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.063+0000 7ff39b280b80 1 Processor -- start 2023-10-11T16:42:57.074 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.063+0000 7ff39b280b80 1 -- start start 2023-10-11T16:42:57.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff3767fc700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff36c0a12c0 0x7ff36c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b25b60 0x564bc4b25f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.075 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b2a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b2ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff39b280b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4a2b060 con 0x564bc4b25b60 2023-10-11T16:42:57.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff39b280b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4a7d8a0 con 0x564bc4b2a8b0 2023-10-11T16:42:57.076 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff39b280b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4a76d30 con 0x564bc4b26470 2023-10-11T16:42:57.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff3767fc700 1 -- 172.21.15.111:0/1998182051 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7ff370014070 con 0x7ff380101660 2023-10-11T16:42:57.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff385f9c700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff36c0a12c0 0x7ff36c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.077 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff38579b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b25b60 0x564bc4b25f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff384f9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b2a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff38579b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b25b60 0x564bc4b25f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45156/0 (socket says 172.21.15.111:45156) 2023-10-11T16:42:57.078 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff38579b700 1 -- 172.21.15.111:0/219895927 learned_addr learned my addr 172.21.15.111:0/219895927 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:57.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff385f9c700 1 --2- 172.21.15.111:0/219895927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b2ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff384f9a700 1 -- 172.21.15.111:0/219895927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 msgr2=0x564bc4b2ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:57.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.064+0000 7ff384f9a700 1 --2- 172.21.15.111:0/219895927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b2ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.079 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.065+0000 7ff384f9a700 1 -- 172.21.15.111:0/219895927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b25b60 msgr2=0x564bc4b25f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:57.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.065+0000 7ff384f9a700 1 --2- 172.21.15.111:0/219895927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b25b60 0x564bc4b25f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.065+0000 7ff384f9a700 1 -- 172.21.15.111:0/219895927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bc48f8630 con 0x564bc4b26470 2023-10-11T16:42:57.080 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.065+0000 7ff385f9c700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff36c0a12c0 0x7ff36c0a3770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff37800b770 tx=0x7ff378008e90 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:57.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.065+0000 7ff384f9a700 1 --2- 172.21.15.111:0/219895927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b2a370 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff3700ca770 tx=0x7ff370099fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=247249fc04e5ff87 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:57.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.065+0000 7ff36b7fe700 1 -- 172.21.15.111:0/219895927 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37006a5b0 con 0x564bc4b26470 2023-10-11T16:42:57.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.065+0000 7ff36b7fe700 1 -- 172.21.15.111:0/219895927 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37009a0f0 con 0x564bc4b26470 2023-10-11T16:42:57.081 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.065+0000 7ff36b7fe700 1 -- 172.21.15.111:0/219895927 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700ca9f0 con 0x564bc4b26470 2023-10-11T16:42:57.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 -- 172.21.15.111:0/219895927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 msgr2=0x564bc4b2a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 --2- 172.21.15.111:0/219895927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b2a370 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff3700ca770 tx=0x7ff370099fc0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.082 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 -- 172.21.15.111:0/219895927 shutdown_connections 2023-10-11T16:42:57.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 --2- 172.21.15.111:0/219895927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b2ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 --2- 172.21.15.111:0/219895927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b25b60 0x564bc4b25f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 --2- 172.21.15.111:0/219895927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b2a370 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.083 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 -- 172.21.15.111:0/219895927 >> 172.21.15.111:0/219895927 conn(0x564bc4b1b360 msgr2=0x564bc4b20b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:57.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 -- 172.21.15.111:0/219895927 shutdown_connections 2023-10-11T16:42:57.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 -- 172.21.15.111:0/219895927 wait complete. 2023-10-11T16:42:57.084 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.066+0000 7ff39b280b80 1 -- start start 2023-10-11T16:42:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff39b280b80 1 Processor -- start 2023-10-11T16:42:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 0x564bc4b31ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b2f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b2a8b0 0x564bc4b2f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff39b280b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4a188e0 con 0x564bc4b26470 2023-10-11T16:42:57.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff39b280b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4a18580 con 0x564bc4b25b60 2023-10-11T16:42:57.086 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff39b280b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc48fa920 con 0x564bc4b2a8b0 2023-10-11T16:42:57.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff39b280b80 10 client.0 _ll_register_callbacks cb 0x564bc4a36780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:57.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff384f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b2f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff384f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b2f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45172/0 (socket says 172.21.15.111:45172) 2023-10-11T16:42:57.087 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff384f9a700 1 -- 172.21.15.111:0/3938955384 learned_addr learned my addr 172.21.15.111:0/3938955384 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:57.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.067+0000 7ff385f9c700 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b2a8b0 0x564bc4b2f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff38579b700 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 0x564bc4b31ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff384f9a700 1 -- 172.21.15.111:0/3938955384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b2a8b0 msgr2=0x564bc4b2f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:57.088 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff384f9a700 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b2a8b0 0x564bc4b2f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff384f9a700 1 -- 172.21.15.111:0/3938955384 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 msgr2=0x564bc4b31ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:57.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff384f9a700 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 0x564bc4b31ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.089 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff384f9a700 1 -- 172.21.15.111:0/3938955384 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff380110250 con 0x564bc4b26470 2023-10-11T16:42:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff384f9a700 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b2f270 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff3700ca7d0 tx=0x7ff3700cd2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3433789d709a38 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff368ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff368ff9700 1 -- 172.21.15.111:0/3938955384 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370067050 con 0x564bc4b26470 2023-10-11T16:42:57.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff368ff9700 1 -- 172.21.15.111:0/3938955384 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7ff3700a2540 con 0x564bc4b26470 2023-10-11T16:42:57.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff39b280b80 1 -- 172.21.15.111:0/3938955384 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564bc4b18b20 con 0x564bc4b26470 2023-10-11T16:42:57.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff368ff9700 1 -- 172.21.15.111:0/3938955384 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff37009a350 con 0x564bc4b26470 2023-10-11T16:42:57.091 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff39b280b80 10 client.17244 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.068+0000 7ff39b280b80 1 -- 172.21.15.111:0/3938955384 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x564bc4a3d190 con 0x564bc4b26470 2023-10-11T16:42:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff368ff9700 1 -- 172.21.15.111:0/3938955384 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564bc4b18b20 con 0x564bc4b26470 2023-10-11T16:42:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff368ff9700 1 -- 172.21.15.111:0/3938955384 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff3700a20b0 con 0x564bc4b26470 2023-10-11T16:42:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff39b280b80 10 client.17244 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff39b280b80 10 client.17244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff39b280b80 20 client.17244 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff39b280b80 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564bc4b3f390 0x564bc4b41770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff39b280b80 4 client.17244 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-11T16:42:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff39b280b80 1 -- 172.21.15.111:0/3938955384 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564bc4a390a0 con 0x564bc4b3f390 2023-10-11T16:42:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.069+0000 7ff38579b700 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564bc4b3f390 0x564bc4b41770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.070+0000 7ff38579b700 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564bc4b3f390 0x564bc4b41770 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa8a87d02f8d6e3f server_cookie=d78ff89cfbd9a1fc in_seq=0 out_seq=0 2023-10-11T16:42:57.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.070+0000 7ff368ff9700 10 client.17244 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:57.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff368ff9700 1 -- 172.21.15.111:0/3938955384 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564bc4a390a0 con 0x564bc4b3f390 2023-10-11T16:42:57.095 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff368ff9700 10 client.17244 handle_command_reply: tid=0 2023-10-11T16:42:57.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 client.17244 shutdown 2023-10-11T16:42:57.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 -- 172.21.15.111:0/3938955384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 msgr2=0x564bc4b2f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b2f270 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff3700ca7d0 tx=0x7ff3700cd2f0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.096 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 -- 172.21.15.111:0/3938955384 shutdown_connections 2023-10-11T16:42:57.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564bc4b3f390 0x564bc4b41770 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b2a8b0 0x564bc4b2f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.097 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 0x564bc4b31ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff368ff9700 0 client.17244 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:57.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 --2- 172.21.15.111:0/3938955384 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b2f270 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 -- 172.21.15.111:0/3938955384 >> 172.21.15.111:0/3938955384 conn(0x564bc4b1b360 msgr2=0x564bc4b2bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:57.098 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 -- 172.21.15.111:0/3938955384 shutdown_connections 2023-10-11T16:42:57.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 1 -- 172.21.15.111:0/3938955384 wait complete. 2023-10-11T16:42:57.099 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.071+0000 7ff39b280b80 20 client.17244 trim_cache size 0 max 16384 2023-10-11T16:42:57.104 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.084+0000 7ff39b280b80 1 Processor -- start 2023-10-11T16:42:57.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff39b280b80 1 -- start start 2023-10-11T16:42:57.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b25b60 0x564bc4b1f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.105 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b289d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b29850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff39b280b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4b26bc0 con 0x564bc4b26470 2023-10-11T16:42:57.106 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff39b280b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4b189a0 con 0x564bc4b2a8b0 2023-10-11T16:42:57.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff39b280b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4b41d00 con 0x564bc4b25b60 2023-10-11T16:42:57.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff384f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b289d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff38579b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b25b60 0x564bc4b1f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.107 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff384f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b289d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45174/0 (socket says 172.21.15.111:45174) 2023-10-11T16:42:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff384f9a700 1 -- 172.21.15.111:0/1562342210 learned_addr learned my addr 172.21.15.111:0/1562342210 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff385f9c700 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b29850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.108 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff384f9a700 1 -- 172.21.15.111:0/1562342210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b25b60 msgr2=0x564bc4b1f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff384f9a700 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b25b60 0x564bc4b1f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff384f9a700 1 -- 172.21.15.111:0/1562342210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 msgr2=0x564bc4b29850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff384f9a700 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b29850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.109 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff384f9a700 1 -- 172.21.15.111:0/1562342210 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bc4b2af50 con 0x564bc4b26470 2023-10-11T16:42:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.085+0000 7ff385f9c700 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b29850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-11T16:42:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.086+0000 7ff384f9a700 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b289d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff37009a080 tx=0x7ff37000efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b3fa513f20231a server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:57.110 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.086+0000 7ff36bfff700 1 -- 172.21.15.111:0/1562342210 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370067070 con 0x564bc4b26470 2023-10-11T16:42:57.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.086+0000 7ff36bfff700 1 -- 172.21.15.111:0/1562342210 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3700995e0 con 0x564bc4b26470 2023-10-11T16:42:57.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.086+0000 7ff36bfff700 1 -- 172.21.15.111:0/1562342210 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700b8500 con 0x564bc4b26470 2023-10-11T16:42:57.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.086+0000 7ff39b280b80 1 -- 172.21.15.111:0/1562342210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 msgr2=0x564bc4b289d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.086+0000 7ff39b280b80 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b289d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff37009a080 tx=0x7ff37000efd0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 -- 172.21.15.111:0/1562342210 shutdown_connections 2023-10-11T16:42:57.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b2a8b0 0x564bc4b29850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b25b60 0x564bc4b1f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 --2- 172.21.15.111:0/1562342210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b26470 0x564bc4b289d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 -- 172.21.15.111:0/1562342210 >> 172.21.15.111:0/1562342210 conn(0x564bc4b1b360 msgr2=0x564bc4b3bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 -- 172.21.15.111:0/1562342210 shutdown_connections 2023-10-11T16:42:57.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 -- 172.21.15.111:0/1562342210 wait complete. 2023-10-11T16:42:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 -- start start 2023-10-11T16:42:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.087+0000 7ff39b280b80 1 Processor -- start 2023-10-11T16:42:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 0x564bc4b8b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b8b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff39b280b80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b2a8b0 0x564bc4b8bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff39b280b80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4b96620 con 0x564bc4b2a8b0 2023-10-11T16:42:57.115 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff39b280b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4b96790 con 0x564bc4b25b60 2023-10-11T16:42:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff39b280b80 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bc4b96950 con 0x564bc4b26470 2023-10-11T16:42:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff39b280b80 10 client.0 _ll_register_callbacks cb 0x564bc4a36780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-11T16:42:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff385f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b2a8b0 0x564bc4b8bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff384f9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b8b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff385f9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b2a8b0 0x564bc4b8bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45190/0 (socket says 172.21.15.111:45190) 2023-10-11T16:42:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff385f9c700 1 -- 172.21.15.111:0/2251827016 learned_addr learned my addr 172.21.15.111:0/2251827016 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:57.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff38579b700 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 0x564bc4b8b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff385f9c700 1 -- 172.21.15.111:0/2251827016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 msgr2=0x564bc4b8b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff385f9c700 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b8b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff385f9c700 1 -- 172.21.15.111:0/2251827016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 msgr2=0x564bc4b8b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-11T16:42:57.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff385f9c700 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 0x564bc4b8b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff385f9c700 1 -- 172.21.15.111:0/2251827016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bc4a3d190 con 0x564bc4b2a8b0 2023-10-11T16:42:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.088+0000 7ff384f9a700 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b8b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:42:57.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff385f9c700 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b2a8b0 0x564bc4b8bf30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff378009fd0 tx=0x7ff37800b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873bdf733aa8c680 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:57.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff368ff9700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2023-10-11T16:42:57.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff368ff9700 1 -- 172.21.15.111:0/2251827016 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37800e040 con 0x564bc4b2a8b0 2023-10-11T16:42:57.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff368ff9700 1 -- 172.21.15.111:0/2251827016 <== mon.0 v2:172.21.15.111:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7ff3780078f0 con 0x564bc4b2a8b0 2023-10-11T16:42:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff39b280b80 1 -- 172.21.15.111:0/2251827016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff37009a0f0 con 0x564bc4b2a8b0 2023-10-11T16:42:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff368ff9700 1 -- 172.21.15.111:0/2251827016 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff37800bea0 con 0x564bc4b2a8b0 2023-10-11T16:42:57.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff39b280b80 10 client.17250 fetch_fsmap learned FSMap version 39 2023-10-11T16:42:57.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff39b280b80 1 -- 172.21.15.111:0/2251827016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x564bc48a4de0 con 0x564bc4b2a8b0 2023-10-11T16:42:57.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff368ff9700 1 -- 172.21.15.111:0/2251827016 <== mon.0 v2:172.21.15.111:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37009a0f0 con 0x564bc4b2a8b0 2023-10-11T16:42:57.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.089+0000 7ff368ff9700 1 -- 172.21.15.111:0/2251827016 <== mon.0 v2:172.21.15.111:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff3780051d0 con 0x564bc4b2a8b0 2023-10-11T16:42:57.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff39b280b80 10 client.17250 fetch_fsmap finished waiting for FSMap version 39 2023-10-11T16:42:57.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff39b280b80 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-11T16:42:57.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff39b280b80 20 client.17250 populate_metadata read hostname 'smithi111' 2023-10-11T16:42:57.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff39b280b80 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564bc4b98060 0x564bc4b9a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff39b280b80 4 client.17250 mds_command: new command op to 24569 tid=0[{"prefix": "damage ls"}] 2023-10-11T16:42:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff39b280b80 1 -- 172.21.15.111:0/2251827016 --> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564bc4a45660 con 0x564bc4b98060 2023-10-11T16:42:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff38579b700 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564bc4b98060 0x564bc4b9a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:57.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff368ff9700 10 client.17250 ms_handle_connect on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.090+0000 7ff38579b700 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564bc4b98060 0x564bc4b9a470 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b468db41ce8c1b8 server_cookie=92c5e6b4b2456fc5 in_seq=0 out_seq=0 2023-10-11T16:42:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff368ff9700 1 -- 172.21.15.111:0/2251827016 <== mds.0 v2:172.21.15.130:6838/2448702777 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564bc4a45660 con 0x564bc4b98060 2023-10-11T16:42:57.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff368ff9700 10 client.17250 handle_command_reply: tid=0 2023-10-11T16:42:57.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 client.17250 shutdown 2023-10-11T16:42:57.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 -- 172.21.15.111:0/2251827016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b2a8b0 msgr2=0x564bc4b8bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b2a8b0 0x564bc4b8bf30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff378009fd0 tx=0x7ff37800b3e0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 -- 172.21.15.111:0/2251827016 shutdown_connections 2023-10-11T16:42:57.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.130:6838/2448702777,v1:172.21.15.130:6839/2448702777] conn(0x564bc4b98060 0x564bc4b9a470 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x564bc4b2a8b0 0x564bc4b8bf30 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.127 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564bc4b25b60 0x564bc4b8b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff368ff9700 0 client.17250 ms_handle_reset on v2:172.21.15.130:6838/2448702777 2023-10-11T16:42:57.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 --2- 172.21.15.111:0/2251827016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564bc4b26470 0x564bc4b8b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 -- 172.21.15.111:0/2251827016 >> 172.21.15.111:0/2251827016 conn(0x564bc4b1b360 msgr2=0x564bc4b1f550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:57.128 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 -- 172.21.15.111:0/2251827016 shutdown_connections 2023-10-11T16:42:57.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 1 -- 172.21.15.111:0/2251827016 wait complete. 2023-10-11T16:42:57.129 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.091+0000 7ff39b280b80 20 client.17250 trim_cache size 0 max 16384 2023-10-11T16:42:57.130 INFO:teuthology.orchestra.run.smithi111.stdout:[] 2023-10-11T16:42:57.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 -- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff36c0a12c0 msgr2=0x7ff36c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.131 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff36c0a12c0 0x7ff36c0a3770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff37800b770 tx=0x7ff378008e90 comp rx=0 tx=0).stop 2023-10-11T16:42:57.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 -- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380101660 msgr2=0x7ff380196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:57.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380101660 0x7ff380196b80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff37000e910 tx=0x7ff37000ec20 comp rx=0 tx=0).stop 2023-10-11T16:42:57.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 -- 172.21.15.111:0/1998182051 shutdown_connections 2023-10-11T16:42:57.132 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff36c0a12c0 0x7ff36c0a3770 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff380105620 0x7ff38019afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff380100cc0 0x7ff380196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.133 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 --2- 172.21.15.111:0/1998182051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff380101660 0x7ff380196b80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:57.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 -- 172.21.15.111:0/1998182051 >> 172.21.15.111:0/1998182051 conn(0x7ff380077560 msgr2=0x7ff380103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:57.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 -- 172.21.15.111:0/1998182051 shutdown_connections 2023-10-11T16:42:57.134 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:57.092+0000 7ff3879ff700 1 -- 172.21.15.111:0/1998182051 wait complete. 2023-10-11T16:42:57.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:57 smithi111 ceph-mon[98527]: pgmap v321: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:42:57.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:57 smithi130 ceph-mon[131859]: pgmap v321: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:42:57.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:57 smithi150 ceph-mon[132217]: pgmap v321: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:42:58.135 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2023-10-11T16:42:58.135 INFO:tasks.fwd_scrub:done joining 2023-10-11T16:42:58.136 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-10-11T16:42:58.148 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:42:58.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 -- 172.21.15.111:0/2147091461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 msgr2=0x7f3b28104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:58.359 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 --2- 172.21.15.111:0/2147091461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 0x7f3b28104e60 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3b180099c0 tx=0x7f3b18009cd0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 -- 172.21.15.111:0/2147091461 shutdown_connections 2023-10-11T16:42:58.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 --2- 172.21.15.111:0/2147091461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b28105540 0x7f3b2810d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.360 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 --2- 172.21.15.111:0/2147091461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28100550 0x7f3b28100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 --2- 172.21.15.111:0/2147091461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 0x7f3b28104e60 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 -- 172.21.15.111:0/2147091461 >> 172.21.15.111:0/2147091461 conn(0x7f3b28077020 msgr2=0x7f3b28077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:58.361 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 -- 172.21.15.111:0/2147091461 shutdown_connections 2023-10-11T16:42:58.362 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.358+0000 7f3b307ff700 1 -- 172.21.15.111:0/2147091461 wait complete. 2023-10-11T16:42:58.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.359+0000 7f3b307ff700 1 Processor -- start 2023-10-11T16:42:58.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.359+0000 7f3b307ff700 1 -- start start 2023-10-11T16:42:58.364 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b307ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b28100550 0x7f3b281963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:58.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b307ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 0x7f3b28196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:58.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b307ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28105540 0x7f3b2819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:58.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b307ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2819b3c0 con 0x7f3b28100ef0 2023-10-11T16:42:58.365 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b307ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2819b530 con 0x7f3b28105540 2023-10-11T16:42:58.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b307ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2819b6f0 con 0x7f3b28100550 2023-10-11T16:42:58.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2dd9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 0x7f3b28196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:58.366 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2dd9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 0x7f3b28196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45208/0 (socket says 172.21.15.111:45208) 2023-10-11T16:42:58.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2dd9a700 1 -- 172.21.15.111:0/1368972914 learned_addr learned my addr 172.21.15.111:0/1368972914 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:58.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2ed9c700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28105540 0x7f3b2819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:58.367 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2e59b700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b28100550 0x7f3b281963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:58.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2dd9a700 1 -- 172.21.15.111:0/1368972914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b28100550 msgr2=0x7f3b281963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:58.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2dd9a700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b28100550 0x7f3b281963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2dd9a700 1 -- 172.21.15.111:0/1368972914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28105540 msgr2=0x7f3b2819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:58.368 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2dd9a700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28105540 0x7f3b2819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.360+0000 7f3b2dd9a700 1 -- 172.21.15.111:0/1368972914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b180095f0 con 0x7f3b28100ef0 2023-10-11T16:42:58.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.361+0000 7f3b2dd9a700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 0x7f3b28196920 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3b180098e0 tx=0x7f3b1800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:58.369 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.361+0000 7f3b1f7fe700 1 -- 172.21.15.111:0/1368972914 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1801d070 con 0x7f3b28100ef0 2023-10-11T16:42:58.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.361+0000 7f3b1f7fe700 1 -- 172.21.15.111:0/1368972914 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b18022470 con 0x7f3b28100ef0 2023-10-11T16:42:58.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.361+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b28110310 con 0x7f3b28100ef0 2023-10-11T16:42:58.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.361+0000 7f3b1f7fe700 1 -- 172.21.15.111:0/1368972914 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18017650 con 0x7f3b28100ef0 2023-10-11T16:42:58.370 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.361+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b28110180 con 0x7f3b28100ef0 2023-10-11T16:42:58.371 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.362+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b281104a0 con 0x7f3b28100ef0 2023-10-11T16:42:58.372 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.364+0000 7f3b1f7fe700 1 -- 172.21.15.111:0/1368972914 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f3b18022850 con 0x7f3b28100ef0 2023-10-11T16:42:58.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.365+0000 7f3b1f7fe700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3b140a12c0 0x7f3b140a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:58.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.365+0000 7f3b1f7fe700 1 -- 172.21.15.111:0/1368972914 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f3b180cdff0 con 0x7f3b28100ef0 2023-10-11T16:42:58.373 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.365+0000 7f3b2e59b700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3b140a12c0 0x7f3b140a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:58.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.366+0000 7f3b2e59b700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3b140a12c0 0x7f3b140a3770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009c60 tx=0x7f3b24009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:58.374 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.369+0000 7f3b1f7fe700 1 -- 172.21.15.111:0/1368972914 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3b18097070 con 0x7f3b28100ef0 2023-10-11T16:42:58.567 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.567+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b281a3450 con 0x7f3b28100ef0 2023-10-11T16:42:58.569 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.568+0000 7f3b1f7fe700 1 -- 172.21.15.111:0/1368972914 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+25411 (secure 0 0 0) 0x7f3b1809a690 con 0x7f3b28100ef0 2023-10-11T16:42:58.570 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:42:58.570 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":101,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:42:55.293687+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/2838247405":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/803529843":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/3816963652":"2023-10-12T16:32:37.683058+0000","172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/2838247405":"2023-10-12T16:32:37.683058+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1856545817":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/444576470":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:42:58.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3b140a12c0 msgr2=0x7f3b140a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:58.572 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3b140a12c0 0x7f3b140a3770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009c60 tx=0x7f3b24009260 comp rx=0 tx=0).stop 2023-10-11T16:42:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 msgr2=0x7f3b28196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 0x7f3b28196920 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f3b180098e0 tx=0x7f3b1800f690 comp rx=0 tx=0).stop 2023-10-11T16:42:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 shutdown_connections 2023-10-11T16:42:58.573 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f3b140a12c0 0x7f3b140a3770 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b28105540 0x7f3b2819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.574 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b28100550 0x7f3b281963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 --2- 172.21.15.111:0/1368972914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b28100ef0 0x7f3b28196920 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.575 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 >> 172.21.15.111:0/1368972914 conn(0x7f3b28077020 msgr2=0x7f3b280fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:58.576 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 shutdown_connections 2023-10-11T16:42:58.577 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.571+0000 7f3b307ff700 1 -- 172.21.15.111:0/1368972914 wait complete. 2023-10-11T16:42:58.592 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 99, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 99, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:42:58.593 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:42:58.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/38549565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 msgr2=0x7f6c9810d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:58.795 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/38549565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 0x7f6c9810d0a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b0a0 tx=0x7f6c9000b3b0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/38549565 shutdown_connections 2023-10-11T16:42:58.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/38549565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 0x7f6c9810d0a0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.796 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/38549565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c981063f0 0x7f6c981067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/38549565 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c98100a50 0x7f6c98100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/38549565 >> 172.21.15.111:0/38549565 conn(0x7f6c98077590 msgr2=0x7f6c98077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:58.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/38549565 shutdown_connections 2023-10-11T16:42:58.797 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.794+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/38549565 wait complete. 2023-10-11T16:42:58.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.796+0000 7f6c9e5ff700 1 Processor -- start 2023-10-11T16:42:58.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.796+0000 7f6c9e5ff700 1 -- start start 2023-10-11T16:42:58.800 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.796+0000 7f6c9e5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c98100a50 0x7f6c98072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:58.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.796+0000 7f6c9e5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 0x7f6c98073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:58.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c9e5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981063f0 0x7f6c981a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:58.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c9e5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98073b60 con 0x7f6c98101400 2023-10-11T16:42:58.801 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c9e5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c981a4be0 con 0x7f6c981063f0 2023-10-11T16:42:58.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c9e5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c981a4d20 con 0x7f6c98100a50 2023-10-11T16:42:58.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 0x7f6c98073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:58.802 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 0x7f6c98073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45224/0 (socket says 172.21.15.111:45224) 2023-10-11T16:42:58.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 -- 172.21.15.111:0/2258911058 learned_addr learned my addr 172.21.15.111:0/2258911058 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:58.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c97fff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c98100a50 0x7f6c98072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:58.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c9cb9c700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981063f0 0x7f6c981a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:58.803 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 -- 172.21.15.111:0/2258911058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c98100a50 msgr2=0x7f6c98072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:58.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c98100a50 0x7f6c98072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 -- 172.21.15.111:0/2258911058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981063f0 msgr2=0x7f6c981a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:58.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981063f0 0x7f6c981a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:58.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 -- 172.21.15.111:0/2258911058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90009d00 con 0x7f6c98101400 2023-10-11T16:42:58.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c9cb9c700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981063f0 0x7f6c981a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:42:58.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.797+0000 7f6c977fe700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 0x7f6c98073330 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00e9a0 tx=0x7f6c8c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:58.805 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.798+0000 7f6c957fa700 1 -- 172.21.15.111:0/2258911058 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c009790 con 0x7f6c98101400 2023-10-11T16:42:58.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.798+0000 7f6c957fa700 1 -- 172.21.15.111:0/2258911058 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c8c00cbd0 con 0x7f6c98101400 2023-10-11T16:42:58.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.798+0000 7f6c957fa700 1 -- 172.21.15.111:0/2258911058 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c018680 con 0x7f6c98101400 2023-10-11T16:42:58.806 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.798+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c98110410 con 0x7f6c98101400 2023-10-11T16:42:58.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.798+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c98110280 con 0x7f6c98101400 2023-10-11T16:42:58.807 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.799+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c981105a0 con 0x7f6c98101400 2023-10-11T16:42:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.801+0000 7f6c957fa700 1 -- 172.21.15.111:0/2258911058 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f6c8c00cd40 con 0x7f6c98101400 2023-10-11T16:42:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.803+0000 7f6c957fa700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6c800a1340 0x7f6c800a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:58.808 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.803+0000 7f6c957fa700 1 -- 172.21.15.111:0/2258911058 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f6c8c014070 con 0x7f6c98101400 2023-10-11T16:42:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.803+0000 7f6c97fff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6c800a1340 0x7f6c800a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.804+0000 7f6c97fff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6c800a1340 0x7f6c800a37f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009be0 tx=0x7f6c88009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:58.804+0000 7f6c957fa700 1 -- 172.21.15.111:0/2258911058 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6c8c096620 con 0x7f6c98101400 2023-10-11T16:42:59.005 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.005+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c98067b00 con 0x7f6c98101400 2023-10-11T16:42:59.006 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.005+0000 7f6c957fa700 1 -- 172.21.15.111:0/2258911058 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c98067b00 con 0x7f6c98101400 2023-10-11T16:42:59.007 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:42:59.008 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.007+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6c800a1340 msgr2=0x7f6c800a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.007+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6c800a1340 0x7f6c800a37f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009be0 tx=0x7f6c88009260 comp rx=0 tx=0).stop 2023-10-11T16:42:59.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 msgr2=0x7f6c98073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 0x7f6c98073330 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00e9a0 tx=0x7f6c8c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.009 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 shutdown_connections 2023-10-11T16:42:59.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f6c800a1340 0x7f6c800a37f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c981063f0 0x7f6c981a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.010 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c98100a50 0x7f6c98072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 --2- 172.21.15.111:0/2258911058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c98101400 0x7f6c98073330 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 >> 172.21.15.111:0/2258911058 conn(0x7f6c98077590 msgr2=0x7f6c980ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:59.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 shutdown_connections 2023-10-11T16:42:59.011 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.008+0000 7f6c9e5ff700 1 -- 172.21.15.111:0/2258911058 wait complete. 2023-10-11T16:42:59.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:42:59.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 -- 172.21.15.111:0/1193326989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 msgr2=0x7f8be8105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.231 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/1193326989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 0x7f8be8105120 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8009990 tx=0x7f8bd8009ca0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 -- 172.21.15.111:0/1193326989 shutdown_connections 2023-10-11T16:42:59.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/1193326989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be8105850 0x7f8be810d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.232 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/1193326989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8be8100cc0 0x7f8be8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/1193326989 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 0x7f8be8105120 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 -- 172.21.15.111:0/1193326989 >> 172.21.15.111:0/1193326989 conn(0x7f8be8077550 msgr2=0x7f8be8077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:59.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 -- 172.21.15.111:0/1193326989 shutdown_connections 2023-10-11T16:42:59.233 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.230+0000 7f8bef3ff700 1 -- 172.21.15.111:0/1193326989 wait complete. 2023-10-11T16:42:59.235 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.231+0000 7f8bef3ff700 1 Processor -- start 2023-10-11T16:42:59.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bef3ff700 1 -- start start 2023-10-11T16:42:59.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bef3ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be8100cc0 0x7f8be819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:59.236 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bef3ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 0x7f8be819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:59.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bef3ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8be8105850 0x7f8be81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:59.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bef3ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be8198c60 con 0x7f8be8101660 2023-10-11T16:42:59.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bef3ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be8198dd0 con 0x7f8be8105850 2023-10-11T16:42:59.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bef3ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be8198f90 con 0x7f8be8100cc0 2023-10-11T16:42:59.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bec99a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 0x7f8be819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:59.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bec99a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 0x7f8be819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45244/0 (socket says 172.21.15.111:45244) 2023-10-11T16:42:59.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bec99a700 1 -- 172.21.15.111:0/3737819646 learned_addr learned my addr 172.21.15.111:0/3737819646 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:59.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bed19b700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be8100cc0 0x7f8be819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:59.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bed99c700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8be8105850 0x7f8be81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:59.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bec99a700 1 -- 172.21.15.111:0/3737819646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be8100cc0 msgr2=0x7f8be819e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bec99a700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be8100cc0 0x7f8be819e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bec99a700 1 -- 172.21.15.111:0/3737819646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8be8105850 msgr2=0x7f8be81985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bec99a700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8be8105850 0x7f8be81985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.232+0000 7f8bec99a700 1 -- 172.21.15.111:0/3737819646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd80095f0 con 0x7f8be8101660 2023-10-11T16:42:59.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.233+0000 7f8bec99a700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 0x7f8be819eac0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8004730 tx=0x7f8bd8004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:59.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.233+0000 7f8bde7fc700 1 -- 172.21.15.111:0/3737819646 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd801d070 con 0x7f8be8101660 2023-10-11T16:42:59.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.233+0000 7f8bde7fc700 1 -- 172.21.15.111:0/3737819646 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8bd8004c90 con 0x7f8be8101660 2023-10-11T16:42:59.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.233+0000 7f8bde7fc700 1 -- 172.21.15.111:0/3737819646 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80054b0 con 0x7f8be8101660 2023-10-11T16:42:59.241 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.233+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be8110560 con 0x7f8be8101660 2023-10-11T16:42:59.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.233+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be81103d0 con 0x7f8be8101660 2023-10-11T16:42:59.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.234+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be81106f0 con 0x7f8be8101660 2023-10-11T16:42:59.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.236+0000 7f8bde7fc700 1 -- 172.21.15.111:0/3737819646 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f8bd800fbc0 con 0x7f8be8101660 2023-10-11T16:42:59.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.237+0000 7f8bde7fc700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8bd40a1340 0x7f8bd40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:59.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.237+0000 7f8bde7fc700 1 -- 172.21.15.111:0/3737819646 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f8bd80ce310 con 0x7f8be8101660 2023-10-11T16:42:59.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.238+0000 7f8bed19b700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8bd40a1340 0x7f8bd40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:59.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.238+0000 7f8bed19b700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8bd40a1340 0x7f8bd40a37f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8be4007c60 tx=0x7f8be40073d0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:59.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.241+0000 7f8bde7fc700 1 -- 172.21.15.111:0/3737819646 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8bd805a730 con 0x7f8be8101660 2023-10-11T16:42:59.444 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.444+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8be8199b50 con 0x7f8be8101660 2023-10-11T16:42:59.445 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.444+0000 7f8bde7fc700 1 -- 172.21.15.111:0/3737819646 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8bd8097350 con 0x7f8be8101660 2023-10-11T16:42:59.446 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:42:59.447 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8bd40a1340 msgr2=0x7f8bd40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8bd40a1340 0x7f8bd40a37f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8be4007c60 tx=0x7f8be40073d0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 msgr2=0x7f8be819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.448 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 0x7f8be819eac0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8004730 tx=0x7f8bd8004760 comp rx=0 tx=0).stop 2023-10-11T16:42:59.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 shutdown_connections 2023-10-11T16:42:59.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f8bd40a1340 0x7f8bd40a37f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8be8105850 0x7f8be81985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.449 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be8100cc0 0x7f8be819e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 --2- 172.21.15.111:0/3737819646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8be8101660 0x7f8be819eac0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 >> 172.21.15.111:0/3737819646 conn(0x7f8be8077550 msgr2=0x7f8be810c670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:59.450 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 shutdown_connections 2023-10-11T16:42:59.451 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.447+0000 7f8bef3ff700 1 -- 172.21.15.111:0/3737819646 wait complete. 2023-10-11T16:42:59.467 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:42:59.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:59 smithi130 ceph-mon[131859]: pgmap v322: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2023-10-11T16:42:59.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:59 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1368972914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:42:59.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:42:59 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2258911058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:42:59.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:59 smithi111 ceph-mon[98527]: pgmap v322: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2023-10-11T16:42:59.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:59 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1368972914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:42:59.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:42:59 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2258911058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:42:59.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 -- 172.21.15.111:0/2829910348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20100a50 msgr2=0x7f0b20100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2829910348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20100a50 0x7f0b20100ec0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009990 tx=0x7f0b0c009ca0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.671 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 -- 172.21.15.111:0/2829910348 shutdown_connections 2023-10-11T16:42:59.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2829910348 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b20101400 0x7f0b2010d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.672 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2829910348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b201063f0 0x7f0b201067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2829910348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20100a50 0x7f0b20100ec0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.673 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 -- 172.21.15.111:0/2829910348 >> 172.21.15.111:0/2829910348 conn(0x7f0b20077590 msgr2=0x7f0b20077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:59.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 -- 172.21.15.111:0/2829910348 shutdown_connections 2023-10-11T16:42:59.674 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.670+0000 7f0b26fff700 1 -- 172.21.15.111:0/2829910348 wait complete. 2023-10-11T16:42:59.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.671+0000 7f0b26fff700 1 Processor -- start 2023-10-11T16:42:59.680 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.671+0000 7f0b26fff700 1 -- start start 2023-10-11T16:42:59.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.671+0000 7f0b26fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b20100a50 0x7f0b20196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:59.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.671+0000 7f0b26fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20101400 0x7f0b20196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:59.681 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.671+0000 7f0b26fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b201063f0 0x7f0b2019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:59.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.671+0000 7f0b26fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2019b400 con 0x7f0b20101400 2023-10-11T16:42:59.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.671+0000 7f0b26fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2019b570 con 0x7f0b201063f0 2023-10-11T16:42:59.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.671+0000 7f0b26fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2019b730 con 0x7f0b20100a50 2023-10-11T16:42:59.682 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20101400 0x7f0b20196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:59.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20101400 0x7f0b20196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45260/0 (socket says 172.21.15.111:45260) 2023-10-11T16:42:59.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 -- 172.21.15.111:0/2355792726 learned_addr learned my addr 172.21.15.111:0/2355792726 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:42:59.683 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b24d9b700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b20100a50 0x7f0b20196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:59.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b2559c700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b201063f0 0x7f0b2019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:59.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 -- 172.21.15.111:0/2355792726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b20100a50 msgr2=0x7f0b20196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b20100a50 0x7f0b20196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.684 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 -- 172.21.15.111:0/2355792726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b201063f0 msgr2=0x7f0b2019adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b201063f0 0x7f0b2019adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 -- 172.21.15.111:0/2355792726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0095f0 con 0x7f0b20101400 2023-10-11T16:42:59.685 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.672+0000 7f0b1ffff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20101400 0x7f0b20196960 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c005910 tx=0x7f0b0c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:59.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.673+0000 7f0b1dffb700 1 -- 172.21.15.111:0/2355792726 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c01d070 con 0x7f0b20101400 2023-10-11T16:42:59.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.673+0000 7f0b1dffb700 1 -- 172.21.15.111:0/2355792726 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b0c022470 con 0x7f0b20101400 2023-10-11T16:42:59.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.673+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b20110410 con 0x7f0b20101400 2023-10-11T16:42:59.686 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.673+0000 7f0b1dffb700 1 -- 172.21.15.111:0/2355792726 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c017650 con 0x7f0b20101400 2023-10-11T16:42:59.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.673+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b20110280 con 0x7f0b20101400 2023-10-11T16:42:59.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.674+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b201105a0 con 0x7f0b20101400 2023-10-11T16:42:59.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.676+0000 7f0b1dffb700 1 -- 172.21.15.111:0/2355792726 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f0b0c0225e0 con 0x7f0b20101400 2023-10-11T16:42:59.687 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.677+0000 7f0b1dffb700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0b100a1340 0x7f0b100a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:42:59.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.677+0000 7f0b1dffb700 1 -- 172.21.15.111:0/2355792726 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7f0b0c0ce3e0 con 0x7f0b20101400 2023-10-11T16:42:59.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.677+0000 7f0b24d9b700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0b100a1340 0x7f0b100a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:42:59.688 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.678+0000 7f0b24d9b700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0b100a1340 0x7f0b100a37f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009be0 tx=0x7f0b14009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:42:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.681+0000 7f0b1dffb700 1 -- 172.21.15.111:0/2355792726 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0b0c097420 con 0x7f0b20101400 2023-10-11T16:42:59.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:59 smithi150 ceph-mon[132217]: pgmap v322: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2023-10-11T16:42:59.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:59 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1368972914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:42:59.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:42:59 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2258911058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:42:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.888+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0b20197810 con 0x7f0b20101400 2023-10-11T16:42:59.889 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.888+0000 7f0b1dffb700 1 -- 172.21.15.111:0/2355792726 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0b0c09aa40 con 0x7f0b20101400 2023-10-11T16:42:59.889 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:42:59.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0b100a1340 msgr2=0x7f0b100a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0b100a1340 0x7f0b100a37f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0b14009be0 tx=0x7f0b14009260 comp rx=0 tx=0).stop 2023-10-11T16:42:59.892 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20101400 msgr2=0x7f0b20196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:42:59.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20101400 0x7f0b20196960 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c005910 tx=0x7f0b0c005720 comp rx=0 tx=0).stop 2023-10-11T16:42:59.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 shutdown_connections 2023-10-11T16:42:59.893 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0b100a1340 0x7f0b100a37f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0b201063f0 0x7f0b2019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b20100a50 0x7f0b20196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 --2- 172.21.15.111:0/2355792726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b20101400 0x7f0b20196960 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:42:59.894 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 >> 172.21.15.111:0/2355792726 conn(0x7f0b20077590 msgr2=0x7f0b200ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:42:59.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 shutdown_connections 2023-10-11T16:42:59.895 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:42:59.891+0000 7f0b26fff700 1 -- 172.21.15.111:0/2355792726 wait complete. 2023-10-11T16:42:59.910 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:43:00.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 -- 172.21.15.111:0/1690413592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c101660 msgr2=0x7fc51c104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:00.111 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1690413592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c101660 0x7fc51c104ef0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0099c0 tx=0x7fc50c009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 -- 172.21.15.111:0/1690413592 shutdown_connections 2023-10-11T16:43:00.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1690413592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc51c105620 0x7fc51c10d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.112 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1690413592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc51c100cc0 0x7fc51c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1690413592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c101660 0x7fc51c104ef0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 -- 172.21.15.111:0/1690413592 >> 172.21.15.111:0/1690413592 conn(0x7fc51c077560 msgr2=0x7fc51c077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:00.113 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 -- 172.21.15.111:0/1690413592 shutdown_connections 2023-10-11T16:43:00.114 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.110+0000 7fc5237ff700 1 -- 172.21.15.111:0/1690413592 wait complete. 2023-10-11T16:43:00.116 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.111+0000 7fc5237ff700 1 Processor -- start 2023-10-11T16:43:00.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.111+0000 7fc5237ff700 1 -- start start 2023-10-11T16:43:00.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc5237ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c100cc0 0x7fc51c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:00.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc5237ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc51c101660 0x7fc51c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:00.117 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc5237ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc51c105620 0x7fc51c19bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:00.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc5237ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c19c5a0 con 0x7fc51c100cc0 2023-10-11T16:43:00.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc5237ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c19c710 con 0x7fc51c101660 2023-10-11T16:43:00.118 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc5237ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c19c8d0 con 0x7fc51c105620 2023-10-11T16:43:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc52159b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c100cc0 0x7fc51c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc52159b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c100cc0 0x7fc51c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45276/0 (socket says 172.21.15.111:45276) 2023-10-11T16:43:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc52159b700 1 -- 172.21.15.111:0/1056440865 learned_addr learned my addr 172.21.15.111:0/1056440865 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc520d9a700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc51c101660 0x7fc51c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc521d9c700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc51c105620 0x7fc51c19bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc52159b700 1 -- 172.21.15.111:0/1056440865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc51c105620 msgr2=0x7fc51c19bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc52159b700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc51c105620 0x7fc51c19bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc52159b700 1 -- 172.21.15.111:0/1056440865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc51c101660 msgr2=0x7fc51c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc52159b700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc51c101660 0x7fc51c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc52159b700 1 -- 172.21.15.111:0/1056440865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c0095f0 con 0x7fc51c100cc0 2023-10-11T16:43:00.121 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.112+0000 7fc521d9c700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc51c105620 0x7fc51c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:43:00.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.113+0000 7fc52159b700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c100cc0 0x7fc51c1986b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc51800e9a0 tx=0x7fc51800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:00.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.113+0000 7fc5127fc700 1 -- 172.21.15.111:0/1056440865 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc518009790 con 0x7fc51c100cc0 2023-10-11T16:43:00.122 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.113+0000 7fc5127fc700 1 -- 172.21.15.111:0/1056440865 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc51800cbd0 con 0x7fc51c100cc0 2023-10-11T16:43:00.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.113+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51c110370 con 0x7fc51c100cc0 2023-10-11T16:43:00.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.113+0000 7fc5127fc700 1 -- 172.21.15.111:0/1056440865 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc518018680 con 0x7fc51c100cc0 2023-10-11T16:43:00.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.113+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51c1101e0 con 0x7fc51c100cc0 2023-10-11T16:43:00.123 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.114+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51c110500 con 0x7fc51c100cc0 2023-10-11T16:43:00.124 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.116+0000 7fc5127fc700 1 -- 172.21.15.111:0/1056440865 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fc51800cd40 con 0x7fc51c100cc0 2023-10-11T16:43:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.117+0000 7fc5127fc700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5080a12c0 0x7fc5080a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.117+0000 7fc5127fc700 1 -- 172.21.15.111:0/1056440865 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7fc518014070 con 0x7fc51c100cc0 2023-10-11T16:43:00.125 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.117+0000 7fc520d9a700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5080a12c0 0x7fc5080a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.118+0000 7fc520d9a700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5080a12c0 0x7fc5080a3770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00b580 tx=0x7fc50c000bc0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:00.126 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.121+0000 7fc5127fc700 1 -- 172.21.15.111:0/1056440865 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc518051730 con 0x7fc51c100cc0 2023-10-11T16:43:00.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.323+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc51c1a5750 con 0x7fc51c100cc0 2023-10-11T16:43:00.324 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.323+0000 7fc5127fc700 1 -- 172.21.15.111:0/1056440865 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v101) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc518096380 con 0x7fc51c100cc0 2023-10-11T16:43:00.325 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:43:00.326 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5080a12c0 msgr2=0x7fc5080a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5080a12c0 0x7fc5080a3770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00b580 tx=0x7fc50c000bc0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c100cc0 msgr2=0x7fc51c1986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c100cc0 0x7fc51c1986b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc51800e9a0 tx=0x7fc51800ecb0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.327 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 shutdown_connections 2023-10-11T16:43:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5080a12c0 0x7fc5080a3770 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc51c105620 0x7fc51c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc51c100cc0 0x7fc51c1986b0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 --2- 172.21.15.111:0/1056440865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc51c101660 0x7fc51c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 >> 172.21.15.111:0/1056440865 conn(0x7fc51c077560 msgr2=0x7fc51c1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 shutdown_connections 2023-10-11T16:43:00.329 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.326+0000 7fc5237ff700 1 -- 172.21.15.111:0/1056440865 wait complete. 2023-10-11T16:43:00.345 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-10-11T16:43:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 -- 172.21.15.111:0/1935573067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 msgr2=0x7ff07c1008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 --2- 172.21.15.111:0/1935573067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 0x7ff07c1008e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff0780099c0 tx=0x7ff078009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 -- 172.21.15.111:0/1935573067 shutdown_connections 2023-10-11T16:43:00.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 --2- 172.21.15.111:0/1935573067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff07c105550 0x7ff07c10d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 --2- 172.21.15.111:0/1935573067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 0x7ff07c1008e0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.545 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 --2- 172.21.15.111:0/1935573067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff07c100eb0 0x7ff07c104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 -- 172.21.15.111:0/1935573067 >> 172.21.15.111:0/1935573067 conn(0x7ff07c077050 msgr2=0x7ff07c077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:00.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 -- 172.21.15.111:0/1935573067 shutdown_connections 2023-10-11T16:43:00.546 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.543+0000 7ff0845ff700 1 -- 172.21.15.111:0/1935573067 wait complete. 2023-10-11T16:43:00.548 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.544+0000 7ff0845ff700 1 Processor -- start 2023-10-11T16:43:00.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.544+0000 7ff0845ff700 1 -- start start 2023-10-11T16:43:00.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.544+0000 7ff0845ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 0x7ff07c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:00.549 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.544+0000 7ff0845ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff07c100eb0 0x7ff07c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:00.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.544+0000 7ff0845ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff07c105550 0x7ff07c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:00.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.544+0000 7ff0845ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c19d560 con 0x7ff07c100510 2023-10-11T16:43:00.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.544+0000 7ff0845ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c19d6d0 con 0x7ff07c100eb0 2023-10-11T16:43:00.550 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.544+0000 7ff0845ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c19d890 con 0x7ff07c105550 2023-10-11T16:43:00.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 0x7ff07c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:00.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 0x7ff07c198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45298/0 (socket says 172.21.15.111:45298) 2023-10-11T16:43:00.551 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 -- 172.21.15.111:0/835224480 learned_addr learned my addr 172.21.15.111:0/835224480 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:00.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff081b9a700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff07c100eb0 0x7ff07c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:00.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff082b9c700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff07c105550 0x7ff07c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:00.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 -- 172.21.15.111:0/835224480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff07c105550 msgr2=0x7ff07c19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:00.552 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff07c105550 0x7ff07c19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 -- 172.21.15.111:0/835224480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff07c100eb0 msgr2=0x7ff07c198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:00.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff07c100eb0 0x7ff07c198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:00.553 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 -- 172.21.15.111:0/835224480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0780095f0 con 0x7ff07c100510 2023-10-11T16:43:00.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff082b9c700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff07c105550 0x7ff07c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:43:00.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.545+0000 7ff08239b700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 0x7ff07c198580 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff078009990 tx=0x7ff07800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:00.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.546+0000 7ff0737fe700 1 -- 172.21.15.111:0/835224480 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07801d070 con 0x7ff07c100510 2023-10-11T16:43:00.554 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.546+0000 7ff0737fe700 1 -- 172.21.15.111:0/835224480 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0780054f0 con 0x7ff07c100510 2023-10-11T16:43:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.546+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c110360 con 0x7ff07c100510 2023-10-11T16:43:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.546+0000 7ff0737fe700 1 -- 172.21.15.111:0/835224480 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078004ae0 con 0x7ff07c100510 2023-10-11T16:43:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.546+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c1101d0 con 0x7ff07c100510 2023-10-11T16:43:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.547+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c1104f0 con 0x7ff07c100510 2023-10-11T16:43:00.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.549+0000 7ff0737fe700 1 -- 172.21.15.111:0/835224480 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7ff07800fb80 con 0x7ff07c100510 2023-10-11T16:43:00.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.550+0000 7ff0737fe700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff0680a1340 0x7ff0680a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:00.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.550+0000 7ff0737fe700 1 -- 172.21.15.111:0/835224480 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9540+0+0 (secure 0 0 0) 0x7ff0780ce280 con 0x7ff07c100510 2023-10-11T16:43:00.557 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.550+0000 7ff081b9a700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff0680a1340 0x7ff0680a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:00.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.551+0000 7ff081b9a700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff0680a1340 0x7ff0680a37f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff07c199b90 tx=0x7ff06c0073d0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:00.558 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.552+0000 7ff0737fe700 1 -- 172.21.15.111:0/835224480 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff0780972c0 con 0x7ff07c100510 2023-10-11T16:43:00.581 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:00 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3737819646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:43:00.582 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:00 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2355792726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:43:00.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:00 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3737819646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:43:00.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:00 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2355792726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:43:00.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:00 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3737819646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:43:00.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:00 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2355792726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:43:00.804 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.804+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7ff07c1a55f0 con 0x7ff07c100510 2023-10-11T16:43:00.814 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.814+0000 7ff0737fe700 1 -- 172.21.15.111:0/835224480 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) v1 ==== 81+0+733785 (secure 0 0 0) 0x7ff07809a8e0 con 0x7ff07c100510 2023-10-11T16:43:00.825 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:43:00.825 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:43:00.826 INFO:teuthology.orchestra.run.smithi111.stdout: "epoch": 25, 2023-10-11T16:43:00.826 INFO:teuthology.orchestra.run.smithi111.stdout: "active_gid": 25544, 2023-10-11T16:43:00.826 INFO:teuthology.orchestra.run.smithi111.stdout: "active_name": "x", 2023-10-11T16:43:00.827 INFO:teuthology.orchestra.run.smithi111.stdout: "active_addrs": { 2023-10-11T16:43:00.827 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2023-10-11T16:43:00.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:00.827 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2023-10-11T16:43:00.828 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6800", 2023-10-11T16:43:00.828 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1233945144 2023-10-11T16:43:00.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:00.829 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2023-10-11T16:43:00.829 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6801", 2023-10-11T16:43:00.829 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1233945144 2023-10-11T16:43:00.830 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:00.830 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:00.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.830 INFO:teuthology.orchestra.run.smithi111.stdout: "active_addr": "172.21.15.111:6801/1233945144", 2023-10-11T16:43:00.831 INFO:teuthology.orchestra.run.smithi111.stdout: "active_change": "2023-10-11T16:32:37.683304+0000", 2023-10-11T16:43:00.831 INFO:teuthology.orchestra.run.smithi111.stdout: "active_mgr_features": 4540138322906710015, 2023-10-11T16:43:00.831 INFO:teuthology.orchestra.run.smithi111.stdout: "available": true, 2023-10-11T16:43:00.832 INFO:teuthology.orchestra.run.smithi111.stdout: "standbys": [ 2023-10-11T16:43:00.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:00.832 INFO:teuthology.orchestra.run.smithi111.stdout: "gid": 25535, 2023-10-11T16:43:00.833 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "y", 2023-10-11T16:43:00.833 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_features": 4540138322906710015, 2023-10-11T16:43:00.833 INFO:teuthology.orchestra.run.smithi111.stdout: "available_modules": [ 2023-10-11T16:43:00.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:00.834 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alerts", 2023-10-11T16:43:00.834 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:00.834 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:00.835 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:00.835 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:00.835 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:00.835 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:00.836 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.836 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.836 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:00.837 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.837 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.837 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.837 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How frequently to reexamine health status", 2023-10-11T16:43:00.838 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.838 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.838 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.839 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:00.839 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:00.839 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.840 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.840 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.840 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:00.841 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.841 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.841 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:00.841 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:00.842 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:00.842 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:00.842 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:00.843 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:00.843 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:00.843 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:00.843 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:00.844 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.844 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.844 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.845 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:00.845 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:00.845 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:00.846 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.846 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.846 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:00.847 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.847 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.847 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.848 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:00.848 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.848 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.848 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.849 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:00.849 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:00.849 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.850 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.850 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.850 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:00.851 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.851 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.851 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:00.851 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:00.852 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:00.852 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:00.852 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:00.852 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:00.853 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:00.853 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:00.853 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:00.854 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.854 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.854 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.855 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:00.855 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:00.855 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:00.856 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.856 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.856 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:00.856 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.857 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.857 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.857 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:00.857 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.858 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.858 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.859 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_destination": { 2023-10-11T16:43:00.859 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_destination", 2023-10-11T16:43:00.859 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.860 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.860 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.860 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:00.860 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.861 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.861 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.861 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Email address to send alerts to", 2023-10-11T16:43:00.861 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.862 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.862 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.863 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_from_name": { 2023-10-11T16:43:00.863 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_from_name", 2023-10-11T16:43:00.863 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.863 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.864 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.864 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "Ceph", 2023-10-11T16:43:00.864 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.865 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.865 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.865 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Email From: name", 2023-10-11T16:43:00.865 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.866 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.866 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.867 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_host": { 2023-10-11T16:43:00.867 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_host", 2023-10-11T16:43:00.867 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.867 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.868 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.868 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:00.868 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.868 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.869 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.869 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP server", 2023-10-11T16:43:00.869 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.870 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.870 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.871 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_password": { 2023-10-11T16:43:00.871 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_password", 2023-10-11T16:43:00.871 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.871 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.872 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.872 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:00.872 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.873 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.873 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.873 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Password to authenticate with", 2023-10-11T16:43:00.873 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.874 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.874 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.875 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_port": { 2023-10-11T16:43:00.875 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_port", 2023-10-11T16:43:00.875 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:00.875 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.876 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.876 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "465", 2023-10-11T16:43:00.876 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.877 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.877 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.877 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP port", 2023-10-11T16:43:00.877 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.878 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.878 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.879 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_sender": { 2023-10-11T16:43:00.879 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_sender", 2023-10-11T16:43:00.879 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.879 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.880 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.880 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:00.880 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.881 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.881 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.881 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP envelope sender", 2023-10-11T16:43:00.881 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.882 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.882 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.883 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_ssl": { 2023-10-11T16:43:00.883 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_ssl", 2023-10-11T16:43:00.883 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:00.885 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.885 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.885 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:00.885 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.886 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.886 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.886 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use SSL to connect to SMTP server", 2023-10-11T16:43:00.887 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.887 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.887 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.888 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_user": { 2023-10-11T16:43:00.888 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_user", 2023-10-11T16:43:00.888 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.889 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.889 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.889 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:00.889 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.890 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.890 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.890 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "User to authenticate as", 2023-10-11T16:43:00.891 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.891 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.891 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.892 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:00.892 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:00.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:00.893 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "balancer", 2023-10-11T16:43:00.893 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:00.893 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:00.894 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:00.894 INFO:teuthology.orchestra.run.smithi111.stdout: "active": { 2023-10-11T16:43:00.894 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "active", 2023-10-11T16:43:00.894 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:00.895 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.895 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.895 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:00.896 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.896 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.896 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.896 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "automatically balance PGs across cluster", 2023-10-11T16:43:00.897 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.897 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.897 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.898 INFO:teuthology.orchestra.run.smithi111.stdout: "begin_time": { 2023-10-11T16:43:00.898 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "begin_time", 2023-10-11T16:43:00.898 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.899 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.899 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.899 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0000", 2023-10-11T16:43:00.899 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.900 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.900 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.900 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "beginning time of day to automatically balance", 2023-10-11T16:43:00.901 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-11T16:43:00.901 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.901 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.902 INFO:teuthology.orchestra.run.smithi111.stdout: "begin_weekday": { 2023-10-11T16:43:00.902 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "begin_weekday", 2023-10-11T16:43:00.902 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:00.903 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.903 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.903 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:00.903 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0", 2023-10-11T16:43:00.904 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "6", 2023-10-11T16:43:00.904 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.904 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-10-11T16:43:00.905 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-11T16:43:00.905 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.905 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.906 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_max_iterations": { 2023-10-11T16:43:00.906 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_max_iterations", 2023-10-11T16:43:00.906 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:00.907 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.907 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.907 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "25", 2023-10-11T16:43:00.910 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:00.911 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "250", 2023-10-11T16:43:00.911 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.911 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-10-11T16:43:00.912 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.912 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.912 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.913 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_metrics": { 2023-10-11T16:43:00.913 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_metrics", 2023-10-11T16:43:00.913 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.914 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.914 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.914 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "pgs,objects,bytes", 2023-10-11T16:43:00.915 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.915 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.915 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.915 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-10-11T16:43:00.916 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-10-11T16:43:00.916 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.916 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.917 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_step": { 2023-10-11T16:43:00.917 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_step", 2023-10-11T16:43:00.918 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:00.918 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.918 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.918 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0.5", 2023-10-11T16:43:00.919 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0.001", 2023-10-11T16:43:00.919 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "0.999", 2023-10-11T16:43:00.919 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.920 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "aggressiveness of optimization", 2023-10-11T16:43:00.920 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-10-11T16:43:00.920 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.920 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.921 INFO:teuthology.orchestra.run.smithi111.stdout: "end_time": { 2023-10-11T16:43:00.921 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "end_time", 2023-10-11T16:43:00.922 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.922 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.922 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.922 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "2359", 2023-10-11T16:43:00.923 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.923 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.923 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.924 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "ending time of day to automatically balance", 2023-10-11T16:43:00.924 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-11T16:43:00.924 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.924 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.925 INFO:teuthology.orchestra.run.smithi111.stdout: "end_weekday": { 2023-10-11T16:43:00.925 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "end_weekday", 2023-10-11T16:43:00.926 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:00.926 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.926 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.926 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:00.927 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0", 2023-10-11T16:43:00.927 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "6", 2023-10-11T16:43:00.927 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.928 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-10-11T16:43:00.928 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-11T16:43:00.928 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.928 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.929 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:00.929 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:00.929 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.930 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.930 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.930 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:00.931 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.931 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.931 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:00.931 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:00.932 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:00.932 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:00.932 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:00.933 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:00.933 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:00.933 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:00.933 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:00.934 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.934 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.934 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:00.936 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.936 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.936 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:00.937 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.937 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.937 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.938 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:00.938 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.938 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.939 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.939 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:00.939 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "min_score": { 2023-10-11T16:43:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "min_score", 2023-10-11T16:43:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-10-11T16:43:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": { 2023-10-11T16:43:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mode", 2023-10-11T16:43:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "upmap", 2023-10-11T16:43:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-compat", 2023-10-11T16:43:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "none", 2023-10-11T16:43:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap" 2023-10-11T16:43:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Balancer mode", 2023-10-11T16:43:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_ids": { 2023-10-11T16:43:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pool_ids", 2023-10-11T16:43:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-10-11T16:43:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-10-11T16:43:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap_max_deviation": { 2023-10-11T16:43:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "upmap_max_deviation", 2023-10-11T16:43:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "deviation below which no optimization is attempted", 2023-10-11T16:43:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-10-11T16:43:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap_max_optimizations": { 2023-10-11T16:43:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "upmap_max_optimizations", 2023-10-11T16:43:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-10-11T16:43:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephadm", 2023-10-11T16:43:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_down_multiplier": { 2023-10-11T16:43:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_down_multiplier", 2023-10-11T16:43:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3.0", 2023-10-11T16:43:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-10-11T16:43:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_refresh_rate": { 2023-10-11T16:43:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_refresh_rate", 2023-10-11T16:43:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "20", 2023-10-11T16:43:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-10-11T16:43:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_starting_port": { 2023-10-11T16:43:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_starting_port", 2023-10-11T16:43:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "4721", 2023-10-11T16:43:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-10-11T16:43:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "alertmanager_web_password": { 2023-10-11T16:43:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alertmanager_web_password", 2023-10-11T16:43:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Alertmanager web password", 2023-10-11T16:43:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "alertmanager_web_user": { 2023-10-11T16:43:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alertmanager_web_user", 2023-10-11T16:43:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Alertmanager web user", 2023-10-11T16:43:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_ptrace": { 2023-10-11T16:43:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_ptrace", 2023-10-11T16:43:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-10-11T16:43:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-10-11T16:43:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "autotune_interval": { 2023-10-11T16:43:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "autotune_interval", 2023-10-11T16:43:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to autotune daemon memory", 2023-10-11T16:43:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "autotune_memory_target_ratio": { 2023-10-11T16:43:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "autotune_memory_target_ratio", 2023-10-11T16:43:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0.7", 2023-10-11T16:43:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-10-11T16:43:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "cgroups_split": { 2023-10-11T16:43:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cgroups_split", 2023-10-11T16:43:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-10-11T16:43:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "config_checks_enabled": { 2023-10-11T16:43:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_checks_enabled", 2023-10-11T16:43:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-10-11T16:43:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "config_dashboard": { 2023-10-11T16:43:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_dashboard", 2023-10-11T16:43:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-10-11T16:43:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_alertmanager": { 2023-10-11T16:43:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_alertmanager", 2023-10-11T16:43:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-10-11T16:43:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_base": { 2023-10-11T16:43:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_base", 2023-10-11T16:43:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/ceph", 2023-10-11T16:43:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.029 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.029 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.029 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Container image name, without the tag", 2023-10-11T16:43:01.029 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.030 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.030 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.031 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_elasticsearch": { 2023-10-11T16:43:01.031 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_elasticsearch", 2023-10-11T16:43:01.031 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.031 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.032 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.032 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-10-11T16:43:01.032 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.033 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.033 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.033 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "elasticsearch container image", 2023-10-11T16:43:01.033 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.034 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.034 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.035 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_grafana": { 2023-10-11T16:43:01.035 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_grafana", 2023-10-11T16:43:01.035 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.036 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.036 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.036 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-10-11T16:43:01.037 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.037 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.037 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.037 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:01.038 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.038 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.038 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.039 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_haproxy": { 2023-10-11T16:43:01.039 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_haproxy", 2023-10-11T16:43:01.040 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.040 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.040 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.040 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-10-11T16:43:01.041 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.041 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.041 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.041 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "HAproxy container image", 2023-10-11T16:43:01.042 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.042 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.042 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.043 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_agent": { 2023-10-11T16:43:01.043 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_agent", 2023-10-11T16:43:01.043 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.044 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.044 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.044 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-10-11T16:43:01.045 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.045 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.045 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.045 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger agent container image", 2023-10-11T16:43:01.046 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.046 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.046 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.047 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_collector": { 2023-10-11T16:43:01.047 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_collector", 2023-10-11T16:43:01.047 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.048 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.048 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.048 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-10-11T16:43:01.049 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.049 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.049 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.049 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger collector container image", 2023-10-11T16:43:01.050 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.050 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.050 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.051 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_query": { 2023-10-11T16:43:01.051 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_query", 2023-10-11T16:43:01.051 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.052 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.052 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.052 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-10-11T16:43:01.053 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.053 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.053 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.053 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger query container image", 2023-10-11T16:43:01.054 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.054 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.054 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.055 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_keepalived": { 2023-10-11T16:43:01.055 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_keepalived", 2023-10-11T16:43:01.055 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.056 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.056 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.056 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-10-11T16:43:01.056 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.057 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.057 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.057 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Keepalived container image", 2023-10-11T16:43:01.058 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.058 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.058 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.059 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_loki": { 2023-10-11T16:43:01.059 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_loki", 2023-10-11T16:43:01.059 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.060 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.060 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.060 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-10-11T16:43:01.060 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.061 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.061 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.061 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Loki container image", 2023-10-11T16:43:01.061 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.062 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.062 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.063 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_node_exporter": { 2023-10-11T16:43:01.063 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_node_exporter", 2023-10-11T16:43:01.063 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.063 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.064 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.064 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-10-11T16:43:01.064 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.065 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.065 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.065 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:01.065 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.066 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.066 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.067 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_prometheus": { 2023-10-11T16:43:01.067 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_prometheus", 2023-10-11T16:43:01.067 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.067 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.068 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.068 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-10-11T16:43:01.068 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.068 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.069 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.069 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:01.069 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.070 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.070 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.070 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_promtail": { 2023-10-11T16:43:01.071 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_promtail", 2023-10-11T16:43:01.071 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.071 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.072 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.072 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-10-11T16:43:01.072 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.072 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.073 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.073 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Promtail container image", 2023-10-11T16:43:01.073 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.073 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.074 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.074 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_snmp_gateway": { 2023-10-11T16:43:01.075 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_snmp_gateway", 2023-10-11T16:43:01.075 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.075 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.076 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.076 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-10-11T16:43:01.076 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.076 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.077 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.077 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SNMP Gateway container image", 2023-10-11T16:43:01.077 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.077 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.078 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.078 INFO:teuthology.orchestra.run.smithi111.stdout: "container_init": { 2023-10-11T16:43:01.079 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_init", 2023-10-11T16:43:01.079 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.079 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.079 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.080 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.080 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.080 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.081 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.081 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Run podman/docker with `--init`", 2023-10-11T16:43:01.081 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.081 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.082 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.082 INFO:teuthology.orchestra.run.smithi111.stdout: "daemon_cache_timeout": { 2023-10-11T16:43:01.083 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "daemon_cache_timeout", 2023-10-11T16:43:01.083 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.083 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.083 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.084 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:01.084 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.084 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.085 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.085 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-10-11T16:43:01.085 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.085 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.086 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.086 INFO:teuthology.orchestra.run.smithi111.stdout: "default_cephadm_command_timeout": { 2023-10-11T16:43:01.087 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "default_cephadm_command_timeout", 2023-10-11T16:43:01.087 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.087 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.087 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.088 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "900", 2023-10-11T16:43:01.088 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.088 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.089 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.089 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-10-11T16:43:01.089 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.089 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.090 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.090 INFO:teuthology.orchestra.run.smithi111.stdout: "default_registry": { 2023-10-11T16:43:01.091 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "default_registry", 2023-10-11T16:43:01.091 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.091 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.091 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.092 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io", 2023-10-11T16:43:01.092 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.092 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.093 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.093 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-10-11T16:43:01.093 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.093 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.094 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.094 INFO:teuthology.orchestra.run.smithi111.stdout: "device_cache_timeout": { 2023-10-11T16:43:01.095 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_cache_timeout", 2023-10-11T16:43:01.095 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.095 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.095 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.096 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1800", 2023-10-11T16:43:01.096 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.096 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.096 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.097 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache device inventory", 2023-10-11T16:43:01.097 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.097 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.098 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.098 INFO:teuthology.orchestra.run.smithi111.stdout: "device_enhanced_scan": { 2023-10-11T16:43:01.098 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_enhanced_scan", 2023-10-11T16:43:01.099 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.099 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.099 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.100 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.100 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.100 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.100 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.101 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use libstoragemgmt during device scans", 2023-10-11T16:43:01.101 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.101 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.101 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.102 INFO:teuthology.orchestra.run.smithi111.stdout: "facts_cache_timeout": { 2023-10-11T16:43:01.102 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "facts_cache_timeout", 2023-10-11T16:43:01.103 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.103 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.103 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.103 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:01.104 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.104 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.104 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.105 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache host facts data", 2023-10-11T16:43:01.105 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.105 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.105 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.106 INFO:teuthology.orchestra.run.smithi111.stdout: "host_check_interval": { 2023-10-11T16:43:01.106 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "host_check_interval", 2023-10-11T16:43:01.106 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.107 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.107 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.107 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:01.107 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.108 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.108 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.108 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to perform a host check", 2023-10-11T16:43:01.109 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.109 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.109 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.110 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.110 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.110 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.111 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.111 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.111 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.111 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.113 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.113 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.113 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.114 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.114 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.114 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.115 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.115 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.115 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.115 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.116 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.116 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.116 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.117 INFO:teuthology.orchestra.run.smithi111.stdout: "log_refresh_metadata": { 2023-10-11T16:43:01.117 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_refresh_metadata", 2023-10-11T16:43:01.117 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.118 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.118 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.118 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.118 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.119 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.119 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.119 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-10-11T16:43:01.119 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.120 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.120 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.121 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.121 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.121 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.121 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.122 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.122 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.122 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.122 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.123 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.123 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-10-11T16:43:01.123 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.124 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.124 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.124 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.125 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.125 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.125 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.125 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.126 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.126 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.126 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.127 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.127 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.127 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.127 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.128 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.128 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.128 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.128 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.129 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.129 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.129 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.130 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.130 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.130 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.131 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.131 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.131 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.131 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.132 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.132 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.132 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.133 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.133 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.133 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.133 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.134 INFO:teuthology.orchestra.run.smithi111.stdout: "manage_etc_ceph_ceph_conf": { 2023-10-11T16:43:01.134 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-10-11T16:43:01.134 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.135 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.135 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.135 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.136 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.136 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.136 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.137 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-10-11T16:43:01.137 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.137 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.137 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.138 INFO:teuthology.orchestra.run.smithi111.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-10-11T16:43:01.138 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-10-11T16:43:01.139 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.139 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.139 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.139 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "*", 2023-10-11T16:43:01.140 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.140 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.140 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.141 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-10-11T16:43:01.141 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.141 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.141 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.142 INFO:teuthology.orchestra.run.smithi111.stdout: "max_count_per_host": { 2023-10-11T16:43:01.142 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_count_per_host", 2023-10-11T16:43:01.143 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.143 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.143 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.143 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:01.144 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.144 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.144 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.145 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "max number of daemons per service per host", 2023-10-11T16:43:01.145 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.145 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.145 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.146 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd_draining_count": { 2023-10-11T16:43:01.146 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_osd_draining_count", 2023-10-11T16:43:01.146 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.147 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.147 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.147 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:01.148 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.148 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.148 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.148 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-10-11T16:43:01.149 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.149 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.149 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.150 INFO:teuthology.orchestra.run.smithi111.stdout: "migration_current": { 2023-10-11T16:43:01.150 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "migration_current", 2023-10-11T16:43:01.150 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.151 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.151 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.151 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.151 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.152 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.152 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.152 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "internal - do not modify", 2023-10-11T16:43:01.153 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.153 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.153 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.154 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": { 2023-10-11T16:43:01.154 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mode", 2023-10-11T16:43:01.154 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.155 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.155 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.155 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "root", 2023-10-11T16:43:01.155 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.156 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.156 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.156 INFO:teuthology.orchestra.run.smithi111.stdout: "cephadm-package", 2023-10-11T16:43:01.156 INFO:teuthology.orchestra.run.smithi111.stdout: "root" 2023-10-11T16:43:01.157 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.157 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "mode for remote execution of cephadm", 2023-10-11T16:43:01.157 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.158 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.158 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.158 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_alerts_path": { 2023-10-11T16:43:01.159 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_alerts_path", 2023-10-11T16:43:01.159 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.159 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.159 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.160 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-10-11T16:43:01.160 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.160 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.161 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.161 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-10-11T16:43:01.161 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.161 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.162 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.162 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_web_password": { 2023-10-11T16:43:01.163 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_web_password", 2023-10-11T16:43:01.163 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.163 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.163 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.164 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:01.164 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.164 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.165 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.165 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus web password", 2023-10-11T16:43:01.165 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.165 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.166 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.166 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_web_user": { 2023-10-11T16:43:01.167 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_web_user", 2023-10-11T16:43:01.167 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.167 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.168 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.168 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:01.168 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.168 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.169 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.169 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus web user", 2023-10-11T16:43:01.169 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.169 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.170 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.170 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_insecure": { 2023-10-11T16:43:01.171 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_insecure", 2023-10-11T16:43:01.171 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.171 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.171 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.172 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.172 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.172 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.172 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.173 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-10-11T16:43:01.173 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.173 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.174 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.174 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_password": { 2023-10-11T16:43:01.174 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_password", 2023-10-11T16:43:01.175 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.175 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.175 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.176 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.176 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.176 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.176 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.177 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-10-11T16:43:01.177 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.177 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.178 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.178 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_url": { 2023-10-11T16:43:01.178 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_url", 2023-10-11T16:43:01.179 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.179 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.179 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.180 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.180 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.180 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.180 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.181 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-10-11T16:43:01.181 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.181 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.181 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.182 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_username": { 2023-10-11T16:43:01.182 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_username", 2023-10-11T16:43:01.183 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.183 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.183 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.183 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.184 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.184 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.184 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.185 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-10-11T16:43:01.185 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.185 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.185 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.186 INFO:teuthology.orchestra.run.smithi111.stdout: "secure_monitoring_stack": { 2023-10-11T16:43:01.186 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "secure_monitoring_stack", 2023-10-11T16:43:01.187 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.187 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.187 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.187 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.188 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.188 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.188 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.188 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-10-11T16:43:01.189 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.189 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.189 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.190 INFO:teuthology.orchestra.run.smithi111.stdout: "service_discovery_port": { 2023-10-11T16:43:01.190 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "service_discovery_port", 2023-10-11T16:43:01.190 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.191 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.191 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.191 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8765", 2023-10-11T16:43:01.192 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.192 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.192 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.192 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "cephadm service discovery port", 2023-10-11T16:43:01.193 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.193 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.193 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.194 INFO:teuthology.orchestra.run.smithi111.stdout: "ssh_config_file": { 2023-10-11T16:43:01.194 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssh_config_file", 2023-10-11T16:43:01.194 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.195 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.195 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.195 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.196 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.196 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.196 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.196 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-10-11T16:43:01.197 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.197 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.197 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.198 INFO:teuthology.orchestra.run.smithi111.stdout: "use_agent": { 2023-10-11T16:43:01.198 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "use_agent", 2023-10-11T16:43:01.198 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.199 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.199 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.199 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.199 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.200 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.200 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.200 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-10-11T16:43:01.200 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.201 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.201 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.202 INFO:teuthology.orchestra.run.smithi111.stdout: "use_repo_digest": { 2023-10-11T16:43:01.202 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "use_repo_digest", 2023-10-11T16:43:01.202 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.202 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.203 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.203 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.203 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.204 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.204 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.204 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-10-11T16:43:01.204 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.205 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.205 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.205 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_failed_host_check": { 2023-10-11T16:43:01.206 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_failed_host_check", 2023-10-11T16:43:01.206 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.206 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.207 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.207 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.207 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.207 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.208 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.208 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if the host check fails", 2023-10-11T16:43:01.208 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.209 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.209 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.209 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_stray_daemons": { 2023-10-11T16:43:01.210 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_stray_daemons", 2023-10-11T16:43:01.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.210 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.210 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.211 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.211 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.211 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.212 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.212 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-10-11T16:43:01.212 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.212 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.213 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.213 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_stray_hosts": { 2023-10-11T16:43:01.213 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_stray_hosts", 2023-10-11T16:43:01.214 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.214 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.214 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.215 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.215 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.215 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.215 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.216 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-10-11T16:43:01.216 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.216 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.216 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.217 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.217 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.218 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crash", 2023-10-11T16:43:01.218 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.218 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.218 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.219 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.219 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.219 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.220 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.220 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.220 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.220 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.221 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.221 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.221 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.221 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.222 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.222 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.222 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.223 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.223 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.223 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.223 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.224 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.224 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.224 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.225 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.225 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.225 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.225 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.226 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.226 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.226 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.226 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.227 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.227 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.227 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.227 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.228 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.228 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.228 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.229 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.229 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.229 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.229 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.230 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.230 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.230 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.231 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.231 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.231 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.231 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.232 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.232 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.232 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.232 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.233 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.233 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.233 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.234 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.234 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.234 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.235 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.235 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.235 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.235 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.236 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.236 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.236 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.236 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.237 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.237 INFO:teuthology.orchestra.run.smithi111.stdout: "retain_interval": { 2023-10-11T16:43:01.237 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "retain_interval", 2023-10-11T16:43:01.238 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.238 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.238 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.239 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "31536000", 2023-10-11T16:43:01.239 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.239 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.239 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.240 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long to retain crashes before pruning them", 2023-10-11T16:43:01.240 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.240 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.240 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.241 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_recent_interval": { 2023-10-11T16:43:01.241 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_recent_interval", 2023-10-11T16:43:01.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.242 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.242 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.242 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1209600", 2023-10-11T16:43:01.243 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.243 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.243 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.243 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "time interval in which to warn about recent crashes", 2023-10-11T16:43:01.244 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.244 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.244 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.245 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.245 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.246 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "dashboard", 2023-10-11T16:43:01.246 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.246 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.246 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.247 INFO:teuthology.orchestra.run.smithi111.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-10-11T16:43:01.247 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-10-11T16:43:01.247 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.247 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.248 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.248 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:01.248 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.249 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.249 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.249 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.249 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.250 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.250 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.250 INFO:teuthology.orchestra.run.smithi111.stdout: "ALERTMANAGER_API_HOST": { 2023-10-11T16:43:01.251 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ALERTMANAGER_API_HOST", 2023-10-11T16:43:01.251 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.251 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.251 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.252 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.252 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.252 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.253 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.253 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.253 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.253 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.254 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.254 INFO:teuthology.orchestra.run.smithi111.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-10-11T16:43:01.254 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-10-11T16:43:01.255 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.255 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.255 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.255 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.256 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.256 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.256 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.256 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.257 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.257 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.257 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.258 INFO:teuthology.orchestra.run.smithi111.stdout: "AUDIT_API_ENABLED": { 2023-10-11T16:43:01.258 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "AUDIT_API_ENABLED", 2023-10-11T16:43:01.258 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.259 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.259 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.259 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.259 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.260 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.260 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.260 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.260 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.260 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.261 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.261 INFO:teuthology.orchestra.run.smithi111.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-10-11T16:43:01.261 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-10-11T16:43:01.262 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.262 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.262 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.263 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.263 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.263 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.263 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.264 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.264 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.264 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.264 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.265 INFO:teuthology.orchestra.run.smithi111.stdout: "ENABLE_BROWSABLE_API": { 2023-10-11T16:43:01.265 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ENABLE_BROWSABLE_API", 2023-10-11T16:43:01.265 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.266 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.266 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.266 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.266 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.267 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.267 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.267 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.268 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.268 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.268 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.269 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-10-11T16:43:01.269 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-10-11T16:43:01.269 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.269 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.270 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.270 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.270 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.270 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.271 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.271 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.271 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.272 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.272 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.272 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-10-11T16:43:01.273 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-10-11T16:43:01.273 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.273 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.273 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.274 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.274 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.274 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.275 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.275 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.275 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.275 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.276 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.276 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-10-11T16:43:01.276 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-10-11T16:43:01.277 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.277 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.277 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.277 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.278 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.278 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.278 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.279 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.279 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.279 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.279 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.280 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-10-11T16:43:01.280 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-10-11T16:43:01.280 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.281 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.281 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.281 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.281 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.282 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.282 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.282 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.283 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.283 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.283 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.284 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_NFS": { 2023-10-11T16:43:01.284 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-10-11T16:43:01.285 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.285 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.285 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.285 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.286 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.286 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.286 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.286 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.287 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.287 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.287 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.288 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_RBD": { 2023-10-11T16:43:01.288 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-10-11T16:43:01.288 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.288 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.289 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.289 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.289 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.289 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.290 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.290 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.290 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.291 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.291 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.291 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_RGW": { 2023-10-11T16:43:01.292 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-10-11T16:43:01.292 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.292 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.292 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.293 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.293 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.293 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.294 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.294 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.294 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.294 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.295 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.295 INFO:teuthology.orchestra.run.smithi111.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-10-11T16:43:01.296 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-10-11T16:43:01.296 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.296 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.296 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.297 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.297 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.297 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.297 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.298 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.298 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.298 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.299 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.299 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_PASSWORD": { 2023-10-11T16:43:01.299 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_PASSWORD", 2023-10-11T16:43:01.300 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.300 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.300 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.300 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:01.301 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.301 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.301 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.302 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.302 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.302 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.302 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.303 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-10-11T16:43:01.303 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-10-11T16:43:01.303 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.304 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.304 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.304 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.304 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.305 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.305 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.305 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.305 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.306 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.306 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.306 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_URL": { 2023-10-11T16:43:01.307 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_URL", 2023-10-11T16:43:01.307 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.307 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.307 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.308 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.308 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.308 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.309 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.309 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.309 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.309 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.310 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.310 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_USERNAME": { 2023-10-11T16:43:01.311 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_USERNAME", 2023-10-11T16:43:01.311 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.311 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.311 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.312 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:01.312 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.312 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.312 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.313 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.313 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.313 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.313 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.314 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-10-11T16:43:01.314 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-10-11T16:43:01.315 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.315 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.315 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.315 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.316 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.316 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.316 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.317 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.317 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.317 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.317 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.318 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-10-11T16:43:01.318 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-10-11T16:43:01.318 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.319 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.319 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.319 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.323 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.323 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.323 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.324 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.324 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.324 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.325 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.325 INFO:teuthology.orchestra.run.smithi111.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-10-11T16:43:01.325 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-10-11T16:43:01.326 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.326 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.326 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.326 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.327 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.327 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.327 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.328 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.328 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.328 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.328 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.329 INFO:teuthology.orchestra.run.smithi111.stdout: "ISSUE_TRACKER_API_KEY": { 2023-10-11T16:43:01.329 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-10-11T16:43:01.329 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.330 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.330 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.330 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.331 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.331 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.331 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.331 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.332 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.332 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.332 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.333 INFO:teuthology.orchestra.run.smithi111.stdout: "PROMETHEUS_API_HOST": { 2023-10-11T16:43:01.333 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PROMETHEUS_API_HOST", 2023-10-11T16:43:01.333 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.333 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.334 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.334 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.334 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.334 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.335 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.335 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.335 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.336 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.336 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.336 INFO:teuthology.orchestra.run.smithi111.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-10-11T16:43:01.337 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-10-11T16:43:01.337 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.337 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.337 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.338 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.338 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.338 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.339 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.339 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.339 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.339 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.340 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.340 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-10-11T16:43:01.340 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-10-11T16:43:01.341 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.341 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.341 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.342 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.342 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.342 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.342 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.343 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.343 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.343 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.344 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.344 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-10-11T16:43:01.344 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-10-11T16:43:01.345 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.345 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.345 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.346 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.346 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.346 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.346 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.347 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.347 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.347 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.347 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.348 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-10-11T16:43:01.348 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-10-11T16:43:01.349 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.349 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.349 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.349 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.350 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.350 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.350 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.350 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.351 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.351 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.351 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.352 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-10-11T16:43:01.352 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-10-11T16:43:01.352 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.353 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.353 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.353 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.353 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.354 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.354 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.354 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.354 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.355 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.355 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.356 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-10-11T16:43:01.356 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-10-11T16:43:01.356 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.356 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.357 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.357 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.357 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.357 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.358 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.358 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.358 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.358 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.359 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.359 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-10-11T16:43:01.360 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-10-11T16:43:01.360 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.360 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.360 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.361 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.361 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.361 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.361 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.362 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.362 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.362 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.362 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.363 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-10-11T16:43:01.363 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-10-11T16:43:01.363 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.364 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.364 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.364 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.365 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.365 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.365 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.365 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.366 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.366 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.366 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.367 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_ENABLED": { 2023-10-11T16:43:01.367 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_ENABLED", 2023-10-11T16:43:01.367 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.368 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.368 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.368 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.368 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.369 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.369 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.369 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.369 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.370 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.370 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.371 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-10-11T16:43:01.371 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-10-11T16:43:01.371 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.371 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.372 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.372 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-10-11T16:43:01.372 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.373 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.373 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.373 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.373 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.374 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.374 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.374 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-10-11T16:43:01.375 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-10-11T16:43:01.375 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.375 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.376 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.376 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:01.376 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.376 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.377 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.377 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.377 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.378 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.378 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.378 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-10-11T16:43:01.379 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-10-11T16:43:01.379 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.379 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.380 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.380 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8", 2023-10-11T16:43:01.380 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.380 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.381 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.381 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.381 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.382 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.382 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.382 INFO:teuthology.orchestra.run.smithi111.stdout: "REST_REQUESTS_TIMEOUT": { 2023-10-11T16:43:01.383 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-10-11T16:43:01.383 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.383 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.383 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.384 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "45", 2023-10-11T16:43:01.384 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.384 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.385 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.385 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.385 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.386 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.386 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.386 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_ACCESS_KEY": { 2023-10-11T16:43:01.387 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_ACCESS_KEY", 2023-10-11T16:43:01.387 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.387 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.387 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.388 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.388 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.388 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.389 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.389 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.389 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.389 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.390 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.390 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-10-11T16:43:01.391 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-10-11T16:43:01.391 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.391 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.391 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.392 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:01.392 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.392 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.392 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.393 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.393 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.393 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.394 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.394 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_SECRET_KEY": { 2023-10-11T16:43:01.394 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_SECRET_KEY", 2023-10-11T16:43:01.395 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.395 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.395 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.396 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.396 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.396 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.396 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.397 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.397 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.397 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.397 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.398 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_SSL_VERIFY": { 2023-10-11T16:43:01.398 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_SSL_VERIFY", 2023-10-11T16:43:01.399 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.399 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.399 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.399 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.400 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.400 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.400 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.400 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.401 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.401 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.401 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.402 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-10-11T16:43:01.402 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-10-11T16:43:01.402 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.403 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.403 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.403 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:01.403 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.404 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.404 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.404 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.404 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.405 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.405 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.405 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-10-11T16:43:01.406 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-10-11T16:43:01.406 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.406 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.407 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.407 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:01.407 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.407 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.408 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.408 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.408 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.409 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.409 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.409 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-10-11T16:43:01.410 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-10-11T16:43:01.410 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.410 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.410 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.411 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:01.411 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.411 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.412 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.412 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.412 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.412 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.413 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.413 INFO:teuthology.orchestra.run.smithi111.stdout: "cross_origin_url": { 2023-10-11T16:43:01.414 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cross_origin_url", 2023-10-11T16:43:01.414 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.414 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.414 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.415 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.415 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.415 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.415 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.416 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.416 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.416 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.417 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.417 INFO:teuthology.orchestra.run.smithi111.stdout: "crt_file": { 2023-10-11T16:43:01.417 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crt_file", 2023-10-11T16:43:01.418 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.418 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.418 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.419 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.419 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.419 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.419 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.420 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.420 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.420 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.420 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.421 INFO:teuthology.orchestra.run.smithi111.stdout: "debug": { 2023-10-11T16:43:01.421 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "debug", 2023-10-11T16:43:01.422 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.422 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.422 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.422 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.423 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.423 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.423 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.424 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable/disable debug options", 2023-10-11T16:43:01.424 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.424 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.424 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.425 INFO:teuthology.orchestra.run.smithi111.stdout: "jwt_token_ttl": { 2023-10-11T16:43:01.425 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "jwt_token_ttl", 2023-10-11T16:43:01.425 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.426 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.426 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.426 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "28800", 2023-10-11T16:43:01.427 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.427 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.427 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.427 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.428 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.428 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.428 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.429 INFO:teuthology.orchestra.run.smithi111.stdout: "key_file": { 2023-10-11T16:43:01.429 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "key_file", 2023-10-11T16:43:01.429 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.430 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.430 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.430 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.430 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.431 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.431 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.431 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.431 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.432 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.432 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.433 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.433 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.433 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.433 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.434 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.434 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.434 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.434 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.435 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.435 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.435 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.436 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.436 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.436 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.436 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.437 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.437 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.437 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.437 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.438 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.438 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.439 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.439 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.439 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.439 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.440 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.440 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.440 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.441 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.441 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.441 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.441 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.442 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.442 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "motd": { 2023-10-11T16:43:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "motd", 2023-10-11T16:43:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "The message of the day", 2023-10-11T16:43:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "redirect_resolve_ip_addr": { 2023-10-11T16:43:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "redirect_resolve_ip_addr", 2023-10-11T16:43:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "::", 2023-10-11T16:43:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8080", 2023-10-11T16:43:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl": { 2023-10-11T16:43:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl", 2023-10-11T16:43:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl_server_port": { 2023-10-11T16:43:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl_server_port", 2023-10-11T16:43:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8443", 2023-10-11T16:43:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_behaviour": { 2023-10-11T16:43:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_behaviour", 2023-10-11T16:43:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "redirect", 2023-10-11T16:43:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: "redirect" 2023-10-11T16:43:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.478 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.478 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.478 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_error_status_code": { 2023-10-11T16:43:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_error_status_code", 2023-10-11T16:43:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "500", 2023-10-11T16:43:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "400", 2023-10-11T16:43:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "599", 2023-10-11T16:43:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "url_prefix": { 2023-10-11T16:43:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "url_prefix", 2023-10-11T16:43:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "devicehealth", 2023-10-11T16:43:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "enable_monitoring": { 2023-10-11T16:43:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enable_monitoring", 2023-10-11T16:43:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "monitor device health metrics", 2023-10-11T16:43:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "mark_out_threshold": { 2023-10-11T16:43:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mark_out_threshold", 2023-10-11T16:43:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "2419200", 2023-10-11T16:43:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-10-11T16:43:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": { 2023-10-11T16:43:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pool_name", 2023-10-11T16:43:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "device_health_metrics", 2023-10-11T16:43:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "name of pool in which to store device health metrics", 2023-10-11T16:43:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "retention_period": { 2023-10-11T16:43:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "retention_period", 2023-10-11T16:43:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15552000", 2023-10-11T16:43:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long to retain device health metrics", 2023-10-11T16:43:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "scrape_frequency": { 2023-10-11T16:43:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "scrape_frequency", 2023-10-11T16:43:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "86400", 2023-10-11T16:43:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to scrape device health metrics", 2023-10-11T16:43:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "self_heal": { 2023-10-11T16:43:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "self_heal", 2023-10-11T16:43:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-10-11T16:43:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.532 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:01.532 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:01.532 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.533 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.533 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.533 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:01.534 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.534 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.534 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.534 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to wake up and check device health", 2023-10-11T16:43:01.535 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.535 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.535 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.536 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_threshold": { 2023-10-11T16:43:01.536 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_threshold", 2023-10-11T16:43:01.536 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.537 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.537 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.537 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "7257600", 2023-10-11T16:43:01.538 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.538 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.538 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.538 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-10-11T16:43:01.539 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.539 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.539 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.539 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.540 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.541 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "diskprediction_local", 2023-10-11T16:43:01.541 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.541 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.541 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.542 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.542 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.542 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.543 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.543 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.543 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.543 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.544 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.544 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.544 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.544 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.545 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.545 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.545 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.546 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.546 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.546 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.546 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.547 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.547 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.547 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.548 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.548 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.548 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.548 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.549 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.549 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.549 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.549 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.550 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.550 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.550 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.551 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.551 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.551 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.552 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.552 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.552 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.553 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.553 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.553 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.553 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.554 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.554 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.554 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.554 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.555 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.555 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.555 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.556 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.556 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.556 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.556 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.557 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.557 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.557 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.558 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.558 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.558 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.558 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.559 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.559 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.559 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.560 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.560 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.560 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.561 INFO:teuthology.orchestra.run.smithi111.stdout: "predict_interval": { 2023-10-11T16:43:01.561 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "predict_interval", 2023-10-11T16:43:01.561 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.561 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.562 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.562 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "86400", 2023-10-11T16:43:01.562 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.562 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.563 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.563 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.563 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.564 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.564 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.564 INFO:teuthology.orchestra.run.smithi111.stdout: "predictor_model": { 2023-10-11T16:43:01.565 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "predictor_model", 2023-10-11T16:43:01.565 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.565 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.565 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.566 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "prophetstor", 2023-10-11T16:43:01.566 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.566 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.567 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.567 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.567 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.567 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.568 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.568 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:01.569 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:01.569 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.569 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.569 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.570 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:01.570 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.570 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.570 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.571 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.571 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.571 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.571 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.572 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.572 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.573 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "influx", 2023-10-11T16:43:01.573 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": false, 2023-10-11T16:43:01.573 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "influxdb python module not found", 2023-10-11T16:43:01.574 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.574 INFO:teuthology.orchestra.run.smithi111.stdout: "batch_size": { 2023-10-11T16:43:01.574 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "batch_size", 2023-10-11T16:43:01.575 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.575 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.575 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.575 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5000", 2023-10-11T16:43:01.576 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.576 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.576 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.576 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-10-11T16:43:01.577 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.577 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.577 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.578 INFO:teuthology.orchestra.run.smithi111.stdout: "database": { 2023-10-11T16:43:01.578 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "database", 2023-10-11T16:43:01.578 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.579 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.579 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.579 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "ceph", 2023-10-11T16:43:01.579 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.580 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.580 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.580 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-10-11T16:43:01.580 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.581 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.581 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.581 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": { 2023-10-11T16:43:01.582 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "hostname", 2023-10-11T16:43:01.582 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.582 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.583 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.583 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.583 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.583 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.584 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.584 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB server hostname", 2023-10-11T16:43:01.584 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.584 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.585 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.585 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:01.585 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:01.586 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.586 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.586 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.587 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "30", 2023-10-11T16:43:01.587 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "5", 2023-10-11T16:43:01.587 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.587 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.588 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-10-11T16:43:01.588 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.588 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.588 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.589 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.589 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.590 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.590 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.590 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.590 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.591 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.591 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.591 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.592 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.592 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.592 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.592 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.593 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.593 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.593 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.594 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.594 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.594 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.594 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.595 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.595 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.596 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.596 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.596 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.596 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.597 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.597 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.597 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.597 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.598 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.598 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.598 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.599 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.599 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.599 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.600 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.600 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.600 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.600 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.601 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.601 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.601 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.601 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.602 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.602 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.602 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.603 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.603 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.603 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.603 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.604 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.604 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.604 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.605 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.605 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.605 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.605 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.606 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.606 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.606 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.607 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.607 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.607 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.607 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.608 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.608 INFO:teuthology.orchestra.run.smithi111.stdout: "password": { 2023-10-11T16:43:01.608 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "password", 2023-10-11T16:43:01.609 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.609 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.609 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.609 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.610 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.610 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.610 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.611 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "password of InfluxDB server user", 2023-10-11T16:43:01.611 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.611 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.611 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.612 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.612 INFO:teuthology.orchestra.run.smithi111.stdout: "port": { 2023-10-11T16:43:01.612 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "port", 2023-10-11T16:43:01.612 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.613 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.613 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.613 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8086", 2023-10-11T16:43:01.614 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.614 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.614 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.614 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB server port", 2023-10-11T16:43:01.615 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.615 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.615 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.616 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl": { 2023-10-11T16:43:01.616 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl", 2023-10-11T16:43:01.616 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.616 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.617 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.617 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "false", 2023-10-11T16:43:01.617 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.617 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.618 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.618 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-10-11T16:43:01.618 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.619 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.619 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.619 INFO:teuthology.orchestra.run.smithi111.stdout: "threads": { 2023-10-11T16:43:01.620 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "threads", 2023-10-11T16:43:01.620 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.620 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.620 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.621 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:01.621 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:01.621 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "32", 2023-10-11T16:43:01.621 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.622 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-10-11T16:43:01.622 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.622 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.622 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.623 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.623 INFO:teuthology.orchestra.run.smithi111.stdout: "username": { 2023-10-11T16:43:01.623 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "username", 2023-10-11T16:43:01.623 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.624 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.624 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.624 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "username of InfluxDB server user", 2023-10-11T16:43:01.626 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.626 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.626 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.627 INFO:teuthology.orchestra.run.smithi111.stdout: "verify_ssl": { 2023-10-11T16:43:01.627 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "verify_ssl", 2023-10-11T16:43:01.627 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.627 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.628 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.628 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "true", 2023-10-11T16:43:01.628 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.629 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.629 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.629 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-10-11T16:43:01.629 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.630 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.630 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.630 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.630 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.631 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "insights", 2023-10-11T16:43:01.632 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.632 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.632 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.632 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.633 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.633 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.633 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.633 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.634 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.634 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.634 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.634 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.635 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.635 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.635 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.635 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.636 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.636 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.636 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.636 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.637 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.637 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.637 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.638 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.638 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.638 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.639 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.639 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.639 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.639 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.640 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.640 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.640 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.640 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.641 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.641 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.641 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.642 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.642 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.642 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.643 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.643 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.643 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.644 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.644 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.644 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.644 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.645 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.645 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.645 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.645 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.646 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.646 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.646 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.646 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.647 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.647 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.648 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.648 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.648 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.648 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.649 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.649 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.649 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.649 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.650 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.650 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.650 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.651 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.651 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.651 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.651 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.652 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.652 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "iostat", 2023-10-11T16:43:01.652 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.652 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.653 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.653 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.653 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.653 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.654 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.654 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.654 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.655 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.655 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.655 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.655 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.656 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.656 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.656 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.656 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.657 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.657 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.657 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.657 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.658 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.658 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.658 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.658 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.659 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.659 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.659 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.660 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.660 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.660 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.660 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.661 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.661 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.661 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.661 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.662 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.662 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.662 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.663 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.663 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.663 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.663 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.664 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.664 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.664 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.664 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.665 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.665 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.665 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.666 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.666 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.666 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.666 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.667 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.667 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.667 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.668 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.668 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.668 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.669 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.669 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.669 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.669 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.670 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.670 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.670 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.671 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.671 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.671 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.671 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.672 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.672 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "k8sevents", 2023-10-11T16:43:01.673 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.673 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.673 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.673 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_event_retention_days": { 2023-10-11T16:43:01.674 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ceph_event_retention_days", 2023-10-11T16:43:01.674 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.674 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.675 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.675 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "7", 2023-10-11T16:43:01.675 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.675 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.676 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.676 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Days to hold ceph event information within local cache", 2023-10-11T16:43:01.676 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.676 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.677 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.677 INFO:teuthology.orchestra.run.smithi111.stdout: "config_check_secs": { 2023-10-11T16:43:01.677 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_check_secs", 2023-10-11T16:43:01.678 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.678 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.678 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.679 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:01.679 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "10", 2023-10-11T16:43:01.679 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.679 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.680 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-10-11T16:43:01.680 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.680 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.681 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.681 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.681 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.682 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.682 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.682 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.682 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.683 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.683 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.683 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.683 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.684 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.684 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.684 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.684 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.685 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.687 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.688 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.689 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.689 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.689 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.689 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.690 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.690 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.690 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.691 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.691 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.691 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.692 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.692 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.692 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.692 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.693 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.693 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.693 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.694 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.694 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.694 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.695 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.695 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.695 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.695 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.696 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.696 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.697 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.697 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.697 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.697 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.698 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.698 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.698 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.698 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.699 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.699 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.699 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.700 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.700 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.700 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.701 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "localpool", 2023-10-11T16:43:01.701 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.701 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.702 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.702 INFO:teuthology.orchestra.run.smithi111.stdout: "failure_domain": { 2023-10-11T16:43:01.702 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "failure_domain", 2023-10-11T16:43:01.703 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.703 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.703 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.703 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "host", 2023-10-11T16:43:01.704 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.704 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.704 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.704 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "failure domain for any created local pool", 2023-10-11T16:43:01.705 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-10-11T16:43:01.705 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.705 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.706 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.706 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.706 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.707 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.707 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.707 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.708 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.708 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.708 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.708 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.709 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.709 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.709 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.709 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.710 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.710 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.710 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.710 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.711 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.711 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.712 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.712 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.712 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.712 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.713 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.713 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.713 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.713 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.714 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.714 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.714 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.715 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.715 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.715 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.715 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.716 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.716 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.716 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.716 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.717 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.717 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.717 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.718 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.718 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.718 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.718 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.719 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.719 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.719 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.719 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.720 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.720 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.720 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.721 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.721 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.721 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.721 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.722 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.722 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.722 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.723 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.723 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.723 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.723 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.724 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.724 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.724 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": { 2023-10-11T16:43:01.725 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "min_size", 2023-10-11T16:43:01.725 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.725 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.725 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.726 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.726 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.726 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.727 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.727 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default min_size for any created local pool", 2023-10-11T16:43:01.727 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-10-11T16:43:01.727 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.729 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.729 INFO:teuthology.orchestra.run.smithi111.stdout: "num_rep": { 2023-10-11T16:43:01.730 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "num_rep", 2023-10-11T16:43:01.730 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.730 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.730 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.731 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3", 2023-10-11T16:43:01.731 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.731 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.731 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.732 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default replica count for any created local pool", 2023-10-11T16:43:01.732 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.732 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.733 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.733 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": { 2023-10-11T16:43:01.733 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pg_num", 2023-10-11T16:43:01.734 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.734 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.734 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.734 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "128", 2023-10-11T16:43:01.735 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.735 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.735 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.736 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default pg_num for any created local pool", 2023-10-11T16:43:01.736 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.736 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.736 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.737 INFO:teuthology.orchestra.run.smithi111.stdout: "prefix": { 2023-10-11T16:43:01.737 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prefix", 2023-10-11T16:43:01.738 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.738 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.738 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.738 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.739 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.739 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.739 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.740 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "name prefix for any created local pool", 2023-10-11T16:43:01.740 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.740 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.740 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.741 INFO:teuthology.orchestra.run.smithi111.stdout: "subtree": { 2023-10-11T16:43:01.741 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "subtree", 2023-10-11T16:43:01.742 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.742 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.742 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.742 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "rack", 2023-10-11T16:43:01.743 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.743 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.743 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.743 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "CRUSH level for which to create a local pool", 2023-10-11T16:43:01.744 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-10-11T16:43:01.744 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.744 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.746 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mds_autoscaler", 2023-10-11T16:43:01.746 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.746 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.747 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.747 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.747 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.747 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.748 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.748 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.748 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.749 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.749 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.749 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.749 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.750 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.750 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.750 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.750 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.751 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.751 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.751 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.752 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.752 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.752 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.753 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.753 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.753 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.753 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.754 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.754 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.754 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.755 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.755 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.755 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.755 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.756 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.756 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.756 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.757 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.757 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.757 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.757 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.758 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.758 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.758 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.758 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.759 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.759 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.759 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.759 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.760 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.760 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.760 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.760 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.761 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.761 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.761 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.762 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.762 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.762 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.763 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.763 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.763 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.763 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.764 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.764 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.764 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.764 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.765 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.765 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.765 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.766 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.766 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mirroring", 2023-10-11T16:43:01.767 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.767 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.767 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.767 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.768 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.768 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.768 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.769 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.769 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.769 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.769 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.770 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.770 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.770 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.771 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.771 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.771 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.771 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.772 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.772 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.772 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.772 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.773 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.773 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.774 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.774 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.774 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.774 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.775 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.775 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.775 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.775 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.776 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.776 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.776 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.776 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.777 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.777 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.777 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.778 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.778 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.778 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.779 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.779 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.779 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.779 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.780 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.780 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.780 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.780 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.781 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.781 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.781 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.782 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.782 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.782 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.782 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.783 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.783 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.783 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.784 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.784 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.784 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.784 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.785 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.785 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.785 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.785 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.786 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.786 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.786 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.786 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.787 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.787 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "nfs", 2023-10-11T16:43:01.787 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.788 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.788 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.788 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.789 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.789 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.789 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.789 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.790 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.790 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.790 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.791 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.791 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.791 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.791 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.792 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.792 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.792 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.793 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.793 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.793 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.793 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.794 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.794 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.794 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.795 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.795 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.795 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.796 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.796 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.796 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.796 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.797 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.797 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.797 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.797 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.798 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.798 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.799 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.799 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.799 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.799 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.800 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.800 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.800 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.800 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.801 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.801 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.801 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.802 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.802 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.802 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.802 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.803 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.803 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.803 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.804 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.804 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.804 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.805 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.805 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.805 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.805 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.806 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.806 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.806 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.807 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.807 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.807 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.807 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.808 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.808 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "orchestrator", 2023-10-11T16:43:01.809 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.809 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.809 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.809 INFO:teuthology.orchestra.run.smithi111.stdout: "fail_fs": { 2023-10-11T16:43:01.810 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "fail_fs", 2023-10-11T16:43:01.810 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.810 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.810 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.811 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.811 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.811 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.812 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.812 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-10-11T16:43:01.812 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.812 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.813 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.813 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.813 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.814 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.814 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.814 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.815 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.815 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.815 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.815 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.816 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.816 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.816 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.816 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.817 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.817 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.817 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.817 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.818 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.818 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.818 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.819 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.819 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.819 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.820 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.820 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.820 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.820 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.821 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.821 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.821 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.822 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.822 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.822 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.823 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.823 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.823 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.823 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.824 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.824 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.824 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.824 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.825 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.825 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.825 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.825 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.826 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.826 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.826 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.827 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.827 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.827 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.827 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.828 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.828 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.828 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.829 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.829 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.829 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.829 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.830 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.830 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.830 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.831 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.831 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.831 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.831 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.832 INFO:teuthology.orchestra.run.smithi111.stdout: "orchestrator": { 2023-10-11T16:43:01.832 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "orchestrator", 2023-10-11T16:43:01.832 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.833 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.833 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.833 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.834 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.834 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.834 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.834 INFO:teuthology.orchestra.run.smithi111.stdout: "cephadm", 2023-10-11T16:43:01.835 INFO:teuthology.orchestra.run.smithi111.stdout: "rook", 2023-10-11T16:43:01.835 INFO:teuthology.orchestra.run.smithi111.stdout: "test_orchestrator" 2023-10-11T16:43:01.835 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.835 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Orchestrator backend", 2023-10-11T16:43:01.836 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.836 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.836 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.838 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "osd_perf_query", 2023-10-11T16:43:01.838 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.838 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.838 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.839 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.839 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.839 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.839 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.840 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.840 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.840 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.840 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.841 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.841 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.841 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.842 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.842 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.842 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.842 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.843 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.843 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.843 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.843 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.844 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.844 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.844 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.845 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.845 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.845 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.845 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.846 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.846 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.846 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.847 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.847 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.847 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.847 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.848 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.848 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.848 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.849 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.849 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.849 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.850 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.850 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.850 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.850 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.851 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.851 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.851 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.851 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.852 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.852 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.852 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.853 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.853 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.853 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.854 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.854 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.854 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.856 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.856 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.856 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.856 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.857 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.857 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.857 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.857 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.858 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "osd_support", 2023-10-11T16:43:01.858 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.859 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.859 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.859 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.859 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.860 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.860 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.860 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.860 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.861 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.861 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.861 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.862 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.862 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.862 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.862 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.863 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.863 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.863 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.863 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.864 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.864 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.864 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.865 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.865 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.865 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.865 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.866 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.866 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.866 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.867 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.867 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.867 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.867 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.868 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.868 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.868 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.869 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.869 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.869 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.870 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.870 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.870 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.870 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.871 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.871 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.871 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.872 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.872 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.872 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.872 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.873 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.873 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.873 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.873 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.874 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.874 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.874 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.875 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.875 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.875 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.875 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.876 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.876 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.876 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.877 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.877 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.877 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.877 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.878 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.878 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.879 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pg_autoscaler", 2023-10-11T16:43:01.879 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.879 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.879 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.880 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.880 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.880 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.880 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.881 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.881 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.881 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.881 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.882 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.882 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.882 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.883 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.883 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.883 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.883 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.884 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.884 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.884 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.884 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.885 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.885 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.886 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.886 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.886 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.886 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.887 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.887 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.887 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.887 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.888 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.888 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.888 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.889 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.889 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.889 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.890 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.890 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.890 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.891 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.891 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.891 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.891 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.892 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.892 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.892 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.893 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.893 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.893 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.893 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.894 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.894 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.894 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.894 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.895 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.895 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.896 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.896 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.896 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.896 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.897 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.897 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.897 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.897 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.898 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.898 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.898 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.899 INFO:teuthology.orchestra.run.smithi111.stdout: "noautoscale": { 2023-10-11T16:43:01.899 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "noautoscale", 2023-10-11T16:43:01.899 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.900 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.900 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.900 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.900 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.901 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.901 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.901 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "global autoscale flag", 2023-10-11T16:43:01.902 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-10-11T16:43:01.902 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.902 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.903 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:01.903 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:01.903 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.903 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.904 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.904 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:01.904 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.904 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.905 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.905 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.905 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.906 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.906 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.906 INFO:teuthology.orchestra.run.smithi111.stdout: "threshold": { 2023-10-11T16:43:01.907 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "threshold", 2023-10-11T16:43:01.907 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:01.907 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.907 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.908 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3.0", 2023-10-11T16:43:01.908 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1.0", 2023-10-11T16:43:01.908 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.908 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.909 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "scaling threshold", 2023-10-11T16:43:01.909 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-10-11T16:43:01.909 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.910 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.910 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.910 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.911 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "progress", 2023-10-11T16:43:01.911 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.911 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.912 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.912 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_pg_recovery_event": { 2023-10-11T16:43:01.912 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_pg_recovery_event", 2023-10-11T16:43:01.912 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.913 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.913 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.913 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.913 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.914 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.914 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.914 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow the module to show pg recovery progress", 2023-10-11T16:43:01.915 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.915 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.915 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.916 INFO:teuthology.orchestra.run.smithi111.stdout: "enabled": { 2023-10-11T16:43:01.916 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enabled", 2023-10-11T16:43:01.916 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.916 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.917 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.917 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.917 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.917 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.918 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.918 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.918 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.919 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.919 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.919 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.920 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.920 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.920 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.920 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.921 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.921 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.921 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.921 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.922 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.922 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.922 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.922 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.923 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.923 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.923 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.923 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.924 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.924 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.924 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.925 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.925 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.925 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.925 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.926 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.926 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.926 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.927 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.927 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.927 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.927 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.928 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.928 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.928 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.929 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.929 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.929 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.929 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.930 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.930 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.930 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.931 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.931 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.931 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.933 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.933 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.933 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.934 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.934 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.934 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.935 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.935 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.935 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.936 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.936 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.937 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.937 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.937 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.937 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.938 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.938 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.938 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.939 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.939 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.939 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.939 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.940 INFO:teuthology.orchestra.run.smithi111.stdout: "max_completed_events": { 2023-10-11T16:43:01.940 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_completed_events", 2023-10-11T16:43:01.940 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.941 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.941 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.941 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "50", 2023-10-11T16:43:01.942 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.942 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.942 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.942 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "number of past completed events to remember", 2023-10-11T16:43:01.943 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.943 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.943 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.944 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:01.944 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:01.944 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:01.945 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.945 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.945 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:01.945 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.946 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.946 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.946 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long the module is going to sleep", 2023-10-11T16:43:01.947 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.947 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.947 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.947 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.948 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:01.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:01.949 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus", 2023-10-11T16:43:01.949 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:01.949 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:01.949 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:01.950 INFO:teuthology.orchestra.run.smithi111.stdout: "cache": { 2023-10-11T16:43:01.950 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cache", 2023-10-11T16:43:01.950 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.951 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.951 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.951 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.951 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.952 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.952 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.952 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.952 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.953 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.953 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.953 INFO:teuthology.orchestra.run.smithi111.stdout: "exclude_perf_counters": { 2023-10-11T16:43:01.954 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "exclude_perf_counters", 2023-10-11T16:43:01.954 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.954 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.955 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.955 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:01.955 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.955 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.956 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.956 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-10-11T16:43:01.956 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-10-11T16:43:01.957 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.957 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.958 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:01.958 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:01.958 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.958 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.959 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.959 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.959 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.960 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.960 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.960 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.960 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.961 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.961 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.961 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.962 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.962 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.962 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.962 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.963 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.963 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.964 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:01.964 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:01.964 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.964 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.965 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.965 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.965 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.965 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.966 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.966 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.966 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.966 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.967 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.967 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:01.968 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:01.968 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.968 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.968 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.969 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:01.969 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.969 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.969 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:01.970 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:01.970 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:01.970 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:01.971 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:01.971 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:01.971 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:01.971 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:01.972 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.972 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.972 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.972 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.973 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:01.973 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:01.973 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:01.974 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.974 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.974 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:01.975 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.975 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.975 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.975 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.976 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.976 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.976 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.977 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_stats_pools": { 2023-10-11T16:43:01.977 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_stats_pools", 2023-10-11T16:43:01.977 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.978 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.978 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.978 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:01.978 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.979 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.979 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.979 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.980 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.980 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.980 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.981 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_stats_pools_refresh_interval": { 2023-10-11T16:43:01.981 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-10-11T16:43:01.981 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.981 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.982 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.982 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "300", 2023-10-11T16:43:01.982 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.983 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.983 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.983 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.983 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.984 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.984 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.984 INFO:teuthology.orchestra.run.smithi111.stdout: "scrape_interval": { 2023-10-11T16:43:01.985 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "scrape_interval", 2023-10-11T16:43:01.985 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:01.985 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.986 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.986 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15.0", 2023-10-11T16:43:01.986 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.986 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.987 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.987 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.987 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.988 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.988 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.988 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:01.989 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:01.989 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.989 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.989 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.990 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "::", 2023-10-11T16:43:01.990 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.990 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.990 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.991 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-10-11T16:43:01.991 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.991 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.992 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.992 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:01.992 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:01.993 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:01.993 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.993 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:01.994 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "9283", 2023-10-11T16:43:01.994 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.994 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.994 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.995 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-10-11T16:43:01.995 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.995 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.995 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:01.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:01.996 INFO:teuthology.orchestra.run.smithi111.stdout: "stale_cache_strategy": { 2023-10-11T16:43:01.996 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "stale_cache_strategy", 2023-10-11T16:43:01.997 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:01.997 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:01.997 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:01.997 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "log", 2023-10-11T16:43:01.998 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:01.998 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:01.998 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:01.998 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:01.999 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:01.999 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:01.999 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.000 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_behaviour": { 2023-10-11T16:43:02.000 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_behaviour", 2023-10-11T16:43:02.000 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.001 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.001 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.001 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "default", 2023-10-11T16:43:02.001 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.002 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.002 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.002 INFO:teuthology.orchestra.run.smithi111.stdout: "default", 2023-10-11T16:43:02.003 INFO:teuthology.orchestra.run.smithi111.stdout: "error" 2023-10-11T16:43:02.003 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.003 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.003 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.004 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.004 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.005 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_error_status_code": { 2023-10-11T16:43:02.005 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_error_status_code", 2023-10-11T16:43:02.005 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.005 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.006 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.006 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "500", 2023-10-11T16:43:02.006 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "400", 2023-10-11T16:43:02.006 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "599", 2023-10-11T16:43:02.007 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.007 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.007 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.008 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.008 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.008 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.009 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.009 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_support", 2023-10-11T16:43:02.010 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.010 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.010 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.010 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.011 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.011 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.011 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.012 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.012 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.012 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.012 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.013 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.013 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.013 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.013 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.014 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.014 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.014 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.015 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.015 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.015 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.015 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.016 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.016 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.016 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.017 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.017 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.017 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.018 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.018 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.018 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.018 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.019 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.019 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.019 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.019 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.020 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.020 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.021 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.021 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.021 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.021 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.022 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.022 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.022 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.023 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.023 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.023 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.023 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.024 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.024 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.024 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.024 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.025 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.025 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.025 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.026 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.026 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.026 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.027 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.027 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.027 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.027 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.028 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.028 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.028 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.029 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.029 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.029 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.030 INFO:teuthology.orchestra.run.smithi111.stdout: "max_concurrent_snap_create": { 2023-10-11T16:43:02.030 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_concurrent_snap_create", 2023-10-11T16:43:02.030 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.031 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.031 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.031 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:02.031 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.032 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.032 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.032 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.032 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.033 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.033 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.034 INFO:teuthology.orchestra.run.smithi111.stdout: "mirror_snapshot_schedule": { 2023-10-11T16:43:02.034 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mirror_snapshot_schedule", 2023-10-11T16:43:02.034 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.034 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.035 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.035 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.035 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.035 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.036 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.036 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.036 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.036 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.037 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.037 INFO:teuthology.orchestra.run.smithi111.stdout: "trash_purge_schedule": { 2023-10-11T16:43:02.038 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "trash_purge_schedule", 2023-10-11T16:43:02.038 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.038 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.038 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.039 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.039 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.039 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.040 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.040 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.040 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.040 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.041 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.041 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.041 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.042 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "restful", 2023-10-11T16:43:02.042 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.043 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.043 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.043 INFO:teuthology.orchestra.run.smithi111.stdout: "enable_auth": { 2023-10-11T16:43:02.043 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enable_auth", 2023-10-11T16:43:02.044 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.044 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.044 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.044 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.045 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.045 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.045 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.046 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.046 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.046 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.046 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.047 INFO:teuthology.orchestra.run.smithi111.stdout: "key_file": { 2023-10-11T16:43:02.047 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "key_file", 2023-10-11T16:43:02.047 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.048 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.048 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.048 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.049 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.049 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.049 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.049 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.050 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.050 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.050 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.051 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.051 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.051 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.052 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.052 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.052 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.052 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.053 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.053 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.053 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.053 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.054 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.054 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.054 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.055 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.055 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.055 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.055 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.056 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.056 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.056 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.057 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.057 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.057 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.058 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.058 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.058 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.058 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.059 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.059 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.059 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.060 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.060 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.060 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.061 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.061 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.061 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.062 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.062 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.062 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.063 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.063 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.063 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.064 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.064 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.064 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.064 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.065 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.065 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.065 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.066 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.066 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.066 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.067 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.067 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.067 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.068 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.068 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.068 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.068 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.069 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.069 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.069 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.069 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.070 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.070 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.071 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:02.071 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:02.071 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.071 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.072 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.072 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.072 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.072 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.073 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.073 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.073 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.074 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.074 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.074 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:02.075 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:02.075 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.075 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.075 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.076 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.076 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.076 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.077 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.077 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.077 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.077 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.079 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rgw", 2023-10-11T16:43:02.079 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.080 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.080 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.080 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.080 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.081 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.081 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.081 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.081 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.082 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.082 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.082 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.083 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.083 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.083 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.084 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.084 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.084 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.084 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.085 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.085 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.085 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.085 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.086 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.086 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.087 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.087 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.087 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.087 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.088 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.088 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.088 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.088 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.089 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.089 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.089 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.090 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.090 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.090 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.091 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.091 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.091 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.091 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.092 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.092 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.092 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.092 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.093 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.093 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.093 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.094 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.094 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.094 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.094 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.095 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.095 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.095 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.096 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.096 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.096 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.097 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.097 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.097 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.097 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.098 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.098 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.098 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.098 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.099 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.099 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.099 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.100 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rook", 2023-10-11T16:43:02.100 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.101 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.101 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.101 INFO:teuthology.orchestra.run.smithi111.stdout: "drive_group_interval": { 2023-10-11T16:43:02.101 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "drive_group_interval", 2023-10-11T16:43:02.102 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:02.102 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.102 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.103 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "300.0", 2023-10-11T16:43:02.103 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.103 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.103 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.104 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-10-11T16:43:02.104 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.104 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.104 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.105 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.105 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.105 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.106 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.106 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.106 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.107 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.107 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.107 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.107 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.108 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.108 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.108 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.108 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.109 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.109 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.109 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.110 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.110 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.110 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.111 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.111 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.111 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.111 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.112 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.112 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.112 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.112 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.113 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.113 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.113 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.114 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.114 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.114 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.115 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.115 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.115 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.116 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.116 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.116 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.116 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.117 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.117 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.117 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.117 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.118 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.118 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.118 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.119 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.119 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.119 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.119 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.120 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.120 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.120 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.121 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.121 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.121 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.122 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.122 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.122 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.122 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.123 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.123 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.123 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.123 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.124 INFO:teuthology.orchestra.run.smithi111.stdout: "storage_class": { 2023-10-11T16:43:02.124 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "storage_class", 2023-10-11T16:43:02.125 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.125 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.125 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.125 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "local", 2023-10-11T16:43:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.126 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-10-11T16:43:02.127 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.127 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.127 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.128 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.128 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "selftest", 2023-10-11T16:43:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.131 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.131 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.131 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.131 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.132 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.132 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.132 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.132 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.133 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.133 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.133 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.133 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.134 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.134 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.134 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.135 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.138 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.138 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.139 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.139 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.139 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.139 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.140 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.140 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.140 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.141 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.141 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.141 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.141 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.142 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.142 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.142 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.143 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.143 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.143 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.144 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.144 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.144 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.144 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.145 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.145 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.145 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.145 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.146 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.146 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.146 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.147 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.147 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.147 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.147 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.148 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.148 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.148 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.149 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.149 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.149 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.150 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.150 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.150 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.150 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.151 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.151 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.151 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.152 INFO:teuthology.orchestra.run.smithi111.stdout: "roption1": { 2023-10-11T16:43:02.152 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "roption1", 2023-10-11T16:43:02.152 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.153 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.153 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.153 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.153 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.154 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.154 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.154 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.154 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.155 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.155 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.156 INFO:teuthology.orchestra.run.smithi111.stdout: "roption2": { 2023-10-11T16:43:02.156 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "roption2", 2023-10-11T16:43:02.156 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.156 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.157 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.157 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "xyz", 2023-10-11T16:43:02.157 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.157 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.158 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.158 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.158 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.159 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.159 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.159 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption1": { 2023-10-11T16:43:02.160 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption1", 2023-10-11T16:43:02.160 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.160 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.160 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.161 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.161 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.161 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.161 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.162 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.162 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.162 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.163 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.163 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption2": { 2023-10-11T16:43:02.163 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption2", 2023-10-11T16:43:02.164 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.164 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.164 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.165 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.165 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.165 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.165 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.166 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.166 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.166 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.167 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.167 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption3": { 2023-10-11T16:43:02.167 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption3", 2023-10-11T16:43:02.168 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:02.168 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.168 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.168 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.169 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.169 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.169 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.169 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.170 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.170 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.170 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.171 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption4": { 2023-10-11T16:43:02.171 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption4", 2023-10-11T16:43:02.171 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.172 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.172 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.172 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.173 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.173 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.173 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.173 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.174 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.174 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.174 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.175 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption5": { 2023-10-11T16:43:02.175 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption5", 2023-10-11T16:43:02.175 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.176 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.176 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.176 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.176 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.177 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.177 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.177 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.177 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.178 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.178 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.179 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption6": { 2023-10-11T16:43:02.179 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption6", 2023-10-11T16:43:02.179 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.179 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.180 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.180 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.180 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.180 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.181 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.181 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.181 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.182 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.182 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.182 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption7": { 2023-10-11T16:43:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption7", 2023-10-11T16:43:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "42", 2023-10-11T16:43:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "testkey": { 2023-10-11T16:43:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testkey", 2023-10-11T16:43:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "testlkey": { 2023-10-11T16:43:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testlkey", 2023-10-11T16:43:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "testnewline": { 2023-10-11T16:43:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testnewline", 2023-10-11T16:43:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "snap_schedule", 2023-10-11T16:43:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_m_granularity": { 2023-10-11T16:43:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_m_granularity", 2023-10-11T16:43:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow minute scheduled snapshots", 2023-10-11T16:43:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "dump_on_update": { 2023-10-11T16:43:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "dump_on_update", 2023-10-11T16:43:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "dump database to debug log on update", 2023-10-11T16:43:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "stats", 2023-10-11T16:43:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "status", 2023-10-11T16:43:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.254 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.254 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.254 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.255 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.255 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.255 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.256 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.256 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.256 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.257 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.257 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.257 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.257 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.258 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.258 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.258 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.259 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.259 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.259 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.260 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.260 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.260 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.260 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.261 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.261 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.261 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.262 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.262 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.262 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.262 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.263 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.263 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.263 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.263 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.264 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.264 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.264 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.265 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.265 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.265 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.266 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.266 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.266 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.266 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.267 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.267 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.267 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.268 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.268 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.268 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.268 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.269 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.269 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.270 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "telegraf", 2023-10-11T16:43:02.270 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.270 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.270 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.271 INFO:teuthology.orchestra.run.smithi111.stdout: "address": { 2023-10-11T16:43:02.271 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "address", 2023-10-11T16:43:02.271 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.272 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.272 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.272 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-10-11T16:43:02.272 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.273 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.273 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.273 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.273 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.274 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.274 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.275 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:02.275 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:02.275 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.275 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.276 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.276 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15", 2023-10-11T16:43:02.276 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.276 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.277 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.277 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.277 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.277 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.278 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.278 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.279 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.279 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.279 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.280 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.280 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.280 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.280 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.281 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.281 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.281 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.281 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.282 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.282 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.282 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.283 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.283 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.283 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.283 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.284 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.284 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.284 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.285 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.285 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.285 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.286 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.286 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.286 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.286 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.287 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.287 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.287 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.287 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.288 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.288 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.289 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.289 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.289 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.289 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.290 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.290 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.290 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.290 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.291 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.291 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.291 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.292 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.292 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.292 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.292 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.293 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.293 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.293 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.294 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.294 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.294 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.295 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.295 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.295 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.295 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.296 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.296 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.296 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.296 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.297 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.298 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "telemetry", 2023-10-11T16:43:02.299 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.299 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.299 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.299 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_basic": { 2023-10-11T16:43:02.300 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_basic", 2023-10-11T16:43:02.300 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.300 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.301 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.301 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.301 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.301 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.302 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.302 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share basic cluster information (size, version)", 2023-10-11T16:43:02.302 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.302 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.303 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.303 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_crash": { 2023-10-11T16:43:02.304 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_crash", 2023-10-11T16:43:02.304 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.304 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.304 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.306 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-10-11T16:43:02.306 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.306 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.307 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.307 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_device": { 2023-10-11T16:43:02.307 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_device", 2023-10-11T16:43:02.308 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.308 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.308 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.308 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.309 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.309 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.309 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.310 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-10-11T16:43:02.310 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.310 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.310 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.311 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_ident": { 2023-10-11T16:43:02.311 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_ident", 2023-10-11T16:43:02.311 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.312 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.312 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.312 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.313 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.313 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.313 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.313 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-10-11T16:43:02.314 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.314 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.314 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.315 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_perf": { 2023-10-11T16:43:02.315 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_perf", 2023-10-11T16:43:02.315 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.316 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.316 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.316 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.316 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.317 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.317 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.317 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share various performance metrics of a cluster", 2023-10-11T16:43:02.317 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.318 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.318 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.319 INFO:teuthology.orchestra.run.smithi111.stdout: "contact": { 2023-10-11T16:43:02.319 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "contact", 2023-10-11T16:43:02.319 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.319 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.320 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.320 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.320 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.320 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.321 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.321 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.321 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.322 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.322 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.322 INFO:teuthology.orchestra.run.smithi111.stdout: "description": { 2023-10-11T16:43:02.323 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "description", 2023-10-11T16:43:02.323 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.323 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.323 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.324 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.324 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.324 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.325 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.325 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.325 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.325 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.326 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.326 INFO:teuthology.orchestra.run.smithi111.stdout: "device_url": { 2023-10-11T16:43:02.326 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_url", 2023-10-11T16:43:02.327 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.327 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.327 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.328 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-10-11T16:43:02.328 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.328 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.328 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.329 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.329 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.329 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.329 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.330 INFO:teuthology.orchestra.run.smithi111.stdout: "enabled": { 2023-10-11T16:43:02.330 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enabled", 2023-10-11T16:43:02.331 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.331 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.331 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.332 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.332 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.332 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.332 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.333 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.333 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.333 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.333 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.334 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:02.334 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:02.335 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.335 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.335 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.335 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "24", 2023-10-11T16:43:02.336 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "8", 2023-10-11T16:43:02.336 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.336 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.336 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.337 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.337 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.337 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.338 INFO:teuthology.orchestra.run.smithi111.stdout: "last_opt_revision": { 2023-10-11T16:43:02.338 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "last_opt_revision", 2023-10-11T16:43:02.338 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.339 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.339 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.339 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1", 2023-10-11T16:43:02.339 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.340 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.340 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.340 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.341 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.341 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.341 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.342 INFO:teuthology.orchestra.run.smithi111.stdout: "leaderboard": { 2023-10-11T16:43:02.342 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "leaderboard", 2023-10-11T16:43:02.342 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.342 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.343 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.343 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.343 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.344 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.344 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.344 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.344 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.345 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.345 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.345 INFO:teuthology.orchestra.run.smithi111.stdout: "leaderboard_description": { 2023-10-11T16:43:02.346 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "leaderboard_description", 2023-10-11T16:43:02.346 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.346 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.347 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.347 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.347 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.347 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.348 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.348 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.348 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.348 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.349 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.349 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.349 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.350 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.350 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.350 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.351 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.351 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.351 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.351 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.352 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.352 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.352 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.352 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.353 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.353 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.353 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.354 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.354 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.354 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.354 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.355 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.355 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.355 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.356 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.356 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.356 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.357 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.357 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.357 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.357 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.358 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.358 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.358 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.359 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.359 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.359 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.360 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.360 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.360 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.360 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.361 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.361 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.361 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.361 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.362 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.362 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.362 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.363 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.363 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.363 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.363 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.364 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.364 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.364 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.365 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.365 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.365 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.365 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.366 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.366 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.366 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.367 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.367 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.367 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.367 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "organization": { 2023-10-11T16:43:02.368 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "organization", 2023-10-11T16:43:02.369 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.369 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.369 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.370 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.370 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.370 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.370 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.371 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.371 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.371 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.371 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.372 INFO:teuthology.orchestra.run.smithi111.stdout: "proxy": { 2023-10-11T16:43:02.372 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "proxy", 2023-10-11T16:43:02.373 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.373 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.373 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.373 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.374 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.374 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.374 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.374 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.375 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.375 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.375 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.376 INFO:teuthology.orchestra.run.smithi111.stdout: "url": { 2023-10-11T16:43:02.376 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "url", 2023-10-11T16:43:02.376 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.377 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.377 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.377 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-10-11T16:43:02.378 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.378 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.378 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.378 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.379 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.379 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.379 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.380 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.380 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.381 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "test_orchestrator", 2023-10-11T16:43:02.381 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.381 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.381 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.382 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.382 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.382 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.383 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.383 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.383 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.383 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.384 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.384 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.384 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.385 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.385 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.385 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.385 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.386 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.386 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.386 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.386 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.387 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.387 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.388 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.388 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.388 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.388 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.389 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.389 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.389 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.389 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.390 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.390 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.390 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.391 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.391 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.391 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.392 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.392 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.392 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.392 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.393 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.393 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.393 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.394 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.394 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.394 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.394 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.395 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.395 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.395 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.395 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.396 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.396 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.396 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.397 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.397 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.397 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.398 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.398 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.398 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.398 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.399 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.399 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.399 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.400 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.400 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.400 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.400 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.401 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.401 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.402 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "volumes", 2023-10-11T16:43:02.402 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.402 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.403 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.403 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.403 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.403 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.404 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.404 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.404 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.404 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.406 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.406 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.406 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.406 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.407 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.407 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.407 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.408 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.408 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.408 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.409 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.409 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.409 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.409 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.410 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.410 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.410 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.411 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.411 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.411 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.411 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.412 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.412 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.413 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.413 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.413 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.414 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.414 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.414 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.414 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.415 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.415 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.415 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.416 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.416 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.416 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.416 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.417 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.417 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.417 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.418 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.418 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.418 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.419 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.419 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.419 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.420 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.420 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.420 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.420 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.421 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.421 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.421 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.422 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.422 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.422 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.423 INFO:teuthology.orchestra.run.smithi111.stdout: "max_concurrent_clones": { 2023-10-11T16:43:02.423 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_concurrent_clones", 2023-10-11T16:43:02.423 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.423 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.424 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.424 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "4", 2023-10-11T16:43:02.424 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.425 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.425 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.425 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Number of asynchronous cloner threads", 2023-10-11T16:43:02.425 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.426 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.426 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.427 INFO:teuthology.orchestra.run.smithi111.stdout: "snapshot_clone_delay": { 2023-10-11T16:43:02.427 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "snapshot_clone_delay", 2023-10-11T16:43:02.427 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.427 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.428 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.429 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:02.430 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.430 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.430 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.431 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-10-11T16:43:02.431 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.431 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.431 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.433 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix", 2023-10-11T16:43:02.433 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.433 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.433 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.434 INFO:teuthology.orchestra.run.smithi111.stdout: "discovery_interval": { 2023-10-11T16:43:02.434 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "discovery_interval", 2023-10-11T16:43:02.434 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:02.435 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.435 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.435 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "100", 2023-10-11T16:43:02.435 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.436 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.436 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.436 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.436 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.437 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.437 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.438 INFO:teuthology.orchestra.run.smithi111.stdout: "identifier": { 2023-10-11T16:43:02.438 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "identifier", 2023-10-11T16:43:02.438 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.439 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.439 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.439 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.439 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.440 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.440 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.440 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.441 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.441 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.441 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.442 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:02.442 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:02.442 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.442 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.443 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.443 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:02.443 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.444 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.444 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.444 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.444 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.445 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.446 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.446 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.446 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.446 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.447 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.447 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.447 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.448 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.448 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.448 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.448 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.449 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.449 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.449 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.449 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.450 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.450 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.450 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.451 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.451 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.451 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.452 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.452 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.452 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.452 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.453 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.453 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.453 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.454 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.454 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.454 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.454 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.455 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.455 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.455 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.456 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.456 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.456 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.457 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.457 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.457 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.457 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.458 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.458 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.458 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.458 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.459 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.459 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.459 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.459 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.460 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.460 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.460 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.461 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.461 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.461 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.461 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.462 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.462 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.462 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.463 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.463 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.463 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.463 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.463 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.464 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_host": { 2023-10-11T16:43:02.464 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_host", 2023-10-11T16:43:02.465 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.465 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.465 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.465 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.466 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.466 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.466 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.466 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.467 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.467 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.467 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.468 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_port": { 2023-10-11T16:43:02.468 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_port", 2023-10-11T16:43:02.468 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.468 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.469 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.469 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10051", 2023-10-11T16:43:02.469 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.469 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.470 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.470 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.470 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.471 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.471 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.471 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_sender": { 2023-10-11T16:43:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_sender", 2023-10-11T16:43:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.472 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.473 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-10-11T16:43:02.473 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.473 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.473 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.474 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.474 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.474 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.474 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.475 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.475 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.475 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.475 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:02.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.476 INFO:teuthology.orchestra.run.smithi111.stdout: "gid": 25558, 2023-10-11T16:43:02.477 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "z", 2023-10-11T16:43:02.477 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_features": 4540138322906710015, 2023-10-11T16:43:02.477 INFO:teuthology.orchestra.run.smithi111.stdout: "available_modules": [ 2023-10-11T16:43:02.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.478 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alerts", 2023-10-11T16:43:02.478 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.478 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.479 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.479 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:02.479 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:02.479 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.480 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.480 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.480 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:02.480 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.481 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.481 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.481 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How frequently to reexamine health status", 2023-10-11T16:43:02.482 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.482 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.482 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.483 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.483 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.483 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.483 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.484 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.484 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.484 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.485 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.485 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.485 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.485 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.486 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.486 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.486 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.486 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.487 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.487 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.487 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.488 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.488 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.488 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.489 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.489 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.489 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.489 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.490 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.490 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.490 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.491 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.491 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.491 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.492 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.492 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.492 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.493 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.493 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.493 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.493 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.494 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.494 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.494 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.494 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.495 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.495 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.495 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.496 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.496 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.496 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.496 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.497 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.497 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.497 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.497 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.498 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.498 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.499 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.499 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.499 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.499 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.500 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.500 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.500 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.500 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.501 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.501 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.501 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.502 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_destination": { 2023-10-11T16:43:02.502 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_destination", 2023-10-11T16:43:02.502 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.503 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.503 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.503 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.503 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.504 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.504 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.504 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Email address to send alerts to", 2023-10-11T16:43:02.504 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.505 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.505 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.505 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_from_name": { 2023-10-11T16:43:02.506 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_from_name", 2023-10-11T16:43:02.506 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.506 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.507 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.507 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "Ceph", 2023-10-11T16:43:02.507 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.507 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.508 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.508 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Email From: name", 2023-10-11T16:43:02.508 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.508 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.509 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.509 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_host": { 2023-10-11T16:43:02.509 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_host", 2023-10-11T16:43:02.510 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.510 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.510 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.510 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.511 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.511 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.511 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.512 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP server", 2023-10-11T16:43:02.512 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.512 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.512 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.513 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_password": { 2023-10-11T16:43:02.513 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_password", 2023-10-11T16:43:02.513 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.514 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.514 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.514 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.515 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.515 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.515 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.515 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Password to authenticate with", 2023-10-11T16:43:02.516 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.516 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.516 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.517 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_port": { 2023-10-11T16:43:02.517 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_port", 2023-10-11T16:43:02.517 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.517 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.518 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.518 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "465", 2023-10-11T16:43:02.518 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.518 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP port", 2023-10-11T16:43:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_sender": { 2023-10-11T16:43:02.520 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_sender", 2023-10-11T16:43:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.521 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.522 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.522 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.522 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.522 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.523 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP envelope sender", 2023-10-11T16:43:02.523 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.523 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.524 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.524 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_ssl": { 2023-10-11T16:43:02.524 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_ssl", 2023-10-11T16:43:02.525 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.525 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.525 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.525 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.526 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.526 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.526 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.527 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use SSL to connect to SMTP server", 2023-10-11T16:43:02.527 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.527 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.527 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.528 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_user": { 2023-10-11T16:43:02.528 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_user", 2023-10-11T16:43:02.528 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.529 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.529 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.529 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.530 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.530 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.530 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.531 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "User to authenticate as", 2023-10-11T16:43:02.531 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.531 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.531 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.533 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "balancer", 2023-10-11T16:43:02.533 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.533 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.533 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.534 INFO:teuthology.orchestra.run.smithi111.stdout: "active": { 2023-10-11T16:43:02.534 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "active", 2023-10-11T16:43:02.534 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.534 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.535 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.535 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.535 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.535 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.536 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.536 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "automatically balance PGs across cluster", 2023-10-11T16:43:02.536 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.536 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.537 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.537 INFO:teuthology.orchestra.run.smithi111.stdout: "begin_time": { 2023-10-11T16:43:02.538 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "begin_time", 2023-10-11T16:43:02.538 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.538 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.538 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.539 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0000", 2023-10-11T16:43:02.539 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.539 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.539 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.540 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "beginning time of day to automatically balance", 2023-10-11T16:43:02.540 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-11T16:43:02.540 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.540 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.541 INFO:teuthology.orchestra.run.smithi111.stdout: "begin_weekday": { 2023-10-11T16:43:02.541 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "begin_weekday", 2023-10-11T16:43:02.542 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:02.542 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.542 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.543 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:02.543 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0", 2023-10-11T16:43:02.543 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "6", 2023-10-11T16:43:02.543 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.544 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-10-11T16:43:02.544 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-11T16:43:02.544 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.545 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.545 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_max_iterations": { 2023-10-11T16:43:02.545 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_max_iterations", 2023-10-11T16:43:02.546 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:02.546 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.546 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.547 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "25", 2023-10-11T16:43:02.547 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:02.547 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "250", 2023-10-11T16:43:02.547 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.548 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-10-11T16:43:02.548 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.548 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.549 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.549 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_metrics": { 2023-10-11T16:43:02.549 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_metrics", 2023-10-11T16:43:02.550 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.550 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.550 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.551 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "pgs,objects,bytes", 2023-10-11T16:43:02.551 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.551 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.551 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.552 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-10-11T16:43:02.552 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-10-11T16:43:02.552 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.553 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.553 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_step": { 2023-10-11T16:43:02.553 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_step", 2023-10-11T16:43:02.554 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:02.554 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.554 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.555 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0.5", 2023-10-11T16:43:02.555 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0.001", 2023-10-11T16:43:02.555 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "0.999", 2023-10-11T16:43:02.555 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.556 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "aggressiveness of optimization", 2023-10-11T16:43:02.556 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-10-11T16:43:02.556 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.557 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.557 INFO:teuthology.orchestra.run.smithi111.stdout: "end_time": { 2023-10-11T16:43:02.557 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "end_time", 2023-10-11T16:43:02.558 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.558 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.558 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.559 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "2359", 2023-10-11T16:43:02.559 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.559 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.559 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.560 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "ending time of day to automatically balance", 2023-10-11T16:43:02.560 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-11T16:43:02.560 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.561 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.561 INFO:teuthology.orchestra.run.smithi111.stdout: "end_weekday": { 2023-10-11T16:43:02.561 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "end_weekday", 2023-10-11T16:43:02.562 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:02.562 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.562 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.562 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:02.563 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0", 2023-10-11T16:43:02.563 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "6", 2023-10-11T16:43:02.563 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.563 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-10-11T16:43:02.564 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-11T16:43:02.564 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.564 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.565 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.565 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.565 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.566 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.566 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.566 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.567 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.567 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.567 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.567 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.568 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.568 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.568 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.568 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.569 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.569 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.569 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.570 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.570 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.570 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.571 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.571 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.571 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.571 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.572 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.572 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.572 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.572 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.573 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.573 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.573 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.573 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.574 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.574 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.575 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.575 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.575 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.575 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.576 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.576 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.576 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.576 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.577 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.577 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.577 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.577 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.578 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.578 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.578 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.579 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.579 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.579 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.579 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.580 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.580 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.580 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.581 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.581 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.581 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.582 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.582 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.582 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.582 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.583 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.583 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.583 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.584 INFO:teuthology.orchestra.run.smithi111.stdout: "min_score": { 2023-10-11T16:43:02.584 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "min_score", 2023-10-11T16:43:02.584 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:02.585 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.585 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.585 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:02.585 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.586 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.586 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.586 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-10-11T16:43:02.586 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.587 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.587 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.588 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": { 2023-10-11T16:43:02.588 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mode", 2023-10-11T16:43:02.588 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.588 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.589 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.589 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "upmap", 2023-10-11T16:43:02.589 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.589 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.590 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.590 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-compat", 2023-10-11T16:43:02.590 INFO:teuthology.orchestra.run.smithi111.stdout: "none", 2023-10-11T16:43:02.591 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap" 2023-10-11T16:43:02.591 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.591 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Balancer mode", 2023-10-11T16:43:02.591 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.592 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.592 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.592 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_ids": { 2023-10-11T16:43:02.593 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pool_ids", 2023-10-11T16:43:02.593 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.593 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.594 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.594 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.594 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.594 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.595 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.595 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-10-11T16:43:02.595 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.595 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.596 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.596 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:02.597 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:02.597 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.597 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.597 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.598 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:02.598 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.598 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.598 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.599 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-10-11T16:43:02.599 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.599 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.599 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.600 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap_max_deviation": { 2023-10-11T16:43:02.600 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "upmap_max_deviation", 2023-10-11T16:43:02.601 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.601 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.601 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.602 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:02.602 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:02.602 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.602 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.603 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "deviation below which no optimization is attempted", 2023-10-11T16:43:02.603 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-10-11T16:43:02.603 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.604 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.604 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap_max_optimizations": { 2023-10-11T16:43:02.604 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "upmap_max_optimizations", 2023-10-11T16:43:02.605 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:02.605 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.605 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.605 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:02.606 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.606 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.606 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.607 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-10-11T16:43:02.607 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.607 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.607 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.608 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.608 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.609 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephadm", 2023-10-11T16:43:02.609 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.609 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.609 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.610 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_down_multiplier": { 2023-10-11T16:43:02.610 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_down_multiplier", 2023-10-11T16:43:02.610 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:02.611 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.611 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.611 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3.0", 2023-10-11T16:43:02.611 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.612 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.612 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.612 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-10-11T16:43:02.613 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.613 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.613 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.614 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_refresh_rate": { 2023-10-11T16:43:02.614 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_refresh_rate", 2023-10-11T16:43:02.614 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.614 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.615 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.615 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "20", 2023-10-11T16:43:02.615 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.615 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.616 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.616 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-10-11T16:43:02.616 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.617 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.617 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.617 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_starting_port": { 2023-10-11T16:43:02.618 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_starting_port", 2023-10-11T16:43:02.618 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.618 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.619 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.619 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "4721", 2023-10-11T16:43:02.619 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.619 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.620 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.620 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-10-11T16:43:02.620 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.620 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.621 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.621 INFO:teuthology.orchestra.run.smithi111.stdout: "alertmanager_web_password": { 2023-10-11T16:43:02.621 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alertmanager_web_password", 2023-10-11T16:43:02.622 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.622 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.622 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.622 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:02.623 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.623 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.623 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.624 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Alertmanager web password", 2023-10-11T16:43:02.624 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.624 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.624 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.625 INFO:teuthology.orchestra.run.smithi111.stdout: "alertmanager_web_user": { 2023-10-11T16:43:02.625 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alertmanager_web_user", 2023-10-11T16:43:02.625 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.626 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.626 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.626 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:02.627 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.627 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.627 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.627 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Alertmanager web user", 2023-10-11T16:43:02.628 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.628 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.628 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.628 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.629 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_ptrace": { 2023-10-11T16:43:02.629 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_ptrace", 2023-10-11T16:43:02.629 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.629 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.630 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.630 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.630 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.631 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.631 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.631 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-10-11T16:43:02.631 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-10-11T16:43:02.632 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.632 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.632 INFO:teuthology.orchestra.run.smithi111.stdout: "autotune_interval": { 2023-10-11T16:43:02.633 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "autotune_interval", 2023-10-11T16:43:02.633 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.633 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.633 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.634 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:02.634 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.634 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.634 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.635 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to autotune daemon memory", 2023-10-11T16:43:02.635 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.635 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.636 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.636 INFO:teuthology.orchestra.run.smithi111.stdout: "autotune_memory_target_ratio": { 2023-10-11T16:43:02.636 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "autotune_memory_target_ratio", 2023-10-11T16:43:02.637 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:02.637 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.637 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.638 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0.7", 2023-10-11T16:43:02.638 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.638 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.638 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.639 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-10-11T16:43:02.639 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.639 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.639 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.640 INFO:teuthology.orchestra.run.smithi111.stdout: "cgroups_split": { 2023-10-11T16:43:02.640 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cgroups_split", 2023-10-11T16:43:02.641 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.641 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.641 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.641 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.642 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.642 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.642 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.643 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-10-11T16:43:02.643 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.643 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.643 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.644 INFO:teuthology.orchestra.run.smithi111.stdout: "config_checks_enabled": { 2023-10-11T16:43:02.644 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_checks_enabled", 2023-10-11T16:43:02.644 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.645 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.645 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.645 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.646 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.646 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.646 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.646 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-10-11T16:43:02.647 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.647 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.647 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.648 INFO:teuthology.orchestra.run.smithi111.stdout: "config_dashboard": { 2023-10-11T16:43:02.648 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_dashboard", 2023-10-11T16:43:02.648 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.649 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.649 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.649 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.649 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.650 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.650 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.650 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-10-11T16:43:02.651 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.651 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.651 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.651 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.652 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_alertmanager": { 2023-10-11T16:43:02.652 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_alertmanager", 2023-10-11T16:43:02.652 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.652 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.653 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.653 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-10-11T16:43:02.653 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.653 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.654 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.654 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:02.654 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.654 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.655 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.655 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.655 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_base": { 2023-10-11T16:43:02.655 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_base", 2023-10-11T16:43:02.656 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.656 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.656 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.657 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/ceph", 2023-10-11T16:43:02.657 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.657 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.657 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.658 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Container image name, without the tag", 2023-10-11T16:43:02.658 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.658 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.658 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.659 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.659 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_elasticsearch": { 2023-10-11T16:43:02.659 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_elasticsearch", 2023-10-11T16:43:02.660 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.660 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.660 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.660 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-10-11T16:43:02.661 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.661 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.661 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.661 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "elasticsearch container image", 2023-10-11T16:43:02.662 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.662 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.662 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.663 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_grafana": { 2023-10-11T16:43:02.663 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_grafana", 2023-10-11T16:43:02.663 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.663 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.664 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.664 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-10-11T16:43:02.664 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.664 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.665 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.665 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:02.665 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.665 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.666 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.666 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_haproxy": { 2023-10-11T16:43:02.667 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_haproxy", 2023-10-11T16:43:02.667 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.667 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.667 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.668 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-10-11T16:43:02.668 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.668 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.668 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.669 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "HAproxy container image", 2023-10-11T16:43:02.669 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.669 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.670 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.670 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_agent": { 2023-10-11T16:43:02.670 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_agent", 2023-10-11T16:43:02.671 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.671 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.671 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.672 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-10-11T16:43:02.672 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.672 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.672 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.673 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger agent container image", 2023-10-11T16:43:02.673 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.673 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.673 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.674 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_collector": { 2023-10-11T16:43:02.674 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_collector", 2023-10-11T16:43:02.674 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.675 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.675 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.675 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-10-11T16:43:02.675 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.676 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.676 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.676 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger collector container image", 2023-10-11T16:43:02.676 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.677 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.677 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.677 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_query": { 2023-10-11T16:43:02.678 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_query", 2023-10-11T16:43:02.678 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.678 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.679 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.679 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-10-11T16:43:02.679 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.679 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.680 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.680 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger query container image", 2023-10-11T16:43:02.680 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.680 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.681 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.681 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_keepalived": { 2023-10-11T16:43:02.682 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_keepalived", 2023-10-11T16:43:02.682 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.682 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.682 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.683 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-10-11T16:43:02.683 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.683 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.684 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.684 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Keepalived container image", 2023-10-11T16:43:02.684 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.684 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.685 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.685 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_loki": { 2023-10-11T16:43:02.685 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_loki", 2023-10-11T16:43:02.686 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.686 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.686 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.687 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-10-11T16:43:02.687 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.687 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.687 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.688 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Loki container image", 2023-10-11T16:43:02.688 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.688 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.688 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.689 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_node_exporter": { 2023-10-11T16:43:02.689 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_node_exporter", 2023-10-11T16:43:02.690 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.690 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.690 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.690 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-10-11T16:43:02.691 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.691 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.691 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.692 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:02.692 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.692 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.692 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.693 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_prometheus": { 2023-10-11T16:43:02.693 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_prometheus", 2023-10-11T16:43:02.693 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.694 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.694 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.694 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-10-11T16:43:02.695 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.695 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.695 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.695 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:02.696 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.696 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.696 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.697 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_promtail": { 2023-10-11T16:43:02.697 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_promtail", 2023-10-11T16:43:02.697 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.698 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.698 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.698 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-10-11T16:43:02.698 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.699 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.699 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.699 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Promtail container image", 2023-10-11T16:43:02.699 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.700 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.700 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.700 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_snmp_gateway": { 2023-10-11T16:43:02.701 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_snmp_gateway", 2023-10-11T16:43:02.701 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.701 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.702 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.702 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-10-11T16:43:02.702 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.702 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.703 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.703 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SNMP Gateway container image", 2023-10-11T16:43:02.703 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.703 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.704 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.704 INFO:teuthology.orchestra.run.smithi111.stdout: "container_init": { 2023-10-11T16:43:02.704 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_init", 2023-10-11T16:43:02.705 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.705 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.705 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.706 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.706 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.706 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.706 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.707 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Run podman/docker with `--init`", 2023-10-11T16:43:02.707 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.707 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.707 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.708 INFO:teuthology.orchestra.run.smithi111.stdout: "daemon_cache_timeout": { 2023-10-11T16:43:02.708 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "daemon_cache_timeout", 2023-10-11T16:43:02.709 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.709 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.709 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.709 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:02.710 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.710 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.710 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.711 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-10-11T16:43:02.711 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.711 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.711 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.712 INFO:teuthology.orchestra.run.smithi111.stdout: "default_cephadm_command_timeout": { 2023-10-11T16:43:02.712 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "default_cephadm_command_timeout", 2023-10-11T16:43:02.712 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.713 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.713 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.713 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "900", 2023-10-11T16:43:02.713 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.714 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.714 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.714 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-10-11T16:43:02.715 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.715 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.715 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.716 INFO:teuthology.orchestra.run.smithi111.stdout: "default_registry": { 2023-10-11T16:43:02.716 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "default_registry", 2023-10-11T16:43:02.716 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.716 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.717 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.717 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io", 2023-10-11T16:43:02.717 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.717 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.718 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.718 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-10-11T16:43:02.718 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.719 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.719 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.719 INFO:teuthology.orchestra.run.smithi111.stdout: "device_cache_timeout": { 2023-10-11T16:43:02.720 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_cache_timeout", 2023-10-11T16:43:02.720 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.720 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.720 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.721 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1800", 2023-10-11T16:43:02.721 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.721 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.721 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.722 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache device inventory", 2023-10-11T16:43:02.722 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.722 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.723 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.723 INFO:teuthology.orchestra.run.smithi111.stdout: "device_enhanced_scan": { 2023-10-11T16:43:02.723 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_enhanced_scan", 2023-10-11T16:43:02.724 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.724 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.724 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.724 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.725 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.725 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.725 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.725 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use libstoragemgmt during device scans", 2023-10-11T16:43:02.726 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.726 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.726 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.727 INFO:teuthology.orchestra.run.smithi111.stdout: "facts_cache_timeout": { 2023-10-11T16:43:02.727 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "facts_cache_timeout", 2023-10-11T16:43:02.727 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.728 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.728 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.728 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:02.728 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.729 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.729 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.729 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache host facts data", 2023-10-11T16:43:02.730 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.730 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.730 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.731 INFO:teuthology.orchestra.run.smithi111.stdout: "host_check_interval": { 2023-10-11T16:43:02.731 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "host_check_interval", 2023-10-11T16:43:02.731 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.732 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.732 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.732 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:02.732 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.733 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.733 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.733 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to perform a host check", 2023-10-11T16:43:02.733 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.734 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.734 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.735 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.735 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.735 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.735 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.736 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.736 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.736 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.736 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.737 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.737 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.737 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.737 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.738 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.738 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.738 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.739 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.739 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.739 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.739 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.740 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.740 INFO:teuthology.orchestra.run.smithi111.stdout: "log_refresh_metadata": { 2023-10-11T16:43:02.740 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_refresh_metadata", 2023-10-11T16:43:02.741 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.741 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.741 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.741 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.742 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.742 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.742 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.743 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-10-11T16:43:02.743 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.743 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.743 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.744 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.744 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.744 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.745 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.745 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.745 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.745 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.746 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.746 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.746 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-10-11T16:43:02.746 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.747 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.747 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.747 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.748 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.748 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.748 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.749 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.749 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.749 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.749 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.750 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.750 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.750 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.750 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.751 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.751 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.751 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.752 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.752 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.752 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.752 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.753 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.753 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.753 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.755 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.755 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.755 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.755 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.757 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.757 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.758 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.758 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.758 INFO:teuthology.orchestra.run.smithi111.stdout: "manage_etc_ceph_ceph_conf": { 2023-10-11T16:43:02.759 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-10-11T16:43:02.759 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.759 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.759 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.760 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.760 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.760 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.761 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.761 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-10-11T16:43:02.761 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.761 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.762 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.762 INFO:teuthology.orchestra.run.smithi111.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-10-11T16:43:02.762 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-10-11T16:43:02.763 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.763 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.763 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.764 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "*", 2023-10-11T16:43:02.764 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.764 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.764 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.765 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-10-11T16:43:02.765 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.765 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.765 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.766 INFO:teuthology.orchestra.run.smithi111.stdout: "max_count_per_host": { 2023-10-11T16:43:02.766 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_count_per_host", 2023-10-11T16:43:02.766 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.767 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.767 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.767 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:02.767 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.768 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.768 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.768 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "max number of daemons per service per host", 2023-10-11T16:43:02.769 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.769 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.769 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.770 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd_draining_count": { 2023-10-11T16:43:02.770 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_osd_draining_count", 2023-10-11T16:43:02.770 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.770 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.771 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.771 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:02.771 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.771 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.772 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.772 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-10-11T16:43:02.772 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.773 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.773 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.773 INFO:teuthology.orchestra.run.smithi111.stdout: "migration_current": { 2023-10-11T16:43:02.774 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "migration_current", 2023-10-11T16:43:02.774 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.774 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.774 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.775 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.775 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.775 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.775 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.776 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "internal - do not modify", 2023-10-11T16:43:02.776 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.776 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.776 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.777 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": { 2023-10-11T16:43:02.777 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mode", 2023-10-11T16:43:02.777 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.778 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.778 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.778 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "root", 2023-10-11T16:43:02.779 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.779 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.779 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.779 INFO:teuthology.orchestra.run.smithi111.stdout: "cephadm-package", 2023-10-11T16:43:02.780 INFO:teuthology.orchestra.run.smithi111.stdout: "root" 2023-10-11T16:43:02.780 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.780 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "mode for remote execution of cephadm", 2023-10-11T16:43:02.780 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.781 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.781 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.781 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_alerts_path": { 2023-10-11T16:43:02.782 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_alerts_path", 2023-10-11T16:43:02.782 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.782 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.783 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.783 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-10-11T16:43:02.783 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.783 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.784 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.784 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-10-11T16:43:02.784 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.784 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.785 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.785 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_web_password": { 2023-10-11T16:43:02.785 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_web_password", 2023-10-11T16:43:02.786 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.786 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.786 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.786 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:02.787 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.787 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.787 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.787 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus web password", 2023-10-11T16:43:02.788 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.788 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.788 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.789 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_web_user": { 2023-10-11T16:43:02.789 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_web_user", 2023-10-11T16:43:02.789 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.789 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.790 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.790 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:02.790 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.791 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.791 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.791 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus web user", 2023-10-11T16:43:02.791 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.792 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.792 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.792 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_insecure": { 2023-10-11T16:43:02.793 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_insecure", 2023-10-11T16:43:02.793 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.793 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.793 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.794 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.794 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.794 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.794 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.795 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-10-11T16:43:02.795 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.795 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.795 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.796 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_password": { 2023-10-11T16:43:02.796 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_password", 2023-10-11T16:43:02.796 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.797 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.797 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.797 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.797 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.798 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.798 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.798 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-10-11T16:43:02.799 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.799 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.799 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.800 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_url": { 2023-10-11T16:43:02.800 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_url", 2023-10-11T16:43:02.800 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.800 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.801 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.801 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.801 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.801 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.802 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.802 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-10-11T16:43:02.802 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.803 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.803 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.803 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_username": { 2023-10-11T16:43:02.804 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_username", 2023-10-11T16:43:02.804 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.804 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.804 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.805 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.805 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.805 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.805 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.806 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-10-11T16:43:02.806 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.806 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: "secure_monitoring_stack": { 2023-10-11T16:43:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "secure_monitoring_stack", 2023-10-11T16:43:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-10-11T16:43:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.811 INFO:teuthology.orchestra.run.smithi111.stdout: "service_discovery_port": { 2023-10-11T16:43:02.811 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "service_discovery_port", 2023-10-11T16:43:02.811 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.812 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.812 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.812 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8765", 2023-10-11T16:43:02.812 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.813 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.813 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.813 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "cephadm service discovery port", 2023-10-11T16:43:02.813 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.814 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.814 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.815 INFO:teuthology.orchestra.run.smithi111.stdout: "ssh_config_file": { 2023-10-11T16:43:02.815 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssh_config_file", 2023-10-11T16:43:02.815 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.815 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.816 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.816 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.816 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.816 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.817 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.817 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-10-11T16:43:02.817 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.817 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.818 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.818 INFO:teuthology.orchestra.run.smithi111.stdout: "use_agent": { 2023-10-11T16:43:02.818 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "use_agent", 2023-10-11T16:43:02.819 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.819 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.819 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.820 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.820 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.820 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.820 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.821 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-10-11T16:43:02.821 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.821 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.821 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.822 INFO:teuthology.orchestra.run.smithi111.stdout: "use_repo_digest": { 2023-10-11T16:43:02.822 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "use_repo_digest", 2023-10-11T16:43:02.823 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.823 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.823 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.823 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.824 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.824 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.824 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.825 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-10-11T16:43:02.825 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.825 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.825 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.826 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_failed_host_check": { 2023-10-11T16:43:02.826 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_failed_host_check", 2023-10-11T16:43:02.826 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.827 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.827 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.827 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.827 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.828 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.828 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.828 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if the host check fails", 2023-10-11T16:43:02.828 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.829 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.829 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.829 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_stray_daemons": { 2023-10-11T16:43:02.830 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_stray_daemons", 2023-10-11T16:43:02.830 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.830 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.831 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.831 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.831 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.831 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.832 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.832 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-10-11T16:43:02.832 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.832 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.833 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.833 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_stray_hosts": { 2023-10-11T16:43:02.834 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_stray_hosts", 2023-10-11T16:43:02.834 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.834 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.834 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.835 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.835 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.835 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.835 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.836 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-10-11T16:43:02.836 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.836 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.836 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.838 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crash", 2023-10-11T16:43:02.838 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.838 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.839 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.839 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:02.839 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:02.839 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.840 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.840 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.840 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.841 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.841 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.841 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.841 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.842 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.842 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.842 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.842 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.843 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.843 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.843 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.844 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.844 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.844 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.845 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:02.845 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:02.845 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.845 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.846 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.846 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.846 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.846 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.847 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.847 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.847 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.847 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.848 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.848 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:02.849 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:02.849 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.849 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.849 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.850 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:02.850 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.850 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.850 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:02.851 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:02.851 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:02.851 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:02.852 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:02.852 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:02.852 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:02.852 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:02.853 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.853 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.853 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.853 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.854 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:02.854 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:02.854 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.855 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.855 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.855 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.855 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.856 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.856 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.856 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.857 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.857 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.857 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.858 INFO:teuthology.orchestra.run.smithi111.stdout: "retain_interval": { 2023-10-11T16:43:02.858 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "retain_interval", 2023-10-11T16:43:02.858 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.858 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.859 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.859 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "31536000", 2023-10-11T16:43:02.859 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.859 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.860 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.860 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long to retain crashes before pruning them", 2023-10-11T16:43:02.860 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.861 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.861 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.861 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_recent_interval": { 2023-10-11T16:43:02.862 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_recent_interval", 2023-10-11T16:43:02.862 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:02.862 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.862 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1209600", 2023-10-11T16:43:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.863 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "time interval in which to warn about recent crashes", 2023-10-11T16:43:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.864 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.865 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.865 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.865 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:02.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:02.866 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "dashboard", 2023-10-11T16:43:02.866 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:02.866 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:02.867 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:02.867 INFO:teuthology.orchestra.run.smithi111.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-10-11T16:43:02.867 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-10-11T16:43:02.867 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.868 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.868 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.868 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:02.869 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.869 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.869 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.869 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.870 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.870 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.870 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.871 INFO:teuthology.orchestra.run.smithi111.stdout: "ALERTMANAGER_API_HOST": { 2023-10-11T16:43:02.871 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ALERTMANAGER_API_HOST", 2023-10-11T16:43:02.871 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.872 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.872 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.872 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.872 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.873 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.873 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.873 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.873 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.874 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.874 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.874 INFO:teuthology.orchestra.run.smithi111.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-10-11T16:43:02.875 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-10-11T16:43:02.875 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.875 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.875 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.876 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.876 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.876 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.876 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.877 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.877 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.877 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.878 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.878 INFO:teuthology.orchestra.run.smithi111.stdout: "AUDIT_API_ENABLED": { 2023-10-11T16:43:02.878 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "AUDIT_API_ENABLED", 2023-10-11T16:43:02.879 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.879 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.879 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.879 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.880 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.880 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.880 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.881 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.881 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.881 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.881 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.882 INFO:teuthology.orchestra.run.smithi111.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-10-11T16:43:02.882 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-10-11T16:43:02.882 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.883 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.883 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.883 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.884 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.884 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.884 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.884 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.885 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.885 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.885 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.886 INFO:teuthology.orchestra.run.smithi111.stdout: "ENABLE_BROWSABLE_API": { 2023-10-11T16:43:02.886 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ENABLE_BROWSABLE_API", 2023-10-11T16:43:02.886 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.887 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.887 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.887 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.887 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.888 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.888 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.888 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.888 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.889 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.889 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.889 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-10-11T16:43:02.890 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-10-11T16:43:02.890 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.890 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.891 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.891 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.891 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.891 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.892 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.892 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.892 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.893 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.893 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.893 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-10-11T16:43:02.894 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-10-11T16:43:02.894 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.894 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.894 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.895 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.895 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.895 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.895 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.896 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.896 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.896 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.896 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.897 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-10-11T16:43:02.897 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-10-11T16:43:02.897 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.898 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.898 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.898 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.899 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.899 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.899 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.899 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.900 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.900 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.900 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.901 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-10-11T16:43:02.901 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-10-11T16:43:02.901 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.901 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.902 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.902 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.902 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.903 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.903 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.903 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.903 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.904 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.904 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.904 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_NFS": { 2023-10-11T16:43:02.905 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-10-11T16:43:02.905 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.905 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.906 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.906 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.906 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.906 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.907 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.907 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.907 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.907 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.908 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.908 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_RBD": { 2023-10-11T16:43:02.908 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-10-11T16:43:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.911 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.911 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.911 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.912 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.912 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_RGW": { 2023-10-11T16:43:02.912 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-10-11T16:43:02.913 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.913 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.913 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.913 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.914 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.914 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.914 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.914 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.915 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.915 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.915 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.916 INFO:teuthology.orchestra.run.smithi111.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-10-11T16:43:02.916 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-10-11T16:43:02.916 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.917 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.917 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.917 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.917 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.918 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.918 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.918 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.918 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.919 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.919 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.920 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_PASSWORD": { 2023-10-11T16:43:02.920 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_PASSWORD", 2023-10-11T16:43:02.920 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.920 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.921 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.921 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:02.921 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.922 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.922 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.922 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.922 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.923 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.923 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.923 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-10-11T16:43:02.924 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-10-11T16:43:02.924 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.924 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.925 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.925 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.925 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.925 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.926 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.926 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.926 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.926 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.927 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.927 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_URL": { 2023-10-11T16:43:02.928 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_URL", 2023-10-11T16:43:02.928 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.928 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.928 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.929 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.929 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.929 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.929 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.930 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.930 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.930 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.931 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.931 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_USERNAME": { 2023-10-11T16:43:02.931 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_USERNAME", 2023-10-11T16:43:02.932 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.932 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.932 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.932 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:02.933 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.933 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.933 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.934 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.934 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.934 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.934 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.935 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-10-11T16:43:02.935 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-10-11T16:43:02.936 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.936 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.936 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.936 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.937 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.937 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.937 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.937 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.938 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.938 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.938 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.939 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-10-11T16:43:02.939 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-10-11T16:43:02.939 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.940 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.940 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.940 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.940 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.941 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.941 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.941 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.942 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.942 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.942 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.943 INFO:teuthology.orchestra.run.smithi111.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-10-11T16:43:02.943 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-10-11T16:43:02.943 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.943 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.944 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.944 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.944 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.944 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.945 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.945 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.945 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.946 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.946 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.946 INFO:teuthology.orchestra.run.smithi111.stdout: "ISSUE_TRACKER_API_KEY": { 2023-10-11T16:43:02.947 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-10-11T16:43:02.947 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.947 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.947 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.948 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.948 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.948 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.949 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.949 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.949 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.949 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.950 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.950 INFO:teuthology.orchestra.run.smithi111.stdout: "PROMETHEUS_API_HOST": { 2023-10-11T16:43:02.950 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PROMETHEUS_API_HOST", 2023-10-11T16:43:02.951 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.951 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.951 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.952 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:02.952 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.952 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.952 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.953 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.953 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.953 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.953 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.954 INFO:teuthology.orchestra.run.smithi111.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-10-11T16:43:02.954 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-10-11T16:43:02.955 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.955 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.955 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.955 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.956 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.956 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.956 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.956 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.957 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.957 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.957 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.958 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-10-11T16:43:02.958 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-10-11T16:43:02.958 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.959 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.959 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.959 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.959 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.960 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.960 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.960 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.960 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.961 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.961 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.961 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-10-11T16:43:02.962 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-10-11T16:43:02.962 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.962 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.963 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.963 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.963 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.963 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.965 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.965 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.965 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.966 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.966 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.967 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-10-11T16:43:02.967 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-10-11T16:43:02.967 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.967 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.968 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.968 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.968 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.968 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.969 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.969 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.969 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.969 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.970 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.970 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-10-11T16:43:02.971 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-10-11T16:43:02.971 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.971 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.971 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.972 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.972 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.972 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.973 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.973 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.973 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.973 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.974 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.974 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-10-11T16:43:02.974 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-10-11T16:43:02.975 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.975 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.975 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.976 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.976 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.976 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.976 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.977 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.977 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.977 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.977 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.978 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-10-11T16:43:02.978 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-10-11T16:43:02.979 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.979 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.979 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.979 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.980 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.980 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.980 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.980 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.981 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.981 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.981 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.982 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-10-11T16:43:02.982 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-10-11T16:43:02.982 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.983 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.983 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.983 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:02.983 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.984 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.984 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.984 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.984 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.985 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.985 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.985 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_ENABLED": { 2023-10-11T16:43:02.986 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_ENABLED", 2023-10-11T16:43:02.986 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:02.986 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.987 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.987 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:02.987 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.987 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.988 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.988 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.988 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.988 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.989 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.989 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-10-11T16:43:02.990 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-10-11T16:43:02.990 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:02.990 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.990 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.991 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-10-11T16:43:02.991 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.991 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.991 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.992 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.992 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.992 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.992 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.993 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-10-11T16:43:02.993 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-10-11T16:43:02.994 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.994 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.994 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.994 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:02.995 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.995 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.995 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.995 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:02.996 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:02.996 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:02.996 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:02.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:02.997 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-10-11T16:43:02.997 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-10-11T16:43:02.997 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:02.998 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:02.998 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:02.998 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8", 2023-10-11T16:43:02.998 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:02.999 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:02.999 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:02.999 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.000 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.000 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.000 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.001 INFO:teuthology.orchestra.run.smithi111.stdout: "REST_REQUESTS_TIMEOUT": { 2023-10-11T16:43:03.001 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-10-11T16:43:03.001 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.001 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.002 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.002 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "45", 2023-10-11T16:43:03.002 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.003 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.003 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.003 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.003 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.004 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.004 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.004 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_ACCESS_KEY": { 2023-10-11T16:43:03.005 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_ACCESS_KEY", 2023-10-11T16:43:03.005 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.005 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.005 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.006 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.006 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.006 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.006 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.007 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.007 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.007 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.008 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.008 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-10-11T16:43:03.008 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-10-11T16:43:03.009 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.009 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.009 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.009 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:03.010 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.010 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.010 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.010 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.011 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.011 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.011 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.012 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.012 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_SECRET_KEY": { 2023-10-11T16:43:03.012 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_SECRET_KEY", 2023-10-11T16:43:03.012 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.013 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.013 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.013 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.013 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.014 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.014 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.014 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.014 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.015 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.015 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.015 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_SSL_VERIFY": { 2023-10-11T16:43:03.016 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_SSL_VERIFY", 2023-10-11T16:43:03.016 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.016 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.016 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.017 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.017 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.017 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.018 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.018 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.018 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.019 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.019 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.019 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-10-11T16:43:03.020 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-10-11T16:43:03.020 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.020 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.020 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.021 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:03.021 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.021 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.022 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.022 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.022 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.022 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.023 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.023 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-10-11T16:43:03.023 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-10-11T16:43:03.024 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.024 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.024 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.025 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:03.025 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.025 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.025 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.026 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.026 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.026 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.027 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.027 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-10-11T16:43:03.027 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-10-11T16:43:03.028 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.028 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.028 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.029 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:03.029 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.029 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.029 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.030 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.030 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.030 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.030 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.031 INFO:teuthology.orchestra.run.smithi111.stdout: "cross_origin_url": { 2023-10-11T16:43:03.031 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cross_origin_url", 2023-10-11T16:43:03.032 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.032 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.032 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.032 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.033 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.033 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.033 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.033 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.034 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.034 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.034 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.035 INFO:teuthology.orchestra.run.smithi111.stdout: "crt_file": { 2023-10-11T16:43:03.035 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crt_file", 2023-10-11T16:43:03.035 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.036 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.036 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.036 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.036 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.037 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.037 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.037 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.037 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.038 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.038 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.039 INFO:teuthology.orchestra.run.smithi111.stdout: "debug": { 2023-10-11T16:43:03.039 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "debug", 2023-10-11T16:43:03.039 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.039 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.040 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.040 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.040 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.040 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.041 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.041 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable/disable debug options", 2023-10-11T16:43:03.041 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.041 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.042 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.042 INFO:teuthology.orchestra.run.smithi111.stdout: "jwt_token_ttl": { 2023-10-11T16:43:03.043 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "jwt_token_ttl", 2023-10-11T16:43:03.043 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.043 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.043 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.044 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "28800", 2023-10-11T16:43:03.044 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.044 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.044 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.045 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.045 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.045 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.045 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.046 INFO:teuthology.orchestra.run.smithi111.stdout: "key_file": { 2023-10-11T16:43:03.046 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "key_file", 2023-10-11T16:43:03.046 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.047 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.047 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.047 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.048 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.048 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.048 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.048 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.049 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.049 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.049 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.050 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.050 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.050 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.050 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.051 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.051 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.051 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.051 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.052 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.052 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.052 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.053 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.053 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.053 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.053 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.054 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.054 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.054 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.054 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.055 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.055 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.055 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.056 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.056 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.056 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.056 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.057 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.057 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.057 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.057 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.058 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.058 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.058 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.059 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.059 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.059 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.060 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.060 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.060 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.060 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.061 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.061 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.061 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.061 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.062 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.062 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.062 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.062 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.063 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.063 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.063 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.063 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.064 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.064 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.064 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.065 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.065 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.065 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.065 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.066 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.066 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.066 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.067 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.067 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.067 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.067 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.068 INFO:teuthology.orchestra.run.smithi111.stdout: "motd": { 2023-10-11T16:43:03.068 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "motd", 2023-10-11T16:43:03.068 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.069 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.069 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.069 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.069 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.070 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.070 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.070 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "The message of the day", 2023-10-11T16:43:03.071 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.071 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.071 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.071 INFO:teuthology.orchestra.run.smithi111.stdout: "redirect_resolve_ip_addr": { 2023-10-11T16:43:03.072 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "redirect_resolve_ip_addr", 2023-10-11T16:43:03.072 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.072 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.073 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.073 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.073 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.073 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.074 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.074 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.074 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.075 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.075 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.075 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:03.076 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:03.076 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.076 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.076 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.077 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "::", 2023-10-11T16:43:03.077 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.077 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.077 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.078 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.078 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.078 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.079 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.079 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:03.079 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:03.080 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.080 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.080 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.081 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8080", 2023-10-11T16:43:03.081 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.081 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.081 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.082 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.082 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.082 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.082 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.083 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl": { 2023-10-11T16:43:03.083 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl", 2023-10-11T16:43:03.083 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.084 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.084 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.084 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.085 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.085 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.085 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.085 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.086 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.086 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.086 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.087 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl_server_port": { 2023-10-11T16:43:03.087 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl_server_port", 2023-10-11T16:43:03.087 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.088 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.088 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.088 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8443", 2023-10-11T16:43:03.088 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.089 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.089 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.089 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.089 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.090 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.090 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.091 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_behaviour": { 2023-10-11T16:43:03.091 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_behaviour", 2023-10-11T16:43:03.091 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.091 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.092 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.092 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "redirect", 2023-10-11T16:43:03.092 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.092 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.093 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.093 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.093 INFO:teuthology.orchestra.run.smithi111.stdout: "redirect" 2023-10-11T16:43:03.093 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.094 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.094 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.094 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.094 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.095 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_error_status_code": { 2023-10-11T16:43:03.095 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_error_status_code", 2023-10-11T16:43:03.095 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.096 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.096 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.096 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "500", 2023-10-11T16:43:03.096 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "400", 2023-10-11T16:43:03.097 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "599", 2023-10-11T16:43:03.097 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.097 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.097 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.098 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.098 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.099 INFO:teuthology.orchestra.run.smithi111.stdout: "url_prefix": { 2023-10-11T16:43:03.099 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "url_prefix", 2023-10-11T16:43:03.099 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.099 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.100 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.100 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.100 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.100 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.101 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.101 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.101 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.101 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.102 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.102 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.102 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.103 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "devicehealth", 2023-10-11T16:43:03.104 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.104 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.104 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.104 INFO:teuthology.orchestra.run.smithi111.stdout: "enable_monitoring": { 2023-10-11T16:43:03.105 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enable_monitoring", 2023-10-11T16:43:03.105 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.105 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.106 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.106 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.106 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.107 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.107 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.107 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "monitor device health metrics", 2023-10-11T16:43:03.108 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.108 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.108 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.109 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.109 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.110 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.110 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.110 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.111 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.111 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.111 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.111 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.112 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.112 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.112 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.113 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.113 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.113 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.113 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.114 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.114 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.114 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.115 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.115 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.116 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.116 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.116 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.116 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.117 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.117 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.117 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.118 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.118 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.118 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.119 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.119 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.119 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.120 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.120 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.120 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.121 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.121 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.121 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.122 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.122 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.122 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.123 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.123 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.123 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.124 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.124 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.124 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.124 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.125 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.125 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.125 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.126 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.126 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.126 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.127 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.127 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.127 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.127 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.128 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.128 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.128 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.128 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.129 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.129 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.130 INFO:teuthology.orchestra.run.smithi111.stdout: "mark_out_threshold": { 2023-10-11T16:43:03.130 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mark_out_threshold", 2023-10-11T16:43:03.130 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.130 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.131 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.131 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "2419200", 2023-10-11T16:43:03.131 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.131 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.132 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.132 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-10-11T16:43:03.132 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.132 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.133 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.133 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": { 2023-10-11T16:43:03.133 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pool_name", 2023-10-11T16:43:03.134 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.134 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.134 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.135 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "device_health_metrics", 2023-10-11T16:43:03.135 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.135 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.135 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.136 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "name of pool in which to store device health metrics", 2023-10-11T16:43:03.136 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.136 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.136 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.137 INFO:teuthology.orchestra.run.smithi111.stdout: "retention_period": { 2023-10-11T16:43:03.137 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "retention_period", 2023-10-11T16:43:03.137 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.138 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.138 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.138 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15552000", 2023-10-11T16:43:03.139 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.139 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.139 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.139 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long to retain device health metrics", 2023-10-11T16:43:03.140 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.140 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.140 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.141 INFO:teuthology.orchestra.run.smithi111.stdout: "scrape_frequency": { 2023-10-11T16:43:03.141 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "scrape_frequency", 2023-10-11T16:43:03.141 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.142 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.142 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.142 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "86400", 2023-10-11T16:43:03.142 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.143 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.143 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.143 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to scrape device health metrics", 2023-10-11T16:43:03.144 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.144 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.144 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.145 INFO:teuthology.orchestra.run.smithi111.stdout: "self_heal": { 2023-10-11T16:43:03.145 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "self_heal", 2023-10-11T16:43:03.145 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.145 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.146 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.146 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.146 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.147 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.147 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.147 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-10-11T16:43:03.147 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.148 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.148 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.148 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:03.149 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:03.149 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.149 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.149 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.150 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:03.150 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.150 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.151 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.151 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to wake up and check device health", 2023-10-11T16:43:03.151 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.151 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.152 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.152 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_threshold": { 2023-10-11T16:43:03.152 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_threshold", 2023-10-11T16:43:03.153 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.153 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.153 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.153 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "7257600", 2023-10-11T16:43:03.154 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.154 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.154 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.155 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-10-11T16:43:03.155 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.155 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.155 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.156 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.156 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.156 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.157 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "diskprediction_local", 2023-10-11T16:43:03.157 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.157 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.157 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.158 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.158 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.158 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.159 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.159 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.159 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.159 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.160 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.160 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.160 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.160 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.161 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.161 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.161 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.161 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.162 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.162 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.162 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.163 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.163 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.163 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.164 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.164 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.164 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.164 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.165 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.165 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.165 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.165 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.166 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.166 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.166 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.167 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.169 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.170 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.170 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.170 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.170 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.171 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.171 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.171 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.171 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.172 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.172 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.172 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.172 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.173 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.173 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.173 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.174 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.174 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.174 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.174 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.175 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.175 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.175 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.176 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.176 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.176 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.176 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.177 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.177 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.177 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.177 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.178 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.178 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.179 INFO:teuthology.orchestra.run.smithi111.stdout: "predict_interval": { 2023-10-11T16:43:03.179 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "predict_interval", 2023-10-11T16:43:03.179 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.179 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.180 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.180 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "86400", 2023-10-11T16:43:03.180 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.180 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.181 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.181 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.181 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.181 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.182 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.182 INFO:teuthology.orchestra.run.smithi111.stdout: "predictor_model": { 2023-10-11T16:43:03.182 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "predictor_model", 2023-10-11T16:43:03.183 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.183 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.183 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.184 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "prophetstor", 2023-10-11T16:43:03.184 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.184 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.184 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.185 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.185 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.185 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.185 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.186 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:03.186 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:03.187 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.187 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.187 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.187 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:03.188 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.188 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.188 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.188 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.189 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.189 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.189 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.190 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.190 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.191 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "influx", 2023-10-11T16:43:03.191 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": false, 2023-10-11T16:43:03.191 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "influxdb python module not found", 2023-10-11T16:43:03.191 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.192 INFO:teuthology.orchestra.run.smithi111.stdout: "batch_size": { 2023-10-11T16:43:03.192 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "batch_size", 2023-10-11T16:43:03.192 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.193 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.193 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.193 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5000", 2023-10-11T16:43:03.193 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.194 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.194 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.194 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-10-11T16:43:03.194 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.195 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.195 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.196 INFO:teuthology.orchestra.run.smithi111.stdout: "database": { 2023-10-11T16:43:03.196 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "database", 2023-10-11T16:43:03.196 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.196 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.197 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.197 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "ceph", 2023-10-11T16:43:03.197 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.197 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.198 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.198 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-10-11T16:43:03.198 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.199 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.199 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.199 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": { 2023-10-11T16:43:03.200 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "hostname", 2023-10-11T16:43:03.200 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.200 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.200 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.200 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.201 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.201 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.201 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.201 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB server hostname", 2023-10-11T16:43:03.202 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.202 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.202 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:03.203 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "30", 2023-10-11T16:43:03.204 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "5", 2023-10-11T16:43:03.205 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.205 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.205 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-10-11T16:43:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.207 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.207 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.207 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.207 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.208 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.208 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.208 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.208 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.209 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.209 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.209 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.209 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.210 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.210 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.210 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.211 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.211 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.211 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.211 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.212 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.213 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.214 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.215 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.216 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.217 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.218 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.219 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.220 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.220 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.220 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.220 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.221 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.221 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.221 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.222 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.222 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.222 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.222 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.223 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.223 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.223 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.223 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.224 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.224 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.224 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.225 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.225 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.225 INFO:teuthology.orchestra.run.smithi111.stdout: "password": { 2023-10-11T16:43:03.226 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "password", 2023-10-11T16:43:03.226 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.226 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.226 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.227 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.227 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.227 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.228 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.228 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "password of InfluxDB server user", 2023-10-11T16:43:03.228 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.228 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.229 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.229 INFO:teuthology.orchestra.run.smithi111.stdout: "port": { 2023-10-11T16:43:03.230 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "port", 2023-10-11T16:43:03.230 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.230 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.230 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.231 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8086", 2023-10-11T16:43:03.231 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.231 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.231 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.232 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB server port", 2023-10-11T16:43:03.232 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.232 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.232 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.233 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl": { 2023-10-11T16:43:03.233 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl", 2023-10-11T16:43:03.233 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.234 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.234 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.234 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "false", 2023-10-11T16:43:03.235 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.235 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.235 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.235 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-10-11T16:43:03.236 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.236 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.236 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.237 INFO:teuthology.orchestra.run.smithi111.stdout: "threads": { 2023-10-11T16:43:03.237 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "threads", 2023-10-11T16:43:03.237 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.238 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.238 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.238 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:03.238 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:03.239 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "32", 2023-10-11T16:43:03.239 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.239 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-10-11T16:43:03.240 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.240 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.240 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.241 INFO:teuthology.orchestra.run.smithi111.stdout: "username": { 2023-10-11T16:43:03.241 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "username", 2023-10-11T16:43:03.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.241 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.242 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.242 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.242 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.242 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.243 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.243 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "username of InfluxDB server user", 2023-10-11T16:43:03.243 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.244 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.244 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.244 INFO:teuthology.orchestra.run.smithi111.stdout: "verify_ssl": { 2023-10-11T16:43:03.245 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "verify_ssl", 2023-10-11T16:43:03.245 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.245 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.245 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.246 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "true", 2023-10-11T16:43:03.246 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.246 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.246 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.247 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-10-11T16:43:03.247 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.247 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.247 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.248 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.248 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.249 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "insights", 2023-10-11T16:43:03.249 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.249 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.250 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.250 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.250 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.250 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.251 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.251 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.251 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.251 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.252 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.252 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.252 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.252 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.253 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.253 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.253 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.254 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.254 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.254 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.254 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.255 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.255 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.255 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.256 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.256 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.256 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.256 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.258 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.258 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.258 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.258 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.259 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.259 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.259 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.260 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.260 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.260 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.260 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.261 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.261 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.261 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.261 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.262 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.262 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.262 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.263 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.263 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.263 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.263 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.264 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.264 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.264 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.265 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.265 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.265 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.265 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.266 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.266 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.266 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.267 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.267 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.267 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.267 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.268 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.269 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "iostat", 2023-10-11T16:43:03.269 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.270 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.270 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.270 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.270 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.271 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.271 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.271 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.272 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.272 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.272 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.272 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.273 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.273 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.273 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.274 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.274 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.274 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.274 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.275 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.275 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.275 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.275 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.276 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.276 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.276 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.277 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.277 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.277 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.277 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.278 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.278 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.278 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.279 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.279 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.279 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.280 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.280 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.280 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.281 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.281 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.281 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.281 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.282 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.282 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.282 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.282 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.283 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.283 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.283 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.283 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.284 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.284 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.284 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.284 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.285 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.285 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.285 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.286 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.286 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.286 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.287 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.287 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.287 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.287 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.288 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.288 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.288 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.288 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.289 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.289 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.290 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "k8sevents", 2023-10-11T16:43:03.290 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.290 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.291 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.291 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_event_retention_days": { 2023-10-11T16:43:03.291 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ceph_event_retention_days", 2023-10-11T16:43:03.291 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.292 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.292 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.292 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "7", 2023-10-11T16:43:03.292 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.293 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.293 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.293 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Days to hold ceph event information within local cache", 2023-10-11T16:43:03.293 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.294 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.294 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.295 INFO:teuthology.orchestra.run.smithi111.stdout: "config_check_secs": { 2023-10-11T16:43:03.295 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_check_secs", 2023-10-11T16:43:03.295 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.295 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.296 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.296 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:03.296 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "10", 2023-10-11T16:43:03.296 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.297 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.297 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-10-11T16:43:03.297 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.297 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.298 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.298 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.299 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.299 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.299 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.299 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.300 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.300 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.300 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.300 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.301 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.301 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.301 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.301 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.302 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.302 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.302 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.302 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.303 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.303 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.303 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.304 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.304 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.304 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.304 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.305 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.305 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.305 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.305 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.306 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.306 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.306 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.306 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.307 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.307 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.308 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.308 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.308 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.308 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.309 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.309 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.309 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.309 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.310 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.310 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.310 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.311 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.311 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.311 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.311 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.312 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.312 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.312 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.312 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.313 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.313 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.313 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.314 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.314 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.314 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.315 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.315 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.315 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.315 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.316 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.316 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.316 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.316 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.317 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.317 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "localpool", 2023-10-11T16:43:03.318 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.318 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.318 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.319 INFO:teuthology.orchestra.run.smithi111.stdout: "failure_domain": { 2023-10-11T16:43:03.319 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "failure_domain", 2023-10-11T16:43:03.319 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.319 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.320 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.320 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "host", 2023-10-11T16:43:03.320 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.320 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.321 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.321 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "failure domain for any created local pool", 2023-10-11T16:43:03.321 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-10-11T16:43:03.321 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.322 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.322 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.323 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.323 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.323 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.323 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.324 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.324 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.324 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.324 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.325 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.325 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.325 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.325 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.326 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.326 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.326 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.326 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.327 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.327 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.327 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.328 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.328 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.328 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.329 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.329 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.329 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.329 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.330 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.330 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.330 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.331 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.331 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.331 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.332 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.333 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.333 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.333 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.334 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.334 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.334 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.334 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.335 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.335 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.335 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.335 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.336 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.336 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.336 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.337 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.337 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.337 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.338 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.338 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.338 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.339 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.339 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.339 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.339 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.340 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.340 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.340 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.340 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.341 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.341 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": { 2023-10-11T16:43:03.341 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "min_size", 2023-10-11T16:43:03.342 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.342 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.342 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.343 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.343 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.343 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.343 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.344 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default min_size for any created local pool", 2023-10-11T16:43:03.344 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-10-11T16:43:03.344 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.344 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.345 INFO:teuthology.orchestra.run.smithi111.stdout: "num_rep": { 2023-10-11T16:43:03.345 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "num_rep", 2023-10-11T16:43:03.345 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.346 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.346 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.346 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3", 2023-10-11T16:43:03.346 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.347 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.347 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.347 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default replica count for any created local pool", 2023-10-11T16:43:03.348 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.348 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.348 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.349 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": { 2023-10-11T16:43:03.349 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pg_num", 2023-10-11T16:43:03.349 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.349 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.350 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.350 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "128", 2023-10-11T16:43:03.350 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.350 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.351 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.351 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default pg_num for any created local pool", 2023-10-11T16:43:03.351 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.352 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.352 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.352 INFO:teuthology.orchestra.run.smithi111.stdout: "prefix": { 2023-10-11T16:43:03.353 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prefix", 2023-10-11T16:43:03.353 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.353 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.353 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.354 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.354 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.354 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.355 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.355 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "name prefix for any created local pool", 2023-10-11T16:43:03.355 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.355 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.356 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.356 INFO:teuthology.orchestra.run.smithi111.stdout: "subtree": { 2023-10-11T16:43:03.356 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "subtree", 2023-10-11T16:43:03.357 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.357 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.357 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.357 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "rack", 2023-10-11T16:43:03.358 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.358 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.358 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.359 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "CRUSH level for which to create a local pool", 2023-10-11T16:43:03.359 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-10-11T16:43:03.359 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.359 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.360 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.360 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.361 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mds_autoscaler", 2023-10-11T16:43:03.361 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.361 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.361 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.362 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.362 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.362 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.363 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.363 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.363 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.363 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.364 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.364 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.364 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.364 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.365 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.365 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.365 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.366 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.366 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.366 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.366 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.367 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.367 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.367 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.368 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.368 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.368 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.369 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.369 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.369 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.369 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.370 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.370 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.370 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.370 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.371 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.371 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.372 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.372 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.372 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.372 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.373 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.373 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.373 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.373 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.374 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.374 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.374 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.375 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.375 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.375 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.375 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.377 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.377 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.378 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.378 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.378 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.379 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.379 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.379 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.379 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.380 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.380 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.380 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.381 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.381 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.381 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.381 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.382 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.382 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.383 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.383 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mirroring", 2023-10-11T16:43:03.384 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.384 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.384 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.384 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.385 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.385 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.385 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.385 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.386 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.386 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.386 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.387 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.388 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.388 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.388 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.388 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.389 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.389 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.389 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.389 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.390 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.390 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.391 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.391 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.391 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.391 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.392 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.392 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.392 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.392 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.393 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.393 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.393 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.394 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.394 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.394 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.395 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.395 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.395 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.396 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.396 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.396 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.396 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.397 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.397 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.397 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.397 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.398 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.398 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.398 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.398 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.399 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.399 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.400 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.400 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.400 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.400 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.401 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.401 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.401 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.401 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.402 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.402 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.402 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.402 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.403 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.403 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.403 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.404 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "nfs", 2023-10-11T16:43:03.404 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.405 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.405 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.405 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.405 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.406 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.406 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.406 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.406 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.407 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.407 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.407 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.407 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.408 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.408 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.408 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.408 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.409 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.409 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.409 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.409 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.410 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.410 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.411 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.411 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.411 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.411 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.412 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.412 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.412 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.412 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.413 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.413 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.413 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.413 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.414 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.414 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.414 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.415 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.415 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.415 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.415 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.416 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.416 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.416 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.416 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.417 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.417 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.417 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.417 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.418 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.418 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.418 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.419 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.419 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.419 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.420 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.420 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.420 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.421 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.421 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.421 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.421 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.422 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.422 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.422 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.422 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.423 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.423 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.424 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "orchestrator", 2023-10-11T16:43:03.424 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.425 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.425 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.425 INFO:teuthology.orchestra.run.smithi111.stdout: "fail_fs": { 2023-10-11T16:43:03.425 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "fail_fs", 2023-10-11T16:43:03.426 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.426 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.426 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.427 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.427 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.427 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.427 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.428 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-10-11T16:43:03.428 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.428 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.428 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.429 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.429 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.430 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.430 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.430 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.430 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.431 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.431 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.431 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.431 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.432 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.432 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.432 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.432 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.433 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.433 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.433 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.434 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.434 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.434 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.435 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.435 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.435 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.436 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.436 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.436 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.437 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.437 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.437 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.437 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.438 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.438 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.438 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.439 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.439 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.439 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.440 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.440 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.440 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.440 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.441 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.441 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.441 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.441 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.442 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.442 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.442 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.443 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.443 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.443 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.443 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.444 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.444 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.444 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.445 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.445 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.445 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.445 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.446 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.446 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.446 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.447 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.447 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.447 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.447 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.448 INFO:teuthology.orchestra.run.smithi111.stdout: "orchestrator": { 2023-10-11T16:43:03.448 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "orchestrator", 2023-10-11T16:43:03.448 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.449 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.449 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.449 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.450 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.450 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.450 INFO:teuthology.orchestra.run.smithi111.stdout: "cephadm", 2023-10-11T16:43:03.451 INFO:teuthology.orchestra.run.smithi111.stdout: "rook", 2023-10-11T16:43:03.451 INFO:teuthology.orchestra.run.smithi111.stdout: "test_orchestrator" 2023-10-11T16:43:03.451 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.451 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Orchestrator backend", 2023-10-11T16:43:03.452 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.452 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.452 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.453 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.453 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "osd_perf_query", 2023-10-11T16:43:03.454 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.454 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.454 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.455 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.455 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.455 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.455 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.456 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.456 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.456 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.456 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.457 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.457 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.457 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.458 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.458 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.458 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.458 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.459 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.459 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.459 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.459 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.460 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.460 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.461 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.461 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.461 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.461 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.462 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.462 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.462 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.462 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.463 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.463 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.463 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.464 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.464 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.464 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.465 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.465 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.465 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.465 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.466 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.466 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.466 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.466 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.467 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.467 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.467 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.468 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.468 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.468 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.468 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.469 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.469 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.469 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.470 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.470 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.470 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.471 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.471 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.471 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.472 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.472 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.472 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.472 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.473 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.473 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.474 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "osd_support", 2023-10-11T16:43:03.474 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.475 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.475 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.475 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.476 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.476 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.476 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.476 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.477 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.477 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.477 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.477 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.478 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.478 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.478 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.478 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.479 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.479 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.479 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.479 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.480 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.480 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.480 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.481 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.481 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.481 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.481 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.482 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.482 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.482 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.483 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.483 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.483 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.483 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.484 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.484 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.484 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.485 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.485 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.485 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.485 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.486 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.486 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.486 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.487 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.487 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.487 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.487 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.488 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.488 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.488 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.488 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.489 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.489 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.489 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.490 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.490 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.490 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.491 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.491 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.491 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.492 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.492 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.492 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.492 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.493 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.493 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.493 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.493 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.495 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pg_autoscaler", 2023-10-11T16:43:03.495 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.495 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.496 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.496 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.496 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.496 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.497 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.497 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.497 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.497 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.498 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.498 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.498 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.499 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.499 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.499 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.499 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.500 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.500 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.500 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.500 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.501 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.501 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.501 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.502 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.502 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.502 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.503 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.503 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.503 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.503 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.504 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.504 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.504 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.504 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.505 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.505 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.505 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.506 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.506 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.506 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.507 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.507 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.507 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.507 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.508 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.508 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.508 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.508 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.509 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.509 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.509 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.510 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.510 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.510 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.510 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.511 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.511 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.511 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.512 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.512 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.512 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.513 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.513 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.513 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.513 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.514 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.514 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.514 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.515 INFO:teuthology.orchestra.run.smithi111.stdout: "noautoscale": { 2023-10-11T16:43:03.515 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "noautoscale", 2023-10-11T16:43:03.515 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.516 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.516 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.516 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.516 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.517 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.517 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.517 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "global autoscale flag", 2023-10-11T16:43:03.517 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-10-11T16:43:03.518 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.518 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.518 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:03.519 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:03.519 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.519 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.520 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.520 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:03.520 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.520 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.521 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.521 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.521 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.521 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.522 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.522 INFO:teuthology.orchestra.run.smithi111.stdout: "threshold": { 2023-10-11T16:43:03.522 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "threshold", 2023-10-11T16:43:03.523 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:03.523 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.523 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.523 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3.0", 2023-10-11T16:43:03.524 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1.0", 2023-10-11T16:43:03.524 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.524 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.524 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "scaling threshold", 2023-10-11T16:43:03.525 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-10-11T16:43:03.525 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.525 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.526 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.526 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.527 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "progress", 2023-10-11T16:43:03.527 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.527 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.527 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.528 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_pg_recovery_event": { 2023-10-11T16:43:03.528 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_pg_recovery_event", 2023-10-11T16:43:03.528 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.528 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.529 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.529 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.529 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.530 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.530 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.530 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow the module to show pg recovery progress", 2023-10-11T16:43:03.530 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.531 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.531 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.531 INFO:teuthology.orchestra.run.smithi111.stdout: "enabled": { 2023-10-11T16:43:03.532 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enabled", 2023-10-11T16:43:03.532 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.532 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.532 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.533 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.533 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.533 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.534 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.534 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.534 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.534 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.535 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.535 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.535 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.536 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.536 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.536 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.536 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.537 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.537 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.537 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.537 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.538 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.538 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.538 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.539 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.539 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.539 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.540 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.540 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.540 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.540 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.541 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.541 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.541 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.542 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.542 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.542 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.542 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.543 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.543 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.543 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.543 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.544 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.544 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.545 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.545 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.545 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.545 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.546 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.546 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.546 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.546 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.547 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.547 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.547 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.547 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.548 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.548 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.548 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.548 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.549 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.549 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.549 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.550 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.550 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.550 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.551 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.551 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.551 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.551 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.552 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.552 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.552 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.552 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.553 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.553 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.553 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.554 INFO:teuthology.orchestra.run.smithi111.stdout: "max_completed_events": { 2023-10-11T16:43:03.554 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_completed_events", 2023-10-11T16:43:03.554 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.555 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.555 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.555 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "50", 2023-10-11T16:43:03.555 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.556 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.556 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.556 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "number of past completed events to remember", 2023-10-11T16:43:03.556 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.557 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.557 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.557 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:03.558 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:03.558 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.558 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.559 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.559 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:03.559 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.559 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.560 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.560 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long the module is going to sleep", 2023-10-11T16:43:03.560 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.560 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.561 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.562 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus", 2023-10-11T16:43:03.562 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.562 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.563 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.563 INFO:teuthology.orchestra.run.smithi111.stdout: "cache": { 2023-10-11T16:43:03.563 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cache", 2023-10-11T16:43:03.563 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.564 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.564 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.564 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.565 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.565 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.565 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.565 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.566 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.566 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.566 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.567 INFO:teuthology.orchestra.run.smithi111.stdout: "exclude_perf_counters": { 2023-10-11T16:43:03.567 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "exclude_perf_counters", 2023-10-11T16:43:03.567 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.567 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.568 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.568 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.568 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.569 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.569 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.569 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-10-11T16:43:03.569 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-10-11T16:43:03.570 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.570 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.570 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.571 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.571 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.571 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.572 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.572 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.572 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.572 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.573 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.573 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.573 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.574 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.574 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.574 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.574 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.575 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.575 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.575 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.575 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.576 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.576 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.577 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.577 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.577 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.577 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.578 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.578 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.578 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.578 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.579 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.579 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.579 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.581 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.582 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.582 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.582 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.583 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.583 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.583 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.583 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.584 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.584 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.584 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.584 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.585 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.585 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.586 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.586 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.586 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.586 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.587 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.587 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.588 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.588 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.588 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.588 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.589 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.589 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.589 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.590 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.590 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.590 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.591 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.591 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.591 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.592 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_stats_pools": { 2023-10-11T16:43:03.592 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_stats_pools", 2023-10-11T16:43:03.592 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.592 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.593 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.593 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.593 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.593 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.594 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.594 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.594 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.595 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.595 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.595 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_stats_pools_refresh_interval": { 2023-10-11T16:43:03.596 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-10-11T16:43:03.596 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.596 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.597 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.597 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "300", 2023-10-11T16:43:03.597 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.597 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.598 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.598 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.598 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.598 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.599 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.599 INFO:teuthology.orchestra.run.smithi111.stdout: "scrape_interval": { 2023-10-11T16:43:03.599 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "scrape_interval", 2023-10-11T16:43:03.600 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:03.600 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.600 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.601 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15.0", 2023-10-11T16:43:03.601 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.601 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.601 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.602 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.602 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.602 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.603 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.603 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:03.603 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:03.604 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.604 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.604 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.604 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "::", 2023-10-11T16:43:03.605 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.605 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.605 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.606 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-10-11T16:43:03.606 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.606 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.606 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.607 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:03.607 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:03.607 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.608 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.608 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.608 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "9283", 2023-10-11T16:43:03.609 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.609 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.609 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.609 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-10-11T16:43:03.610 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.610 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.610 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.611 INFO:teuthology.orchestra.run.smithi111.stdout: "stale_cache_strategy": { 2023-10-11T16:43:03.611 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "stale_cache_strategy", 2023-10-11T16:43:03.611 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.612 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.612 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.612 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "log", 2023-10-11T16:43:03.612 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.613 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.613 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.613 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.614 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.614 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.614 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.615 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_behaviour": { 2023-10-11T16:43:03.615 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_behaviour", 2023-10-11T16:43:03.615 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.616 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.616 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.616 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "default", 2023-10-11T16:43:03.616 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.617 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.617 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.617 INFO:teuthology.orchestra.run.smithi111.stdout: "default", 2023-10-11T16:43:03.617 INFO:teuthology.orchestra.run.smithi111.stdout: "error" 2023-10-11T16:43:03.618 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.618 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.618 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.619 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.619 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.619 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_error_status_code": { 2023-10-11T16:43:03.620 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_error_status_code", 2023-10-11T16:43:03.620 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.620 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.620 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.621 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "500", 2023-10-11T16:43:03.621 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "400", 2023-10-11T16:43:03.621 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "599", 2023-10-11T16:43:03.622 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.622 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.622 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.622 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.623 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.623 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.623 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.624 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.624 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.624 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_support", 2023-10-11T16:43:03.624 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.625 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.625 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.625 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.625 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.626 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.626 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.626 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.627 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.627 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.627 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.628 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.628 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.628 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.628 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.629 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.629 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.629 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.629 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.630 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.630 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.630 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.631 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.631 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.631 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.632 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.632 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.632 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.632 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.633 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.633 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.633 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.634 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.634 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.634 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.634 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.635 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.635 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.636 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.636 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.636 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.636 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.637 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.637 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.637 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.637 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.638 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.638 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.638 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.638 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.639 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.639 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.639 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.640 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.640 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.640 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.641 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.641 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.641 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.641 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.642 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.642 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.642 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.643 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.643 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.643 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.643 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.644 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.644 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.644 INFO:teuthology.orchestra.run.smithi111.stdout: "max_concurrent_snap_create": { 2023-10-11T16:43:03.645 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_concurrent_snap_create", 2023-10-11T16:43:03.645 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.645 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.645 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.646 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:03.646 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.646 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.647 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.647 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.647 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.647 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.648 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mirror_snapshot_schedule": { 2023-10-11T16:43:03.648 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mirror_snapshot_schedule", 2023-10-11T16:43:03.649 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.649 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.649 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.650 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.650 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.650 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.650 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.651 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.651 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.651 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.652 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.652 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.652 INFO:teuthology.orchestra.run.smithi111.stdout: "trash_purge_schedule": { 2023-10-11T16:43:03.652 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "trash_purge_schedule", 2023-10-11T16:43:03.653 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.653 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.653 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.654 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.654 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.654 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.654 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.655 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.655 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.655 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.655 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.656 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.656 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.657 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.657 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "restful", 2023-10-11T16:43:03.657 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.657 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.658 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.658 INFO:teuthology.orchestra.run.smithi111.stdout: "enable_auth": { 2023-10-11T16:43:03.658 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enable_auth", 2023-10-11T16:43:03.659 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.659 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.659 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.659 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.660 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.660 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.660 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.660 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.661 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.661 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.661 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.662 INFO:teuthology.orchestra.run.smithi111.stdout: "key_file": { 2023-10-11T16:43:03.662 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "key_file", 2023-10-11T16:43:03.662 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.663 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.663 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.663 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.663 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.664 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.664 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.664 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.665 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.665 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.665 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.666 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.666 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.666 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.666 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.667 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.667 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.667 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.668 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.668 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.668 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.668 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.669 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.669 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.669 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.670 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.670 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.670 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.670 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.671 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.671 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.671 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.672 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.672 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.672 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.673 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.673 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.673 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.673 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.674 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.674 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.674 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.675 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.675 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.675 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.676 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.676 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.676 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.676 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.677 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.677 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.677 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.678 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.678 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.678 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.678 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.679 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.679 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.679 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.679 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.680 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.680 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.680 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.681 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.681 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.681 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.682 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.682 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.682 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.683 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.683 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.683 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.683 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.684 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.684 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.684 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.685 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.685 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:03.685 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:03.686 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.686 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.686 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.686 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.687 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.687 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.687 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.688 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.688 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.688 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.688 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.689 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:03.689 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:03.689 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.690 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.690 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.690 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.691 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.691 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.691 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.691 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.692 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.692 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.692 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.692 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.693 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.694 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rgw", 2023-10-11T16:43:03.694 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.694 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.694 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.695 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.695 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.695 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.696 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.696 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.696 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.696 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.697 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.697 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.697 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.697 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.698 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.698 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.698 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.699 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.699 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.699 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.699 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.700 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.700 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.701 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.701 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.701 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.701 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.702 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.702 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.702 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.702 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.703 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.703 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.703 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.704 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.704 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.704 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.705 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.705 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.705 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.706 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.706 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.706 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.706 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.707 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.707 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.707 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.707 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.708 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.708 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.708 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.709 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.709 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.709 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.709 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.710 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.710 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.711 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.711 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.711 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.711 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.712 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.712 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.712 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.712 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.713 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.713 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.713 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.714 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.714 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.714 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.715 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rook", 2023-10-11T16:43:03.715 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.715 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.716 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.716 INFO:teuthology.orchestra.run.smithi111.stdout: "drive_group_interval": { 2023-10-11T16:43:03.716 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "drive_group_interval", 2023-10-11T16:43:03.717 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:03.717 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.717 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.717 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "300.0", 2023-10-11T16:43:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.718 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.719 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-10-11T16:43:03.719 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.719 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.719 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.720 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.720 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.721 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.721 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.721 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.721 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.722 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.722 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.722 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.723 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.723 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.723 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.723 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.724 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.724 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.724 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.724 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.725 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.725 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.725 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.726 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.726 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.726 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.727 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.727 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.727 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.728 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.728 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.728 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.728 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.729 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.729 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.729 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.730 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.730 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.730 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.731 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.731 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.731 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.731 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.732 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.732 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.732 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.733 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.733 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.733 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.733 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.734 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.734 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.734 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.735 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.735 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.735 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.736 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.736 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.736 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.737 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.737 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.737 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.737 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.738 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.738 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.738 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.739 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.739 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.739 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.740 INFO:teuthology.orchestra.run.smithi111.stdout: "storage_class": { 2023-10-11T16:43:03.740 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "storage_class", 2023-10-11T16:43:03.740 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.740 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.741 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.741 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "local", 2023-10-11T16:43:03.741 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.742 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.742 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.742 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-10-11T16:43:03.742 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.743 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.743 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.743 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.744 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.744 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "selftest", 2023-10-11T16:43:03.745 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.745 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.745 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.745 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.746 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.746 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.746 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.747 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.747 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.747 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.747 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.748 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.748 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.748 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.748 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.749 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.749 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.749 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.749 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.750 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.750 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.750 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.751 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.751 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.751 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.752 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.752 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.752 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.753 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.753 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.753 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.753 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.754 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.754 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.754 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.755 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.755 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.755 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.756 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.756 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.756 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.756 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.757 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.757 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.757 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.758 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.758 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.758 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.758 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.759 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.759 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.759 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.760 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.760 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.760 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.760 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.761 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.761 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.762 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.762 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.762 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.762 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.763 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.763 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.763 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.764 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.764 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.764 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.764 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.765 INFO:teuthology.orchestra.run.smithi111.stdout: "roption1": { 2023-10-11T16:43:03.765 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "roption1", 2023-10-11T16:43:03.765 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.766 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.766 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.766 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.767 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.767 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.767 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.767 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.768 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.768 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.768 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.769 INFO:teuthology.orchestra.run.smithi111.stdout: "roption2": { 2023-10-11T16:43:03.769 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "roption2", 2023-10-11T16:43:03.769 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.770 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.770 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.770 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "xyz", 2023-10-11T16:43:03.771 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.771 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.771 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.771 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.772 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.772 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.772 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.773 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption1": { 2023-10-11T16:43:03.773 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption1", 2023-10-11T16:43:03.773 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.774 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.774 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.774 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.774 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.775 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.775 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.775 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.776 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.776 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.776 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.777 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption2": { 2023-10-11T16:43:03.777 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption2", 2023-10-11T16:43:03.777 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.777 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.778 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.778 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.778 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.782 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.782 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.782 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.783 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.783 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.783 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.784 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption3": { 2023-10-11T16:43:03.784 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption3", 2023-10-11T16:43:03.784 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:03.784 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.785 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.785 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.785 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.786 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.786 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.786 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.786 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.787 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.787 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.787 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption4": { 2023-10-11T16:43:03.788 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption4", 2023-10-11T16:43:03.788 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.788 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.789 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.789 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.789 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.789 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.790 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.790 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.790 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.791 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.791 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.791 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption5": { 2023-10-11T16:43:03.792 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption5", 2023-10-11T16:43:03.792 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.792 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.792 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.793 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.793 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.793 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.794 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.794 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.794 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.794 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.795 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.795 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption6": { 2023-10-11T16:43:03.796 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption6", 2023-10-11T16:43:03.796 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.796 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.796 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.797 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.797 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.797 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.797 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.798 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.798 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.798 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.799 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.799 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption7": { 2023-10-11T16:43:03.799 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption7", 2023-10-11T16:43:03.800 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.800 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.800 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.801 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.801 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:03.801 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "42", 2023-10-11T16:43:03.801 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.802 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.802 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.802 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.803 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.803 INFO:teuthology.orchestra.run.smithi111.stdout: "testkey": { 2023-10-11T16:43:03.803 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testkey", 2023-10-11T16:43:03.804 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.804 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.804 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.805 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.805 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.805 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.805 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.806 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.806 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.806 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.806 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.807 INFO:teuthology.orchestra.run.smithi111.stdout: "testlkey": { 2023-10-11T16:43:03.807 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testlkey", 2023-10-11T16:43:03.808 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.808 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.808 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.809 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.809 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.809 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.809 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.810 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.810 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.810 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.810 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.811 INFO:teuthology.orchestra.run.smithi111.stdout: "testnewline": { 2023-10-11T16:43:03.811 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testnewline", 2023-10-11T16:43:03.812 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.812 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.812 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.812 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.813 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.813 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.813 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.814 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.814 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.814 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.814 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.815 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.815 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.816 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "snap_schedule", 2023-10-11T16:43:03.816 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.816 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.817 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.817 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_m_granularity": { 2023-10-11T16:43:03.817 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_m_granularity", 2023-10-11T16:43:03.817 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.818 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.818 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.818 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.819 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.819 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.819 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.820 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow minute scheduled snapshots", 2023-10-11T16:43:03.820 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.820 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.820 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.821 INFO:teuthology.orchestra.run.smithi111.stdout: "dump_on_update": { 2023-10-11T16:43:03.821 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "dump_on_update", 2023-10-11T16:43:03.821 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.822 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.822 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.822 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.823 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.823 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.823 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.824 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "dump database to debug log on update", 2023-10-11T16:43:03.824 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.824 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.824 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.825 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.825 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.825 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.826 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.826 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.826 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.827 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.827 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.827 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.827 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.828 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.828 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.828 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.829 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.829 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.829 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.829 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.830 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.830 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.830 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.831 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.831 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.831 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.832 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.832 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.832 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.833 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.833 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.833 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.833 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.834 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.834 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.834 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.835 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.835 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.835 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.836 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.836 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.836 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.836 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.837 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.837 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.837 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.838 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.838 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.838 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.838 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.839 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.839 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.839 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.840 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.840 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.840 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.841 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.841 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.841 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.841 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.842 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.842 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.842 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.843 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.843 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.843 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.843 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.844 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.844 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.844 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.845 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.845 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "stats", 2023-10-11T16:43:03.846 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.846 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.846 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.847 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.847 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.847 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.847 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.848 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.848 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.848 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.848 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.849 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.849 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.849 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.849 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.850 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.850 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.850 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.851 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.851 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.851 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.851 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.852 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.852 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.853 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.853 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.853 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.853 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.854 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.854 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.854 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.855 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.855 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.855 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.855 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.856 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.856 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.856 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.857 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.857 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.857 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.858 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.858 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.858 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.858 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.859 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.859 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.859 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.860 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.860 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.860 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.860 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.861 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.861 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.861 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.862 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.862 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.862 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.863 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.863 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.863 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.864 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.864 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.864 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.864 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.865 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.865 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.865 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.865 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.866 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.866 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.867 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "status", 2023-10-11T16:43:03.867 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.867 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.868 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.868 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.868 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.869 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.869 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.869 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.869 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.870 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.870 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.870 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.870 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.871 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.871 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.871 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.872 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.872 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.872 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.872 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.873 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.873 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.873 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.874 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.874 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.874 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.875 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.875 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.875 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.875 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.876 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.876 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.876 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.877 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.877 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.877 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.878 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.878 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.878 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.879 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.879 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.879 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.879 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.880 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.880 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.880 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.881 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.881 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.881 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.881 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.882 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.882 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.882 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.883 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.883 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.883 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.884 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.884 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.884 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.884 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.885 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.885 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.885 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.886 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.886 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.886 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.886 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.887 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.887 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.887 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.888 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.888 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "telegraf", 2023-10-11T16:43:03.889 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.889 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.889 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.889 INFO:teuthology.orchestra.run.smithi111.stdout: "address": { 2023-10-11T16:43:03.890 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "address", 2023-10-11T16:43:03.890 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.890 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.891 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.891 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-10-11T16:43:03.891 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.891 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.892 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.892 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.892 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.892 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.893 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.893 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:03.894 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:03.894 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:03.894 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.894 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.895 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15", 2023-10-11T16:43:03.895 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.895 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.895 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.896 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.896 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.896 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.897 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.897 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.897 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.898 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.898 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.898 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.898 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.899 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.899 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.899 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.900 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.900 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.900 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.900 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.901 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.901 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.901 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.901 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.902 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.902 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.902 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.903 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.903 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.903 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.904 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.904 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.904 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.905 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.905 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.905 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.905 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.906 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.906 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.906 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.907 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.907 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.907 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.908 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.908 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.908 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.908 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.909 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.909 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.909 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.909 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.910 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.910 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.910 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.911 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.911 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.911 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.911 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.912 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.912 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.912 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.913 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.913 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.913 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.914 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.914 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.914 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.914 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.915 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.915 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.915 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.916 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.916 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.916 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.916 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:03.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:03.917 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "telemetry", 2023-10-11T16:43:03.917 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:03.918 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:03.918 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:03.918 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_basic": { 2023-10-11T16:43:03.918 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_basic", 2023-10-11T16:43:03.919 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.919 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.919 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.920 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.920 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.920 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.920 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.921 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share basic cluster information (size, version)", 2023-10-11T16:43:03.921 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.921 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.921 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.922 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_crash": { 2023-10-11T16:43:03.922 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_crash", 2023-10-11T16:43:03.922 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.923 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.923 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.923 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.923 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.924 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.924 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.924 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-10-11T16:43:03.925 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.925 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.925 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.926 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_device": { 2023-10-11T16:43:03.926 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_device", 2023-10-11T16:43:03.926 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.927 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.927 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.927 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:03.927 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.928 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.928 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.928 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-10-11T16:43:03.928 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.929 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.929 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.930 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_ident": { 2023-10-11T16:43:03.930 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_ident", 2023-10-11T16:43:03.930 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.930 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.931 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.931 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.931 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.931 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.932 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.932 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-10-11T16:43:03.932 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.933 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.933 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.933 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_perf": { 2023-10-11T16:43:03.934 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_perf", 2023-10-11T16:43:03.934 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.934 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.934 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.935 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.935 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.935 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.936 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.936 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share various performance metrics of a cluster", 2023-10-11T16:43:03.936 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.936 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.937 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.937 INFO:teuthology.orchestra.run.smithi111.stdout: "contact": { 2023-10-11T16:43:03.937 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "contact", 2023-10-11T16:43:03.938 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.938 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.938 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.939 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.939 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.939 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.939 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.940 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.940 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.940 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.941 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.941 INFO:teuthology.orchestra.run.smithi111.stdout: "description": { 2023-10-11T16:43:03.941 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "description", 2023-10-11T16:43:03.942 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.942 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.942 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.942 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.943 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.943 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.943 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.944 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.944 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.944 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.944 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: "device_url": { 2023-10-11T16:43:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_url", 2023-10-11T16:43:03.945 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.946 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-10-11T16:43:03.947 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.947 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.947 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.947 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.948 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.948 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.948 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.949 INFO:teuthology.orchestra.run.smithi111.stdout: "enabled": { 2023-10-11T16:43:03.949 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enabled", 2023-10-11T16:43:03.949 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.950 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.950 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.950 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.950 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.951 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.951 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.951 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.951 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.952 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.952 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.953 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:03.953 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:03.953 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.953 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.954 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.954 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "24", 2023-10-11T16:43:03.954 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "8", 2023-10-11T16:43:03.955 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.955 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.955 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.955 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.956 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.956 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.956 INFO:teuthology.orchestra.run.smithi111.stdout: "last_opt_revision": { 2023-10-11T16:43:03.957 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "last_opt_revision", 2023-10-11T16:43:03.957 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:03.957 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.957 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.958 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1", 2023-10-11T16:43:03.958 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.958 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.959 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.959 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.959 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.959 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.960 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.960 INFO:teuthology.orchestra.run.smithi111.stdout: "leaderboard": { 2023-10-11T16:43:03.961 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "leaderboard", 2023-10-11T16:43:03.961 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.961 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.961 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.962 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.962 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.962 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.962 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.963 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.963 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.963 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.964 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.964 INFO:teuthology.orchestra.run.smithi111.stdout: "leaderboard_description": { 2023-10-11T16:43:03.964 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "leaderboard_description", 2023-10-11T16:43:03.965 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.965 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.965 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.965 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.966 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.966 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.966 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.967 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.967 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.967 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.967 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.968 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:03.968 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:03.969 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.969 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.969 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.969 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.970 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.970 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.970 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.971 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.971 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.971 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.971 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.972 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.972 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.972 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.972 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.973 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.973 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.973 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.974 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:03.974 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:03.974 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.975 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.975 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.975 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.975 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.976 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.976 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.976 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.977 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.977 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.977 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.978 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:03.978 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:03.978 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.979 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.979 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.979 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:03.979 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.980 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.980 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:03.980 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:03.980 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:03.981 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:03.981 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:03.981 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:03.982 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:03.982 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:03.982 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.982 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.983 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.983 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.984 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:03.984 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:03.984 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:03.984 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.985 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:03.985 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:03.985 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.985 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.986 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.986 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.986 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.987 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.987 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.987 INFO:teuthology.orchestra.run.smithi111.stdout: "organization": { 2023-10-11T16:43:03.988 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "organization", 2023-10-11T16:43:03.988 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.989 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.989 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.990 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.990 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.990 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.990 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.991 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.991 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.991 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.992 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.992 INFO:teuthology.orchestra.run.smithi111.stdout: "proxy": { 2023-10-11T16:43:03.992 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "proxy", 2023-10-11T16:43:03.993 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.993 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.993 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.993 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:03.994 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.994 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.994 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.995 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.995 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.995 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.995 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:03.996 INFO:teuthology.orchestra.run.smithi111.stdout: "url": { 2023-10-11T16:43:03.996 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "url", 2023-10-11T16:43:03.996 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:03.997 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:03.997 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:03.997 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-10-11T16:43:03.998 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:03.998 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:03.998 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:03.998 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:03.999 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:03.999 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:03.999 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:03.999 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.000 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.001 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "test_orchestrator", 2023-10-11T16:43:04.001 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.001 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.001 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.002 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.002 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.002 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.002 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.003 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.003 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.003 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.004 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.004 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.004 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.004 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.005 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.005 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.005 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.005 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.006 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.006 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.006 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.007 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.007 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.007 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.008 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.008 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.008 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.009 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.009 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.009 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.009 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.010 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.010 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.010 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.011 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.011 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.011 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.011 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.012 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.012 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.012 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.013 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.013 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.013 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.013 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.014 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.014 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.014 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.014 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.015 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.015 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.015 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.015 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.016 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.016 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.016 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.016 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.017 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.017 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.017 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.018 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.018 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.018 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.019 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.019 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.019 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.019 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.020 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.020 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.020 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.020 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.021 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.022 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "volumes", 2023-10-11T16:43:04.022 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.022 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.023 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.023 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.023 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.023 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.024 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.024 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.024 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.025 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.025 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.025 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.025 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.026 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.026 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.026 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.026 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.027 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.027 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.027 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.028 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.028 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.028 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.029 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.029 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.029 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.029 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.030 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.030 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.030 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.031 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.031 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.031 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.031 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.032 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.032 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.033 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.033 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.033 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.033 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.034 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.034 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.034 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.035 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.035 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.035 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.035 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.036 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.036 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.036 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.037 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.037 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.037 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.037 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.038 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.038 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.038 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.039 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.039 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.039 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.040 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.040 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.040 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.040 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.041 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.041 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.041 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.041 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.042 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.042 INFO:teuthology.orchestra.run.smithi111.stdout: "max_concurrent_clones": { 2023-10-11T16:43:04.043 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_concurrent_clones", 2023-10-11T16:43:04.043 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.043 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.043 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.044 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "4", 2023-10-11T16:43:04.044 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.044 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.045 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.045 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Number of asynchronous cloner threads", 2023-10-11T16:43:04.045 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.045 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.046 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.046 INFO:teuthology.orchestra.run.smithi111.stdout: "snapshot_clone_delay": { 2023-10-11T16:43:04.047 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "snapshot_clone_delay", 2023-10-11T16:43:04.047 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.047 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.047 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.048 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:04.048 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.048 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.048 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.049 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-10-11T16:43:04.049 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.049 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.050 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.050 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.050 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.051 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix", 2023-10-11T16:43:04.051 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.052 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.052 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.052 INFO:teuthology.orchestra.run.smithi111.stdout: "discovery_interval": { 2023-10-11T16:43:04.052 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "discovery_interval", 2023-10-11T16:43:04.053 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:04.053 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.053 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.053 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "100", 2023-10-11T16:43:04.054 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.054 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.054 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.055 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.055 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.055 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.055 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.056 INFO:teuthology.orchestra.run.smithi111.stdout: "identifier": { 2023-10-11T16:43:04.056 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "identifier", 2023-10-11T16:43:04.056 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.057 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.057 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.057 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.057 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.058 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.058 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.058 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.059 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.059 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.059 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.060 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:04.060 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:04.060 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.061 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.061 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.061 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:04.061 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.062 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.062 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.062 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.062 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.063 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.063 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.064 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.064 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.064 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.064 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.065 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.065 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.065 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.065 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.066 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.066 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.066 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.067 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.067 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.067 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.068 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.068 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.068 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.068 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.069 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.069 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.069 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.070 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.070 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.070 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.071 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.071 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.071 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.071 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.072 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.072 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.072 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.073 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.073 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.073 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.074 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.074 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.074 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.075 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.075 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.075 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.075 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.076 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.076 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.076 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.077 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.077 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.077 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.077 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.078 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.078 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.078 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.079 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.079 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.079 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.080 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.080 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.080 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.080 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.081 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.081 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.081 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.082 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.082 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.082 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.082 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.083 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.083 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_host": { 2023-10-11T16:43:04.084 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_host", 2023-10-11T16:43:04.084 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.084 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.084 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.085 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.085 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.085 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.085 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.086 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.086 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.086 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.087 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.087 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_port": { 2023-10-11T16:43:04.087 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_port", 2023-10-11T16:43:04.088 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.088 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.088 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.089 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10051", 2023-10-11T16:43:04.089 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.089 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.089 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.090 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.090 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.090 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.091 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.091 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_sender": { 2023-10-11T16:43:04.091 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_sender", 2023-10-11T16:43:04.092 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.092 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.092 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.092 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-10-11T16:43:04.093 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.093 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.093 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.094 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.094 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.094 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.094 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.095 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.095 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.095 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.095 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:04.096 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.096 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.096 INFO:teuthology.orchestra.run.smithi111.stdout: "modules": [ 2023-10-11T16:43:04.097 INFO:teuthology.orchestra.run.smithi111.stdout: "cephadm", 2023-10-11T16:43:04.097 INFO:teuthology.orchestra.run.smithi111.stdout: "dashboard", 2023-10-11T16:43:04.097 INFO:teuthology.orchestra.run.smithi111.stdout: "iostat", 2023-10-11T16:43:04.097 INFO:teuthology.orchestra.run.smithi111.stdout: "nfs", 2023-10-11T16:43:04.098 INFO:teuthology.orchestra.run.smithi111.stdout: "restful", 2023-10-11T16:43:04.098 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_schedule" 2023-10-11T16:43:04.098 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.098 INFO:teuthology.orchestra.run.smithi111.stdout: "available_modules": [ 2023-10-11T16:43:04.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.099 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alerts", 2023-10-11T16:43:04.099 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.100 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.100 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.100 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:04.100 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:04.101 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.101 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.101 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.101 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:04.102 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.102 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.102 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.103 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How frequently to reexamine health status", 2023-10-11T16:43:04.103 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.103 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.103 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.104 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.104 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.104 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.105 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.105 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.105 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.105 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.106 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.106 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.106 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.107 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.107 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.107 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.107 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.108 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.108 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.108 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.109 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.109 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.109 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.110 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.110 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.110 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.111 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.111 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.111 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.111 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.112 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.112 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.112 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.112 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.113 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.113 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.114 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.114 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.114 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.114 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.115 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.115 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.115 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.115 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.116 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.116 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.116 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.117 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.117 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.117 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.117 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.118 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.118 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.118 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.119 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.119 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.119 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.120 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.120 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.120 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.120 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.121 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.121 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.121 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.121 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.122 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.122 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.122 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.123 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.123 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_destination": { 2023-10-11T16:43:04.123 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_destination", 2023-10-11T16:43:04.124 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.124 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.124 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.124 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.125 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.125 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.125 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.125 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Email address to send alerts to", 2023-10-11T16:43:04.126 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.126 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.126 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.127 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_from_name": { 2023-10-11T16:43:04.127 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_from_name", 2023-10-11T16:43:04.127 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.128 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.128 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.128 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "Ceph", 2023-10-11T16:43:04.128 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.129 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.129 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.129 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Email From: name", 2023-10-11T16:43:04.129 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.130 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.130 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.131 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_host": { 2023-10-11T16:43:04.131 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_host", 2023-10-11T16:43:04.131 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.131 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.132 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.132 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.132 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.132 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.133 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.133 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP server", 2023-10-11T16:43:04.133 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.133 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.134 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.134 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_password": { 2023-10-11T16:43:04.135 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_password", 2023-10-11T16:43:04.135 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.135 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.135 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.136 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.136 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.136 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.137 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.137 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Password to authenticate with", 2023-10-11T16:43:04.137 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.137 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.138 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.138 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_port": { 2023-10-11T16:43:04.138 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_port", 2023-10-11T16:43:04.139 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.139 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.139 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.140 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "465", 2023-10-11T16:43:04.140 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.140 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.140 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.141 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP port", 2023-10-11T16:43:04.141 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.141 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.142 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.142 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_sender": { 2023-10-11T16:43:04.142 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_sender", 2023-10-11T16:43:04.143 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.143 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.143 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.143 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.144 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.144 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.144 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.144 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SMTP envelope sender", 2023-10-11T16:43:04.145 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.145 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.145 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.146 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_ssl": { 2023-10-11T16:43:04.146 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_ssl", 2023-10-11T16:43:04.146 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.147 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.147 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.147 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.147 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.148 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.148 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.148 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use SSL to connect to SMTP server", 2023-10-11T16:43:04.149 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.149 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.149 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.150 INFO:teuthology.orchestra.run.smithi111.stdout: "smtp_user": { 2023-10-11T16:43:04.150 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "smtp_user", 2023-10-11T16:43:04.150 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.151 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.151 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.151 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.151 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.152 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.152 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.152 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "User to authenticate as", 2023-10-11T16:43:04.152 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.153 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.153 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.153 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.154 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.154 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "balancer", 2023-10-11T16:43:04.155 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.155 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.155 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.156 INFO:teuthology.orchestra.run.smithi111.stdout: "active": { 2023-10-11T16:43:04.156 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "active", 2023-10-11T16:43:04.156 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.156 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.157 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.157 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.157 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.157 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.158 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.158 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "automatically balance PGs across cluster", 2023-10-11T16:43:04.158 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.158 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.159 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.159 INFO:teuthology.orchestra.run.smithi111.stdout: "begin_time": { 2023-10-11T16:43:04.160 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "begin_time", 2023-10-11T16:43:04.160 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.160 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.160 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.161 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0000", 2023-10-11T16:43:04.161 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.161 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.161 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.162 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "beginning time of day to automatically balance", 2023-10-11T16:43:04.162 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-11T16:43:04.162 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.163 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.163 INFO:teuthology.orchestra.run.smithi111.stdout: "begin_weekday": { 2023-10-11T16:43:04.163 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "begin_weekday", 2023-10-11T16:43:04.164 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:04.164 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.164 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.164 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:04.165 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0", 2023-10-11T16:43:04.165 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "6", 2023-10-11T16:43:04.165 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.165 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-10-11T16:43:04.166 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-11T16:43:04.166 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.166 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.167 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_max_iterations": { 2023-10-11T16:43:04.167 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_max_iterations", 2023-10-11T16:43:04.167 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:04.168 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.168 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.168 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "25", 2023-10-11T16:43:04.169 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:04.169 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "250", 2023-10-11T16:43:04.169 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.169 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-10-11T16:43:04.170 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.170 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.170 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.171 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_metrics": { 2023-10-11T16:43:04.171 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_metrics", 2023-10-11T16:43:04.171 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.172 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.172 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.172 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "pgs,objects,bytes", 2023-10-11T16:43:04.172 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.173 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.173 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.173 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-10-11T16:43:04.173 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-10-11T16:43:04.174 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.174 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.175 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_compat_step": { 2023-10-11T16:43:04.175 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crush_compat_step", 2023-10-11T16:43:04.175 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:04.175 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.176 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.176 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0.5", 2023-10-11T16:43:04.176 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0.001", 2023-10-11T16:43:04.176 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "0.999", 2023-10-11T16:43:04.177 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.177 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "aggressiveness of optimization", 2023-10-11T16:43:04.177 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-10-11T16:43:04.178 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.178 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.178 INFO:teuthology.orchestra.run.smithi111.stdout: "end_time": { 2023-10-11T16:43:04.179 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "end_time", 2023-10-11T16:43:04.179 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.179 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.180 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.180 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "2359", 2023-10-11T16:43:04.180 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.180 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.181 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.181 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "ending time of day to automatically balance", 2023-10-11T16:43:04.181 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-11T16:43:04.181 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.182 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.182 INFO:teuthology.orchestra.run.smithi111.stdout: "end_weekday": { 2023-10-11T16:43:04.183 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "end_weekday", 2023-10-11T16:43:04.183 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:04.183 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.183 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.184 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:04.184 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "0", 2023-10-11T16:43:04.184 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "6", 2023-10-11T16:43:04.184 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.185 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-10-11T16:43:04.185 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-11T16:43:04.185 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.186 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.186 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.186 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.187 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.187 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.187 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.187 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.188 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.188 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.188 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.189 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.189 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.189 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.189 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.190 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.190 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.190 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.191 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.191 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.191 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.191 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.192 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.192 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.192 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.193 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.193 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.193 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.193 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.194 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.194 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.194 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.195 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.195 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.195 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.196 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.196 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.196 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.198 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.198 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.199 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.199 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.199 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.199 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.200 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.200 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.200 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.200 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.201 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.201 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.201 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.202 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.202 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.202 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.202 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.203 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.203 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.203 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.204 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.204 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.204 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.204 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.205 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.205 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.205 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.205 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.206 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.206 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.206 INFO:teuthology.orchestra.run.smithi111.stdout: "min_score": { 2023-10-11T16:43:04.207 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "min_score", 2023-10-11T16:43:04.207 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:04.207 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.207 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.208 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:04.208 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.208 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.209 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.209 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-10-11T16:43:04.209 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.209 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.210 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.210 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": { 2023-10-11T16:43:04.211 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mode", 2023-10-11T16:43:04.211 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.211 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.211 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.212 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "upmap", 2023-10-11T16:43:04.212 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.212 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.213 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.213 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-compat", 2023-10-11T16:43:04.213 INFO:teuthology.orchestra.run.smithi111.stdout: "none", 2023-10-11T16:43:04.213 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap" 2023-10-11T16:43:04.214 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.214 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Balancer mode", 2023-10-11T16:43:04.214 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.214 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.215 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.215 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_ids": { 2023-10-11T16:43:04.216 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pool_ids", 2023-10-11T16:43:04.216 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.216 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.216 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.217 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.217 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.217 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.217 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.218 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-10-11T16:43:04.218 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.218 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.219 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.219 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:04.219 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:04.220 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.220 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.220 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.220 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:04.221 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.221 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.221 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.222 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-10-11T16:43:04.222 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.222 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.222 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.223 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap_max_deviation": { 2023-10-11T16:43:04.223 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "upmap_max_deviation", 2023-10-11T16:43:04.224 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.224 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.224 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.224 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:04.225 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:04.225 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.225 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.225 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "deviation below which no optimization is attempted", 2023-10-11T16:43:04.226 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-10-11T16:43:04.226 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.226 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.227 INFO:teuthology.orchestra.run.smithi111.stdout: "upmap_max_optimizations": { 2023-10-11T16:43:04.227 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "upmap_max_optimizations", 2023-10-11T16:43:04.227 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:04.228 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.228 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.228 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:04.228 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.229 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.229 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.229 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-10-11T16:43:04.229 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.230 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.230 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.230 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.231 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.231 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cephadm", 2023-10-11T16:43:04.232 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.232 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.232 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.233 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_down_multiplier": { 2023-10-11T16:43:04.233 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_down_multiplier", 2023-10-11T16:43:04.233 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:04.233 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.234 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.234 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3.0", 2023-10-11T16:43:04.234 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.234 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.235 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.235 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-10-11T16:43:04.235 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.236 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.236 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.236 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_refresh_rate": { 2023-10-11T16:43:04.237 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_refresh_rate", 2023-10-11T16:43:04.237 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.237 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.237 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.238 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "20", 2023-10-11T16:43:04.238 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.238 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.238 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.239 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-10-11T16:43:04.239 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.239 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.239 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.240 INFO:teuthology.orchestra.run.smithi111.stdout: "agent_starting_port": { 2023-10-11T16:43:04.240 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "agent_starting_port", 2023-10-11T16:43:04.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.241 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.241 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.241 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "4721", 2023-10-11T16:43:04.242 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.242 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.242 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.243 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-10-11T16:43:04.243 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.243 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.243 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.244 INFO:teuthology.orchestra.run.smithi111.stdout: "alertmanager_web_password": { 2023-10-11T16:43:04.244 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alertmanager_web_password", 2023-10-11T16:43:04.244 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.245 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.245 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.245 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:04.245 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.246 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.246 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.246 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Alertmanager web password", 2023-10-11T16:43:04.247 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.247 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.247 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.248 INFO:teuthology.orchestra.run.smithi111.stdout: "alertmanager_web_user": { 2023-10-11T16:43:04.248 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "alertmanager_web_user", 2023-10-11T16:43:04.248 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.248 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.249 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.249 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:04.249 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.249 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.249 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.250 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Alertmanager web user", 2023-10-11T16:43:04.250 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.250 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.251 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.251 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_ptrace": { 2023-10-11T16:43:04.251 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_ptrace", 2023-10-11T16:43:04.252 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.252 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.252 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.252 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.253 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.253 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.253 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.253 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-10-11T16:43:04.254 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-10-11T16:43:04.254 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.254 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.255 INFO:teuthology.orchestra.run.smithi111.stdout: "autotune_interval": { 2023-10-11T16:43:04.255 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "autotune_interval", 2023-10-11T16:43:04.255 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.256 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.256 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.256 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:04.256 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.257 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.257 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.257 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to autotune daemon memory", 2023-10-11T16:43:04.257 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.258 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.258 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.259 INFO:teuthology.orchestra.run.smithi111.stdout: "autotune_memory_target_ratio": { 2023-10-11T16:43:04.259 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "autotune_memory_target_ratio", 2023-10-11T16:43:04.259 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:04.259 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0.7", 2023-10-11T16:43:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.260 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-10-11T16:43:04.261 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.262 INFO:teuthology.orchestra.run.smithi111.stdout: "cgroups_split": { 2023-10-11T16:43:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cgroups_split", 2023-10-11T16:43:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.263 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.264 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-10-11T16:43:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.265 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "config_checks_enabled": { 2023-10-11T16:43:04.266 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_checks_enabled", 2023-10-11T16:43:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.267 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.268 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-10-11T16:43:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "config_dashboard": { 2023-10-11T16:43:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_dashboard", 2023-10-11T16:43:04.270 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.271 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.272 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.272 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.272 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-10-11T16:43:04.273 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.273 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.273 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.274 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_alertmanager": { 2023-10-11T16:43:04.274 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_alertmanager", 2023-10-11T16:43:04.274 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.274 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.275 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.275 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-10-11T16:43:04.275 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.275 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.276 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.276 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:04.276 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.276 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.277 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.277 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_base": { 2023-10-11T16:43:04.277 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_base", 2023-10-11T16:43:04.278 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.278 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.278 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.278 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/ceph", 2023-10-11T16:43:04.279 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.279 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.279 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.279 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Container image name, without the tag", 2023-10-11T16:43:04.280 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.280 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.280 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.281 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_elasticsearch": { 2023-10-11T16:43:04.281 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_elasticsearch", 2023-10-11T16:43:04.281 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.281 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.282 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.282 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-10-11T16:43:04.282 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.283 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.283 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.283 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "elasticsearch container image", 2023-10-11T16:43:04.283 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.284 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.284 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.284 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_grafana": { 2023-10-11T16:43:04.285 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_grafana", 2023-10-11T16:43:04.285 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.285 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.286 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.286 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-10-11T16:43:04.286 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.286 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.287 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.287 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:04.287 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.288 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.288 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.288 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_haproxy": { 2023-10-11T16:43:04.289 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_haproxy", 2023-10-11T16:43:04.289 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.289 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.289 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.290 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-10-11T16:43:04.290 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.290 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.290 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.291 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "HAproxy container image", 2023-10-11T16:43:04.291 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.291 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.291 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.292 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_agent": { 2023-10-11T16:43:04.292 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_agent", 2023-10-11T16:43:04.293 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.293 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.293 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.293 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-10-11T16:43:04.294 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.294 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.294 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.295 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger agent container image", 2023-10-11T16:43:04.295 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.295 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.295 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.296 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_collector": { 2023-10-11T16:43:04.296 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_collector", 2023-10-11T16:43:04.296 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.297 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.297 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.297 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-10-11T16:43:04.297 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.298 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.298 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.298 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger collector container image", 2023-10-11T16:43:04.299 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.299 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.299 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.300 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_jaeger_query": { 2023-10-11T16:43:04.300 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_jaeger_query", 2023-10-11T16:43:04.300 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.301 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.301 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.301 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-10-11T16:43:04.301 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.302 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.302 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.302 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Jaeger query container image", 2023-10-11T16:43:04.302 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.303 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.303 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.304 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_keepalived": { 2023-10-11T16:43:04.304 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_keepalived", 2023-10-11T16:43:04.304 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.304 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.305 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.305 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-10-11T16:43:04.305 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.305 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.306 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.306 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Keepalived container image", 2023-10-11T16:43:04.306 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.306 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.307 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.307 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_loki": { 2023-10-11T16:43:04.308 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_loki", 2023-10-11T16:43:04.308 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.308 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.308 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.309 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-10-11T16:43:04.309 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.309 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.310 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.310 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Loki container image", 2023-10-11T16:43:04.310 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.310 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.311 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.311 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_node_exporter": { 2023-10-11T16:43:04.312 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_node_exporter", 2023-10-11T16:43:04.312 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.312 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.312 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.313 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-10-11T16:43:04.313 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.313 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.313 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.314 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:04.314 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.314 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.315 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.315 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_prometheus": { 2023-10-11T16:43:04.315 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_prometheus", 2023-10-11T16:43:04.316 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.316 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.316 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.316 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-10-11T16:43:04.317 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.317 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.317 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.317 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus container image", 2023-10-11T16:43:04.318 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.318 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.318 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.319 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_promtail": { 2023-10-11T16:43:04.319 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_promtail", 2023-10-11T16:43:04.319 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.320 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.320 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.320 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-10-11T16:43:04.320 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.321 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.321 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.321 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Promtail container image", 2023-10-11T16:43:04.321 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.322 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.322 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.323 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image_snmp_gateway": { 2023-10-11T16:43:04.323 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_image_snmp_gateway", 2023-10-11T16:43:04.323 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.323 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.324 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.324 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-10-11T16:43:04.324 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.324 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.325 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.325 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "SNMP Gateway container image", 2023-10-11T16:43:04.325 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.325 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.326 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.326 INFO:teuthology.orchestra.run.smithi111.stdout: "container_init": { 2023-10-11T16:43:04.326 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "container_init", 2023-10-11T16:43:04.327 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.327 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.327 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.327 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.328 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.328 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.328 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.329 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Run podman/docker with `--init`", 2023-10-11T16:43:04.329 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.329 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.329 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.330 INFO:teuthology.orchestra.run.smithi111.stdout: "daemon_cache_timeout": { 2023-10-11T16:43:04.330 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "daemon_cache_timeout", 2023-10-11T16:43:04.330 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.331 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.331 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.331 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:04.332 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.332 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.332 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.332 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-10-11T16:43:04.333 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.333 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.333 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.334 INFO:teuthology.orchestra.run.smithi111.stdout: "default_cephadm_command_timeout": { 2023-10-11T16:43:04.334 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "default_cephadm_command_timeout", 2023-10-11T16:43:04.334 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.335 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.335 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.335 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "900", 2023-10-11T16:43:04.335 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.336 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.336 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.336 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-10-11T16:43:04.336 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.337 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.337 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "default_registry": { 2023-10-11T16:43:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "default_registry", 2023-10-11T16:43:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "docker.io", 2023-10-11T16:43:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-10-11T16:43:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "device_cache_timeout": { 2023-10-11T16:43:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_cache_timeout", 2023-10-11T16:43:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1800", 2023-10-11T16:43:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache device inventory", 2023-10-11T16:43:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "device_enhanced_scan": { 2023-10-11T16:43:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_enhanced_scan", 2023-10-11T16:43:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use libstoragemgmt during device scans", 2023-10-11T16:43:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "facts_cache_timeout": { 2023-10-11T16:43:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "facts_cache_timeout", 2023-10-11T16:43:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:04.351 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.351 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.351 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.352 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "seconds to cache host facts data", 2023-10-11T16:43:04.352 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.352 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.352 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.353 INFO:teuthology.orchestra.run.smithi111.stdout: "host_check_interval": { 2023-10-11T16:43:04.353 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "host_check_interval", 2023-10-11T16:43:04.353 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.354 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.354 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.354 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:04.355 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.355 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.355 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.355 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to perform a host check", 2023-10-11T16:43:04.356 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.356 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.356 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.357 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.357 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.357 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.357 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.358 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.359 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.360 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.361 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.362 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.362 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.363 INFO:teuthology.orchestra.run.smithi111.stdout: "log_refresh_metadata": { 2023-10-11T16:43:04.363 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_refresh_metadata", 2023-10-11T16:43:04.363 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.363 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.364 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.364 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.364 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.364 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.365 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.365 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-10-11T16:43:04.365 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.366 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.366 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.367 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.367 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.367 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.367 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.368 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.368 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.368 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.368 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.369 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.369 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-10-11T16:43:04.369 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.370 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.370 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.370 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.371 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.371 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.371 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.372 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.372 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.372 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.372 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.373 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.373 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.373 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.373 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.374 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.374 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.374 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.375 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.375 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.375 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.375 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.376 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.376 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.376 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.377 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.377 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.377 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.378 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.378 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.378 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.378 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.379 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.379 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.379 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.380 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.380 INFO:teuthology.orchestra.run.smithi111.stdout: "manage_etc_ceph_ceph_conf": { 2023-10-11T16:43:04.380 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-10-11T16:43:04.381 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.381 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.381 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.381 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.382 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.382 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.382 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.383 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-10-11T16:43:04.383 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.383 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.383 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.384 INFO:teuthology.orchestra.run.smithi111.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-10-11T16:43:04.384 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-10-11T16:43:04.384 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.385 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.385 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.385 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "*", 2023-10-11T16:43:04.386 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.386 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.386 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.386 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-10-11T16:43:04.387 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.387 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.387 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.388 INFO:teuthology.orchestra.run.smithi111.stdout: "max_count_per_host": { 2023-10-11T16:43:04.388 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_count_per_host", 2023-10-11T16:43:04.388 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.389 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.389 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.389 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:04.389 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.390 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.390 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.390 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "max number of daemons per service per host", 2023-10-11T16:43:04.390 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.391 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.391 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.392 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd_draining_count": { 2023-10-11T16:43:04.392 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_osd_draining_count", 2023-10-11T16:43:04.392 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.392 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.393 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.393 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:04.393 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.393 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.394 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.394 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-10-11T16:43:04.394 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.395 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.395 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.395 INFO:teuthology.orchestra.run.smithi111.stdout: "migration_current": { 2023-10-11T16:43:04.396 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "migration_current", 2023-10-11T16:43:04.396 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.396 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.397 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.397 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.397 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.397 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.398 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.398 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "internal - do not modify", 2023-10-11T16:43:04.398 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.399 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.399 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.399 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": { 2023-10-11T16:43:04.400 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mode", 2023-10-11T16:43:04.400 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.400 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.400 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.401 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "root", 2023-10-11T16:43:04.401 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.401 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.402 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.402 INFO:teuthology.orchestra.run.smithi111.stdout: "cephadm-package", 2023-10-11T16:43:04.402 INFO:teuthology.orchestra.run.smithi111.stdout: "root" 2023-10-11T16:43:04.402 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.403 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "mode for remote execution of cephadm", 2023-10-11T16:43:04.403 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.403 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.404 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.404 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_alerts_path": { 2023-10-11T16:43:04.404 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_alerts_path", 2023-10-11T16:43:04.405 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.405 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.405 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.405 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-10-11T16:43:04.406 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.406 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.406 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.407 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-10-11T16:43:04.407 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.407 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.407 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.408 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_web_password": { 2023-10-11T16:43:04.408 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_web_password", 2023-10-11T16:43:04.409 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.409 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.409 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.409 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus web password", 2023-10-11T16:43:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "prometheus_web_user": { 2023-10-11T16:43:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus_web_user", 2023-10-11T16:43:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Prometheus web user", 2023-10-11T16:43:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_insecure": { 2023-10-11T16:43:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_insecure", 2023-10-11T16:43:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-10-11T16:43:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_password": { 2023-10-11T16:43:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_password", 2023-10-11T16:43:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-10-11T16:43:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_url": { 2023-10-11T16:43:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_url", 2023-10-11T16:43:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-10-11T16:43:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "registry_username": { 2023-10-11T16:43:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "registry_username", 2023-10-11T16:43:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-10-11T16:43:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "secure_monitoring_stack": { 2023-10-11T16:43:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "secure_monitoring_stack", 2023-10-11T16:43:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-10-11T16:43:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "service_discovery_port": { 2023-10-11T16:43:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "service_discovery_port", 2023-10-11T16:43:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8765", 2023-10-11T16:43:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "cephadm service discovery port", 2023-10-11T16:43:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "ssh_config_file": { 2023-10-11T16:43:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssh_config_file", 2023-10-11T16:43:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-10-11T16:43:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "use_agent": { 2023-10-11T16:43:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "use_agent", 2023-10-11T16:43:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-10-11T16:43:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "use_repo_digest": { 2023-10-11T16:43:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "use_repo_digest", 2023-10-11T16:43:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-10-11T16:43:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_failed_host_check": { 2023-10-11T16:43:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_failed_host_check", 2023-10-11T16:43:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if the host check fails", 2023-10-11T16:43:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_stray_daemons": { 2023-10-11T16:43:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_stray_daemons", 2023-10-11T16:43:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-10-11T16:43:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_on_stray_hosts": { 2023-10-11T16:43:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_on_stray_hosts", 2023-10-11T16:43:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-10-11T16:43:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crash", 2023-10-11T16:43:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.482 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.482 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.482 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.482 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.483 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.483 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.483 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.484 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.484 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.484 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.484 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.485 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.485 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.485 INFO:teuthology.orchestra.run.smithi111.stdout: "retain_interval": { 2023-10-11T16:43:04.486 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "retain_interval", 2023-10-11T16:43:04.486 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.486 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.487 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.487 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "31536000", 2023-10-11T16:43:04.487 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.487 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.488 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.488 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long to retain crashes before pruning them", 2023-10-11T16:43:04.488 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.488 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.489 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.489 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_recent_interval": { 2023-10-11T16:43:04.490 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_recent_interval", 2023-10-11T16:43:04.490 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.490 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.490 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.491 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1209600", 2023-10-11T16:43:04.491 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.491 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.492 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.492 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "time interval in which to warn about recent crashes", 2023-10-11T16:43:04.492 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.492 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.493 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.493 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.493 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.494 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "dashboard", 2023-10-11T16:43:04.494 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.495 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.495 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.495 INFO:teuthology.orchestra.run.smithi111.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-10-11T16:43:04.495 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-10-11T16:43:04.496 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.496 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.496 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.497 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:04.497 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.497 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.497 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.498 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.498 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.498 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.499 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.499 INFO:teuthology.orchestra.run.smithi111.stdout: "ALERTMANAGER_API_HOST": { 2023-10-11T16:43:04.499 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ALERTMANAGER_API_HOST", 2023-10-11T16:43:04.500 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.500 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.500 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.500 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.501 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.501 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.501 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.502 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.502 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.502 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.502 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.503 INFO:teuthology.orchestra.run.smithi111.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-10-11T16:43:04.503 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-10-11T16:43:04.504 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.504 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.504 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.504 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.505 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.505 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.505 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.505 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.506 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.506 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.506 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.507 INFO:teuthology.orchestra.run.smithi111.stdout: "AUDIT_API_ENABLED": { 2023-10-11T16:43:04.507 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "AUDIT_API_ENABLED", 2023-10-11T16:43:04.507 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.508 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.508 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.508 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.509 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.509 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.509 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.509 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.510 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.510 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.510 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.511 INFO:teuthology.orchestra.run.smithi111.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-10-11T16:43:04.511 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-10-11T16:43:04.511 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.512 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.512 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.512 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.512 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.513 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.513 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.513 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.514 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.514 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.514 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.515 INFO:teuthology.orchestra.run.smithi111.stdout: "ENABLE_BROWSABLE_API": { 2023-10-11T16:43:04.515 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ENABLE_BROWSABLE_API", 2023-10-11T16:43:04.515 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.515 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.516 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.516 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.516 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.517 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.517 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.517 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.517 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.518 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.518 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.518 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-10-11T16:43:04.519 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-10-11T16:43:04.519 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.519 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.520 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.520 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.520 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.520 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.521 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.521 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.521 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.522 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.522 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.522 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-10-11T16:43:04.523 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-10-11T16:43:04.523 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.523 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.523 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.524 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.524 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.524 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.525 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.525 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.525 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.525 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.526 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.526 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-10-11T16:43:04.527 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-10-11T16:43:04.527 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.527 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.527 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.528 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.528 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.528 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.529 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.529 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.529 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.530 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.530 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.530 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-10-11T16:43:04.531 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-10-11T16:43:04.531 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.531 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.532 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.532 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.532 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.532 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.533 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.533 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.533 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.534 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.534 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.534 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_NFS": { 2023-10-11T16:43:04.535 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-10-11T16:43:04.535 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.535 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.535 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.536 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.536 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.536 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.537 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.537 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.537 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.537 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.538 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.538 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_RBD": { 2023-10-11T16:43:04.538 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-10-11T16:43:04.539 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.539 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.539 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.540 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.540 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.540 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.540 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.541 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.541 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.541 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.541 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.542 INFO:teuthology.orchestra.run.smithi111.stdout: "FEATURE_TOGGLE_RGW": { 2023-10-11T16:43:04.542 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-10-11T16:43:04.543 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.543 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.543 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.543 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.544 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.544 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.544 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.544 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.545 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.545 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.545 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.546 INFO:teuthology.orchestra.run.smithi111.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-10-11T16:43:04.546 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-10-11T16:43:04.546 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.547 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.547 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.547 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.547 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.548 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.548 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.548 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.548 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.549 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.549 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.550 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_PASSWORD": { 2023-10-11T16:43:04.550 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_PASSWORD", 2023-10-11T16:43:04.550 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.550 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.551 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.551 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:04.551 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.551 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.552 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.552 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.552 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.553 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.553 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.553 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-10-11T16:43:04.554 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-10-11T16:43:04.554 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.554 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.555 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.555 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.555 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.555 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.556 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.556 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.556 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.556 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.557 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.557 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_URL": { 2023-10-11T16:43:04.557 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_URL", 2023-10-11T16:43:04.558 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.558 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.558 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.559 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.559 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.559 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.559 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.560 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.560 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.560 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.560 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.561 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_API_USERNAME": { 2023-10-11T16:43:04.561 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_API_USERNAME", 2023-10-11T16:43:04.562 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.562 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.562 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.562 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:04.563 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.563 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.563 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.563 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.564 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.564 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.564 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.565 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-10-11T16:43:04.565 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-10-11T16:43:04.565 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.566 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.566 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.566 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.567 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.567 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.567 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.567 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.568 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.568 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.568 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.569 INFO:teuthology.orchestra.run.smithi111.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-10-11T16:43:04.569 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-10-11T16:43:04.569 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.570 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.570 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.570 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.570 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.571 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.571 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.571 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.572 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.572 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.572 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.573 INFO:teuthology.orchestra.run.smithi111.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-10-11T16:43:04.573 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-10-11T16:43:04.573 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.573 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.574 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.574 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.574 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.575 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.575 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.575 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.575 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.576 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.576 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.576 INFO:teuthology.orchestra.run.smithi111.stdout: "ISSUE_TRACKER_API_KEY": { 2023-10-11T16:43:04.577 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-10-11T16:43:04.577 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.577 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.578 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.578 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.578 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.578 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.579 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.579 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.579 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.580 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.580 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.580 INFO:teuthology.orchestra.run.smithi111.stdout: "PROMETHEUS_API_HOST": { 2023-10-11T16:43:04.581 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PROMETHEUS_API_HOST", 2023-10-11T16:43:04.581 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.581 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.581 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.582 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.582 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.582 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.583 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.583 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.583 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.583 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.584 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.584 INFO:teuthology.orchestra.run.smithi111.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-10-11T16:43:04.584 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-10-11T16:43:04.585 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.585 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.585 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.586 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.586 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.586 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.586 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.587 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.587 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.587 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.587 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.588 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-10-11T16:43:04.588 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-10-11T16:43:04.589 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.589 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.589 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.590 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.590 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.591 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.591 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.591 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.592 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.592 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.593 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.593 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-10-11T16:43:04.594 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-10-11T16:43:04.594 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.594 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.594 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.595 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.595 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.595 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.596 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.596 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.596 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.596 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.597 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.597 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-10-11T16:43:04.597 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-10-11T16:43:04.598 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.598 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.598 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.599 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.599 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.599 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.599 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.600 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.600 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.600 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.601 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.601 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-10-11T16:43:04.601 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-10-11T16:43:04.602 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.602 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.602 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.603 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.603 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.603 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.603 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.605 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.605 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.606 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.606 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.607 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-10-11T16:43:04.607 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-10-11T16:43:04.607 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.607 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.608 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.608 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.608 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.608 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.609 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.609 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.609 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.610 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.610 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.610 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-10-11T16:43:04.611 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-10-11T16:43:04.611 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.611 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.611 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.612 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.612 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.612 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.613 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.613 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.613 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.613 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.614 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.614 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-10-11T16:43:04.615 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-10-11T16:43:04.615 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.615 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.615 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.616 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.616 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.616 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.616 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.617 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.617 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.617 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.618 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.618 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.618 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_ENABLED": { 2023-10-11T16:43:04.618 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_ENABLED", 2023-10-11T16:43:04.619 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.619 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.619 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.620 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.620 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.620 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.620 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.621 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.621 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.621 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.621 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.622 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-10-11T16:43:04.622 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-10-11T16:43:04.623 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.623 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.623 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.623 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-10-11T16:43:04.624 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.624 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.624 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.624 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.625 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.625 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.625 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.626 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-10-11T16:43:04.626 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-10-11T16:43:04.626 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.626 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.627 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.627 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:04.627 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.627 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.628 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.628 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.628 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.628 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.629 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.629 INFO:teuthology.orchestra.run.smithi111.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-10-11T16:43:04.629 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-10-11T16:43:04.630 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.630 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.630 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.630 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8", 2023-10-11T16:43:04.631 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.631 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.631 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.632 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.632 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.632 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.632 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.633 INFO:teuthology.orchestra.run.smithi111.stdout: "REST_REQUESTS_TIMEOUT": { 2023-10-11T16:43:04.633 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-10-11T16:43:04.633 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.634 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.634 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.634 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "45", 2023-10-11T16:43:04.635 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.635 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.635 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.635 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.636 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.636 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.636 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.637 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_ACCESS_KEY": { 2023-10-11T16:43:04.637 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_ACCESS_KEY", 2023-10-11T16:43:04.637 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.637 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.638 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.638 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.638 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.639 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.639 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.639 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.639 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.640 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.640 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.640 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-10-11T16:43:04.641 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-10-11T16:43:04.641 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.641 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.642 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.642 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "admin", 2023-10-11T16:43:04.642 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.642 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.643 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.643 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.643 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.643 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.644 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.644 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_SECRET_KEY": { 2023-10-11T16:43:04.645 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_SECRET_KEY", 2023-10-11T16:43:04.645 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.645 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.645 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.646 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.646 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.646 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.647 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.647 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.647 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.647 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.648 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.648 INFO:teuthology.orchestra.run.smithi111.stdout: "RGW_API_SSL_VERIFY": { 2023-10-11T16:43:04.649 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "RGW_API_SSL_VERIFY", 2023-10-11T16:43:04.649 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.649 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.649 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.650 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.650 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.650 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.651 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.651 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.651 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.651 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.652 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.652 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.652 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-10-11T16:43:04.652 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-10-11T16:43:04.653 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.653 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.653 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.654 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:04.654 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.654 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.654 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.655 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.655 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.655 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.656 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.656 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-10-11T16:43:04.656 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-10-11T16:43:04.657 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.657 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.657 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.657 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:04.658 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.658 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.658 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.659 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.659 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.659 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.659 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.660 INFO:teuthology.orchestra.run.smithi111.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-10-11T16:43:04.660 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-10-11T16:43:04.661 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.661 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.661 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.661 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:04.662 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.662 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.662 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.663 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.663 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.663 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.663 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.664 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.664 INFO:teuthology.orchestra.run.smithi111.stdout: "cross_origin_url": { 2023-10-11T16:43:04.664 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cross_origin_url", 2023-10-11T16:43:04.665 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.665 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.665 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.665 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.666 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.666 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.666 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.667 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.667 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.667 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.667 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.668 INFO:teuthology.orchestra.run.smithi111.stdout: "crt_file": { 2023-10-11T16:43:04.668 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "crt_file", 2023-10-11T16:43:04.669 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.669 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.669 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.669 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.670 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.670 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.670 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.671 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.671 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.671 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.671 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.672 INFO:teuthology.orchestra.run.smithi111.stdout: "debug": { 2023-10-11T16:43:04.672 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "debug", 2023-10-11T16:43:04.672 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.673 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.673 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.673 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.673 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.674 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.674 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.674 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Enable/disable debug options", 2023-10-11T16:43:04.675 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.675 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.675 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.676 INFO:teuthology.orchestra.run.smithi111.stdout: "jwt_token_ttl": { 2023-10-11T16:43:04.676 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "jwt_token_ttl", 2023-10-11T16:43:04.676 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.676 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.677 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.677 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "28800", 2023-10-11T16:43:04.677 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.678 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.678 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.678 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.678 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.679 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.679 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.680 INFO:teuthology.orchestra.run.smithi111.stdout: "key_file": { 2023-10-11T16:43:04.680 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "key_file", 2023-10-11T16:43:04.680 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.680 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.681 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.681 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.681 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.681 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.682 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.682 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.682 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.682 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.683 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.683 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.684 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.684 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.684 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.684 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.685 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.685 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.685 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.685 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.686 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.686 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.686 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.687 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.687 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.687 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.687 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.688 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.688 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.688 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.688 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.689 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.689 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.689 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.690 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.690 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.690 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.691 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.691 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.691 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.691 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.692 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.692 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.692 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.693 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.693 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.693 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.694 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.694 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.694 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.694 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.695 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.695 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.695 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.696 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.696 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.696 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.696 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.697 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.697 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.697 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.697 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.698 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.698 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.699 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.699 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.699 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.699 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.700 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.700 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.700 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.700 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.701 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.701 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.701 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.702 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.702 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.702 INFO:teuthology.orchestra.run.smithi111.stdout: "motd": { 2023-10-11T16:43:04.703 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "motd", 2023-10-11T16:43:04.703 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.703 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.703 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.704 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.704 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.704 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.705 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.705 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "The message of the day", 2023-10-11T16:43:04.705 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.705 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.706 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.706 INFO:teuthology.orchestra.run.smithi111.stdout: "redirect_resolve_ip_addr": { 2023-10-11T16:43:04.707 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "redirect_resolve_ip_addr", 2023-10-11T16:43:04.707 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.707 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.707 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.708 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.708 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.708 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.709 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.709 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.709 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.709 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.710 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.710 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:04.710 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:04.711 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.711 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.711 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.712 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "::", 2023-10-11T16:43:04.712 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.712 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.712 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.713 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.713 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.713 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.714 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.714 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:04.714 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:04.715 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.715 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.715 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.716 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8080", 2023-10-11T16:43:04.716 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.716 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.716 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.717 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.717 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.717 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.718 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.718 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl": { 2023-10-11T16:43:04.718 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl", 2023-10-11T16:43:04.719 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.719 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.719 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.720 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.720 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.720 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.720 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.721 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.721 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.721 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.721 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.722 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl_server_port": { 2023-10-11T16:43:04.722 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl_server_port", 2023-10-11T16:43:04.723 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.723 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.723 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.723 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8443", 2023-10-11T16:43:04.724 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.724 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.724 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.724 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.725 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.725 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.725 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.726 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_behaviour": { 2023-10-11T16:43:04.726 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_behaviour", 2023-10-11T16:43:04.726 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.727 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.727 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.727 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "redirect", 2023-10-11T16:43:04.727 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.728 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.728 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.728 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.728 INFO:teuthology.orchestra.run.smithi111.stdout: "redirect" 2023-10-11T16:43:04.729 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.729 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.729 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.729 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.730 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.730 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_error_status_code": { 2023-10-11T16:43:04.731 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_error_status_code", 2023-10-11T16:43:04.731 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.731 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.731 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.732 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "500", 2023-10-11T16:43:04.732 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "400", 2023-10-11T16:43:04.732 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "599", 2023-10-11T16:43:04.733 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.733 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.733 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.733 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.733 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.734 INFO:teuthology.orchestra.run.smithi111.stdout: "url_prefix": { 2023-10-11T16:43:04.734 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "url_prefix", 2023-10-11T16:43:04.735 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.735 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.735 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.735 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.736 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.736 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.736 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.736 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.737 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.737 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.737 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.738 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.739 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "devicehealth", 2023-10-11T16:43:04.739 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.739 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.739 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.740 INFO:teuthology.orchestra.run.smithi111.stdout: "enable_monitoring": { 2023-10-11T16:43:04.740 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enable_monitoring", 2023-10-11T16:43:04.740 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.741 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.741 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.741 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.741 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.742 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.742 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.742 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "monitor device health metrics", 2023-10-11T16:43:04.742 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.743 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.743 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.744 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.744 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.744 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.744 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.745 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.745 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.745 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.745 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.746 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.746 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.746 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.747 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.747 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.747 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.747 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.748 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.748 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.748 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.749 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.749 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.749 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.750 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.750 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.750 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.751 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.751 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.751 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.751 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.752 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.752 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.752 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.752 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.753 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.753 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.754 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.754 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.754 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.754 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.755 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.755 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.755 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.755 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.756 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.756 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.756 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.757 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.757 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.757 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.757 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.758 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.758 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.758 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.759 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.759 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.759 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.760 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.760 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.760 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.760 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.761 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.761 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.761 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.761 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.762 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.762 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.762 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.763 INFO:teuthology.orchestra.run.smithi111.stdout: "mark_out_threshold": { 2023-10-11T16:43:04.763 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mark_out_threshold", 2023-10-11T16:43:04.763 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.764 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.764 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.764 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "2419200", 2023-10-11T16:43:04.765 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.765 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.765 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.765 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-10-11T16:43:04.766 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.766 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.766 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.767 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": { 2023-10-11T16:43:04.767 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pool_name", 2023-10-11T16:43:04.767 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.768 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.768 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.768 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "device_health_metrics", 2023-10-11T16:43:04.768 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.769 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.769 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.769 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "name of pool in which to store device health metrics", 2023-10-11T16:43:04.769 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.770 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.770 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.771 INFO:teuthology.orchestra.run.smithi111.stdout: "retention_period": { 2023-10-11T16:43:04.771 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "retention_period", 2023-10-11T16:43:04.771 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.771 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.772 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.772 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15552000", 2023-10-11T16:43:04.772 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.773 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.773 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.773 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long to retain device health metrics", 2023-10-11T16:43:04.773 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.774 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.774 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.775 INFO:teuthology.orchestra.run.smithi111.stdout: "scrape_frequency": { 2023-10-11T16:43:04.775 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "scrape_frequency", 2023-10-11T16:43:04.775 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.775 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.776 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.776 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "86400", 2023-10-11T16:43:04.776 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.777 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.777 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.777 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to scrape device health metrics", 2023-10-11T16:43:04.777 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.778 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.778 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.779 INFO:teuthology.orchestra.run.smithi111.stdout: "self_heal": { 2023-10-11T16:43:04.779 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "self_heal", 2023-10-11T16:43:04.779 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.779 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.780 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.780 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:04.780 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.781 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.781 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.781 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-10-11T16:43:04.781 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.782 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.782 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.782 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.782 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:04.783 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:04.783 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.783 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.783 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.784 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:04.784 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.784 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.785 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.785 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how frequently to wake up and check device health", 2023-10-11T16:43:04.785 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.785 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.786 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.786 INFO:teuthology.orchestra.run.smithi111.stdout: "warn_threshold": { 2023-10-11T16:43:04.787 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "warn_threshold", 2023-10-11T16:43:04.787 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.787 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.787 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.788 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "7257600", 2023-10-11T16:43:04.788 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.788 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.788 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.789 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-10-11T16:43:04.789 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.789 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.790 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.790 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.790 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.791 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "diskprediction_local", 2023-10-11T16:43:04.791 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.792 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.792 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.792 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.792 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.793 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.793 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.793 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.793 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.794 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.794 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.794 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.795 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.795 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.795 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.795 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.796 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.796 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.796 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.796 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.797 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.797 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.797 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.798 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.798 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.798 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.798 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.799 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.799 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.799 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.799 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.800 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.800 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.800 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.800 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.801 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.801 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.801 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.802 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.802 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.802 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.803 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.803 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.803 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.803 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.804 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.804 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.804 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.804 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.805 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.805 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.805 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.805 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.806 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.806 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.806 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.807 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.807 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.807 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.808 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.808 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.808 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.808 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.809 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.809 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.809 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.810 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.810 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.810 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "predict_interval": { 2023-10-11T16:43:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "predict_interval", 2023-10-11T16:43:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.812 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.812 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.812 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "86400", 2023-10-11T16:43:04.812 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.813 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.813 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.813 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.814 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.814 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.814 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.815 INFO:teuthology.orchestra.run.smithi111.stdout: "predictor_model": { 2023-10-11T16:43:04.815 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "predictor_model", 2023-10-11T16:43:04.815 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.815 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.816 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.816 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "prophetstor", 2023-10-11T16:43:04.816 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.817 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.817 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.817 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.817 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.818 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.818 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.818 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:04.819 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:04.819 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.819 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.820 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.820 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "600", 2023-10-11T16:43:04.820 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.820 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.821 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.821 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.821 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.821 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.822 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.822 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.822 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.823 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "influx", 2023-10-11T16:43:04.823 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": false, 2023-10-11T16:43:04.824 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "influxdb python module not found", 2023-10-11T16:43:04.824 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.824 INFO:teuthology.orchestra.run.smithi111.stdout: "batch_size": { 2023-10-11T16:43:04.824 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "batch_size", 2023-10-11T16:43:04.825 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.825 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.825 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.826 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5000", 2023-10-11T16:43:04.826 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.826 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.826 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.827 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-10-11T16:43:04.827 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.827 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.828 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.828 INFO:teuthology.orchestra.run.smithi111.stdout: "database": { 2023-10-11T16:43:04.828 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "database", 2023-10-11T16:43:04.829 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.829 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.829 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.829 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "ceph", 2023-10-11T16:43:04.830 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.830 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.830 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.831 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-10-11T16:43:04.831 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.831 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.831 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.832 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": { 2023-10-11T16:43:04.832 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "hostname", 2023-10-11T16:43:04.833 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.833 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.833 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.833 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.834 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.834 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.834 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.835 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB server hostname", 2023-10-11T16:43:04.835 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.835 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.835 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.836 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:04.836 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:04.836 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:04.837 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.837 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.837 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "30", 2023-10-11T16:43:04.838 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "5", 2023-10-11T16:43:04.838 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.838 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.838 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-10-11T16:43:04.839 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.839 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.839 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.840 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.840 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.840 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.841 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.842 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.842 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.842 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.843 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.843 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.843 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.843 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.844 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.844 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.844 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.845 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.845 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.845 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.846 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.846 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.846 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.846 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.847 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.847 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.847 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.848 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.848 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.848 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.848 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.849 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.849 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.849 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.850 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.850 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.850 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.851 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.851 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.851 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.851 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.852 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.852 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.852 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.852 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.853 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.853 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.853 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.854 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.854 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.854 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.854 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.855 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.855 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.856 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.856 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.856 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.856 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.857 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.857 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.857 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.858 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.858 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.858 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.858 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.859 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.859 INFO:teuthology.orchestra.run.smithi111.stdout: "password": { 2023-10-11T16:43:04.859 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "password", 2023-10-11T16:43:04.860 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.860 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.860 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.861 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.861 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.861 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.862 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.862 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "password of InfluxDB server user", 2023-10-11T16:43:04.862 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.862 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.863 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.863 INFO:teuthology.orchestra.run.smithi111.stdout: "port": { 2023-10-11T16:43:04.864 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "port", 2023-10-11T16:43:04.864 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.864 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.864 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.865 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "8086", 2023-10-11T16:43:04.865 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.866 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.866 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.866 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "InfluxDB server port", 2023-10-11T16:43:04.866 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.867 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.867 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.867 INFO:teuthology.orchestra.run.smithi111.stdout: "ssl": { 2023-10-11T16:43:04.868 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ssl", 2023-10-11T16:43:04.868 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.868 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.868 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.869 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "false", 2023-10-11T16:43:04.869 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.869 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.870 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.870 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-10-11T16:43:04.870 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.871 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.871 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.871 INFO:teuthology.orchestra.run.smithi111.stdout: "threads": { 2023-10-11T16:43:04.872 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "threads", 2023-10-11T16:43:04.872 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.872 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.872 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.873 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:04.873 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:04.873 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "32", 2023-10-11T16:43:04.873 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.874 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-10-11T16:43:04.874 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.874 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.875 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.875 INFO:teuthology.orchestra.run.smithi111.stdout: "username": { 2023-10-11T16:43:04.875 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "username", 2023-10-11T16:43:04.876 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.876 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.876 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.876 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.877 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.877 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.877 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.877 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "username of InfluxDB server user", 2023-10-11T16:43:04.878 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.878 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.878 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.879 INFO:teuthology.orchestra.run.smithi111.stdout: "verify_ssl": { 2023-10-11T16:43:04.879 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "verify_ssl", 2023-10-11T16:43:04.879 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.880 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.880 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.880 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "true", 2023-10-11T16:43:04.881 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.881 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.881 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.881 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-10-11T16:43:04.882 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.882 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.882 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.883 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.883 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.884 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "insights", 2023-10-11T16:43:04.884 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.884 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.885 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.885 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.885 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.885 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.886 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.886 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.886 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.886 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.887 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.887 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.887 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.888 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.888 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.888 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.888 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.889 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.889 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.889 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "iostat", 2023-10-11T16:43:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "k8sevents", 2023-10-11T16:43:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_event_retention_days": { 2023-10-11T16:43:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "ceph_event_retention_days", 2023-10-11T16:43:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "7", 2023-10-11T16:43:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Days to hold ceph event information within local cache", 2023-10-11T16:43:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "config_check_secs": { 2023-10-11T16:43:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "config_check_secs", 2023-10-11T16:43:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "10", 2023-10-11T16:43:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-10-11T16:43:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "localpool", 2023-10-11T16:43:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "failure_domain": { 2023-10-11T16:43:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "failure_domain", 2023-10-11T16:43:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "host", 2023-10-11T16:43:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "failure domain for any created local pool", 2023-10-11T16:43:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-10-11T16:43:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.975 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.975 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.975 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.976 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:04.976 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:04.976 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:04.976 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.977 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.977 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:04.977 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.977 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.978 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.978 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:04.978 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.978 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.979 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.979 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": { 2023-10-11T16:43:04.980 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "min_size", 2023-10-11T16:43:04.980 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.980 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.980 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.981 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.981 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.981 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.981 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.982 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default min_size for any created local pool", 2023-10-11T16:43:04.982 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-10-11T16:43:04.982 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.982 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.983 INFO:teuthology.orchestra.run.smithi111.stdout: "num_rep": { 2023-10-11T16:43:04.983 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "num_rep", 2023-10-11T16:43:04.983 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.984 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.984 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.984 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3", 2023-10-11T16:43:04.984 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.985 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.985 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.985 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default replica count for any created local pool", 2023-10-11T16:43:04.985 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.986 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.986 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.987 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": { 2023-10-11T16:43:04.987 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pg_num", 2023-10-11T16:43:04.987 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:04.987 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.988 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.988 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "128", 2023-10-11T16:43:04.988 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.989 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.989 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.989 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "default pg_num for any created local pool", 2023-10-11T16:43:04.989 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.990 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.990 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.991 INFO:teuthology.orchestra.run.smithi111.stdout: "prefix": { 2023-10-11T16:43:04.991 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prefix", 2023-10-11T16:43:04.991 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.991 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.992 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.992 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:04.992 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.993 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.993 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.993 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "name prefix for any created local pool", 2023-10-11T16:43:04.993 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:04.994 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.994 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.995 INFO:teuthology.orchestra.run.smithi111.stdout: "subtree": { 2023-10-11T16:43:04.995 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "subtree", 2023-10-11T16:43:04.995 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:04.995 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:04.996 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:04.996 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "rack", 2023-10-11T16:43:04.996 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:04.997 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:04.997 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:04.997 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "CRUSH level for which to create a local pool", 2023-10-11T16:43:04.997 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-10-11T16:43:04.998 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:04.998 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:04.998 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.999 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:04.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:04.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:04.999 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mds_autoscaler", 2023-10-11T16:43:05.000 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.000 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.000 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.001 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.001 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.001 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.001 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.002 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.002 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.002 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.003 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.003 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.003 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.003 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.004 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.004 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.004 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.004 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.005 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.005 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.005 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.006 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.006 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.006 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.007 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.007 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.007 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.008 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.008 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.008 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.008 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.009 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.009 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.009 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.010 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.010 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.011 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.011 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.011 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.011 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.012 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.012 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.012 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.012 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.013 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.013 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.013 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.014 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.014 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.014 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.014 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.015 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.015 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.017 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.017 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.018 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.018 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.019 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.019 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.019 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.019 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.020 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.020 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.020 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.021 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.021 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.021 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.021 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.022 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.022 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.022 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.023 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mirroring", 2023-10-11T16:43:05.023 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.024 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.024 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.024 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.024 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.025 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.025 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.025 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.025 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.026 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.026 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.026 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.027 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.027 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.027 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.027 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.028 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.028 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.028 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.029 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.029 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.029 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.029 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.030 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.030 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.031 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.031 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.031 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.031 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.032 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.032 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.032 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.032 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.033 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.033 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.033 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.034 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.034 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.034 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.035 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.035 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.035 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.036 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.036 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.036 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.036 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.037 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.037 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.037 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.037 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.038 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.038 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.038 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.038 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.039 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.039 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.040 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.040 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.040 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.040 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.041 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.041 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.041 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.041 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.042 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.042 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.042 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.043 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.043 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.043 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.043 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.044 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "nfs", 2023-10-11T16:43:05.045 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.045 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.045 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.045 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.046 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.046 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.046 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.047 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.047 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.047 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.047 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.048 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.048 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.048 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.048 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.049 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.049 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.049 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.050 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.050 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.050 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.050 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.051 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.051 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.052 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.052 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.052 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.052 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.053 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.053 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.053 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.053 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.054 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.054 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.054 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.054 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.055 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.055 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.055 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.056 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.056 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.056 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.057 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.057 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.057 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.057 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.058 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.058 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.058 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.058 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.059 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.059 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.059 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.060 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.060 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.060 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.061 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.061 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.061 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.062 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.062 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.062 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.062 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.063 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.063 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.063 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.064 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.064 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.064 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.064 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.065 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.066 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "orchestrator", 2023-10-11T16:43:05.066 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.066 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.067 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.067 INFO:teuthology.orchestra.run.smithi111.stdout: "fail_fs": { 2023-10-11T16:43:05.067 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "fail_fs", 2023-10-11T16:43:05.067 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.068 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.068 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.068 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.068 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.069 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.069 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.069 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-10-11T16:43:05.070 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.070 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.070 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.071 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.071 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.071 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.072 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.072 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.072 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.073 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.073 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.073 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.073 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.074 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.074 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.074 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.075 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.075 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.075 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.075 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.076 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.076 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.076 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.077 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.077 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.077 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.078 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.078 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.078 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.078 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.079 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.079 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.079 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.080 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.080 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.080 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.081 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.081 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.081 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.081 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.082 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.082 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.082 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.083 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.083 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.083 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.083 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.084 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.084 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.084 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.085 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.085 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.085 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.085 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.086 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.086 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.087 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.087 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.087 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.088 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.088 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.088 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.088 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.089 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.089 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.089 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.089 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.090 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.090 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.091 INFO:teuthology.orchestra.run.smithi111.stdout: "orchestrator": { 2023-10-11T16:43:05.091 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "orchestrator", 2023-10-11T16:43:05.091 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.091 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.092 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.092 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.092 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.093 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.093 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.093 INFO:teuthology.orchestra.run.smithi111.stdout: "cephadm", 2023-10-11T16:43:05.093 INFO:teuthology.orchestra.run.smithi111.stdout: "rook", 2023-10-11T16:43:05.094 INFO:teuthology.orchestra.run.smithi111.stdout: "test_orchestrator" 2023-10-11T16:43:05.094 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.094 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Orchestrator backend", 2023-10-11T16:43:05.095 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.095 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.095 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.095 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.096 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.097 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "osd_perf_query", 2023-10-11T16:43:05.097 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.097 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.097 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.098 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.098 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.098 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.098 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.099 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.099 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.099 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.100 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.100 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.100 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.100 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.101 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.101 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.101 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.102 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.102 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.102 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.102 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.103 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.103 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.103 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.104 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.104 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.104 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.104 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.105 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.105 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.105 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.106 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.106 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.106 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.106 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.107 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.107 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.108 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.108 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.108 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.108 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.109 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.109 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.109 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.110 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.110 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.110 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.110 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.111 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.111 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.111 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.111 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.112 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.112 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.112 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.113 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.113 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.113 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.114 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.114 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.114 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.115 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.115 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.115 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.115 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.116 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.116 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.116 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.116 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.118 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "osd_support", 2023-10-11T16:43:05.118 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.118 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.119 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.119 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.119 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.119 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.120 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.120 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.120 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.121 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.121 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.121 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.121 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.122 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.122 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.122 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.123 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.123 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.123 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.123 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.124 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.124 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.124 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.125 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.125 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.125 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.126 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.126 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.126 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.126 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.127 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.127 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.127 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.127 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.128 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.128 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.129 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.129 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.129 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.129 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.130 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.130 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.130 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.131 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.131 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.131 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.132 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.132 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.132 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.132 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.133 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.133 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.133 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.133 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.134 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.134 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.135 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.135 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.135 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.135 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.136 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.136 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.136 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.136 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.137 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.137 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.137 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.137 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.138 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.139 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "pg_autoscaler", 2023-10-11T16:43:05.139 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.140 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.140 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.140 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.141 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.141 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.141 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.141 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.142 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.142 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.142 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.143 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.143 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.143 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.143 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.144 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.144 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.144 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.144 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.145 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.145 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.145 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.146 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.146 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.146 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.147 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.147 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.147 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.148 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.148 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.148 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.148 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.149 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.149 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.149 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.149 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.150 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.150 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.151 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.151 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.151 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.151 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.152 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.152 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.152 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.152 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.153 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.153 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.153 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.153 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.154 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.154 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.154 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.155 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.155 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.155 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.156 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.156 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.156 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.157 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.157 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.157 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.157 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.158 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.158 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.158 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.159 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.159 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.159 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.160 INFO:teuthology.orchestra.run.smithi111.stdout: "noautoscale": { 2023-10-11T16:43:05.160 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "noautoscale", 2023-10-11T16:43:05.160 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.160 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.161 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.161 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.161 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.162 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.162 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.162 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "global autoscale flag", 2023-10-11T16:43:05.162 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-10-11T16:43:05.163 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.163 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.163 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:05.164 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:05.164 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:05.164 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.165 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.165 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:05.165 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.165 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.166 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.166 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.166 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.167 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.167 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.167 INFO:teuthology.orchestra.run.smithi111.stdout: "threshold": { 2023-10-11T16:43:05.168 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "threshold", 2023-10-11T16:43:05.168 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:05.168 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.168 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.169 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "3.0", 2023-10-11T16:43:05.169 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1.0", 2023-10-11T16:43:05.169 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.170 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.170 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "scaling threshold", 2023-10-11T16:43:05.170 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-10-11T16:43:05.170 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.172 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "progress", 2023-10-11T16:43:05.172 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.172 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.173 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.173 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_pg_recovery_event": { 2023-10-11T16:43:05.173 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_pg_recovery_event", 2023-10-11T16:43:05.173 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.174 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.174 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.174 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.175 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.175 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.175 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.175 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow the module to show pg recovery progress", 2023-10-11T16:43:05.176 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.176 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.176 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.177 INFO:teuthology.orchestra.run.smithi111.stdout: "enabled": { 2023-10-11T16:43:05.177 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enabled", 2023-10-11T16:43:05.177 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.177 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.178 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.178 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:05.178 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.178 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.179 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.179 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.179 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.180 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.180 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.180 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.181 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.181 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.181 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.182 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.182 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.182 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.183 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.183 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.183 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.183 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.184 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.184 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.184 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.185 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.185 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.185 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.185 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.186 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.186 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.186 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.187 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.187 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.187 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.188 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.188 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.188 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.188 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.189 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.189 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.189 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.190 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.190 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.190 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.191 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.191 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.191 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.192 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.192 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.192 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.192 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.193 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.193 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.193 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.194 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.194 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.194 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.194 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.195 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.195 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.195 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.196 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.196 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.196 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.197 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.197 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.197 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.198 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.198 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.198 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.198 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.199 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.199 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.199 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.200 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.200 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.200 INFO:teuthology.orchestra.run.smithi111.stdout: "max_completed_events": { 2023-10-11T16:43:05.201 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_completed_events", 2023-10-11T16:43:05.201 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.201 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.201 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.202 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "50", 2023-10-11T16:43:05.202 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.202 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.203 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.203 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "number of past completed events to remember", 2023-10-11T16:43:05.203 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.203 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.204 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.204 INFO:teuthology.orchestra.run.smithi111.stdout: "sleep_interval": { 2023-10-11T16:43:05.204 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "sleep_interval", 2023-10-11T16:43:05.205 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:05.205 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.205 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.205 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "5", 2023-10-11T16:43:05.206 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.206 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.206 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.206 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "how long the module is going to sleep", 2023-10-11T16:43:05.207 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.207 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.207 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.208 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.208 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.209 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "prometheus", 2023-10-11T16:43:05.209 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.209 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.210 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.210 INFO:teuthology.orchestra.run.smithi111.stdout: "cache": { 2023-10-11T16:43:05.210 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "cache", 2023-10-11T16:43:05.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.211 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.211 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.211 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:05.212 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.212 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.212 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.212 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.213 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.213 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.213 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.214 INFO:teuthology.orchestra.run.smithi111.stdout: "exclude_perf_counters": { 2023-10-11T16:43:05.214 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "exclude_perf_counters", 2023-10-11T16:43:05.214 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.215 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.215 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.215 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:05.215 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.216 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.216 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.216 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-10-11T16:43:05.216 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-10-11T16:43:05.217 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.217 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.218 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.218 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.218 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.218 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.219 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.219 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.219 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.221 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.221 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.221 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.222 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.222 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.222 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.223 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.223 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.223 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.223 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.224 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.224 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.224 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.225 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.225 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.225 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.226 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.226 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.226 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.227 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.227 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.227 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.227 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.228 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.228 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.228 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.229 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.229 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.229 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.230 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.230 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.230 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.230 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.231 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.231 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.231 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.232 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.232 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.232 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.232 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.233 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.233 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.233 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.233 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.234 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.234 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.235 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.235 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.235 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.236 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.236 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.236 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.236 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.237 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.237 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.237 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.238 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.238 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.238 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.239 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_stats_pools": { 2023-10-11T16:43:05.239 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_stats_pools", 2023-10-11T16:43:05.239 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.239 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.240 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.240 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.240 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.240 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.241 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.241 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.241 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.241 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.242 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.242 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_stats_pools_refresh_interval": { 2023-10-11T16:43:05.243 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-10-11T16:43:05.243 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.243 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.243 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.244 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "300", 2023-10-11T16:43:05.244 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.244 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.245 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.245 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.245 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.245 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.246 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.246 INFO:teuthology.orchestra.run.smithi111.stdout: "scrape_interval": { 2023-10-11T16:43:05.246 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "scrape_interval", 2023-10-11T16:43:05.247 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:05.247 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.247 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.247 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15.0", 2023-10-11T16:43:05.248 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.248 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.248 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.249 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.249 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.249 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.249 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.250 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:05.250 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:05.251 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.251 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.251 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.251 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "::", 2023-10-11T16:43:05.252 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.252 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.252 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.252 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-10-11T16:43:05.253 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.253 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.253 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.254 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:05.254 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:05.254 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.255 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.255 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.255 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "9283", 2023-10-11T16:43:05.256 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.256 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.256 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.256 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-10-11T16:43:05.257 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.257 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.257 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.258 INFO:teuthology.orchestra.run.smithi111.stdout: "stale_cache_strategy": { 2023-10-11T16:43:05.258 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "stale_cache_strategy", 2023-10-11T16:43:05.258 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.259 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.259 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.259 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "log", 2023-10-11T16:43:05.260 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.260 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.260 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.260 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.261 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.261 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.261 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.262 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_behaviour": { 2023-10-11T16:43:05.262 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_behaviour", 2023-10-11T16:43:05.262 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.263 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.263 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.263 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "default", 2023-10-11T16:43:05.263 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.264 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.264 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.264 INFO:teuthology.orchestra.run.smithi111.stdout: "default", 2023-10-11T16:43:05.265 INFO:teuthology.orchestra.run.smithi111.stdout: "error" 2023-10-11T16:43:05.265 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.265 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.265 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.266 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.266 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.267 INFO:teuthology.orchestra.run.smithi111.stdout: "standby_error_status_code": { 2023-10-11T16:43:05.267 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "standby_error_status_code", 2023-10-11T16:43:05.267 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.267 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.268 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.268 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "500", 2023-10-11T16:43:05.268 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "400", 2023-10-11T16:43:05.269 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "599", 2023-10-11T16:43:05.269 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.269 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.269 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.270 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.270 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.270 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.271 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.271 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_support", 2023-10-11T16:43:05.272 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.272 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.272 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.273 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.273 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.273 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.273 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.274 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.274 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.274 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.275 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.275 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.275 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.275 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.276 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.276 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.276 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.276 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.277 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.277 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.277 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.278 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.278 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.279 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.279 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.279 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.279 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.280 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.280 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.280 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.281 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.281 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.281 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.281 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.282 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.282 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.282 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.283 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.283 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.283 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.284 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.284 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.284 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.284 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.285 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.285 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.285 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.286 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.286 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.286 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.286 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.287 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.287 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.287 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.288 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.288 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.288 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.289 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.289 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.289 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.289 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.290 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.290 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.290 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.291 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.291 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.291 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.291 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.292 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.292 INFO:teuthology.orchestra.run.smithi111.stdout: "max_concurrent_snap_create": { 2023-10-11T16:43:05.292 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_concurrent_snap_create", 2023-10-11T16:43:05.293 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.293 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.293 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.294 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10", 2023-10-11T16:43:05.294 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.294 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.294 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.295 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.295 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.295 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.296 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.296 INFO:teuthology.orchestra.run.smithi111.stdout: "mirror_snapshot_schedule": { 2023-10-11T16:43:05.296 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mirror_snapshot_schedule", 2023-10-11T16:43:05.297 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.297 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.297 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.298 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.298 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.298 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.299 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.299 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.299 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.299 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.300 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.300 INFO:teuthology.orchestra.run.smithi111.stdout: "trash_purge_schedule": { 2023-10-11T16:43:05.300 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "trash_purge_schedule", 2023-10-11T16:43:05.301 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.301 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.301 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.302 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.302 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.302 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.302 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.303 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.303 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.303 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.304 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.304 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.304 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.305 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "restful", 2023-10-11T16:43:05.305 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.305 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.306 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.306 INFO:teuthology.orchestra.run.smithi111.stdout: "enable_auth": { 2023-10-11T16:43:05.306 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enable_auth", 2023-10-11T16:43:05.307 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.307 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.307 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.307 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:05.308 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.308 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.308 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.308 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.309 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.309 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.309 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.310 INFO:teuthology.orchestra.run.smithi111.stdout: "key_file": { 2023-10-11T16:43:05.310 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "key_file", 2023-10-11T16:43:05.311 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.311 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.311 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.311 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.312 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.312 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.312 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.312 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.313 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.313 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.313 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.314 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.314 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.314 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.314 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.315 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.315 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.315 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.316 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.316 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.316 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.316 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.317 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.317 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.317 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.318 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.318 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.318 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.318 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.319 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.319 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.320 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.320 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.320 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.320 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.321 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.321 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.321 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.322 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.322 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.322 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.322 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.323 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.323 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.323 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.324 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.324 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.324 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.324 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.325 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.325 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.325 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.326 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.326 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.326 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.326 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.327 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.327 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.327 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.328 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.328 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.328 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.328 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.329 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.329 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.330 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.330 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.330 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.330 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.331 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.331 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.331 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.331 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.332 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.332 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.332 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.333 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.333 INFO:teuthology.orchestra.run.smithi111.stdout: "server_addr": { 2023-10-11T16:43:05.333 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_addr", 2023-10-11T16:43:05.334 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.334 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.334 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.335 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.335 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.335 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.335 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.336 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.336 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.336 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.337 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.337 INFO:teuthology.orchestra.run.smithi111.stdout: "server_port": { 2023-10-11T16:43:05.337 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "server_port", 2023-10-11T16:43:05.338 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.338 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.338 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.339 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.339 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.339 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.339 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.340 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.340 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.340 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.340 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.341 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.341 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.342 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rgw", 2023-10-11T16:43:05.342 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.342 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.343 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.343 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.343 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.343 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.344 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.344 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.344 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.344 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.345 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.345 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.345 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.346 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.346 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.346 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.346 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.347 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.347 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.347 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.348 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.348 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.348 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.349 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.349 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.349 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.349 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.350 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.350 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.350 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.351 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.351 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.351 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.351 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.352 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.352 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.353 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.353 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.353 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.353 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.354 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.354 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.354 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.355 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.355 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.355 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.355 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.356 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.356 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.356 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.356 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.357 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.357 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.357 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.357 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.358 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.358 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.359 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.359 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.359 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.359 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.360 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.360 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.360 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.360 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.361 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.361 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.361 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.361 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.362 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.362 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.363 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rook", 2023-10-11T16:43:05.363 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.363 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.364 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.364 INFO:teuthology.orchestra.run.smithi111.stdout: "drive_group_interval": { 2023-10-11T16:43:05.364 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "drive_group_interval", 2023-10-11T16:43:05.365 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:05.365 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.365 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.365 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "300.0", 2023-10-11T16:43:05.366 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.366 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.366 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.367 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-10-11T16:43:05.367 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.367 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.368 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.368 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.369 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.369 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.369 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.369 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.370 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.370 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.370 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.371 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.371 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.371 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.371 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.372 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.372 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.372 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.373 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.373 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.373 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.373 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.374 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.374 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.375 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.375 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.375 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.375 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.376 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.376 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.376 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.377 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.377 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.377 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.377 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.378 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.378 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.378 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.379 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.379 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.379 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.380 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.380 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.380 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.380 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.381 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.381 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.381 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.381 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.382 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.382 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.382 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.383 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.383 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.383 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.383 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.384 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.384 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.385 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.385 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.385 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.385 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.386 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.386 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.386 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.386 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.387 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.387 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.387 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.388 INFO:teuthology.orchestra.run.smithi111.stdout: "storage_class": { 2023-10-11T16:43:05.388 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "storage_class", 2023-10-11T16:43:05.388 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.389 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.389 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.389 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "local", 2023-10-11T16:43:05.390 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.390 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.390 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.390 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-10-11T16:43:05.391 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.391 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.391 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.392 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.392 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.393 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "selftest", 2023-10-11T16:43:05.393 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.393 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.393 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.394 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.394 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.394 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.395 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.395 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.395 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.395 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.396 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.396 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.396 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.396 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "roption1": { 2023-10-11T16:43:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "roption1", 2023-10-11T16:43:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "roption2": { 2023-10-11T16:43:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "roption2", 2023-10-11T16:43:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "xyz", 2023-10-11T16:43:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption1": { 2023-10-11T16:43:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption1", 2023-10-11T16:43:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption2": { 2023-10-11T16:43:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption2", 2023-10-11T16:43:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption3": { 2023-10-11T16:43:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption3", 2023-10-11T16:43:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "float", 2023-10-11T16:43:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.432 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption4": { 2023-10-11T16:43:05.432 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption4", 2023-10-11T16:43:05.432 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.433 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.433 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.433 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption5": { 2023-10-11T16:43:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption5", 2023-10-11T16:43:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption6": { 2023-10-11T16:43:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption6", 2023-10-11T16:43:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "rwoption7": { 2023-10-11T16:43:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rwoption7", 2023-10-11T16:43:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "1", 2023-10-11T16:43:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "42", 2023-10-11T16:43:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "testkey": { 2023-10-11T16:43:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testkey", 2023-10-11T16:43:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "testlkey": { 2023-10-11T16:43:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testlkey", 2023-10-11T16:43:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "testnewline": { 2023-10-11T16:43:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "testnewline", 2023-10-11T16:43:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "snap_schedule", 2023-10-11T16:43:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_m_granularity": { 2023-10-11T16:43:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "allow_m_granularity", 2023-10-11T16:43:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "allow minute scheduled snapshots", 2023-10-11T16:43:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: "dump_on_update": { 2023-10-11T16:43:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "dump_on_update", 2023-10-11T16:43:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "dump database to debug log on update", 2023-10-11T16:43:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.537 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.538 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.538 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.538 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.539 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.539 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.539 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.539 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.540 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.540 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.540 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.541 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.542 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.542 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.543 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.543 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.543 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.543 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.544 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.544 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.544 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.545 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.545 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.545 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.546 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.546 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.547 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.547 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.547 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.548 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.548 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.548 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.548 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.549 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.549 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.549 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.550 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.550 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.550 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.550 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.551 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.551 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.551 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.552 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.552 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.552 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.553 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.553 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.554 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.554 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.554 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.555 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.555 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.555 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.556 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.556 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.556 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.556 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.557 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.557 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.557 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.558 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "stats", 2023-10-11T16:43:05.559 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.559 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.559 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.559 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.560 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.560 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.560 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.561 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.561 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.561 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.561 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.562 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.562 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.562 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.563 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.563 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.563 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.563 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.564 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.564 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.564 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.565 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.565 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.565 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.566 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.566 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.566 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.567 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.567 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.567 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.567 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.568 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.568 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.568 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.569 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.569 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.569 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.570 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.570 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.570 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.571 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.571 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.571 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.572 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.572 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.572 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.573 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.573 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.573 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.573 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.574 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.574 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.574 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.575 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.575 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.575 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.576 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.576 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.576 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.577 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.577 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.577 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.577 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.578 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.578 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.578 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.579 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.579 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.579 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.579 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.580 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.581 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "status", 2023-10-11T16:43:05.581 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.581 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.581 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.582 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.582 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.582 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.583 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.583 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.583 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.583 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.584 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.584 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.584 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.584 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.585 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.585 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.585 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.586 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.586 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.586 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.586 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.587 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.587 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.588 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.588 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.588 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.588 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.589 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.589 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.589 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.590 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.590 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.590 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.590 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.591 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.591 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.592 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.592 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.592 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.592 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.593 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.593 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.593 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.593 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.594 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.594 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.594 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.595 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.595 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.595 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.595 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.596 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.596 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.596 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.597 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.597 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.597 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.598 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.598 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.598 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.599 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.599 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.599 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.599 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.600 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.600 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.600 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.600 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.601 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.601 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.601 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.602 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "telegraf", 2023-10-11T16:43:05.602 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.603 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.603 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.603 INFO:teuthology.orchestra.run.smithi111.stdout: "address": { 2023-10-11T16:43:05.604 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "address", 2023-10-11T16:43:05.604 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.604 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.604 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.605 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-10-11T16:43:05.605 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.605 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.606 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.606 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.606 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.606 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.607 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.607 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:05.608 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:05.608 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:05.608 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.608 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.609 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "15", 2023-10-11T16:43:05.609 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.609 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.610 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.610 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.610 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.610 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.611 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.611 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.612 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.612 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.612 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.612 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.613 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.613 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.613 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.614 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.614 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.614 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.614 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.615 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.615 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.615 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.615 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.616 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.616 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.616 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.617 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.617 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.617 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.618 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.618 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.618 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.619 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.619 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.619 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.619 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.620 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.620 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.620 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.621 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.621 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.622 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.622 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.622 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.622 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.623 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.623 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.623 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.624 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.624 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.624 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.624 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.625 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.626 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.627 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.627 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.628 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.628 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.628 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.628 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.629 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.629 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.629 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.630 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.631 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.631 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.631 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.631 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.634 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "telemetry", 2023-10-11T16:43:05.635 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.635 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.635 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.636 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_basic": { 2023-10-11T16:43:05.636 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_basic", 2023-10-11T16:43:05.636 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.637 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.637 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.637 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:05.638 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.638 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.638 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.639 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share basic cluster information (size, version)", 2023-10-11T16:43:05.639 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.639 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.639 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.640 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_crash": { 2023-10-11T16:43:05.640 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_crash", 2023-10-11T16:43:05.641 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.641 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.641 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.642 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:05.642 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.642 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.642 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.643 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-10-11T16:43:05.643 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.643 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.643 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.644 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_device": { 2023-10-11T16:43:05.644 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_device", 2023-10-11T16:43:05.645 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.645 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.645 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.645 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "True", 2023-10-11T16:43:05.646 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.646 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.646 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.646 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-10-11T16:43:05.647 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.647 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.647 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.648 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_ident": { 2023-10-11T16:43:05.648 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_ident", 2023-10-11T16:43:05.648 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.649 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.649 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.649 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.649 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.650 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.650 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.650 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-10-11T16:43:05.651 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.651 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.651 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.651 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.652 INFO:teuthology.orchestra.run.smithi111.stdout: "channel_perf": { 2023-10-11T16:43:05.652 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "channel_perf", 2023-10-11T16:43:05.652 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.653 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.653 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.653 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.654 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.654 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.654 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.654 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Share various performance metrics of a cluster", 2023-10-11T16:43:05.655 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.655 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.655 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.656 INFO:teuthology.orchestra.run.smithi111.stdout: "contact": { 2023-10-11T16:43:05.656 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "contact", 2023-10-11T16:43:05.656 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.657 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.657 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.657 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.657 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.658 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.658 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.658 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.659 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.659 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.659 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.659 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.660 INFO:teuthology.orchestra.run.smithi111.stdout: "description": { 2023-10-11T16:43:05.660 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "description", 2023-10-11T16:43:05.660 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.661 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.661 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.661 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.661 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.662 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.662 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.662 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.663 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.663 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.663 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.663 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.664 INFO:teuthology.orchestra.run.smithi111.stdout: "device_url": { 2023-10-11T16:43:05.664 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "device_url", 2023-10-11T16:43:05.664 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.664 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.665 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.665 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-10-11T16:43:05.665 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.666 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "enabled": { 2023-10-11T16:43:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "enabled", 2023-10-11T16:43:05.668 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.669 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.670 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.671 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:05.672 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.673 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "24", 2023-10-11T16:43:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "8", 2023-10-11T16:43:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.674 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.675 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "last_opt_revision": { 2023-10-11T16:43:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "last_opt_revision", 2023-10-11T16:43:05.676 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "1", 2023-10-11T16:43:05.677 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.678 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "leaderboard": { 2023-10-11T16:43:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "leaderboard", 2023-10-11T16:43:05.680 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.681 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.682 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.684 INFO:teuthology.orchestra.run.smithi111.stdout: "leaderboard_description": { 2023-10-11T16:43:05.684 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "leaderboard_description", 2023-10-11T16:43:05.684 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.684 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.685 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.685 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.685 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.686 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.686 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.686 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.686 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.687 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.687 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.688 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.688 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.688 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.688 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.689 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.689 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.689 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.690 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.690 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.690 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.690 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.691 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.691 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.691 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.692 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.692 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.692 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.692 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.693 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.693 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.694 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.694 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.694 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.694 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.695 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.695 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.695 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.696 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.696 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.696 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.697 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.697 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.697 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.698 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.698 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.698 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.699 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.699 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.699 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.700 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.700 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.700 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.701 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.701 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.701 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.702 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.702 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.702 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.703 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.703 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.703 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.704 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.704 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.705 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.705 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.705 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.705 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.706 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.706 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.706 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.707 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.707 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.707 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.708 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.708 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.708 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.709 INFO:teuthology.orchestra.run.smithi111.stdout: "organization": { 2023-10-11T16:43:05.709 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "organization", 2023-10-11T16:43:05.709 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.709 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.710 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.710 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.710 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.711 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.711 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.711 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.711 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.712 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.712 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.713 INFO:teuthology.orchestra.run.smithi111.stdout: "proxy": { 2023-10-11T16:43:05.713 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "proxy", 2023-10-11T16:43:05.713 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.713 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.714 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.714 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.714 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.715 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.715 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.715 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.715 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.716 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.716 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.717 INFO:teuthology.orchestra.run.smithi111.stdout: "url": { 2023-10-11T16:43:05.717 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "url", 2023-10-11T16:43:05.717 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.717 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.718 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.718 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-10-11T16:43:05.718 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.719 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.719 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.719 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.719 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.720 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.720 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.721 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.721 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "test_orchestrator", 2023-10-11T16:43:05.722 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.722 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.722 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.723 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.723 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.723 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.723 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.724 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.724 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.724 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.725 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.725 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.725 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.725 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.726 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.726 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.726 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.727 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.727 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.727 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.727 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.728 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.728 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.729 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.729 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.729 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.729 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.730 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.730 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.730 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.731 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.731 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.731 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.731 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.732 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.732 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.733 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.733 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.733 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.733 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.734 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.734 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.734 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.735 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.735 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.735 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.735 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.736 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.736 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.736 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.737 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.737 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.737 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.737 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.738 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.738 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.739 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.739 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.739 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.740 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.740 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.740 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.740 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.741 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.741 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.741 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.742 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.742 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.742 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.743 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.743 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.744 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "volumes", 2023-10-11T16:43:05.744 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.744 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.745 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.745 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.745 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.746 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.746 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.746 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.746 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.747 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.747 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.747 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.748 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.748 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.748 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.748 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.749 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.749 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.749 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.749 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.750 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.750 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.750 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.751 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.751 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.751 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.752 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.752 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.752 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.753 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.753 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.753 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.754 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.754 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.754 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.755 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.755 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.756 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.756 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.756 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.757 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.757 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.757 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.757 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.758 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.758 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.758 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.759 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.759 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.759 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.759 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.760 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.760 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.760 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.761 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.761 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.761 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.762 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.762 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.762 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.763 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.763 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.763 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.764 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.764 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.764 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.764 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.765 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.765 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.766 INFO:teuthology.orchestra.run.smithi111.stdout: "max_concurrent_clones": { 2023-10-11T16:43:05.766 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "max_concurrent_clones", 2023-10-11T16:43:05.766 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.767 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.767 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.767 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "4", 2023-10-11T16:43:05.767 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.768 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.768 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.768 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Number of asynchronous cloner threads", 2023-10-11T16:43:05.769 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.769 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.769 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.770 INFO:teuthology.orchestra.run.smithi111.stdout: "snapshot_clone_delay": { 2023-10-11T16:43:05.770 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "snapshot_clone_delay", 2023-10-11T16:43:05.770 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.771 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.771 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.771 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "0", 2023-10-11T16:43:05.771 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.772 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.772 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.772 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-10-11T16:43:05.773 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.773 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.773 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.773 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.774 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.774 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix", 2023-10-11T16:43:05.775 INFO:teuthology.orchestra.run.smithi111.stdout: "can_run": true, 2023-10-11T16:43:05.775 INFO:teuthology.orchestra.run.smithi111.stdout: "error_string": "", 2023-10-11T16:43:05.775 INFO:teuthology.orchestra.run.smithi111.stdout: "module_options": { 2023-10-11T16:43:05.776 INFO:teuthology.orchestra.run.smithi111.stdout: "discovery_interval": { 2023-10-11T16:43:05.776 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "discovery_interval", 2023-10-11T16:43:05.776 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "uint", 2023-10-11T16:43:05.777 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.777 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.777 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "100", 2023-10-11T16:43:05.777 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.778 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.778 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.778 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.779 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.779 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.779 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.780 INFO:teuthology.orchestra.run.smithi111.stdout: "identifier": { 2023-10-11T16:43:05.780 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "identifier", 2023-10-11T16:43:05.781 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.781 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.781 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.781 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.782 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.782 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.782 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.783 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.783 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.783 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.784 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.784 INFO:teuthology.orchestra.run.smithi111.stdout: "interval": { 2023-10-11T16:43:05.785 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "interval", 2023-10-11T16:43:05.785 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "secs", 2023-10-11T16:43:05.785 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.785 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.786 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "60", 2023-10-11T16:43:05.786 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.786 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.787 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.787 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.787 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.788 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.788 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.788 INFO:teuthology.orchestra.run.smithi111.stdout: "log_level": { 2023-10-11T16:43:05.789 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_level", 2023-10-11T16:43:05.789 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.789 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.790 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.790 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.790 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.790 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.791 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.791 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.791 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.792 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.792 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.792 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.792 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.793 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.793 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.793 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.794 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.794 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.795 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster": { 2023-10-11T16:43:05.795 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster", 2023-10-11T16:43:05.795 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.795 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.796 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.796 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.796 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.797 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.797 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.797 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.797 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.798 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.798 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.799 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_cluster_level": { 2023-10-11T16:43:05.799 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_cluster_level", 2023-10-11T16:43:05.799 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.799 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.800 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.800 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "info", 2023-10-11T16:43:05.800 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.801 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.801 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [ 2023-10-11T16:43:05.801 INFO:teuthology.orchestra.run.smithi111.stdout: "", 2023-10-11T16:43:05.801 INFO:teuthology.orchestra.run.smithi111.stdout: "critical", 2023-10-11T16:43:05.802 INFO:teuthology.orchestra.run.smithi111.stdout: "debug", 2023-10-11T16:43:05.802 INFO:teuthology.orchestra.run.smithi111.stdout: "error", 2023-10-11T16:43:05.802 INFO:teuthology.orchestra.run.smithi111.stdout: "info", 2023-10-11T16:43:05.803 INFO:teuthology.orchestra.run.smithi111.stdout: "warning" 2023-10-11T16:43:05.803 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.803 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.803 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.804 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.804 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.805 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": { 2023-10-11T16:43:05.805 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "log_to_file", 2023-10-11T16:43:05.805 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "bool", 2023-10-11T16:43:05.806 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.806 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2023-10-11T16:43:05.806 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "False", 2023-10-11T16:43:05.806 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.807 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.807 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.807 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.807 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.808 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.808 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.809 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_host": { 2023-10-11T16:43:05.809 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_host", 2023-10-11T16:43:05.809 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.809 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.810 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.810 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "", 2023-10-11T16:43:05.810 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.811 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.811 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.811 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.811 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.812 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.812 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.813 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_port": { 2023-10-11T16:43:05.813 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_port", 2023-10-11T16:43:05.813 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "int", 2023-10-11T16:43:05.813 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.814 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.814 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "10051", 2023-10-11T16:43:05.814 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.815 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.815 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.815 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.815 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.816 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.816 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.817 INFO:teuthology.orchestra.run.smithi111.stdout: "zabbix_sender": { 2023-10-11T16:43:05.817 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "zabbix_sender", 2023-10-11T16:43:05.817 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "str", 2023-10-11T16:43:05.817 INFO:teuthology.orchestra.run.smithi111.stdout: "level": "advanced", 2023-10-11T16:43:05.818 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 0, 2023-10-11T16:43:05.818 INFO:teuthology.orchestra.run.smithi111.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-10-11T16:43:05.818 INFO:teuthology.orchestra.run.smithi111.stdout: "min": "", 2023-10-11T16:43:05.818 INFO:teuthology.orchestra.run.smithi111.stdout: "max": "", 2023-10-11T16:43:05.819 INFO:teuthology.orchestra.run.smithi111.stdout: "enum_allowed": [], 2023-10-11T16:43:05.819 INFO:teuthology.orchestra.run.smithi111.stdout: "desc": "", 2023-10-11T16:43:05.819 INFO:teuthology.orchestra.run.smithi111.stdout: "long_desc": "", 2023-10-11T16:43:05.820 INFO:teuthology.orchestra.run.smithi111.stdout: "tags": [], 2023-10-11T16:43:05.820 INFO:teuthology.orchestra.run.smithi111.stdout: "see_also": [] 2023-10-11T16:43:05.820 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.821 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.821 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.821 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.821 INFO:teuthology.orchestra.run.smithi111.stdout: "services": { 2023-10-11T16:43:05.822 INFO:teuthology.orchestra.run.smithi111.stdout: "dashboard": "https://172.21.15.111:8443/" 2023-10-11T16:43:05.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.822 INFO:teuthology.orchestra.run.smithi111.stdout: "always_on_modules": { 2023-10-11T16:43:05.823 INFO:teuthology.orchestra.run.smithi111.stdout: "octopus": [ 2023-10-11T16:43:05.823 INFO:teuthology.orchestra.run.smithi111.stdout: "balancer", 2023-10-11T16:43:05.823 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2023-10-11T16:43:05.823 INFO:teuthology.orchestra.run.smithi111.stdout: "devicehealth", 2023-10-11T16:43:05.824 INFO:teuthology.orchestra.run.smithi111.stdout: "orchestrator", 2023-10-11T16:43:05.824 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscaler", 2023-10-11T16:43:05.824 INFO:teuthology.orchestra.run.smithi111.stdout: "progress", 2023-10-11T16:43:05.824 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_support", 2023-10-11T16:43:05.825 INFO:teuthology.orchestra.run.smithi111.stdout: "status", 2023-10-11T16:43:05.825 INFO:teuthology.orchestra.run.smithi111.stdout: "telemetry", 2023-10-11T16:43:05.825 INFO:teuthology.orchestra.run.smithi111.stdout: "volumes" 2023-10-11T16:43:05.826 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.826 INFO:teuthology.orchestra.run.smithi111.stdout: "pacific": [ 2023-10-11T16:43:05.826 INFO:teuthology.orchestra.run.smithi111.stdout: "balancer", 2023-10-11T16:43:05.827 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2023-10-11T16:43:05.827 INFO:teuthology.orchestra.run.smithi111.stdout: "devicehealth", 2023-10-11T16:43:05.827 INFO:teuthology.orchestra.run.smithi111.stdout: "orchestrator", 2023-10-11T16:43:05.827 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscaler", 2023-10-11T16:43:05.828 INFO:teuthology.orchestra.run.smithi111.stdout: "progress", 2023-10-11T16:43:05.828 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_support", 2023-10-11T16:43:05.828 INFO:teuthology.orchestra.run.smithi111.stdout: "status", 2023-10-11T16:43:05.829 INFO:teuthology.orchestra.run.smithi111.stdout: "telemetry", 2023-10-11T16:43:05.829 INFO:teuthology.orchestra.run.smithi111.stdout: "volumes" 2023-10-11T16:43:05.829 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.829 INFO:teuthology.orchestra.run.smithi111.stdout: "quincy": [ 2023-10-11T16:43:05.830 INFO:teuthology.orchestra.run.smithi111.stdout: "balancer", 2023-10-11T16:43:05.830 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2023-10-11T16:43:05.830 INFO:teuthology.orchestra.run.smithi111.stdout: "devicehealth", 2023-10-11T16:43:05.831 INFO:teuthology.orchestra.run.smithi111.stdout: "orchestrator", 2023-10-11T16:43:05.831 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscaler", 2023-10-11T16:43:05.831 INFO:teuthology.orchestra.run.smithi111.stdout: "progress", 2023-10-11T16:43:05.832 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_support", 2023-10-11T16:43:05.832 INFO:teuthology.orchestra.run.smithi111.stdout: "status", 2023-10-11T16:43:05.832 INFO:teuthology.orchestra.run.smithi111.stdout: "telemetry", 2023-10-11T16:43:05.832 INFO:teuthology.orchestra.run.smithi111.stdout: "volumes" 2023-10-11T16:43:05.833 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2023-10-11T16:43:05.833 INFO:teuthology.orchestra.run.smithi111.stdout: "reef": [ 2023-10-11T16:43:05.833 INFO:teuthology.orchestra.run.smithi111.stdout: "balancer", 2023-10-11T16:43:05.833 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2023-10-11T16:43:05.834 INFO:teuthology.orchestra.run.smithi111.stdout: "devicehealth", 2023-10-11T16:43:05.834 INFO:teuthology.orchestra.run.smithi111.stdout: "orchestrator", 2023-10-11T16:43:05.834 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscaler", 2023-10-11T16:43:05.835 INFO:teuthology.orchestra.run.smithi111.stdout: "progress", 2023-10-11T16:43:05.835 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_support", 2023-10-11T16:43:05.835 INFO:teuthology.orchestra.run.smithi111.stdout: "status", 2023-10-11T16:43:05.836 INFO:teuthology.orchestra.run.smithi111.stdout: "telemetry", 2023-10-11T16:43:05.836 INFO:teuthology.orchestra.run.smithi111.stdout: "volumes" 2023-10-11T16:43:05.836 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:05.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.837 INFO:teuthology.orchestra.run.smithi111.stdout: "last_failure_osd_epoch": 92, 2023-10-11T16:43:05.837 INFO:teuthology.orchestra.run.smithi111.stdout: "active_clients": [ 2023-10-11T16:43:05.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.838 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "devicehealth", 2023-10-11T16:43:05.838 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2023-10-11T16:43:05.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.839 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2023-10-11T16:43:05.841 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:0", 2023-10-11T16:43:05.841 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 879160294 2023-10-11T16:43:05.842 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.842 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:05.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.843 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "libcephsqlite", 2023-10-11T16:43:05.843 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2023-10-11T16:43:05.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.844 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2023-10-11T16:43:05.844 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:0", 2023-10-11T16:43:05.844 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1104121023 2023-10-11T16:43:05.845 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.845 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:05.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.846 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "rbd_support", 2023-10-11T16:43:05.846 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2023-10-11T16:43:05.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.847 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2023-10-11T16:43:05.847 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:0", 2023-10-11T16:43:05.847 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1789253424 2023-10-11T16:43:05.848 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.848 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:05.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.849 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "snap_schedule", 2023-10-11T16:43:05.849 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2023-10-11T16:43:05.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.850 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2023-10-11T16:43:05.850 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:0", 2023-10-11T16:43:05.850 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1571315532 2023-10-11T16:43:05.851 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.851 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:05.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.852 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "snap_schedule", 2023-10-11T16:43:05.852 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2023-10-11T16:43:05.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.853 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2023-10-11T16:43:05.853 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:0", 2023-10-11T16:43:05.853 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1352557956 2023-10-11T16:43:05.853 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.854 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:05.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:05.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.855 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "volumes", 2023-10-11T16:43:05.855 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2023-10-11T16:43:05.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2023-10-11T16:43:05.856 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2023-10-11T16:43:05.856 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:0", 2023-10-11T16:43:05.856 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2291413636 2023-10-11T16:43:05.857 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.857 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:05.857 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:05.858 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2023-10-11T16:43:05.858 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:43:05.860 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:01 smithi150 ceph-mon[132217]: pgmap v323: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-11T16:43:05.861 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:01 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1056440865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:43:05.861 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:01 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/835224480' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-10-11T16:43:05.861 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:03 smithi150 ceph-mon[132217]: pgmap v324: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:05.862 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:05 smithi150 ceph-mon[132217]: pgmap v325: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:43:05.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:01 smithi130 ceph-mon[131859]: pgmap v323: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-11T16:43:05.862 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:01 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1056440865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:43:05.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:01 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/835224480' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-10-11T16:43:05.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:03 smithi130 ceph-mon[131859]: pgmap v324: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:05.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:05 smithi130 ceph-mon[131859]: pgmap v325: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:43:05.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff0680a1340 msgr2=0x7ff0680a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:05.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff0680a1340 0x7ff0680a37f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff07c199b90 tx=0x7ff06c0073d0 comp rx=0 tx=0).stop 2023-10-11T16:43:05.864 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 msgr2=0x7ff07c198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:05.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 0x7ff07c198580 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff078009990 tx=0x7ff07800f690 comp rx=0 tx=0).stop 2023-10-11T16:43:05.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 shutdown_connections 2023-10-11T16:43:05.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7ff0680a1340 0x7ff0680a37f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:05.865 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff07c105550 0x7ff07c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:05.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff07c100510 0x7ff07c198580 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:05.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 --2- 172.21.15.111:0/835224480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff07c100eb0 0x7ff07c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:05.866 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 >> 172.21.15.111:0/835224480 conn(0x7ff07c077050 msgr2=0x7ff07c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:05.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 shutdown_connections 2023-10-11T16:43:05.867 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:00.821+0000 7ff0845ff700 1 -- 172.21.15.111:0/835224480 wait complete. 2023-10-11T16:43:05.876 DEBUG:tasks.check_counter:Getting stats from a 2023-10-11T16:43:05.877 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.a perf dump 2023-10-11T16:43:05.878 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:01 smithi111 ceph-mon[98527]: pgmap v323: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-11T16:43:05.878 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:01 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1056440865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:43:05.879 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:01 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/835224480' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-10-11T16:43:05.879 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:03 smithi111 ceph-mon[98527]: pgmap v324: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:05.879 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:05 smithi111 ceph-mon[98527]: pgmap v325: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-11T16:43:06.762 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:43:07.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:07 smithi111 ceph-mon[98527]: pgmap v326: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2023-10-11T16:43:07.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:07 smithi150 ceph-mon[132217]: pgmap v326: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2023-10-11T16:43:07.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:07 smithi130 ceph-mon[131859]: pgmap v326: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 50 KiB/s wr, 22 op/s 2023-10-11T16:43:09.507 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:43:09.508 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:09.508 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 0, 2023-10-11T16:43:09.508 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 2, 2023-10-11T16:43:09.508 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 0, 2023-10-11T16:43:09.509 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 0, 2023-10-11T16:43:09.509 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2023-10-11T16:43:09.509 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 0, 2023-10-11T16:43:09.510 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.003311391, 2023-10-11T16:43:09.510 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.000000000, 2023-10-11T16:43:09.510 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.000409762, 2023-10-11T16:43:09.510 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:09.511 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:09.511 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:09.511 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:09.512 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:09.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.512 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:09.512 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:09.513 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:09.513 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:09.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.514 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-11T16:43:09.514 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2023-10-11T16:43:09.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.515 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:09.515 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 221, 2023-10-11T16:43:09.515 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 211, 2023-10-11T16:43:09.515 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1421440, 2023-10-11T16:43:09.516 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 120352, 2023-10-11T16:43:09.516 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 2, 2023-10-11T16:43:09.516 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 1, 2023-10-11T16:43:09.516 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.050369990, 2023-10-11T16:43:09.517 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.018895725, 2023-10-11T16:43:09.517 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.035231511, 2023-10-11T16:43:09.517 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.005008930, 2023-10-11T16:43:09.518 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:09.518 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 210, 2023-10-11T16:43:09.518 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.016106783, 2023-10-11T16:43:09.518 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000076698 2023-10-11T16:43:09.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.519 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:09.519 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:09.520 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:09.520 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:09.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.520 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1421440, 2023-10-11T16:43:09.521 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 120352 2023-10-11T16:43:09.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.521 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:09.521 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 6, 2023-10-11T16:43:09.522 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 164, 2023-10-11T16:43:09.522 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 3312, 2023-10-11T16:43:09.523 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 180688, 2023-10-11T16:43:09.523 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:09.524 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2023-10-11T16:43:09.524 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.019034510, 2023-10-11T16:43:09.524 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.012365123, 2023-10-11T16:43:09.524 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.003157895, 2023-10-11T16:43:09.525 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:09.525 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:09.525 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 162, 2023-10-11T16:43:09.526 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.020364302, 2023-10-11T16:43:09.526 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000125705 2023-10-11T16:43:09.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.526 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:09.527 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:09.527 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:09.527 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:09.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.528 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-10-11T16:43:09.528 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 180688 2023-10-11T16:43:09.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.529 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2023-10-11T16:43:09.529 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 0, 2023-10-11T16:43:09.529 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2023-10-11T16:43:09.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.530 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2023-10-11T16:43:09.530 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:09.531 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:09.531 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:09.531 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:09.532 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:09.532 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:09.532 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:09.533 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:09.533 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:09.533 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:09.533 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:09.534 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:09.534 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:09.534 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:09.535 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:09.535 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:09.535 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:09.535 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:09.536 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:09.536 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:09.536 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:09.536 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:09.537 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:09.537 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:09.537 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:09.537 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:09.538 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:09.538 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:09.538 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:09.541 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:09.542 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:09.542 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2023-10-11T16:43:09.542 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:09.542 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:09.543 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:09.543 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:09.543 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 150048, 2023-10-11T16:43:09.544 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 32, 2023-10-11T16:43:09.544 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:09.544 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:09.544 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2023-10-11T16:43:09.545 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2023-10-11T16:43:09.545 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:09.545 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:09.545 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:09.546 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:09.546 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 0, 2023-10-11T16:43:09.546 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 0, 2023-10-11T16:43:09.547 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:09.547 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:09.547 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:09.547 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2023-10-11T16:43:09.548 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 928, 2023-10-11T16:43:09.548 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 2, 2023-10-11T16:43:09.548 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:09.549 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2023-10-11T16:43:09.549 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:09.549 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2023-10-11T16:43:09.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:09.550 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:09.550 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2023-10-11T16:43:09.550 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2023-10-11T16:43:09.550 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2023-10-11T16:43:09.551 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 223, 2023-10-11T16:43:09.551 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1390255, 2023-10-11T16:43:09.551 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:09.552 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 223, 2023-10-11T16:43:09.552 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2023-10-11T16:43:09.552 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2023-10-11T16:43:09.552 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 223, 2023-10-11T16:43:09.553 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1390255, 2023-10-11T16:43:09.553 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2023-10-11T16:43:09.553 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:09.554 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:09.554 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:09.554 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:09.554 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:09.555 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:43:09.626 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:09 smithi111 ceph-mon[98527]: pgmap v327: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2023-10-11T16:43:09.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:09 smithi150 ceph-mon[132217]: pgmap v327: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2023-10-11T16:43:09.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:09 smithi130 ceph-mon[131859]: pgmap v327: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 19 op/s 2023-10-11T16:43:09.965 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2023-10-11T16:43:09.966 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2023-10-11T16:43:09.966 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2023-10-11T16:43:09.967 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2023-10-11T16:43:09.967 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2023-10-11T16:43:09.967 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2023-10-11T16:43:09.968 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2023-10-11T16:43:09.968 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2023-10-11T16:43:09.968 DEBUG:tasks.check_counter:Getting stats from d 2023-10-11T16:43:09.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.d perf dump 2023-10-11T16:43:10.813 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:43:11.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:11 smithi111 ceph-mon[98527]: pgmap v328: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2023-10-11T16:43:11.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:11 smithi150 ceph-mon[132217]: pgmap v328: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2023-10-11T16:43:11.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:11 smithi130 ceph-mon[131859]: pgmap v328: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2023-10-11T16:43:12.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:12 smithi111 ceph-mon[98527]: osdmap e102: 12 total, 12 up, 12 in 2023-10-11T16:43:12.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:12 smithi150 ceph-mon[132217]: osdmap e102: 12 total, 12 up, 12 in 2023-10-11T16:43:12.792 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:43:12.792 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:12.792 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 0, 2023-10-11T16:43:12.793 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 2, 2023-10-11T16:43:12.793 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 0, 2023-10-11T16:43:12.793 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 0, 2023-10-11T16:43:12.794 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2023-10-11T16:43:12.794 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 0, 2023-10-11T16:43:12.794 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.003405363, 2023-10-11T16:43:12.795 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.000000000, 2023-10-11T16:43:12.795 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.000778039, 2023-10-11T16:43:12.795 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:12.796 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:12.796 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:12.796 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:12.796 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:12.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.797 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:12.797 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:12.797 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:12.798 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:12.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.798 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-11T16:43:12.799 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2023-10-11T16:43:12.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.799 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:12.799 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 230, 2023-10-11T16:43:12.800 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 218, 2023-10-11T16:43:12.800 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1425792, 2023-10-11T16:43:12.800 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 123760, 2023-10-11T16:43:12.801 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 2, 2023-10-11T16:43:12.801 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 1, 2023-10-11T16:43:12.801 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.053860435, 2023-10-11T16:43:12.801 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.021220110, 2023-10-11T16:43:12.802 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.035467055, 2023-10-11T16:43:12.802 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.004935506, 2023-10-11T16:43:12.802 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:12.802 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 218, 2023-10-11T16:43:12.803 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.019450345, 2023-10-11T16:43:12.803 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000089221 2023-10-11T16:43:12.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.804 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:12.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:12.804 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:12.804 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:12.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.805 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1425792, 2023-10-11T16:43:12.805 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 123760 2023-10-11T16:43:12.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.806 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:12.806 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 2, 2023-10-11T16:43:12.806 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 165, 2023-10-11T16:43:12.807 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 320, 2023-10-11T16:43:12.807 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 180768, 2023-10-11T16:43:12.807 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:12.808 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2023-10-11T16:43:12.808 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.018637401, 2023-10-11T16:43:12.808 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.012342015, 2023-10-11T16:43:12.808 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.003096467, 2023-10-11T16:43:12.809 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:12.809 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:12.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 162, 2023-10-11T16:43:12.809 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.023728282, 2023-10-11T16:43:12.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000146470 2023-10-11T16:43:12.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.810 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:12.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:12.811 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:12.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:12.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.812 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-11T16:43:12.812 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 180768 2023-10-11T16:43:12.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.812 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2023-10-11T16:43:12.813 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 0, 2023-10-11T16:43:12.813 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2023-10-11T16:43:12.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.813 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2023-10-11T16:43:12.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:12.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:12.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:12.814 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:12.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:12.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:12.815 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:12.816 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:12.818 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:12.818 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:12.818 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:12.819 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:12.819 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:12.819 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:12.819 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:12.820 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:12.820 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:12.820 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:12.820 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:12.821 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:12.821 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:12.821 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:12.821 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:12.822 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:12.822 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:12.822 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:12.823 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:12.823 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:12.823 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:12.823 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:12.824 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:12.824 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2023-10-11T16:43:12.824 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:12.824 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:12.825 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:12.825 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:12.825 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 148977, 2023-10-11T16:43:12.825 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 30, 2023-10-11T16:43:12.826 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:12.826 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:12.826 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2023-10-11T16:43:12.827 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2023-10-11T16:43:12.827 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:12.827 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:12.827 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:12.828 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:12.828 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 0, 2023-10-11T16:43:12.828 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 0, 2023-10-11T16:43:12.829 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:12.829 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:12.829 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:12.829 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2023-10-11T16:43:12.830 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 928, 2023-10-11T16:43:12.830 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 2, 2023-10-11T16:43:12.830 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:12.831 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2023-10-11T16:43:12.831 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:12.831 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2023-10-11T16:43:12.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:12.832 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:12.832 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2023-10-11T16:43:12.832 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2023-10-11T16:43:12.833 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2023-10-11T16:43:12.833 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 228, 2023-10-11T16:43:12.833 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1390905, 2023-10-11T16:43:12.833 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:12.834 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 228, 2023-10-11T16:43:12.834 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2023-10-11T16:43:12.834 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2023-10-11T16:43:12.835 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 228, 2023-10-11T16:43:12.835 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1390905, 2023-10-11T16:43:12.835 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2023-10-11T16:43:12.835 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:12.836 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:12.836 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:12.836 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:12.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:12.837 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:43:12.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:12 smithi130 ceph-mon[131859]: osdmap e102: 12 total, 12 up, 12 in 2023-10-11T16:43:13.303 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2023-10-11T16:43:13.303 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2023-10-11T16:43:13.303 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2023-10-11T16:43:13.304 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2023-10-11T16:43:13.304 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2023-10-11T16:43:13.304 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2023-10-11T16:43:13.305 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2023-10-11T16:43:13.305 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2023-10-11T16:43:13.305 DEBUG:tasks.check_counter:Getting stats from g 2023-10-11T16:43:13.306 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.g perf dump 2023-10-11T16:43:13.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:13 smithi111 ceph-mon[98527]: pgmap v330: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:43:13.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:13 smithi150 ceph-mon[132217]: pgmap v330: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:43:13.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:13 smithi130 ceph-mon[131859]: pgmap v330: 129 pgs: 129 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:43:14.599 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:43:15.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:15 smithi111 ceph-mon[98527]: pgmap v331: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:43:15.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:15 smithi150 ceph-mon[132217]: pgmap v331: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:43:15.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:15 smithi130 ceph-mon[131859]: pgmap v331: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-11T16:43:16.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:16 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:43:16.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:16 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:43:16.804 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:43:16.804 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:16.805 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 0, 2023-10-11T16:43:16.805 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 3, 2023-10-11T16:43:16.806 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 0, 2023-10-11T16:43:16.806 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 0, 2023-10-11T16:43:16.806 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:16.807 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 0, 2023-10-11T16:43:16.807 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.003860398, 2023-10-11T16:43:16.807 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.000000000, 2023-10-11T16:43:16.807 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.001054566, 2023-10-11T16:43:16.808 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:16.808 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:16.808 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:16.809 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:16.809 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:16.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.810 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:16.810 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:16.811 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:16.811 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:16.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.811 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-11T16:43:16.812 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2023-10-11T16:43:16.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.812 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:16.813 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 232, 2023-10-11T16:43:16.813 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 220, 2023-10-11T16:43:16.813 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1563680, 2023-10-11T16:43:16.813 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 123056, 2023-10-11T16:43:16.814 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2023-10-11T16:43:16.814 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 2, 2023-10-11T16:43:16.814 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.056101964, 2023-10-11T16:43:16.815 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.020309228, 2023-10-11T16:43:16.815 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.040873127, 2023-10-11T16:43:16.815 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.005388951, 2023-10-11T16:43:16.815 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:16.816 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 219, 2023-10-11T16:43:16.816 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.018138032, 2023-10-11T16:43:16.816 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000082822 2023-10-11T16:43:16.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.817 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:16.817 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:16.817 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:16.818 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:16.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.818 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1563680, 2023-10-11T16:43:16.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 123056 2023-10-11T16:43:16.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.819 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:16.819 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 2, 2023-10-11T16:43:16.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 163, 2023-10-11T16:43:16.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 320, 2023-10-11T16:43:16.820 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 182592, 2023-10-11T16:43:16.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:16.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 2, 2023-10-11T16:43:16.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.018638281, 2023-10-11T16:43:16.821 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.012658244, 2023-10-11T16:43:16.822 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.002544313, 2023-10-11T16:43:16.822 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:16.822 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:16.822 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 160, 2023-10-11T16:43:16.823 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.021997030, 2023-10-11T16:43:16.823 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000137481 2023-10-11T16:43:16.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.824 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:16.824 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:16.824 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:16.824 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:16.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.825 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-11T16:43:16.825 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 182592 2023-10-11T16:43:16.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.826 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2023-10-11T16:43:16.826 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 0, 2023-10-11T16:43:16.826 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2023-10-11T16:43:16.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.827 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2023-10-11T16:43:16.827 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:16.827 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:16.828 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:16.828 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:16.828 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:16.829 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:16.829 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:16.829 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:16.829 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:16.830 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:16.830 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:16.830 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:16.830 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:16.831 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:16.831 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:16.831 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:16.832 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:16.832 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:16.832 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:16.832 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:16.833 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:16.833 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:16.833 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:16.833 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:16.834 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:16.834 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:16.834 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:16.835 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:16.835 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:16.835 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:16.835 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:16.836 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2023-10-11T16:43:16.836 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:16.836 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:16.836 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:16.837 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:16.837 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 148939, 2023-10-11T16:43:16.837 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 29, 2023-10-11T16:43:16.838 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:16.838 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:16.838 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2023-10-11T16:43:16.838 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2023-10-11T16:43:16.839 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:16.839 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:16.839 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:16.840 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:16.840 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 0, 2023-10-11T16:43:16.840 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 0, 2023-10-11T16:43:16.840 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:16.841 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:16.841 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:16.841 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2023-10-11T16:43:16.841 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 928, 2023-10-11T16:43:16.842 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 2, 2023-10-11T16:43:16.842 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:16.842 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2023-10-11T16:43:16.843 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:16.843 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2023-10-11T16:43:16.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:16.843 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:16.844 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2023-10-11T16:43:16.844 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2023-10-11T16:43:16.844 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2023-10-11T16:43:16.844 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 230, 2023-10-11T16:43:16.845 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1528532, 2023-10-11T16:43:16.845 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:16.845 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 230, 2023-10-11T16:43:16.846 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2023-10-11T16:43:16.846 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2023-10-11T16:43:16.846 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 230, 2023-10-11T16:43:16.846 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1528532, 2023-10-11T16:43:16.847 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2023-10-11T16:43:16.847 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:16.847 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:16.847 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:16.848 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:16.848 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:16.848 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:43:16.849 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:16 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-11T16:43:17.299 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2023-10-11T16:43:17.300 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2023-10-11T16:43:17.300 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2023-10-11T16:43:17.300 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2023-10-11T16:43:17.301 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2023-10-11T16:43:17.301 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2023-10-11T16:43:17.301 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2023-10-11T16:43:17.301 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2023-10-11T16:43:17.302 DEBUG:tasks.check_counter:Getting stats from j 2023-10-11T16:43:17.302 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.j perf dump 2023-10-11T16:43:17.637 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:17 smithi111 ceph-mon[98527]: pgmap v332: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:17.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:17 smithi150 ceph-mon[132217]: pgmap v332: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:17.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:17 smithi130 ceph-mon[131859]: pgmap v332: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:18.651 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-11T16:43:19.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:19 smithi130 ceph-mon[131859]: pgmap v333: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:19.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:19 smithi111 ceph-mon[98527]: pgmap v333: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:19.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:19 smithi150 ceph-mon[132217]: pgmap v333: 129 pgs: 72 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:20.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:20 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:20 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:20 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:20 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:20 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:20 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:20 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:20 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:20 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:20 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:20 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:20.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:20 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:22.060 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: pgmap v334: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:22.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.061 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.062 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.062 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.062 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.063 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.063 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.063 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:21 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.085 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: pgmap v334: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:22.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.087 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.087 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.087 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.088 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.088 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:21 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: pgmap v334: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-11T16:43:22.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:21 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:22.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:22 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi130 to 196.2M 2023-10-11T16:43:22.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:22 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi150 to 196.2M 2023-10-11T16:43:22.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:22 smithi111 ceph-mon[98527]: Unable to set osd_memory_target on smithi130 to 205751705: error parsing value: Value '205751705' is below minimum 939524096 2023-10-11T16:43:22.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:22 smithi111 ceph-mon[98527]: Unable to set osd_memory_target on smithi150 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-11T16:43:23.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:22 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi130 to 196.2M 2023-10-11T16:43:23.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:22 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi150 to 196.2M 2023-10-11T16:43:23.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:22 smithi130 ceph-mon[131859]: Unable to set osd_memory_target on smithi130 to 205751705: error parsing value: Value '205751705' is below minimum 939524096 2023-10-11T16:43:23.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:22 smithi130 ceph-mon[131859]: Unable to set osd_memory_target on smithi150 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-11T16:43:23.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:22 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi130 to 196.2M 2023-10-11T16:43:23.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:22 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi150 to 196.2M 2023-10-11T16:43:23.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:22 smithi150 ceph-mon[132217]: Unable to set osd_memory_target on smithi130 to 205751705: error parsing value: Value '205751705' is below minimum 939524096 2023-10-11T16:43:23.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:22 smithi150 ceph-mon[132217]: Unable to set osd_memory_target on smithi150 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-11T16:43:23.596 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2023-10-11T16:43:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 0, 2023-10-11T16:43:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 2, 2023-10-11T16:43:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 0, 2023-10-11T16:43:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 0, 2023-10-11T16:43:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 4, 2023-10-11T16:43:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 0, 2023-10-11T16:43:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.003092442, 2023-10-11T16:43:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.000000000, 2023-10-11T16:43:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.000664749, 2023-10-11T16:43:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-11T16:43:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 0 2023-10-11T16:43:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 231, 2023-10-11T16:43:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 221, 2023-10-11T16:43:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 1424224, 2023-10-11T16:43:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 125840, 2023-10-11T16:43:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 2, 2023-10-11T16:43:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 1, 2023-10-11T16:43:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.053067013, 2023-10-11T16:43:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.019571205, 2023-10-11T16:43:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.037157276, 2023-10-11T16:43:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.005707638, 2023-10-11T16:43:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 220, 2023-10-11T16:43:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.017651152, 2023-10-11T16:43:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000080232 2023-10-11T16:43:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 1424224, 2023-10-11T16:43:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 125840 2023-10-11T16:43:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_messages": 6, 2023-10-11T16:43:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages": 170, 2023-10-11T16:43:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_bytes": 3312, 2023-10-11T16:43:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_bytes": 181744, 2023-10-11T16:43:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_active_connections": 3, 2023-10-11T16:43:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_total_time": 0.021209293, 2023-10-11T16:43:23.612 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_send_time": 0.014270114, 2023-10-11T16:43:23.612 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_recv_time": 0.003094321, 2023-10-11T16:43:23.612 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:23.613 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:23.613 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 168, 2023-10-11T16:43:23.613 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.022205886, 2023-10-11T16:43:23.613 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000132177 2023-10-11T16:43:23.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.614 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:23.614 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:23.615 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:23.615 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:23.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.615 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-10-11T16:43:23.616 INFO:teuthology.orchestra.run.smithi111.stdout: "msgr_send_encrypted_bytes": 181744 2023-10-11T16:43:23.616 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.616 INFO:teuthology.orchestra.run.smithi111.stdout: "cct": { 2023-10-11T16:43:23.616 INFO:teuthology.orchestra.run.smithi111.stdout: "total_workers": 0, 2023-10-11T16:43:23.617 INFO:teuthology.orchestra.run.smithi111.stdout: "unhealthy_workers": 0 2023-10-11T16:43:23.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.617 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool": { 2023-10-11T16:43:23.618 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:23.618 INFO:teuthology.orchestra.run.smithi111.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:23.618 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:23.618 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:23.619 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:23.619 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:23.619 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:23.619 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:23.620 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:23.620 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:23.620 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:23.621 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:23.621 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:23.621 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:23.621 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:23.622 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:23.622 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:23.622 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:23.623 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:23.623 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:23.623 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:23.623 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:23.624 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:23.624 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:23.624 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:23.624 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:23.625 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:23.625 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:23.625 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:23.626 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:23.626 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:23.626 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_items": 0, 2023-10-11T16:43:23.626 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:23.627 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:23.627 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:23.627 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:23.628 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_bytes": 148977, 2023-10-11T16:43:23.628 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_anon_items": 30, 2023-10-11T16:43:23.628 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:23.628 INFO:teuthology.orchestra.run.smithi111.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:23.629 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bytes": 0, 2023-10-11T16:43:23.629 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_items": 0, 2023-10-11T16:43:23.629 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:23.630 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:23.630 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:23.630 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:23.630 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_bytes": 0, 2023-10-11T16:43:23.631 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_items": 0, 2023-10-11T16:43:23.631 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:23.631 INFO:teuthology.orchestra.run.smithi111.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:23.632 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:23.632 INFO:teuthology.orchestra.run.smithi111.stdout: "pgmap_items": 0, 2023-10-11T16:43:23.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_bytes": 928, 2023-10-11T16:43:23.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_co_items": 2, 2023-10-11T16:43:23.633 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:23.633 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_1_items": 0, 2023-10-11T16:43:23.633 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:23.634 INFO:teuthology.orchestra.run.smithi111.stdout: "unittest_2_items": 0 2023-10-11T16:43:23.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2023-10-11T16:43:23.634 INFO:teuthology.orchestra.run.smithi111.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:23.634 INFO:teuthology.orchestra.run.smithi111.stdout: "val": 0, 2023-10-11T16:43:23.635 INFO:teuthology.orchestra.run.smithi111.stdout: "max": 104857600, 2023-10-11T16:43:23.635 INFO:teuthology.orchestra.run.smithi111.stdout: "get_started": 0, 2023-10-11T16:43:23.635 INFO:teuthology.orchestra.run.smithi111.stdout: "get": 233, 2023-10-11T16:43:23.636 INFO:teuthology.orchestra.run.smithi111.stdout: "get_sum": 1391619, 2023-10-11T16:43:23.636 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:23.636 INFO:teuthology.orchestra.run.smithi111.stdout: "get_or_fail_success": 233, 2023-10-11T16:43:23.636 INFO:teuthology.orchestra.run.smithi111.stdout: "take": 0, 2023-10-11T16:43:23.637 INFO:teuthology.orchestra.run.smithi111.stdout: "take_sum": 0, 2023-10-11T16:43:23.637 INFO:teuthology.orchestra.run.smithi111.stdout: "put": 233, 2023-10-11T16:43:23.637 INFO:teuthology.orchestra.run.smithi111.stdout: "put_sum": 1391619, 2023-10-11T16:43:23.637 INFO:teuthology.orchestra.run.smithi111.stdout: "wait": { 2023-10-11T16:43:23.638 INFO:teuthology.orchestra.run.smithi111.stdout: "avgcount": 0, 2023-10-11T16:43:23.638 INFO:teuthology.orchestra.run.smithi111.stdout: "sum": 0.000000000, 2023-10-11T16:43:23.638 INFO:teuthology.orchestra.run.smithi111.stdout: "avgtime": 0.000000000 2023-10-11T16:43:23.639 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:23.639 INFO:teuthology.orchestra.run.smithi111.stdout: } 2023-10-11T16:43:23.639 INFO:teuthology.orchestra.run.smithi111.stdout:} 2023-10-11T16:43:23.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:23 smithi111 ceph-mon[98527]: pgmap v335: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 193 B/s wr, 5 op/s 2023-10-11T16:43:24.081 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:23 smithi130 ceph-mon[131859]: pgmap v335: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 193 B/s wr, 5 op/s 2023-10-11T16:43:24.131 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2023-10-11T16:43:24.132 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2023-10-11T16:43:24.132 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2023-10-11T16:43:24.132 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2023-10-11T16:43:24.133 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2023-10-11T16:43:24.133 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2023-10-11T16:43:24.133 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2023-10-11T16:43:24.134 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2023-10-11T16:43:24.134 DEBUG:tasks.check_counter:Getting stats from b 2023-10-11T16:43:24.135 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.b perf dump 2023-10-11T16:43:24.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:23 smithi150 ceph-mon[132217]: pgmap v335: 129 pgs: 70 active+clean+snaptrim_wait, 23 active+clean+snaptrim, 36 active+clean; 663 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 193 B/s wr, 5 op/s 2023-10-11T16:43:24.944 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:43:25.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:25 smithi130 ceph-mon[131859]: pgmap v336: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:43:25.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:25 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:25.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:25 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:26.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:25 smithi111 ceph-mon[98527]: pgmap v336: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:43:26.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:25 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:26.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:25 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:26.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:25 smithi150 ceph-mon[132217]: pgmap v336: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:43:26.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:25 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:26.207 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:25 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:27.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:26 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:26 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:26 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:26 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:26 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:43:27.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:26 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:43:27.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:26 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:27.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:26 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:26 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:26 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:26 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:26 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:43:27.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:26 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:43:27.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:26 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:27.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:26 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:26 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:26 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.205 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:26 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-11T16:43:27.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:26 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-11T16:43:27.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:26 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-11T16:43:27.206 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:26 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' 2023-10-11T16:43:27.822 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2023-10-11T16:43:27.823 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:27.823 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 35873, 2023-10-11T16:43:27.823 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 36301, 2023-10-11T16:43:27.823 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 22128235, 2023-10-11T16:43:27.824 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 158877686, 2023-10-11T16:43:27.824 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 29, 2023-10-11T16:43:27.824 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2023-10-11T16:43:27.825 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 2.300612232, 2023-10-11T16:43:27.825 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.766444205, 2023-10-11T16:43:27.825 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.703646731, 2023-10-11T16:43:27.825 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.843008952, 2023-10-11T16:43:27.826 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:27.826 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 36301, 2023-10-11T16:43:27.826 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 40.411977623, 2023-10-11T16:43:27.827 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001113246 2023-10-11T16:43:27.827 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.827 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:27.827 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 17114, 2023-10-11T16:43:27.828 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.002679748, 2023-10-11T16:43:27.828 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000156 2023-10-11T16:43:27.828 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.829 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1546288, 2023-10-11T16:43:27.829 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 467440 2023-10-11T16:43:27.829 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.829 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:27.830 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 103709, 2023-10-11T16:43:27.830 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 166662, 2023-10-11T16:43:27.830 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 39858702, 2023-10-11T16:43:27.831 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 187793408, 2023-10-11T16:43:27.831 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 148, 2023-10-11T16:43:27.831 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2023-10-11T16:43:27.831 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 6.123638586, 2023-10-11T16:43:27.832 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 2.374844202, 2023-10-11T16:43:27.832 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 14.383874335, 2023-10-11T16:43:27.832 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.961333879, 2023-10-11T16:43:27.832 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:27.833 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 166659, 2023-10-11T16:43:27.833 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 411.185288609, 2023-10-11T16:43:27.833 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002467225 2023-10-11T16:43:27.834 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.834 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:27.834 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 172628, 2023-10-11T16:43:27.834 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.028392495, 2023-10-11T16:43:27.835 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000164 2023-10-11T16:43:27.835 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.835 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 144496, 2023-10-11T16:43:27.836 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 168096 2023-10-11T16:43:27.836 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.836 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:27.836 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 44295, 2023-10-11T16:43:27.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 44555, 2023-10-11T16:43:27.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 33984584, 2023-10-11T16:43:27.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 146644813, 2023-10-11T16:43:27.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 18, 2023-10-11T16:43:27.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2023-10-11T16:43:27.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 2.634240686, 2023-10-11T16:43:27.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.826292084, 2023-10-11T16:43:27.838 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.726453860, 2023-10-11T16:43:27.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 1.058984464, 2023-10-11T16:43:27.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:27.839 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 44553, 2023-10-11T16:43:27.840 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 52.364822804, 2023-10-11T16:43:27.840 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001175337 2023-10-11T16:43:27.840 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.840 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:27.841 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 16895, 2023-10-11T16:43:27.841 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.002523057, 2023-10-11T16:43:27.841 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000149 2023-10-11T16:43:27.841 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.842 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 329040, 2023-10-11T16:43:27.842 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 97248 2023-10-11T16:43:27.842 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.843 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2023-10-11T16:43:27.843 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2023-10-11T16:43:27.843 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2023-10-11T16:43:27.843 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.844 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2023-10-11T16:43:27.844 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2023-10-11T16:43:27.844 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2023-10-11T16:43:27.844 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 12274, 2023-10-11T16:43:27.845 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 59.536167654, 2023-10-11T16:43:27.845 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.004850592 2023-10-11T16:43:27.845 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.846 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.846 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2023-10-11T16:43:27.846 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2023-10-11T16:43:27.846 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2023-10-11T16:43:27.847 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 21759, 2023-10-11T16:43:27.847 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 4.780960028, 2023-10-11T16:43:27.847 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000219723 2023-10-11T16:43:27.847 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.848 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.848 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2023-10-11T16:43:27.848 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 48945, 2023-10-11T16:43:27.849 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 39649, 2023-10-11T16:43:27.849 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2023-10-11T16:43:27.849 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 39649, 2023-10-11T16:43:27.849 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 107.936911272, 2023-10-11T16:43:27.850 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002722311 2023-10-11T16:43:27.850 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.850 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2023-10-11T16:43:27.850 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 9296, 2023-10-11T16:43:27.851 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 473, 2023-10-11T16:43:27.851 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2023-10-11T16:43:27.851 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 10431, 2023-10-11T16:43:27.852 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 136, 2023-10-11T16:43:27.852 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 22, 2023-10-11T16:43:27.852 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 14163, 2023-10-11T16:43:27.852 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 9901, 2023-10-11T16:43:27.853 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 4251, 2023-10-11T16:43:27.853 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 11, 2023-10-11T16:43:27.853 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 16, 2023-10-11T16:43:27.853 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 39970, 2023-10-11T16:43:27.854 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 1, 2023-10-11T16:43:27.854 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 2, 2023-10-11T16:43:27.854 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 5, 2023-10-11T16:43:27.855 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 61573, 2023-10-11T16:43:27.855 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 52203, 2023-10-11T16:43:27.855 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 9038, 2023-10-11T16:43:27.855 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 128, 2023-10-11T16:43:27.856 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 111, 2023-10-11T16:43:27.856 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2023-10-11T16:43:27.856 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2023-10-11T16:43:27.857 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 93, 2023-10-11T16:43:27.857 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2023-10-11T16:43:27.857 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 514, 2023-10-11T16:43:27.857 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 30939, 2023-10-11T16:43:27.858 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 386, 2023-10-11T16:43:27.858 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2023-10-11T16:43:27.858 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2023-10-11T16:43:27.859 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 13771, 2023-10-11T16:43:27.859 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2023-10-11T16:43:27.859 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 4, 2023-10-11T16:43:27.859 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 246, 2023-10-11T16:43:27.860 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2023-10-11T16:43:27.860 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 13771, 2023-10-11T16:43:27.860 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2023-10-11T16:43:27.860 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2023-10-11T16:43:27.861 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2023-10-11T16:43:27.861 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 42, 2023-10-11T16:43:27.861 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 12, 2023-10-11T16:43:27.862 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 30, 2023-10-11T16:43:27.862 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 13729, 2023-10-11T16:43:27.862 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 6, 2023-10-11T16:43:27.862 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 98, 2023-10-11T16:43:27.863 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 27580, 2023-10-11T16:43:27.863 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2023-10-11T16:43:27.863 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-11T16:43:27.863 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-11T16:43:27.864 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 22703, 2023-10-11T16:43:27.864 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 22591, 2023-10-11T16:43:27.864 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 242, 2023-10-11T16:43:27.864 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 24245 2023-10-11T16:43:27.865 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.865 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2023-10-11T16:43:27.865 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 1546, 2023-10-11T16:43:27.865 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 2889, 2023-10-11T16:43:27.866 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 1446, 2023-10-11T16:43:27.866 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 2075, 2023-10-11T16:43:27.866 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 1826, 2023-10-11T16:43:27.867 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 1, 2023-10-11T16:43:27.867 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2023-10-11T16:43:27.867 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2023-10-11T16:43:27.868 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 12042, 2023-10-11T16:43:27.868 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 12041, 2023-10-11T16:43:27.868 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2023-10-11T16:43:27.868 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2023-10-11T16:43:27.869 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2023-10-11T16:43:27.869 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2023-10-11T16:43:27.869 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2023-10-11T16:43:27.870 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2023-10-11T16:43:27.870 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2023-10-11T16:43:27.870 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 12, 2023-10-11T16:43:27.870 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 517, 2023-10-11T16:43:27.871 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2023-10-11T16:43:27.871 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 169, 2023-10-11T16:43:27.871 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2023-10-11T16:43:27.872 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2023-10-11T16:43:27.872 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.872 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2023-10-11T16:43:27.872 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 64589, 2023-10-11T16:43:27.873 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 64540, 2023-10-11T16:43:27.873 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 64540, 2023-10-11T16:43:27.873 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 49, 2023-10-11T16:43:27.873 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2023-10-11T16:43:27.874 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 3, 2023-10-11T16:43:27.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 117, 2023-10-11T16:43:27.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 116, 2023-10-11T16:43:27.875 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 116, 2023-10-11T16:43:27.875 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2023-10-11T16:43:27.875 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2023-10-11T16:43:27.875 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 1, 2023-10-11T16:43:27.876 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 253252842, 2023-10-11T16:43:27.876 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 253321127, 2023-10-11T16:43:27.876 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2023-10-11T16:43:27.877 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2023-10-11T16:43:27.877 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1653, 2023-10-11T16:43:27.877 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 73.905497391, 2023-10-11T16:43:27.877 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.044709919 2023-10-11T16:43:27.878 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.878 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2023-10-11T16:43:27.878 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.878 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2023-10-11T16:43:27.879 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 13533, 2023-10-11T16:43:27.879 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 55402, 2023-10-11T16:43:27.879 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 41869, 2023-10-11T16:43:27.879 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 513, 2023-10-11T16:43:27.880 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 2948, 2023-10-11T16:43:27.880 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 2435, 2023-10-11T16:43:27.880 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 14163, 2023-10-11T16:43:27.881 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 74351, 2023-10-11T16:43:27.881 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 60188, 2023-10-11T16:43:27.881 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 2, 2023-10-11T16:43:27.881 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 34701, 2023-10-11T16:43:27.882 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 34699, 2023-10-11T16:43:27.882 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 254992, 2023-10-11T16:43:27.882 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207156 2023-10-11T16:43:27.882 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.883 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2023-10-11T16:43:27.883 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 50051, 2023-10-11T16:43:27.883 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 32, 2023-10-11T16:43:27.884 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 48945, 2023-10-11T16:43:27.884 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 318, 2023-10-11T16:43:27.884 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 78, 2023-10-11T16:43:27.884 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2023-10-11T16:43:27.885 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 15376, 2023-10-11T16:43:27.885 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 35.845965508, 2023-10-11T16:43:27.885 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002331293 2023-10-11T16:43:27.885 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.886 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2023-10-11T16:43:27.886 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 232, 2023-10-11T16:43:27.886 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.154631688, 2023-10-11T16:43:27.887 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000666515 2023-10-11T16:43:27.887 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.887 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2023-10-11T16:43:27.887 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.888 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.888 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.888 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.888 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2023-10-11T16:43:27.889 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.889 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.889 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.890 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.890 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2023-10-11T16:43:27.890 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 10694, 2023-10-11T16:43:27.890 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 13.281807878, 2023-10-11T16:43:27.891 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001241986 2023-10-11T16:43:27.891 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.891 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2023-10-11T16:43:27.891 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.892 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.892 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.892 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.893 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2023-10-11T16:43:27.893 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2023-10-11T16:43:27.893 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000288324, 2023-10-11T16:43:27.893 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000144162 2023-10-11T16:43:27.894 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.894 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2023-10-11T16:43:27.894 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.895 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.895 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.895 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.895 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2023-10-11T16:43:27.896 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.896 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.896 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.896 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.897 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2023-10-11T16:43:27.897 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.897 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.897 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.898 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.898 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2023-10-11T16:43:27.898 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 11, 2023-10-11T16:43:27.899 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.001832435, 2023-10-11T16:43:27.899 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000166585 2023-10-11T16:43:27.899 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.899 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2023-10-11T16:43:27.900 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 6, 2023-10-11T16:43:27.900 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.007983100, 2023-10-11T16:43:27.900 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001330516 2023-10-11T16:43:27.900 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.901 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2023-10-11T16:43:27.901 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.901 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.901 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.902 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.902 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2023-10-11T16:43:27.902 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 11, 2023-10-11T16:43:27.903 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 3.476846007, 2023-10-11T16:43:27.903 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.316076909 2023-10-11T16:43:27.903 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.903 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2023-10-11T16:43:27.904 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.904 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.904 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.905 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.905 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2023-10-11T16:43:27.905 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1260, 2023-10-11T16:43:27.905 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 13.220861873, 2023-10-11T16:43:27.906 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.010492747 2023-10-11T16:43:27.906 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.906 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2023-10-11T16:43:27.907 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2023-10-11T16:43:27.907 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.002532399, 2023-10-11T16:43:27.907 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001266199 2023-10-11T16:43:27.907 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.908 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2023-10-11T16:43:27.908 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.908 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.909 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.909 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.909 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2023-10-11T16:43:27.909 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.910 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.910 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.910 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.911 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2023-10-11T16:43:27.911 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1, 2023-10-11T16:43:27.911 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.085271609, 2023-10-11T16:43:27.911 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.085271609 2023-10-11T16:43:27.912 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.912 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2023-10-11T16:43:27.912 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 5, 2023-10-11T16:43:27.912 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.024364295, 2023-10-11T16:43:27.913 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.004872859 2023-10-11T16:43:27.913 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.913 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2023-10-11T16:43:27.914 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.914 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.914 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.914 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.915 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2023-10-11T16:43:27.915 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1, 2023-10-11T16:43:27.915 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000407999, 2023-10-11T16:43:27.915 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000407999 2023-10-11T16:43:27.916 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.916 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2023-10-11T16:43:27.916 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.917 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.917 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.917 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.917 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2023-10-11T16:43:27.918 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.918 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.918 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.919 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.919 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2023-10-11T16:43:27.919 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.919 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.920 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.920 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.920 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2023-10-11T16:43:27.920 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 7, 2023-10-11T16:43:27.921 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.019985840, 2023-10-11T16:43:27.921 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.002855120 2023-10-11T16:43:27.921 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.922 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2023-10-11T16:43:27.922 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.922 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.922 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.923 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.923 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2023-10-11T16:43:27.923 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 12041, 2023-10-11T16:43:27.923 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 41.814132317, 2023-10-11T16:43:27.924 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.003472646 2023-10-11T16:43:27.924 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.924 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2023-10-11T16:43:27.925 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2023-10-11T16:43:27.925 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2023-10-11T16:43:27.925 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.925 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.926 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.926 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.926 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.927 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2023-10-11T16:43:27.927 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2023-10-11T16:43:27.927 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 6, 2023-10-11T16:43:27.927 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 4, 2023-10-11T16:43:27.928 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 2, 2023-10-11T16:43:27.928 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2023-10-11T16:43:27.928 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 537, 2023-10-11T16:43:27.929 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 268, 2023-10-11T16:43:27.929 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 635 2023-10-11T16:43:27.929 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.929 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2023-10-11T16:43:27.930 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:27.930 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:27.930 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:27.931 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:27.931 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:27.931 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:27.931 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:27.932 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:27.932 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:27.932 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:27.933 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:27.933 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:27.933 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:27.933 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:27.934 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:27.934 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:27.934 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:27.935 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:27.935 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:27.935 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:27.936 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:27.936 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:27.936 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:27.936 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:27.937 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:27.937 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:27.937 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:27.937 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:27.938 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:27.938 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:27.938 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:27.938 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2023-10-11T16:43:27.939 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:27.939 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:27.939 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:27.940 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:27.940 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 761461, 2023-10-11T16:43:27.940 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 13493, 2023-10-11T16:43:27.940 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 1584, 2023-10-11T16:43:27.941 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 18, 2023-10-11T16:43:27.941 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2023-10-11T16:43:27.941 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2023-10-11T16:43:27.942 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:27.942 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:27.942 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:27.943 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:27.943 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 10152, 2023-10-11T16:43:27.943 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 198, 2023-10-11T16:43:27.944 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:27.944 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:27.944 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:27.944 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2023-10-11T16:43:27.945 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 41170196, 2023-10-11T16:43:27.945 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 370523, 2023-10-11T16:43:27.945 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:27.945 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2023-10-11T16:43:27.946 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:27.946 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2023-10-11T16:43:27.946 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.947 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2023-10-11T16:43:27.947 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2023-10-11T16:43:27.947 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2023-10-11T16:43:27.947 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 93689, 2023-10-11T16:43:27.948 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 321885092, 2023-10-11T16:43:27.948 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 37, 2023-10-11T16:43:27.948 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 93689, 2023-10-11T16:43:27.948 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2023-10-11T16:43:27.949 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 93689, 2023-10-11T16:43:27.949 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 40745.167286371, 2023-10-11T16:43:27.949 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.434898091 2023-10-11T16:43:27.950 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.950 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2023-10-11T16:43:27.950 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2023-10-11T16:43:27.950 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 93689, 2023-10-11T16:43:27.951 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 173364 2023-10-11T16:43:27.951 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.951 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 93689, 2023-10-11T16:43:27.951 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 14452, 2023-10-11T16:43:27.952 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 79237, 2023-10-11T16:43:27.952 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2023-10-11T16:43:27.952 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2023-10-11T16:43:27.953 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 9305, 2023-10-11T16:43:27.953 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 41745, 2023-10-11T16:43:27.953 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 196, 2023-10-11T16:43:27.953 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 1837, 2023-10-11T16:43:27.954 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 186, 2023-10-11T16:43:27.954 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2023-10-11T16:43:27.954 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2023-10-11T16:43:27.954 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2023-10-11T16:43:27.955 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2023-10-11T16:43:27.955 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 24944, 2023-10-11T16:43:27.955 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2023-10-11T16:43:27.956 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2023-10-11T16:43:27.956 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2023-10-11T16:43:27.956 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 13783, 2023-10-11T16:43:27.956 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 63362, 2023-10-11T16:43:27.957 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2023-10-11T16:43:27.957 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2023-10-11T16:43:27.957 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2023-10-11T16:43:27.957 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2023-10-11T16:43:27.958 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2023-10-11T16:43:27.958 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2023-10-11T16:43:27.958 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2023-10-11T16:43:27.959 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2023-10-11T16:43:27.959 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2023-10-11T16:43:27.959 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2023-10-11T16:43:27.959 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2023-10-11T16:43:27.960 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2023-10-11T16:43:27.960 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2023-10-11T16:43:27.960 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2023-10-11T16:43:27.960 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2023-10-11T16:43:27.961 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2023-10-11T16:43:27.961 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2023-10-11T16:43:27.961 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2023-10-11T16:43:27.962 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2023-10-11T16:43:27.962 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2023-10-11T16:43:27.962 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2023-10-11T16:43:27.962 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2023-10-11T16:43:27.963 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2023-10-11T16:43:27.963 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2023-10-11T16:43:27.963 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2023-10-11T16:43:27.963 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2023-10-11T16:43:27.964 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 102, 2023-10-11T16:43:27.964 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2023-10-11T16:43:27.964 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 20, 2023-10-11T16:43:27.965 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 12, 2023-10-11T16:43:27.965 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 12, 2023-10-11T16:43:27.965 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2023-10-11T16:43:27.965 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2023-10-11T16:43:27.966 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 16210, 2023-10-11T16:43:27.966 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 946, 2023-10-11T16:43:27.966 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 850 2023-10-11T16:43:27.966 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.967 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2023-10-11T16:43:27.967 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2023-10-11T16:43:27.967 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2023-10-11T16:43:27.968 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2023-10-11T16:43:27.968 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2023-10-11T16:43:27.968 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.968 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2023-10-11T16:43:27.969 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 49703 2023-10-11T16:43:27.969 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.969 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2023-10-11T16:43:27.970 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2023-10-11T16:43:27.970 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 16, 2023-10-11T16:43:27.970 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2023-10-11T16:43:27.970 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 8, 2023-10-11T16:43:27.971 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 12041, 2023-10-11T16:43:27.971 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2023-10-11T16:43:27.971 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.971 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:27.972 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:27.972 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2023-10-11T16:43:27.972 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:27.973 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 183873, 2023-10-11T16:43:27.973 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 79353621, 2023-10-11T16:43:27.973 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:27.973 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 183873, 2023-10-11T16:43:27.974 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:27.974 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:27.974 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 183873, 2023-10-11T16:43:27.975 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 79353621, 2023-10-11T16:43:27.975 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:27.975 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.975 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.976 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.976 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.976 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.976 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2023-10-11T16:43:27.977 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:27.977 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2023-10-11T16:43:27.977 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:27.977 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:27.978 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:27.978 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:27.978 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:27.979 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 93689, 2023-10-11T16:43:27.979 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 323347609, 2023-10-11T16:43:27.979 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 93689, 2023-10-11T16:43:27.979 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 323347609, 2023-10-11T16:43:27.980 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:27.980 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.980 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.981 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.981 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.981 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.981 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2023-10-11T16:43:27.982 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:27.982 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2023-10-11T16:43:27.982 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:27.983 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:27.983 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:27.983 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:27.983 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:27.984 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 93689, 2023-10-11T16:43:27.984 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 93689, 2023-10-11T16:43:27.984 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 93689, 2023-10-11T16:43:27.984 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 93689, 2023-10-11T16:43:27.985 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:27.985 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.985 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.986 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.986 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.986 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.986 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2023-10-11T16:43:27.987 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:27.987 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2023-10-11T16:43:27.987 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:27.988 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 12041, 2023-10-11T16:43:27.988 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1470085, 2023-10-11T16:43:27.988 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:27.988 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 12041, 2023-10-11T16:43:27.989 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:27.989 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:27.989 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 184, 2023-10-11T16:43:27.989 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1470085, 2023-10-11T16:43:27.990 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:27.990 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.990 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.991 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.991 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.991 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:27.991 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x55b283c400a0": { 2023-10-11T16:43:27.992 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:27.992 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2023-10-11T16:43:27.992 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:27.992 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 64589, 2023-10-11T16:43:27.993 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 249126823, 2023-10-11T16:43:27.993 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:27.993 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 64589, 2023-10-11T16:43:27.994 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:27.994 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:27.994 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 1653, 2023-10-11T16:43:27.994 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 249126823, 2023-10-11T16:43:27.995 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:27.995 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:27.995 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:27.996 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:27.996 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.996 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:27.996 INFO:teuthology.orchestra.run.smithi130.stdout:} 2023-10-11T16:43:28.082 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:27 smithi130 ceph-mon[131859]: pgmap v337: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:43:28.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:27 smithi130 ceph-mon[131859]: Adjusting osd_memory_target on smithi111 to 221.9M 2023-10-11T16:43:28.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:27 smithi130 ceph-mon[131859]: Unable to set osd_memory_target on smithi111 to 232767897: error parsing value: Value '232767897' is below minimum 939524096 2023-10-11T16:43:28.083 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:27 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:43:28.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:27 smithi130 ceph-mon[131859]: Service mds.all has no related daemon already placed. 2023-10-11T16:43:28.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:27 smithi130 ceph-mon[131859]: Service mgr has no related daemon already placed. 2023-10-11T16:43:28.084 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:27 smithi130 ceph-mon[131859]: Service mon has no related daemon already placed. 2023-10-11T16:43:28.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:27 smithi111 ceph-mon[98527]: pgmap v337: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:43:28.136 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:27 smithi111 ceph-mon[98527]: Adjusting osd_memory_target on smithi111 to 221.9M 2023-10-11T16:43:28.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:27 smithi111 ceph-mon[98527]: Unable to set osd_memory_target on smithi111 to 232767897: error parsing value: Value '232767897' is below minimum 939524096 2023-10-11T16:43:28.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:27 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:43:28.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:27 smithi111 ceph-mon[98527]: Service mds.all has no related daemon already placed. 2023-10-11T16:43:28.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:27 smithi111 ceph-mon[98527]: Service mgr has no related daemon already placed. 2023-10-11T16:43:28.138 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:27 smithi111 ceph-mon[98527]: Service mon has no related daemon already placed. 2023-10-11T16:43:28.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:27 smithi150 ceph-mon[132217]: pgmap v337: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2023-10-11T16:43:28.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:27 smithi150 ceph-mon[132217]: Adjusting osd_memory_target on smithi111 to 221.9M 2023-10-11T16:43:28.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:27 smithi150 ceph-mon[132217]: Unable to set osd_memory_target on smithi111 to 232767897: error parsing value: Value '232767897' is below minimum 939524096 2023-10-11T16:43:28.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:27 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:43:28.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:27 smithi150 ceph-mon[132217]: Service mds.all has no related daemon already placed. 2023-10-11T16:43:28.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:27 smithi150 ceph-mon[132217]: Service mgr has no related daemon already placed. 2023-10-11T16:43:28.204 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:27 smithi150 ceph-mon[132217]: Service mon has no related daemon already placed. 2023-10-11T16:43:28.293 INFO:tasks.check_counter:Daemon mds.b mds.exported=514 2023-10-11T16:43:28.293 INFO:tasks.check_counter:Daemon mds.b mds.imported=386 2023-10-11T16:43:28.294 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=1546 2023-10-11T16:43:28.294 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=2889 2023-10-11T16:43:28.294 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2023-10-11T16:43:28.295 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2023-10-11T16:43:28.295 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2023-10-11T16:43:28.295 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=136 2023-10-11T16:43:28.296 DEBUG:tasks.check_counter:Getting stats from e 2023-10-11T16:43:28.296 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.e perf dump 2023-10-11T16:43:29.125 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:43:29.950 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:29 smithi130 ceph-mon[131859]: pgmap v338: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:43:30.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:29 smithi111 ceph-mon[98527]: pgmap v338: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:43:30.203 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:29 smithi150 ceph-mon[132217]: pgmap v338: 129 pgs: 70 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 37 active+clean; 663 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-11T16:43:30.421 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2023-10-11T16:43:30.422 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:30.422 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 941, 2023-10-11T16:43:30.422 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 941, 2023-10-11T16:43:30.423 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 125276426, 2023-10-11T16:43:30.423 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 291919, 2023-10-11T16:43:30.423 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 8, 2023-10-11T16:43:30.424 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2023-10-11T16:43:30.424 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.140617960, 2023-10-11T16:43:30.424 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.027222095, 2023-10-11T16:43:30.424 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.076402569, 2023-10-11T16:43:30.425 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.028691943, 2023-10-11T16:43:30.425 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:30.425 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 940, 2023-10-11T16:43:30.426 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.053524912, 2023-10-11T16:43:30.426 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000056941 2023-10-11T16:43:30.426 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.427 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:30.427 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.427 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.427 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.428 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.428 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-11T16:43:30.428 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 448 2023-10-11T16:43:30.428 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.429 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:30.429 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 912, 2023-10-11T16:43:30.429 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1078, 2023-10-11T16:43:30.429 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 151975488, 2023-10-11T16:43:30.430 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 719367, 2023-10-11T16:43:30.430 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 9, 2023-10-11T16:43:30.430 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2023-10-11T16:43:30.431 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.153608619, 2023-10-11T16:43:30.431 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.026254418, 2023-10-11T16:43:30.431 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.092152663, 2023-10-11T16:43:30.431 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.026112022, 2023-10-11T16:43:30.432 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:30.432 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1075, 2023-10-11T16:43:30.433 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.077786769, 2023-10-11T16:43:30.433 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000072359 2023-10-11T16:43:30.433 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.433 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:30.434 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.434 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.434 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.435 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.435 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-11T16:43:30.435 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 436352 2023-10-11T16:43:30.436 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.436 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:30.436 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 1270, 2023-10-11T16:43:30.436 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1245, 2023-10-11T16:43:30.437 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 79755960, 2023-10-11T16:43:30.437 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 441540, 2023-10-11T16:43:30.437 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:30.438 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2023-10-11T16:43:30.438 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.188030325, 2023-10-11T16:43:30.438 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.042781693, 2023-10-11T16:43:30.438 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.099690927, 2023-10-11T16:43:30.439 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.044953857, 2023-10-11T16:43:30.439 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:30.439 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1244, 2023-10-11T16:43:30.439 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.074449128, 2023-10-11T16:43:30.440 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000059846 2023-10-11T16:43:30.440 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.440 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:30.441 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.441 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.441 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.441 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.442 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1514432, 2023-10-11T16:43:30.442 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 129296 2023-10-11T16:43:30.442 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.443 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2023-10-11T16:43:30.443 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2023-10-11T16:43:30.443 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2023-10-11T16:43:30.444 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.444 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2023-10-11T16:43:30.444 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2023-10-11T16:43:30.444 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2023-10-11T16:43:30.445 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4481, 2023-10-11T16:43:30.445 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.147980479, 2023-10-11T16:43:30.445 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000033023 2023-10-11T16:43:30.446 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.446 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.446 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2023-10-11T16:43:30.446 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2023-10-11T16:43:30.447 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2023-10-11T16:43:30.447 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.447 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.448 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.448 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.448 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.448 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2023-10-11T16:43:30.449 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 0, 2023-10-11T16:43:30.449 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 0, 2023-10-11T16:43:30.449 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2023-10-11T16:43:30.449 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.450 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.450 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.450 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.451 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2023-10-11T16:43:30.451 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2023-10-11T16:43:30.451 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2023-10-11T16:43:30.452 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2023-10-11T16:43:30.452 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 0, 2023-10-11T16:43:30.452 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2023-10-11T16:43:30.453 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2023-10-11T16:43:30.453 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 42754, 2023-10-11T16:43:30.453 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 8330, 2023-10-11T16:43:30.453 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 34424, 2023-10-11T16:43:30.454 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2023-10-11T16:43:30.454 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 30854, 2023-10-11T16:43:30.454 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2023-10-11T16:43:30.454 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 0, 2023-10-11T16:43:30.455 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 0, 2023-10-11T16:43:30.455 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 1, 2023-10-11T16:43:30.455 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 0, 2023-10-11T16:43:30.456 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 0, 2023-10-11T16:43:30.456 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2023-10-11T16:43:30.456 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 0, 2023-10-11T16:43:30.456 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2023-10-11T16:43:30.457 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2023-10-11T16:43:30.457 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2023-10-11T16:43:30.457 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2023-10-11T16:43:30.458 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2023-10-11T16:43:30.458 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2023-10-11T16:43:30.458 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2023-10-11T16:43:30.458 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2023-10-11T16:43:30.459 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2023-10-11T16:43:30.459 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2023-10-11T16:43:30.459 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2023-10-11T16:43:30.460 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2023-10-11T16:43:30.460 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 0, 2023-10-11T16:43:30.460 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 0, 2023-10-11T16:43:30.460 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 0, 2023-10-11T16:43:30.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2023-10-11T16:43:30.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2023-10-11T16:43:30.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2023-10-11T16:43:30.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2023-10-11T16:43:30.462 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2023-10-11T16:43:30.462 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2023-10-11T16:43:30.462 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2023-10-11T16:43:30.463 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2023-10-11T16:43:30.463 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2023-10-11T16:43:30.463 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2023-10-11T16:43:30.463 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2023-10-11T16:43:30.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2023-10-11T16:43:30.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-11T16:43:30.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-11T16:43:30.465 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2023-10-11T16:43:30.465 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2023-10-11T16:43:30.465 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2023-10-11T16:43:30.465 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2023-10-11T16:43:30.466 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.466 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2023-10-11T16:43:30.466 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2023-10-11T16:43:30.467 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2023-10-11T16:43:30.467 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2023-10-11T16:43:30.467 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 0, 2023-10-11T16:43:30.468 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2023-10-11T16:43:30.468 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2023-10-11T16:43:30.468 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2023-10-11T16:43:30.468 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2023-10-11T16:43:30.469 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2023-10-11T16:43:30.469 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2023-10-11T16:43:30.469 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2023-10-11T16:43:30.470 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2023-10-11T16:43:30.470 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2023-10-11T16:43:30.470 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2023-10-11T16:43:30.470 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2023-10-11T16:43:30.471 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2023-10-11T16:43:30.471 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2023-10-11T16:43:30.471 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2023-10-11T16:43:30.472 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2023-10-11T16:43:30.472 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2023-10-11T16:43:30.472 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2023-10-11T16:43:30.472 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2023-10-11T16:43:30.473 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2023-10-11T16:43:30.473 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.473 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2023-10-11T16:43:30.474 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 0, 2023-10-11T16:43:30.474 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2023-10-11T16:43:30.474 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2023-10-11T16:43:30.474 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 0, 2023-10-11T16:43:30.475 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2023-10-11T16:43:30.475 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2023-10-11T16:43:30.475 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 0, 2023-10-11T16:43:30.476 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2023-10-11T16:43:30.476 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2023-10-11T16:43:30.476 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 131, 2023-10-11T16:43:30.476 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2023-10-11T16:43:30.477 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2023-10-11T16:43:30.477 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 61802902, 2023-10-11T16:43:30.477 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 0, 2023-10-11T16:43:30.478 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 348463616, 2023-10-11T16:43:30.478 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2023-10-11T16:43:30.478 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.478 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.479 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.479 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.479 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 97442 2023-10-11T16:43:30.479 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.480 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2023-10-11T16:43:30.480 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 12379, 2023-10-11T16:43:30.480 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 61713, 2023-10-11T16:43:30.480 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 49334, 2023-10-11T16:43:30.481 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 523, 2023-10-11T16:43:30.481 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 1740, 2023-10-11T16:43:30.481 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 1217, 2023-10-11T16:43:30.482 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 42754, 2023-10-11T16:43:30.482 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 93903, 2023-10-11T16:43:30.482 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 51149, 2023-10-11T16:43:30.482 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 0, 2023-10-11T16:43:30.483 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 0, 2023-10-11T16:43:30.483 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2023-10-11T16:43:30.483 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 160780, 2023-10-11T16:43:30.484 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207156 2023-10-11T16:43:30.484 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.484 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2023-10-11T16:43:30.484 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 0, 2023-10-11T16:43:30.485 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2023-10-11T16:43:30.485 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 0, 2023-10-11T16:43:30.485 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 0, 2023-10-11T16:43:30.485 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2023-10-11T16:43:30.486 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2023-10-11T16:43:30.486 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.487 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.487 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.487 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.488 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2023-10-11T16:43:30.488 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.488 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.488 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.489 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.489 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2023-10-11T16:43:30.489 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.490 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.490 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.490 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.491 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2023-10-11T16:43:30.491 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.491 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.491 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.492 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.492 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2023-10-11T16:43:30.492 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.493 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.493 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.493 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.493 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2023-10-11T16:43:30.494 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.494 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.494 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.495 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.495 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2023-10-11T16:43:30.495 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.495 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.496 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.496 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.496 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2023-10-11T16:43:30.496 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.497 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.497 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.497 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.498 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2023-10-11T16:43:30.498 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.498 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.499 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.499 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.499 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2023-10-11T16:43:30.499 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.500 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.500 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.500 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.500 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2023-10-11T16:43:30.501 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.501 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.501 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.502 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.502 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2023-10-11T16:43:30.502 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.502 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.503 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.503 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.503 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2023-10-11T16:43:30.504 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.504 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.504 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.504 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.505 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2023-10-11T16:43:30.505 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.505 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.505 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.506 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.506 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2023-10-11T16:43:30.506 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.507 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.507 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.507 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.507 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2023-10-11T16:43:30.508 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.508 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.508 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.509 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.509 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2023-10-11T16:43:30.509 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.510 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.510 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.510 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.511 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2023-10-11T16:43:30.511 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.511 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.511 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.512 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.512 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2023-10-11T16:43:30.512 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.513 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.513 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.513 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.513 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2023-10-11T16:43:30.514 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.514 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.514 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.515 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.515 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2023-10-11T16:43:30.515 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.515 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.516 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.516 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.516 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2023-10-11T16:43:30.516 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.517 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.517 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.517 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.518 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2023-10-11T16:43:30.518 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.518 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.518 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.519 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.519 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2023-10-11T16:43:30.519 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.520 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.520 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.520 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.520 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2023-10-11T16:43:30.521 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.521 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.521 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.522 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.522 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2023-10-11T16:43:30.522 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.523 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.523 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.523 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.523 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2023-10-11T16:43:30.524 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.524 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.524 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.525 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.525 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2023-10-11T16:43:30.525 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.525 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.526 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.526 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.526 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2023-10-11T16:43:30.527 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.527 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.527 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.527 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.528 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2023-10-11T16:43:30.528 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2023-10-11T16:43:30.528 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2023-10-11T16:43:30.529 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.529 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.529 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.529 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.530 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.530 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2023-10-11T16:43:30.530 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2023-10-11T16:43:30.530 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 4, 2023-10-11T16:43:30.531 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 2, 2023-10-11T16:43:30.531 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 4, 2023-10-11T16:43:30.531 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2023-10-11T16:43:30.532 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 0, 2023-10-11T16:43:30.532 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 0, 2023-10-11T16:43:30.532 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 4555 2023-10-11T16:43:30.532 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.533 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2023-10-11T16:43:30.533 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:30.533 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:30.534 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:30.534 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:30.534 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:30.534 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:30.535 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:30.535 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:30.535 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:30.536 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:30.536 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:30.536 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:30.536 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:30.537 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:30.537 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:30.537 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:30.538 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:30.538 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:30.538 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:30.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:30.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:30.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:30.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:30.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:30.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:30.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:30.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:30.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:30.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:30.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:30.542 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:30.542 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2023-10-11T16:43:30.542 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:30.542 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:30.543 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:30.543 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:30.543 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 660016, 2023-10-11T16:43:30.544 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 12450, 2023-10-11T16:43:30.544 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:30.544 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:30.544 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2023-10-11T16:43:30.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2023-10-11T16:43:30.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:30.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:30.546 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:30.546 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:30.546 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 10152, 2023-10-11T16:43:30.547 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 198, 2023-10-11T16:43:30.547 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:30.547 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:30.547 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:30.548 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2023-10-11T16:43:30.548 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 58455580, 2023-10-11T16:43:30.548 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 425582, 2023-10-11T16:43:30.549 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:30.549 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2023-10-11T16:43:30.549 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:30.549 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2023-10-11T16:43:30.550 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.550 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2023-10-11T16:43:30.550 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2023-10-11T16:43:30.551 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2023-10-11T16:43:30.551 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 2851, 2023-10-11T16:43:30.551 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 21, 2023-10-11T16:43:30.551 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 3, 2023-10-11T16:43:30.552 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 2851, 2023-10-11T16:43:30.552 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2023-10-11T16:43:30.552 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2851, 2023-10-11T16:43:30.553 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 5.355900109, 2023-10-11T16:43:30.553 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001878604 2023-10-11T16:43:30.553 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.554 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2023-10-11T16:43:30.554 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2023-10-11T16:43:30.554 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2851, 2023-10-11T16:43:30.554 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2852 2023-10-11T16:43:30.555 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.555 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 2851, 2023-10-11T16:43:30.555 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 2851, 2023-10-11T16:43:30.556 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 0, 2023-10-11T16:43:30.556 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2023-10-11T16:43:30.556 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2023-10-11T16:43:30.556 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 1846, 2023-10-11T16:43:30.557 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 0, 2023-10-11T16:43:30.557 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 1003, 2023-10-11T16:43:30.557 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 0, 2023-10-11T16:43:30.557 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 0, 2023-10-11T16:43:30.558 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2023-10-11T16:43:30.558 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2023-10-11T16:43:30.558 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2023-10-11T16:43:30.559 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2023-10-11T16:43:30.559 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 0, 2023-10-11T16:43:30.559 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2023-10-11T16:43:30.560 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2023-10-11T16:43:30.560 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2023-10-11T16:43:30.560 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 1, 2023-10-11T16:43:30.560 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 0, 2023-10-11T16:43:30.561 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2023-10-11T16:43:30.561 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2023-10-11T16:43:30.561 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2023-10-11T16:43:30.562 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2023-10-11T16:43:30.562 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2023-10-11T16:43:30.562 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2023-10-11T16:43:30.562 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2023-10-11T16:43:30.563 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2023-10-11T16:43:30.563 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2023-10-11T16:43:30.563 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2023-10-11T16:43:30.564 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2023-10-11T16:43:30.564 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2023-10-11T16:43:30.564 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2023-10-11T16:43:30.564 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2023-10-11T16:43:30.565 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2023-10-11T16:43:30.565 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2023-10-11T16:43:30.565 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2023-10-11T16:43:30.566 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2023-10-11T16:43:30.566 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2023-10-11T16:43:30.566 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2023-10-11T16:43:30.567 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2023-10-11T16:43:30.567 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2023-10-11T16:43:30.567 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2023-10-11T16:43:30.567 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2023-10-11T16:43:30.568 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2023-10-11T16:43:30.568 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2023-10-11T16:43:30.568 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 102, 2023-10-11T16:43:30.568 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2023-10-11T16:43:30.569 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 15, 2023-10-11T16:43:30.569 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 11, 2023-10-11T16:43:30.569 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 11, 2023-10-11T16:43:30.570 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2023-10-11T16:43:30.570 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2023-10-11T16:43:30.570 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 0, 2023-10-11T16:43:30.570 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 2, 2023-10-11T16:43:30.571 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 0 2023-10-11T16:43:30.571 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.571 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2023-10-11T16:43:30.572 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 0, 2023-10-11T16:43:30.572 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2023-10-11T16:43:30.572 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2023-10-11T16:43:30.572 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2023-10-11T16:43:30.573 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.573 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2023-10-11T16:43:30.573 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2023-10-11T16:43:30.574 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.574 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2023-10-11T16:43:30.574 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2023-10-11T16:43:30.574 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2023-10-11T16:43:30.575 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2023-10-11T16:43:30.575 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2023-10-11T16:43:30.575 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2023-10-11T16:43:30.576 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2023-10-11T16:43:30.576 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.576 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:30.576 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:30.577 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2023-10-11T16:43:30.577 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:30.577 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 3119, 2023-10-11T16:43:30.578 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 356710563, 2023-10-11T16:43:30.578 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:30.578 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 3119, 2023-10-11T16:43:30.578 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:30.579 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:30.579 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3119, 2023-10-11T16:43:30.579 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 356710563, 2023-10-11T16:43:30.580 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:30.580 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.580 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.580 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.581 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.581 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.581 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2023-10-11T16:43:30.581 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:30.582 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2023-10-11T16:43:30.582 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:30.582 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:30.583 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:30.583 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:30.583 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:30.584 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 2851, 2023-10-11T16:43:30.584 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 354698896, 2023-10-11T16:43:30.584 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2851, 2023-10-11T16:43:30.585 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 354698896, 2023-10-11T16:43:30.585 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:30.585 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.585 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.586 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.586 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.586 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.587 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2023-10-11T16:43:30.587 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:30.587 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2023-10-11T16:43:30.588 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:30.588 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:30.588 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:30.588 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:30.589 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:30.589 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 2851, 2023-10-11T16:43:30.589 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 2851, 2023-10-11T16:43:30.590 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2851, 2023-10-11T16:43:30.590 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 2851, 2023-10-11T16:43:30.590 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:30.590 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.591 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.591 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.591 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.591 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.592 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2023-10-11T16:43:30.592 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:30.592 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2023-10-11T16:43:30.593 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:30.593 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:30.593 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:30.593 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:30.594 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:30.594 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:30.594 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:30.595 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2023-10-11T16:43:30.595 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2023-10-11T16:43:30.595 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:30.596 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.596 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.596 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.596 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.597 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:30.597 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x560b54fac460": { 2023-10-11T16:43:30.597 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:30.598 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2023-10-11T16:43:30.598 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:30.598 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:30.598 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:30.599 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:30.599 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:30.599 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:30.600 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:30.600 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2023-10-11T16:43:30.600 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2023-10-11T16:43:30.600 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:30.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:30.601 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:30.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:30.602 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.602 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:30.602 INFO:teuthology.orchestra.run.smithi130.stdout:} 2023-10-11T16:43:30.817 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2023-10-11T16:43:30.818 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2023-10-11T16:43:30.818 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2023-10-11T16:43:30.818 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2023-10-11T16:43:30.819 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2023-10-11T16:43:30.819 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2023-10-11T16:43:30.819 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2023-10-11T16:43:30.820 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2023-10-11T16:43:30.820 DEBUG:tasks.check_counter:Getting stats from h 2023-10-11T16:43:30.821 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.h perf dump 2023-10-11T16:43:31.444 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:43:32.032 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:31 smithi130 ceph-mon[131859]: pgmap v339: 129 pgs: 58 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 53 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-11T16:43:32.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:31 smithi111 ceph-mon[98527]: pgmap v339: 129 pgs: 58 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 53 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-11T16:43:32.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:31 smithi150 ceph-mon[132217]: pgmap v339: 129 pgs: 58 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 53 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-11T16:43:32.818 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2023-10-11T16:43:32.819 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:32.819 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 1442, 2023-10-11T16:43:32.819 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1443, 2023-10-11T16:43:32.819 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 105475394, 2023-10-11T16:43:32.820 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 448161, 2023-10-11T16:43:32.820 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 8, 2023-10-11T16:43:32.820 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2023-10-11T16:43:32.821 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.173825287, 2023-10-11T16:43:32.821 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.046300005, 2023-10-11T16:43:32.821 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.073473151, 2023-10-11T16:43:32.822 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.041720617, 2023-10-11T16:43:32.822 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:32.822 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1441, 2023-10-11T16:43:32.822 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.072756699, 2023-10-11T16:43:32.823 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000050490 2023-10-11T16:43:32.823 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.823 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:32.824 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.824 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.824 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.825 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.825 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-11T16:43:32.825 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2023-10-11T16:43:32.825 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.826 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:32.826 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 512, 2023-10-11T16:43:32.826 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 677, 2023-10-11T16:43:32.827 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 76472221, 2023-10-11T16:43:32.827 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 594852, 2023-10-11T16:43:32.827 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 9, 2023-10-11T16:43:32.827 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 7, 2023-10-11T16:43:32.828 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.087674290, 2023-10-11T16:43:32.828 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.015872555, 2023-10-11T16:43:32.828 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.049497184, 2023-10-11T16:43:32.828 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.017045093, 2023-10-11T16:43:32.829 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:32.829 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 674, 2023-10-11T16:43:32.829 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.042497443, 2023-10-11T16:43:32.830 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000063052 2023-10-11T16:43:32.830 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.830 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:32.830 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.831 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.831 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.831 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.832 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-11T16:43:32.832 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 436176 2023-10-11T16:43:32.832 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.833 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:32.833 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 1140, 2023-10-11T16:43:32.833 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 1113, 2023-10-11T16:43:32.833 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 70514242, 2023-10-11T16:43:32.834 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 399883, 2023-10-11T16:43:32.834 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:32.834 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2023-10-11T16:43:32.835 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.150601055, 2023-10-11T16:43:32.835 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.031888501, 2023-10-11T16:43:32.835 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.089484729, 2023-10-11T16:43:32.836 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.032223460, 2023-10-11T16:43:32.836 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:32.836 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1113, 2023-10-11T16:43:32.836 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.050037246, 2023-10-11T16:43:32.837 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000044957 2023-10-11T16:43:32.837 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.837 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:32.837 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.838 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.838 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.838 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1517312, 2023-10-11T16:43:32.839 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 129312 2023-10-11T16:43:32.839 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.840 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2023-10-11T16:43:32.840 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2023-10-11T16:43:32.840 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2023-10-11T16:43:32.840 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.841 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2023-10-11T16:43:32.841 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2023-10-11T16:43:32.841 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2023-10-11T16:43:32.842 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4487, 2023-10-11T16:43:32.842 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.747623796, 2023-10-11T16:43:32.842 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000166619 2023-10-11T16:43:32.842 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:32.843 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.843 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2023-10-11T16:43:32.843 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2023-10-11T16:43:32.844 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2023-10-11T16:43:32.844 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.844 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.844 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.845 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:32.845 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.845 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2023-10-11T16:43:32.846 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 0, 2023-10-11T16:43:32.846 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 0, 2023-10-11T16:43:32.846 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2023-10-11T16:43:32.847 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.847 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.847 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.847 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.848 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2023-10-11T16:43:32.848 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2023-10-11T16:43:32.848 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2023-10-11T16:43:32.848 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2023-10-11T16:43:32.849 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 0, 2023-10-11T16:43:32.849 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2023-10-11T16:43:32.849 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2023-10-11T16:43:32.850 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 34655, 2023-10-11T16:43:32.850 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 24250, 2023-10-11T16:43:32.850 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 10404, 2023-10-11T16:43:32.850 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 1, 2023-10-11T16:43:32.851 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 12, 2023-10-11T16:43:32.851 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 457, 2023-10-11T16:43:32.851 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 0, 2023-10-11T16:43:32.852 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 0, 2023-10-11T16:43:32.852 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 3, 2023-10-11T16:43:32.852 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 0, 2023-10-11T16:43:32.852 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 0, 2023-10-11T16:43:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2023-10-11T16:43:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 0, 2023-10-11T16:43:32.853 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2023-10-11T16:43:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2023-10-11T16:43:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2023-10-11T16:43:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2023-10-11T16:43:32.854 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2023-10-11T16:43:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2023-10-11T16:43:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2023-10-11T16:43:32.855 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2023-10-11T16:43:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2023-10-11T16:43:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2023-10-11T16:43:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2023-10-11T16:43:32.856 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2023-10-11T16:43:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 4, 2023-10-11T16:43:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 246, 2023-10-11T16:43:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2023-10-11T16:43:32.857 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2023-10-11T16:43:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2023-10-11T16:43:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2023-10-11T16:43:32.858 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2023-10-11T16:43:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2023-10-11T16:43:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2023-10-11T16:43:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2023-10-11T16:43:32.859 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2023-10-11T16:43:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2023-10-11T16:43:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2023-10-11T16:43:32.860 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2023-10-11T16:43:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2023-10-11T16:43:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-11T16:43:32.861 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-11T16:43:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2023-10-11T16:43:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2023-10-11T16:43:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2023-10-11T16:43:32.862 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2023-10-11T16:43:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2023-10-11T16:43:32.863 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2023-10-11T16:43:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2023-10-11T16:43:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2023-10-11T16:43:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 0, 2023-10-11T16:43:32.864 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2023-10-11T16:43:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2023-10-11T16:43:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2023-10-11T16:43:32.865 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2023-10-11T16:43:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2023-10-11T16:43:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2023-10-11T16:43:32.866 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2023-10-11T16:43:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2023-10-11T16:43:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2023-10-11T16:43:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2023-10-11T16:43:32.867 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2023-10-11T16:43:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2023-10-11T16:43:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2023-10-11T16:43:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2023-10-11T16:43:32.868 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2023-10-11T16:43:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2023-10-11T16:43:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2023-10-11T16:43:32.869 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2023-10-11T16:43:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2023-10-11T16:43:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.870 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2023-10-11T16:43:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 0, 2023-10-11T16:43:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2023-10-11T16:43:32.871 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2023-10-11T16:43:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 0, 2023-10-11T16:43:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2023-10-11T16:43:32.872 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2023-10-11T16:43:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 0, 2023-10-11T16:43:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2023-10-11T16:43:32.873 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2023-10-11T16:43:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2023-10-11T16:43:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2023-10-11T16:43:32.874 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2023-10-11T16:43:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 253252842, 2023-10-11T16:43:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 0, 2023-10-11T16:43:32.875 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 253320991, 2023-10-11T16:43:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2023-10-11T16:43:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.876 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 64589 2023-10-11T16:43:32.877 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2023-10-11T16:43:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 13110, 2023-10-11T16:43:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 46948, 2023-10-11T16:43:32.878 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 33838, 2023-10-11T16:43:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 500, 2023-10-11T16:43:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 1392, 2023-10-11T16:43:32.879 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 892, 2023-10-11T16:43:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 34655, 2023-10-11T16:43:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 67170, 2023-10-11T16:43:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 32515, 2023-10-11T16:43:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 0, 2023-10-11T16:43:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 0, 2023-10-11T16:43:32.881 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2023-10-11T16:43:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 171456, 2023-10-11T16:43:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 207156 2023-10-11T16:43:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.882 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2023-10-11T16:43:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 0, 2023-10-11T16:43:32.885 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2023-10-11T16:43:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 0, 2023-10-11T16:43:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 0, 2023-10-11T16:43:32.886 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2023-10-11T16:43:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2023-10-11T16:43:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.887 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2023-10-11T16:43:32.888 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.889 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2023-10-11T16:43:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.890 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2023-10-11T16:43:32.891 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.892 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2023-10-11T16:43:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.893 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.894 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2023-10-11T16:43:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.895 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2023-10-11T16:43:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.896 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2023-10-11T16:43:32.897 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.898 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2023-10-11T16:43:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.899 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2023-10-11T16:43:32.900 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.901 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2023-10-11T16:43:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.902 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2023-10-11T16:43:32.903 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.904 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2023-10-11T16:43:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2023-10-11T16:43:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.907 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2023-10-11T16:43:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.908 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2023-10-11T16:43:32.909 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.910 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2023-10-11T16:43:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.911 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2023-10-11T16:43:32.912 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.913 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2023-10-11T16:43:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.914 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2023-10-11T16:43:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.915 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.916 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2023-10-11T16:43:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.917 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.918 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2023-10-11T16:43:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.919 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2023-10-11T16:43:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.920 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.921 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2023-10-11T16:43:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.922 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2023-10-11T16:43:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.923 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.924 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2023-10-11T16:43:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.925 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2023-10-11T16:43:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.926 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2023-10-11T16:43:32.927 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.928 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2023-10-11T16:43:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.929 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.930 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.930 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2023-10-11T16:43:32.930 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2023-10-11T16:43:32.930 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2023-10-11T16:43:32.931 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.931 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.931 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.932 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:32.932 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.932 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2023-10-11T16:43:32.932 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 2, 2023-10-11T16:43:32.933 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 6, 2023-10-11T16:43:32.933 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 4, 2023-10-11T16:43:32.933 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 4, 2023-10-11T16:43:32.933 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2023-10-11T16:43:32.934 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 0, 2023-10-11T16:43:32.934 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 0, 2023-10-11T16:43:32.934 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 4557 2023-10-11T16:43:32.935 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.935 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2023-10-11T16:43:32.935 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:32.935 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:32.936 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:32.936 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:32.936 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:32.937 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:32.937 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:32.937 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:32.938 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:32.938 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:32.938 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:32.938 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:32.939 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:32.939 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:32.939 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:32.940 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:32.940 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:32.940 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:32.941 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:32.941 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:32.941 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:32.941 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:32.942 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:32.942 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:32.942 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:32.943 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:32.943 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:32.943 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:32.943 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:32.944 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:32.944 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:32.944 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2023-10-11T16:43:32.945 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:32.945 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:32.945 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:32.945 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:32.946 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 684898, 2023-10-11T16:43:32.946 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 13181, 2023-10-11T16:43:32.946 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:32.947 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:32.947 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2023-10-11T16:43:32.947 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2023-10-11T16:43:32.947 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:32.948 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:32.948 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:32.948 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:32.949 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 10152, 2023-10-11T16:43:32.949 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 198, 2023-10-11T16:43:32.949 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:32.949 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:32.950 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:32.950 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2023-10-11T16:43:32.950 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 54543721, 2023-10-11T16:43:32.951 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 434116, 2023-10-11T16:43:32.951 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:32.951 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2023-10-11T16:43:32.952 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:32.952 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2023-10-11T16:43:32.952 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.953 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2023-10-11T16:43:32.953 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2023-10-11T16:43:32.953 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2023-10-11T16:43:32.954 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 2821, 2023-10-11T16:43:32.954 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 26, 2023-10-11T16:43:32.954 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 4, 2023-10-11T16:43:32.955 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 2821, 2023-10-11T16:43:32.955 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2023-10-11T16:43:32.955 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2821, 2023-10-11T16:43:32.955 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 10.823797965, 2023-10-11T16:43:32.956 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.003836865 2023-10-11T16:43:32.956 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.956 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2023-10-11T16:43:32.957 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2023-10-11T16:43:32.957 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2821, 2023-10-11T16:43:32.957 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2822 2023-10-11T16:43:32.957 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.958 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 2821, 2023-10-11T16:43:32.958 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 2821, 2023-10-11T16:43:32.958 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 0, 2023-10-11T16:43:32.959 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2023-10-11T16:43:32.959 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2023-10-11T16:43:32.959 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 1780, 2023-10-11T16:43:32.959 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 0, 2023-10-11T16:43:32.960 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 1038, 2023-10-11T16:43:32.960 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 0, 2023-10-11T16:43:32.960 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 0, 2023-10-11T16:43:32.961 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2023-10-11T16:43:32.961 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2023-10-11T16:43:32.961 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2023-10-11T16:43:32.961 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2023-10-11T16:43:32.962 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 0, 2023-10-11T16:43:32.962 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2023-10-11T16:43:32.962 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2023-10-11T16:43:32.962 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2023-10-11T16:43:32.963 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 2, 2023-10-11T16:43:32.963 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 0, 2023-10-11T16:43:32.963 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2023-10-11T16:43:32.964 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2023-10-11T16:43:32.964 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2023-10-11T16:43:32.964 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2023-10-11T16:43:32.964 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2023-10-11T16:43:32.965 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2023-10-11T16:43:32.965 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2023-10-11T16:43:32.965 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2023-10-11T16:43:32.966 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2023-10-11T16:43:32.966 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2023-10-11T16:43:32.966 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2023-10-11T16:43:32.966 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2023-10-11T16:43:32.967 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2023-10-11T16:43:32.967 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2023-10-11T16:43:32.967 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2023-10-11T16:43:32.967 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2023-10-11T16:43:32.968 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2023-10-11T16:43:32.968 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2023-10-11T16:43:32.968 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2023-10-11T16:43:32.969 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2023-10-11T16:43:32.969 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2023-10-11T16:43:32.969 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2023-10-11T16:43:32.970 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2023-10-11T16:43:32.970 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2023-10-11T16:43:32.970 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2023-10-11T16:43:32.971 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2023-10-11T16:43:32.971 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 102, 2023-10-11T16:43:32.971 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2023-10-11T16:43:32.971 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 15, 2023-10-11T16:43:32.972 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 11, 2023-10-11T16:43:32.972 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 11, 2023-10-11T16:43:32.972 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2023-10-11T16:43:32.973 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2023-10-11T16:43:32.973 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 0, 2023-10-11T16:43:32.973 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 2, 2023-10-11T16:43:32.973 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 0 2023-10-11T16:43:32.974 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.974 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2023-10-11T16:43:32.974 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 0, 2023-10-11T16:43:32.975 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2023-10-11T16:43:32.975 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2023-10-11T16:43:32.975 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2023-10-11T16:43:32.976 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.976 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2023-10-11T16:43:32.976 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2023-10-11T16:43:32.976 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.977 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2023-10-11T16:43:32.977 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2023-10-11T16:43:32.977 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2023-10-11T16:43:32.978 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2023-10-11T16:43:32.978 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2023-10-11T16:43:32.978 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2023-10-11T16:43:32.978 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2023-10-11T16:43:32.979 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.979 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:32.979 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:32.980 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2023-10-11T16:43:32.980 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:32.980 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 3090, 2023-10-11T16:43:32.981 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 252167162, 2023-10-11T16:43:32.981 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:32.981 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 3090, 2023-10-11T16:43:32.982 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:32.982 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:32.982 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 3090, 2023-10-11T16:43:32.982 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 252167162, 2023-10-11T16:43:32.983 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:32.983 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.983 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.983 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.984 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:32.984 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.984 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2023-10-11T16:43:32.985 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:32.985 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2023-10-11T16:43:32.985 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:32.985 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:32.986 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:32.986 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:32.986 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:32.987 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 2821, 2023-10-11T16:43:32.987 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 250162043, 2023-10-11T16:43:32.987 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2821, 2023-10-11T16:43:32.987 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 250162043, 2023-10-11T16:43:32.988 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:32.988 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.988 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.989 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.989 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:32.989 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.989 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2023-10-11T16:43:32.990 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:32.990 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2023-10-11T16:43:32.990 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:32.991 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:32.991 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:32.991 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:32.991 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:32.992 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 2821, 2023-10-11T16:43:32.992 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 2821, 2023-10-11T16:43:32.992 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2821, 2023-10-11T16:43:32.992 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 2821, 2023-10-11T16:43:32.993 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:32.993 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.993 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.994 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.994 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:32.994 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:32.995 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2023-10-11T16:43:32.995 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:32.995 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2023-10-11T16:43:32.995 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:32.996 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:32.996 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:32.996 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:32.997 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:32.997 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:32.997 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:32.997 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2023-10-11T16:43:32.998 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2023-10-11T16:43:32.998 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:32.998 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:32.999 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:32.999 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:32.999 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:33.000 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:33.000 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x5647d9474460": { 2023-10-11T16:43:33.000 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:33.000 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2023-10-11T16:43:33.001 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:33.001 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2023-10-11T16:43:33.001 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2023-10-11T16:43:33.002 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:33.002 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:33.002 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:33.003 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:33.003 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2023-10-11T16:43:33.003 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2023-10-11T16:43:33.003 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:33.005 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:33.005 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:33.005 INFO:teuthology.orchestra.run.smithi130.stdout:} 2023-10-11T16:43:33.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:32 smithi130 ceph-mon[131859]: pgmap v340: 129 pgs: 58 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 53 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:33.130 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2023-10-11T16:43:33.131 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2023-10-11T16:43:33.131 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2023-10-11T16:43:33.132 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2023-10-11T16:43:33.132 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2023-10-11T16:43:33.132 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2023-10-11T16:43:33.133 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2023-10-11T16:43:33.133 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2023-10-11T16:43:33.133 DEBUG:tasks.check_counter:Getting stats from k 2023-10-11T16:43:33.134 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.k perf dump 2023-10-11T16:43:33.137 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:32 smithi111 ceph-mon[98527]: pgmap v340: 129 pgs: 58 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 53 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:33.202 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:32 smithi150 ceph-mon[132217]: pgmap v340: 129 pgs: 58 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 53 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:33.847 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-11T16:43:35.115 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2023-10-11T16:43:35.116 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:35.116 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 0, 2023-10-11T16:43:35.116 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 2, 2023-10-11T16:43:35.117 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 0, 2023-10-11T16:43:35.117 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 0, 2023-10-11T16:43:35.117 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 4, 2023-10-11T16:43:35.117 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 0, 2023-10-11T16:43:35.118 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.001223573, 2023-10-11T16:43:35.118 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.000000000, 2023-10-11T16:43:35.118 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.000285770, 2023-10-11T16:43:35.119 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:35.119 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:35.119 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:35.119 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:35.120 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:35.120 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.120 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:35.121 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:35.121 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:35.121 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:35.122 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.122 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-11T16:43:35.122 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2023-10-11T16:43:35.123 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.123 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:35.123 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 234, 2023-10-11T16:43:35.123 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 224, 2023-10-11T16:43:35.124 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1425040, 2023-10-11T16:43:35.124 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 127600, 2023-10-11T16:43:35.124 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 2, 2023-10-11T16:43:35.125 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 1, 2023-10-11T16:43:35.125 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.044800999, 2023-10-11T16:43:35.125 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.013532074, 2023-10-11T16:43:35.125 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.032623292, 2023-10-11T16:43:35.126 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.006493871, 2023-10-11T16:43:35.126 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:35.126 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 223, 2023-10-11T16:43:35.127 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.013479945, 2023-10-11T16:43:35.127 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000060448 2023-10-11T16:43:35.127 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.127 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:35.128 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:35.128 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:35.128 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:35.129 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.129 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1425040, 2023-10-11T16:43:35.129 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 127600 2023-10-11T16:43:35.129 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.130 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:35.130 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 6, 2023-10-11T16:43:35.130 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 173, 2023-10-11T16:43:35.131 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 3312, 2023-10-11T16:43:35.131 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 182272, 2023-10-11T16:43:35.131 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:35.132 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 3, 2023-10-11T16:43:35.132 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.011347781, 2023-10-11T16:43:35.132 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.006817689, 2023-10-11T16:43:35.133 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.002391719, 2023-10-11T16:43:35.133 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:35.133 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:35.133 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 171, 2023-10-11T16:43:35.134 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.018724493, 2023-10-11T16:43:35.134 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000109499 2023-10-11T16:43:35.134 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.135 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:35.137 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:35.138 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:35.138 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:35.138 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.138 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-10-11T16:43:35.139 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 182272 2023-10-11T16:43:35.139 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.139 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2023-10-11T16:43:35.140 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 0, 2023-10-11T16:43:35.140 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2023-10-11T16:43:35.140 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.140 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2023-10-11T16:43:35.141 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:35.141 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:35.141 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:35.141 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:35.142 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:35.142 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:35.142 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:35.143 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:35.143 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:35.143 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:35.143 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:35.144 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:35.144 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:35.144 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:35.145 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:35.145 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:35.145 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:35.145 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:35.146 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:35.146 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:35.146 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:35.147 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:35.147 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:35.147 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:35.147 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:35.148 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:35.148 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:35.148 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:35.148 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:35.149 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:35.149 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:35.149 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2023-10-11T16:43:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 149017, 2023-10-11T16:43:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 31, 2023-10-11T16:43:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2023-10-11T16:43:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2023-10-11T16:43:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:35.153 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:35.153 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:35.153 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:35.154 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 0, 2023-10-11T16:43:35.154 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 0, 2023-10-11T16:43:35.154 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:35.154 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:35.155 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:35.155 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2023-10-11T16:43:35.155 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 928, 2023-10-11T16:43:35.155 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 2, 2023-10-11T16:43:35.156 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:35.156 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2023-10-11T16:43:35.156 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:35.157 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2023-10-11T16:43:35.157 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2023-10-11T16:43:35.157 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:35.157 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2023-10-11T16:43:35.158 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2023-10-11T16:43:35.158 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2023-10-11T16:43:35.158 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 236, 2023-10-11T16:43:35.159 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1392009, 2023-10-11T16:43:35.159 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:35.159 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 236, 2023-10-11T16:43:35.159 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2023-10-11T16:43:35.160 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2023-10-11T16:43:35.160 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 236, 2023-10-11T16:43:35.161 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1392009, 2023-10-11T16:43:35.161 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2023-10-11T16:43:35.161 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2023-10-11T16:43:35.161 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2023-10-11T16:43:35.162 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2023-10-11T16:43:35.162 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:35.162 INFO:teuthology.orchestra.run.smithi130.stdout: } 2023-10-11T16:43:35.163 INFO:teuthology.orchestra.run.smithi130.stdout:} 2023-10-11T16:43:35.163 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:34 smithi130 ceph-mon[131859]: pgmap v341: 129 pgs: 44 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 69 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2023-10-11T16:43:35.385 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:34 smithi111 ceph-mon[98527]: pgmap v341: 129 pgs: 44 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 69 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2023-10-11T16:43:35.452 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:34 smithi150 ceph-mon[132217]: pgmap v341: 129 pgs: 44 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 69 active+clean; 663 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2023-10-11T16:43:35.517 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2023-10-11T16:43:35.517 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2023-10-11T16:43:35.518 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2023-10-11T16:43:35.518 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2023-10-11T16:43:35.518 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2023-10-11T16:43:35.518 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2023-10-11T16:43:35.519 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2023-10-11T16:43:35.519 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2023-10-11T16:43:35.519 DEBUG:tasks.check_counter:Getting stats from c 2023-10-11T16:43:35.520 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.c perf dump 2023-10-11T16:43:36.543 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:43:37.609 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:37 smithi150 ceph-mon[132217]: pgmap v342: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:43:37.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:37 smithi111 ceph-mon[98527]: pgmap v342: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:43:37.799 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-10-11T16:43:37.799 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:37.800 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 155632, 2023-10-11T16:43:37.800 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 168021, 2023-10-11T16:43:37.800 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 48698680, 2023-10-11T16:43:37.800 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 165910432, 2023-10-11T16:43:37.801 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 12, 2023-10-11T16:43:37.801 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 5, 2023-10-11T16:43:37.801 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 8.730659090, 2023-10-11T16:43:37.801 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 3.693523515, 2023-10-11T16:43:37.802 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 5.157865214, 2023-10-11T16:43:37.802 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.904278948, 2023-10-11T16:43:37.802 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:37.803 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 168019, 2023-10-11T16:43:37.803 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 25.495650004, 2023-10-11T16:43:37.803 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000151742 2023-10-11T16:43:37.803 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.804 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:37.804 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 273857, 2023-10-11T16:43:37.804 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.039155366, 2023-10-11T16:43:37.805 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000142 2023-10-11T16:43:37.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.805 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1712, 2023-10-11T16:43:37.805 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 5824 2023-10-11T16:43:37.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.806 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:37.806 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 104620, 2023-10-11T16:43:37.807 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 170930, 2023-10-11T16:43:37.807 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 36852926, 2023-10-11T16:43:37.807 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 195895453, 2023-10-11T16:43:37.807 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 37, 2023-10-11T16:43:37.808 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 9, 2023-10-11T16:43:37.808 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 6.437823086, 2023-10-11T16:43:37.808 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 2.573287515, 2023-10-11T16:43:37.809 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 23.552206698, 2023-10-11T16:43:37.809 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.995618167, 2023-10-11T16:43:37.809 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:37.809 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 170928, 2023-10-11T16:43:37.810 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 37.740869765, 2023-10-11T16:43:37.810 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000220799 2023-10-11T16:43:37.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.811 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:37.811 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 189372, 2023-10-11T16:43:37.811 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.035718272, 2023-10-11T16:43:37.813 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000188 2023-10-11T16:43:37.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.814 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 5872, 2023-10-11T16:43:37.814 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 131344 2023-10-11T16:43:37.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.815 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:37.815 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 22381, 2023-10-11T16:43:37.815 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 22171, 2023-10-11T16:43:37.815 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 59969105, 2023-10-11T16:43:37.816 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 107238187, 2023-10-11T16:43:37.816 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 9, 2023-10-11T16:43:37.816 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 6, 2023-10-11T16:43:37.816 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 1.876883282, 2023-10-11T16:43:37.817 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.633001214, 2023-10-11T16:43:37.817 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.647203233, 2023-10-11T16:43:37.817 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.667332492, 2023-10-11T16:43:37.818 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:37.818 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 22170, 2023-10-11T16:43:37.818 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 16.859627237, 2023-10-11T16:43:37.818 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000760470 2023-10-11T16:43:37.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.819 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:37.819 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 16927, 2023-10-11T16:43:37.820 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.002943751, 2023-10-11T16:43:37.820 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000173 2023-10-11T16:43:37.820 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.821 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1528288, 2023-10-11T16:43:37.821 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 503888 2023-10-11T16:43:37.821 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.821 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2023-10-11T16:43:37.822 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 1, 2023-10-11T16:43:37.822 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2023-10-11T16:43:37.822 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.823 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-MDSRank": { 2023-10-11T16:43:37.823 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2023-10-11T16:43:37.823 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2023-10-11T16:43:37.823 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 4591, 2023-10-11T16:43:37.824 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 65.238397820, 2023-10-11T16:43:37.824 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.014210062 2023-10-11T16:43:37.824 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.824 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.825 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-PurgeQueue": { 2023-10-11T16:43:37.825 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2023-10-11T16:43:37.825 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2023-10-11T16:43:37.826 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 34123, 2023-10-11T16:43:37.826 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 14.556976182, 2023-10-11T16:43:37.826 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000426603 2023-10-11T16:43:37.826 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.827 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.827 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": { 2023-10-11T16:43:37.827 INFO:teuthology.orchestra.run.smithi150.stdout: "request": 56829, 2023-10-11T16:43:37.827 INFO:teuthology.orchestra.run.smithi150.stdout: "reply": 50812, 2023-10-11T16:43:37.828 INFO:teuthology.orchestra.run.smithi150.stdout: "reply_latency": { 2023-10-11T16:43:37.828 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 50812, 2023-10-11T16:43:37.828 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 194.194837139, 2023-10-11T16:43:37.829 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.003821830 2023-10-11T16:43:37.829 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.829 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_reply": 0, 2023-10-11T16:43:37.829 INFO:teuthology.orchestra.run.smithi150.stdout: "forward": 6017, 2023-10-11T16:43:37.830 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_complete": 477, 2023-10-11T16:43:37.830 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_keys": 0, 2023-10-11T16:43:37.830 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_commit": 887, 2023-10-11T16:43:37.830 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_split": 82, 2023-10-11T16:43:37.831 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_merge": 0, 2023-10-11T16:43:37.831 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes": 16747, 2023-10-11T16:43:37.831 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_top": 5710, 2023-10-11T16:43:37.832 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_bottom": 8971, 2023-10-11T16:43:37.832 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pin_tail": 2066, 2023-10-11T16:43:37.832 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pinned": 8587, 2023-10-11T16:43:37.832 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_expired": 62440, 2023-10-11T16:43:37.833 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_with_caps": 0, 2023-10-11T16:43:37.833 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": 0, 2023-10-11T16:43:37.833 INFO:teuthology.orchestra.run.smithi150.stdout: "subtrees": 5, 2023-10-11T16:43:37.833 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse": 80505, 2023-10-11T16:43:37.834 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_hit": 72691, 2023-10-11T16:43:37.834 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_forward": 5867, 2023-10-11T16:43:37.834 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_discover": 530, 2023-10-11T16:43:37.834 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_dir_fetch": 431, 2023-10-11T16:43:37.835 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_remote_ino": 0, 2023-10-11T16:43:37.835 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_lock": 0, 2023-10-11T16:43:37.835 INFO:teuthology.orchestra.run.smithi150.stdout: "load_cent": 0, 2023-10-11T16:43:37.835 INFO:teuthology.orchestra.run.smithi150.stdout: "q": 0, 2023-10-11T16:43:37.836 INFO:teuthology.orchestra.run.smithi150.stdout: "exported": 1106, 2023-10-11T16:43:37.836 INFO:teuthology.orchestra.run.smithi150.stdout: "exported_inodes": 49827, 2023-10-11T16:43:37.836 INFO:teuthology.orchestra.run.smithi150.stdout: "imported": 651, 2023-10-11T16:43:37.836 INFO:teuthology.orchestra.run.smithi150.stdout: "imported_inodes": 0, 2023-10-11T16:43:37.837 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_dir_fetch": 0, 2023-10-11T16:43:37.837 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_backtrace_fetch": 1591, 2023-10-11T16:43:37.837 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_peer_discover": 0, 2023-10-11T16:43:37.837 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rfiles": 4, 2023-10-11T16:43:37.838 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rbytes": 246, 2023-10-11T16:43:37.838 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rsnaps": 6, 2023-10-11T16:43:37.838 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_fetch": 1591, 2023-10-11T16:43:37.839 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_set_tag": 0, 2023-10-11T16:43:37.839 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_repaired": 0, 2023-10-11T16:43:37.839 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_inotable_repaired": 0, 2023-10-11T16:43:37.839 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_inodes": 0, 2023-10-11T16:43:37.840 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_base_inodes": 0, 2023-10-11T16:43:37.840 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dirfrag_rstats": 0, 2023-10-11T16:43:37.840 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_file_inodes": 1591, 2023-10-11T16:43:37.840 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_inode_file_caps": 3, 2023-10-11T16:43:37.841 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_revoke": 20, 2023-10-11T16:43:37.841 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_grant": 29927, 2023-10-11T16:43:37.841 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_trunc": 0, 2023-10-11T16:43:37.842 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-11T16:43:37.842 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-11T16:43:37.842 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps": 23497, 2023-10-11T16:43:37.843 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps_dirty": 23465, 2023-10-11T16:43:37.843 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_cap_release": 396, 2023-10-11T16:43:37.843 INFO:teuthology.orchestra.run.smithi150.stdout: "process_request_cap_release": 41661 2023-10-11T16:43:37.843 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache": { 2023-10-11T16:43:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update": 826, 2023-10-11T16:43:37.844 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update_receipt": 3327, 2023-10-11T16:43:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_try_discover": 1666, 2023-10-11T16:43:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_send_discover": 1694, 2023-10-11T16:43:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_handle_discover": 911, 2023-10-11T16:43:37.845 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays": 1, 2023-10-11T16:43:37.846 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_delayed": 0, 2023-10-11T16:43:37.846 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_enqueuing": 0, 2023-10-11T16:43:37.846 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_created": 20379, 2023-10-11T16:43:37.847 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_enqueued": 20378, 2023-10-11T16:43:37.847 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_reintegrated": 0, 2023-10-11T16:43:37.847 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_migrated": 0, 2023-10-11T16:43:37.848 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_processing": 0, 2023-10-11T16:43:37.848 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_enqueued": 0, 2023-10-11T16:43:37.848 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_prioritized": 0, 2023-10-11T16:43:37.848 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_started": 0, 2023-10-11T16:43:37.849 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_completed": 0, 2023-10-11T16:43:37.849 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_enqueue_scrub": 0, 2023-10-11T16:43:37.849 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_exportdir": 1154, 2023-10-11T16:43:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_flush": 0, 2023-10-11T16:43:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragmentdir": 82, 2023-10-11T16:43:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragstats": 0, 2023-10-11T16:43:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_inodestats": 0 2023-10-11T16:43:37.851 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.851 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log": { 2023-10-11T16:43:37.851 INFO:teuthology.orchestra.run.smithi150.stdout: "evadd": 97442, 2023-10-11T16:43:37.851 INFO:teuthology.orchestra.run.smithi150.stdout: "evex": 20436, 2023-10-11T16:43:37.852 INFO:teuthology.orchestra.run.smithi150.stdout: "evtrm": 20436, 2023-10-11T16:43:37.852 INFO:teuthology.orchestra.run.smithi150.stdout: "ev": 77006, 2023-10-11T16:43:37.852 INFO:teuthology.orchestra.run.smithi150.stdout: "evexg": 0, 2023-10-11T16:43:37.853 INFO:teuthology.orchestra.run.smithi150.stdout: "evexd": 2066, 2023-10-11T16:43:37.853 INFO:teuthology.orchestra.run.smithi150.stdout: "segadd": 158, 2023-10-11T16:43:37.853 INFO:teuthology.orchestra.run.smithi150.stdout: "segex": 30, 2023-10-11T16:43:37.853 INFO:teuthology.orchestra.run.smithi150.stdout: "segtrm": 30, 2023-10-11T16:43:37.854 INFO:teuthology.orchestra.run.smithi150.stdout: "seg": 131, 2023-10-11T16:43:37.854 INFO:teuthology.orchestra.run.smithi150.stdout: "segexg": 0, 2023-10-11T16:43:37.854 INFO:teuthology.orchestra.run.smithi150.stdout: "segexd": 3, 2023-10-11T16:43:37.854 INFO:teuthology.orchestra.run.smithi150.stdout: "expos": 61802902, 2023-10-11T16:43:37.855 INFO:teuthology.orchestra.run.smithi150.stdout: "wrpos": 348468590, 2023-10-11T16:43:37.855 INFO:teuthology.orchestra.run.smithi150.stdout: "rdpos": 0, 2023-10-11T16:43:37.856 INFO:teuthology.orchestra.run.smithi150.stdout: "jlat": { 2023-10-11T16:43:37.856 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 2905, 2023-10-11T16:43:37.856 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 206.997883254, 2023-10-11T16:43:37.856 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.071255725 2023-10-11T16:43:37.857 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.857 INFO:teuthology.orchestra.run.smithi150.stdout: "replayed": 0 2023-10-11T16:43:37.857 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.857 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_mem": { 2023-10-11T16:43:37.858 INFO:teuthology.orchestra.run.smithi150.stdout: "ino": 9482, 2023-10-11T16:43:37.858 INFO:teuthology.orchestra.run.smithi150.stdout: "ino+": 68856, 2023-10-11T16:43:37.858 INFO:teuthology.orchestra.run.smithi150.stdout: "ino-": 59374, 2023-10-11T16:43:37.859 INFO:teuthology.orchestra.run.smithi150.stdout: "dir": 533, 2023-10-11T16:43:37.859 INFO:teuthology.orchestra.run.smithi150.stdout: "dir+": 2758, 2023-10-11T16:43:37.859 INFO:teuthology.orchestra.run.smithi150.stdout: "dir-": 2225, 2023-10-11T16:43:37.859 INFO:teuthology.orchestra.run.smithi150.stdout: "dn": 16747, 2023-10-11T16:43:37.860 INFO:teuthology.orchestra.run.smithi150.stdout: "dn+": 99565, 2023-10-11T16:43:37.860 INFO:teuthology.orchestra.run.smithi150.stdout: "dn-": 82818, 2023-10-11T16:43:37.860 INFO:teuthology.orchestra.run.smithi150.stdout: "cap": 0, 2023-10-11T16:43:37.860 INFO:teuthology.orchestra.run.smithi150.stdout: "cap+": 42127, 2023-10-11T16:43:37.861 INFO:teuthology.orchestra.run.smithi150.stdout: "cap-": 42127, 2023-10-11T16:43:37.861 INFO:teuthology.orchestra.run.smithi150.stdout: "rss": 245124, 2023-10-11T16:43:37.861 INFO:teuthology.orchestra.run.smithi150.stdout: "heap": 198964 2023-10-11T16:43:37.861 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.862 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_server": { 2023-10-11T16:43:37.862 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_client_request": 60478, 2023-10-11T16:43:37.862 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_server_request": 2, 2023-10-11T16:43:37.863 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_request": 56829, 2023-10-11T16:43:37.863 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_session": 26, 2023-10-11T16:43:37.863 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_peer_request": 256, 2023-10-11T16:43:37.863 INFO:teuthology.orchestra.run.smithi150.stdout: "req_create_latency": { 2023-10-11T16:43:37.864 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 9473, 2023-10-11T16:43:37.864 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 27.505865386, 2023-10-11T16:43:37.864 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002903606 2023-10-11T16:43:37.864 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.865 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getattr_latency": { 2023-10-11T16:43:37.865 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 38, 2023-10-11T16:43:37.865 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.048423710, 2023-10-11T16:43:37.865 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.001274308 2023-10-11T16:43:37.866 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.866 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getfilelock_latency": { 2023-10-11T16:43:37.866 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.867 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.867 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.867 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.867 INFO:teuthology.orchestra.run.smithi150.stdout: "req_link_latency": { 2023-10-11T16:43:37.868 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.868 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.868 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.868 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.869 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookup_latency": { 2023-10-11T16:43:37.869 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 20028, 2023-10-11T16:43:37.869 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 59.602407581, 2023-10-11T16:43:37.870 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002975954 2023-10-11T16:43:37.870 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.870 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookuphash_latency": { 2023-10-11T16:43:37.871 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.871 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.871 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.871 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.872 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupino_latency": { 2023-10-11T16:43:37.872 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.872 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.872 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.873 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.873 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupname_latency": { 2023-10-11T16:43:37.873 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.873 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.874 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.874 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.874 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupparent_latency": { 2023-10-11T16:43:37.875 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.875 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.875 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.875 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.876 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupsnap_latency": { 2023-10-11T16:43:37.876 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.876 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.877 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.877 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.877 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lssnap_latency": { 2023-10-11T16:43:37.877 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.878 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.878 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.878 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.879 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mkdir_latency": { 2023-10-11T16:43:37.879 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.879 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.879 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.880 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.880 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mknod_latency": { 2023-10-11T16:43:37.880 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.880 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.881 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.881 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.881 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mksnap_latency": { 2023-10-11T16:43:37.882 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.882 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.882 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.883 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.883 INFO:teuthology.orchestra.run.smithi150.stdout: "req_open_latency": { 2023-10-11T16:43:37.883 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.883 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.884 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.884 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.884 INFO:teuthology.orchestra.run.smithi150.stdout: "req_readdir_latency": { 2023-10-11T16:43:37.884 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 894, 2023-10-11T16:43:37.885 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 2.525350348, 2023-10-11T16:43:37.885 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002824776 2023-10-11T16:43:37.885 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.885 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rename_latency": { 2023-10-11T16:43:37.886 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.886 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.886 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.887 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.887 INFO:teuthology.orchestra.run.smithi150.stdout: "req_renamesnap_latency": { 2023-10-11T16:43:37.887 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.887 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.888 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.888 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.888 INFO:teuthology.orchestra.run.smithi150.stdout: "req_snapdiff_latency": { 2023-10-11T16:43:37.889 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.889 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.889 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.889 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.890 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmdir_latency": { 2023-10-11T16:43:37.890 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 1, 2023-10-11T16:43:37.890 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.031019215, 2023-10-11T16:43:37.890 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.031019215 2023-10-11T16:43:37.891 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.891 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmsnap_latency": { 2023-10-11T16:43:37.891 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.892 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.892 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.892 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.892 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmxattr_latency": { 2023-10-11T16:43:37.893 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.893 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.893 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.893 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.894 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setattr_latency": { 2023-10-11T16:43:37.894 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.894 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.895 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.895 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.895 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setdirlayout_latency": { 2023-10-11T16:43:37.895 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.896 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.896 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.896 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.896 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setfilelock_latency": { 2023-10-11T16:43:37.897 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.897 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.897 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.898 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.898 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setlayout_latency": { 2023-10-11T16:43:37.898 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.898 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.899 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.899 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.899 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setxattr_latency": { 2023-10-11T16:43:37.900 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.900 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.900 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.900 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.901 INFO:teuthology.orchestra.run.smithi150.stdout: "req_symlink_latency": { 2023-10-11T16:43:37.901 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.901 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.902 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.902 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.902 INFO:teuthology.orchestra.run.smithi150.stdout: "req_unlink_latency": { 2023-10-11T16:43:37.902 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 20378, 2023-10-11T16:43:37.903 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 104.481770899, 2023-10-11T16:43:37.903 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.005127184 2023-10-11T16:43:37.903 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.903 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_revoke_eviction": 0, 2023-10-11T16:43:37.904 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_acquisition_throttle": 0, 2023-10-11T16:43:37.904 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getvxattr_latency": { 2023-10-11T16:43:37.904 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.904 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.905 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.905 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.905 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.906 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_sessions": { 2023-10-11T16:43:37.906 INFO:teuthology.orchestra.run.smithi150.stdout: "session_count": 2, 2023-10-11T16:43:37.906 INFO:teuthology.orchestra.run.smithi150.stdout: "session_add": 4, 2023-10-11T16:43:37.906 INFO:teuthology.orchestra.run.smithi150.stdout: "session_remove": 2, 2023-10-11T16:43:37.907 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_open": 2, 2023-10-11T16:43:37.907 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_stale": 0, 2023-10-11T16:43:37.907 INFO:teuthology.orchestra.run.smithi150.stdout: "total_load": 22172, 2023-10-11T16:43:37.908 INFO:teuthology.orchestra.run.smithi150.stdout: "average_load": 11086, 2023-10-11T16:43:37.908 INFO:teuthology.orchestra.run.smithi150.stdout: "avg_session_uptime": 623 2023-10-11T16:43:37.908 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.909 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2023-10-11T16:43:37.909 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:37.909 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:37.909 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:37.910 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:37.910 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:37.910 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:37.910 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:37.911 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:37.911 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:37.911 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:37.912 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:37.912 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:37.912 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:37.912 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:37.913 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:37.913 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:37.913 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:37.913 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:37.914 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:37.914 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:37.914 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:37.915 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:37.915 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:37.915 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:37.915 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:37.916 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:37.916 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:37.916 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:37.916 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:37.917 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:37.917 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:37.917 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2023-10-11T16:43:37.918 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:37.918 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:37.918 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:37.918 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:37.919 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 599447, 2023-10-11T16:43:37.919 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 8793, 2023-10-11T16:43:37.919 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 176, 2023-10-11T16:43:37.919 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 2, 2023-10-11T16:43:37.920 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2023-10-11T16:43:37.920 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2023-10-11T16:43:37.920 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:37.921 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:37.921 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:37.921 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:37.921 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 10152, 2023-10-11T16:43:37.922 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 198, 2023-10-11T16:43:37.922 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:37.922 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:37.922 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:37.923 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2023-10-11T16:43:37.923 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 34029510, 2023-10-11T16:43:37.923 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 308339, 2023-10-11T16:43:37.924 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:37.924 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2023-10-11T16:43:37.924 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:37.924 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2023-10-11T16:43:37.925 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.925 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter": { 2023-10-11T16:43:37.925 INFO:teuthology.orchestra.run.smithi150.stdout: "op_active": 0, 2023-10-11T16:43:37.926 INFO:teuthology.orchestra.run.smithi150.stdout: "op_laggy": 0, 2023-10-11T16:43:37.926 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send": 55838, 2023-10-11T16:43:37.926 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send_bytes": 361283661, 2023-10-11T16:43:37.926 INFO:teuthology.orchestra.run.smithi150.stdout: "op_resend": 27, 2023-10-11T16:43:37.927 INFO:teuthology.orchestra.run.smithi150.stdout: "op_reply": 55838, 2023-10-11T16:43:37.927 INFO:teuthology.orchestra.run.smithi150.stdout: "op_latency": { 2023-10-11T16:43:37.927 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 55838, 2023-10-11T16:43:37.928 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 3367.253259980, 2023-10-11T16:43:37.928 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.060303973 2023-10-11T16:43:37.928 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.928 INFO:teuthology.orchestra.run.smithi150.stdout: "op_inflight": 0, 2023-10-11T16:43:37.929 INFO:teuthology.orchestra.run.smithi150.stdout: "oplen_avg": { 2023-10-11T16:43:37.929 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 55838, 2023-10-11T16:43:37.929 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 69636 2023-10-11T16:43:37.930 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.930 INFO:teuthology.orchestra.run.smithi150.stdout: "op": 55838, 2023-10-11T16:43:37.930 INFO:teuthology.orchestra.run.smithi150.stdout: "op_r": 2453, 2023-10-11T16:43:37.930 INFO:teuthology.orchestra.run.smithi150.stdout: "op_w": 53385, 2023-10-11T16:43:37.931 INFO:teuthology.orchestra.run.smithi150.stdout: "op_rmw": 0, 2023-10-11T16:43:37.931 INFO:teuthology.orchestra.run.smithi150.stdout: "op_pg": 0, 2023-10-11T16:43:37.931 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_stat": 220, 2023-10-11T16:43:37.932 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_create": 8068, 2023-10-11T16:43:37.932 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_read": 385, 2023-10-11T16:43:37.932 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_write": 3290, 2023-10-11T16:43:37.932 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writefull": 148, 2023-10-11T16:43:37.933 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writesame": 0, 2023-10-11T16:43:37.933 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_append": 0, 2023-10-11T16:43:37.933 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_zero": 0, 2023-10-11T16:43:37.933 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_truncate": 0, 2023-10-11T16:43:37.934 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_delete": 40945, 2023-10-11T16:43:37.934 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_mapext": 0, 2023-10-11T16:43:37.934 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_sparse_read": 0, 2023-10-11T16:43:37.934 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_clonerange": 0, 2023-10-11T16:43:37.935 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_getxattr": 1591, 2023-10-11T16:43:37.935 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_setxattr": 12107, 2023-10-11T16:43:37.935 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_cmpxattr": 0, 2023-10-11T16:43:37.936 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_rmxattr": 0, 2023-10-11T16:43:37.936 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_resetxattrs": 0, 2023-10-11T16:43:37.936 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_call": 0, 2023-10-11T16:43:37.936 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_watch": 0, 2023-10-11T16:43:37.937 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_notify": 0, 2023-10-11T16:43:37.937 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_src_cmpxattr": 0, 2023-10-11T16:43:37.937 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls": 0, 2023-10-11T16:43:37.938 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls_filter": 0, 2023-10-11T16:43:37.938 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_other": 0, 2023-10-11T16:43:37.938 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_active": 0, 2023-10-11T16:43:37.938 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_send": 0, 2023-10-11T16:43:37.939 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_resend": 0, 2023-10-11T16:43:37.939 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_ping": 0, 2023-10-11T16:43:37.939 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_active": 0, 2023-10-11T16:43:37.939 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_send": 0, 2023-10-11T16:43:37.940 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_resend": 0, 2023-10-11T16:43:37.940 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_active": 0, 2023-10-11T16:43:37.940 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_send": 0, 2023-10-11T16:43:37.941 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_resend": 0, 2023-10-11T16:43:37.941 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_active": 0, 2023-10-11T16:43:37.941 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_send": 0, 2023-10-11T16:43:37.941 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_resend": 0, 2023-10-11T16:43:37.942 INFO:teuthology.orchestra.run.smithi150.stdout: "command_active": 0, 2023-10-11T16:43:37.942 INFO:teuthology.orchestra.run.smithi150.stdout: "command_send": 0, 2023-10-11T16:43:37.942 INFO:teuthology.orchestra.run.smithi150.stdout: "command_resend": 0, 2023-10-11T16:43:37.942 INFO:teuthology.orchestra.run.smithi150.stdout: "map_epoch": 102, 2023-10-11T16:43:37.943 INFO:teuthology.orchestra.run.smithi150.stdout: "map_full": 0, 2023-10-11T16:43:37.943 INFO:teuthology.orchestra.run.smithi150.stdout: "map_inc": 15, 2023-10-11T16:43:37.943 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_sessions": 12, 2023-10-11T16:43:37.944 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_open": 12, 2023-10-11T16:43:37.944 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_close": 0, 2023-10-11T16:43:37.944 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_laggy": 0, 2023-10-11T16:43:37.944 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_wr": 1853, 2023-10-11T16:43:37.945 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_rd": 954, 2023-10-11T16:43:37.945 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_del": 75 2023-10-11T16:43:37.945 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.945 INFO:teuthology.orchestra.run.smithi150.stdout: "oft": { 2023-10-11T16:43:37.946 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_objs": 1, 2023-10-11T16:43:37.946 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_kv_pairs": 0, 2023-10-11T16:43:37.946 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_updates": 0, 2023-10-11T16:43:37.946 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_removes": 0 2023-10-11T16:43:37.947 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.947 INFO:teuthology.orchestra.run.smithi150.stdout: "osd-slow-ops": { 2023-10-11T16:43:37.947 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_ops_count": 58070 2023-10-11T16:43:37.948 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.948 INFO:teuthology.orchestra.run.smithi150.stdout: "purge_queue": { 2023-10-11T16:43:37.948 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops": 0, 2023-10-11T16:43:37.948 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops_high_water": 16, 2023-10-11T16:43:37.949 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing": 0, 2023-10-11T16:43:37.949 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_high_water": 8, 2023-10-11T16:43:37.949 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executed": 20378, 2023-10-11T16:43:37.950 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_item_in_journal": 0 2023-10-11T16:43:37.950 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.950 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:37.950 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:37.951 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2023-10-11T16:43:37.951 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:37.951 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 282629, 2023-10-11T16:43:37.951 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 120064119, 2023-10-11T16:43:37.952 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:37.952 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 282629, 2023-10-11T16:43:37.952 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:37.953 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:37.953 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 282629, 2023-10-11T16:43:37.953 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 120064119, 2023-10-11T16:43:37.953 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:37.954 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.954 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.954 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.955 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.955 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.955 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_bytes": { 2023-10-11T16:43:37.955 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:37.956 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2023-10-11T16:43:37.956 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:37.956 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2023-10-11T16:43:37.956 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2023-10-11T16:43:37.957 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:37.957 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:37.957 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 55838, 2023-10-11T16:43:37.958 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 363818863, 2023-10-11T16:43:37.958 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 55838, 2023-10-11T16:43:37.958 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 363818863, 2023-10-11T16:43:37.958 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:37.959 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.959 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.959 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.960 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.960 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.960 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_ops": { 2023-10-11T16:43:37.961 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:37.961 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 1024, 2023-10-11T16:43:37.961 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:37.961 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2023-10-11T16:43:37.962 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2023-10-11T16:43:37.962 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:37.962 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:37.962 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 55838, 2023-10-11T16:43:37.963 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 55838, 2023-10-11T16:43:37.963 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 55838, 2023-10-11T16:43:37.963 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 55838, 2023-10-11T16:43:37.964 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:37.964 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.964 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.964 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.965 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.965 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.965 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle": { 2023-10-11T16:43:37.966 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:37.966 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2023-10-11T16:43:37.966 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:37.966 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 20378, 2023-10-11T16:43:37.967 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 2542834, 2023-10-11T16:43:37.967 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:37.967 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 20378, 2023-10-11T16:43:37.968 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:37.968 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:37.968 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 385, 2023-10-11T16:43:37.968 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 2542834, 2023-10-11T16:43:37.969 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:37.969 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.969 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.970 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.970 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.970 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:37.970 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle-0x562e8f6b00a0": { 2023-10-11T16:43:37.971 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:37.971 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2023-10-11T16:43:37.971 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:37.972 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 97442, 2023-10-11T16:43:37.972 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 344274286, 2023-10-11T16:43:37.972 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:37.972 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 97442, 2023-10-11T16:43:37.973 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:37.973 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:37.973 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 2905, 2023-10-11T16:43:37.974 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 344274286, 2023-10-11T16:43:37.974 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:37.974 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:37.974 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:37.975 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:37.975 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.975 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:37.975 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-10-11T16:43:37.976 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:37 smithi130 ceph-mon[131859]: pgmap v342: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-11T16:43:38.200 INFO:tasks.check_counter:Daemon mds.c mds.exported=1106 2023-10-11T16:43:38.201 INFO:tasks.check_counter:Daemon mds.c mds.imported=651 2023-10-11T16:43:38.201 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=826 2023-10-11T16:43:38.201 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=3327 2023-10-11T16:43:38.202 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2023-10-11T16:43:38.202 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2023-10-11T16:43:38.202 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2023-10-11T16:43:38.203 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=82 2023-10-11T16:43:38.203 DEBUG:tasks.check_counter:Getting stats from f 2023-10-11T16:43:38.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.f perf dump 2023-10-11T16:43:38.847 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:43:39.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:39 smithi111 ceph-mon[98527]: pgmap v343: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:39.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:39 smithi150 ceph-mon[132217]: pgmap v343: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:39.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:39 smithi130 ceph-mon[131859]: pgmap v343: 129 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 100 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:40.089 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-10-11T16:43:40.090 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:40.090 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 1636, 2023-10-11T16:43:40.091 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 1636, 2023-10-11T16:43:40.091 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 159915910, 2023-10-11T16:43:40.091 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 508007, 2023-10-11T16:43:40.091 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 8, 2023-10-11T16:43:40.092 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 4, 2023-10-11T16:43:40.092 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.221751868, 2023-10-11T16:43:40.092 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.044242786, 2023-10-11T16:43:40.093 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.108603661, 2023-10-11T16:43:40.093 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.055189524, 2023-10-11T16:43:40.093 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:40.094 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 1635, 2023-10-11T16:43:40.094 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.075684165, 2023-10-11T16:43:40.094 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000046290 2023-10-11T16:43:40.094 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.095 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:40.095 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.095 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.096 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.096 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.096 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-11T16:43:40.097 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 448 2023-10-11T16:43:40.097 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.097 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:40.098 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 1199, 2023-10-11T16:43:40.098 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 1174, 2023-10-11T16:43:40.098 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 101052339, 2023-10-11T16:43:40.098 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 420083, 2023-10-11T16:43:40.099 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 6, 2023-10-11T16:43:40.099 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 5, 2023-10-11T16:43:40.099 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.168631264, 2023-10-11T16:43:40.100 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.030131487, 2023-10-11T16:43:40.100 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.105716791, 2023-10-11T16:43:40.100 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.036111479, 2023-10-11T16:43:40.101 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:40.102 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 1173, 2023-10-11T16:43:40.102 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.079094243, 2023-10-11T16:43:40.102 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000067429 2023-10-11T16:43:40.103 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.103 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:40.103 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.104 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.104 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.104 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.105 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1514976, 2023-10-11T16:43:40.105 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 130480 2023-10-11T16:43:40.105 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.105 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:40.106 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 430, 2023-10-11T16:43:40.106 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 598, 2023-10-11T16:43:40.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 142747637, 2023-10-11T16:43:40.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 572916, 2023-10-11T16:43:40.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 8, 2023-10-11T16:43:40.107 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 6, 2023-10-11T16:43:40.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.110343175, 2023-10-11T16:43:40.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.013373909, 2023-10-11T16:43:40.108 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.081469117, 2023-10-11T16:43:40.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.010802405, 2023-10-11T16:43:40.109 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:40.109 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 595, 2023-10-11T16:43:40.109 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.049011215, 2023-10-11T16:43:40.110 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000082371 2023-10-11T16:43:40.110 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.111 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:40.112 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.112 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.112 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.113 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.113 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-11T16:43:40.113 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 439808 2023-10-11T16:43:40.114 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.114 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2023-10-11T16:43:40.115 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 1, 2023-10-11T16:43:40.115 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2023-10-11T16:43:40.116 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.116 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-MDSRank": { 2023-10-11T16:43:40.116 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2023-10-11T16:43:40.116 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2023-10-11T16:43:40.117 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 4701, 2023-10-11T16:43:40.117 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.135546433, 2023-10-11T16:43:40.117 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000028833 2023-10-11T16:43:40.118 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.118 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.118 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-PurgeQueue": { 2023-10-11T16:43:40.118 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2023-10-11T16:43:40.119 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2023-10-11T16:43:40.119 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.119 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.120 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.120 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.120 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.121 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": { 2023-10-11T16:43:40.121 INFO:teuthology.orchestra.run.smithi150.stdout: "request": 0, 2023-10-11T16:43:40.121 INFO:teuthology.orchestra.run.smithi150.stdout: "reply": 0, 2023-10-11T16:43:40.122 INFO:teuthology.orchestra.run.smithi150.stdout: "reply_latency": { 2023-10-11T16:43:40.122 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.122 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.122 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.123 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.123 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_reply": 0, 2023-10-11T16:43:40.124 INFO:teuthology.orchestra.run.smithi150.stdout: "forward": 0, 2023-10-11T16:43:40.124 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_complete": 0, 2023-10-11T16:43:40.125 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_keys": 0, 2023-10-11T16:43:40.125 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_commit": 0, 2023-10-11T16:43:40.125 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_split": 0, 2023-10-11T16:43:40.126 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_merge": 0, 2023-10-11T16:43:40.126 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes": 86320, 2023-10-11T16:43:40.126 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_top": 1589, 2023-10-11T16:43:40.126 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_bottom": 71720, 2023-10-11T16:43:40.127 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pin_tail": 13011, 2023-10-11T16:43:40.127 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pinned": 84049, 2023-10-11T16:43:40.127 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_expired": 0, 2023-10-11T16:43:40.128 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_with_caps": 0, 2023-10-11T16:43:40.128 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": 0, 2023-10-11T16:43:40.128 INFO:teuthology.orchestra.run.smithi150.stdout: "subtrees": 3, 2023-10-11T16:43:40.128 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse": 0, 2023-10-11T16:43:40.129 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_hit": 0, 2023-10-11T16:43:40.129 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_forward": 0, 2023-10-11T16:43:40.129 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_discover": 0, 2023-10-11T16:43:40.129 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_dir_fetch": 0, 2023-10-11T16:43:40.130 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_remote_ino": 0, 2023-10-11T16:43:40.130 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_lock": 0, 2023-10-11T16:43:40.130 INFO:teuthology.orchestra.run.smithi150.stdout: "load_cent": 0, 2023-10-11T16:43:40.130 INFO:teuthology.orchestra.run.smithi150.stdout: "q": 0, 2023-10-11T16:43:40.132 INFO:teuthology.orchestra.run.smithi150.stdout: "exported": 0, 2023-10-11T16:43:40.132 INFO:teuthology.orchestra.run.smithi150.stdout: "exported_inodes": 0, 2023-10-11T16:43:40.132 INFO:teuthology.orchestra.run.smithi150.stdout: "imported": 0, 2023-10-11T16:43:40.132 INFO:teuthology.orchestra.run.smithi150.stdout: "imported_inodes": 0, 2023-10-11T16:43:40.133 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_dir_fetch": 0, 2023-10-11T16:43:40.133 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_backtrace_fetch": 0, 2023-10-11T16:43:40.133 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_peer_discover": 0, 2023-10-11T16:43:40.133 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rfiles": 0, 2023-10-11T16:43:40.134 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rbytes": 0, 2023-10-11T16:43:40.134 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rsnaps": 0, 2023-10-11T16:43:40.134 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_fetch": 0, 2023-10-11T16:43:40.135 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_set_tag": 0, 2023-10-11T16:43:40.135 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_repaired": 0, 2023-10-11T16:43:40.135 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_inotable_repaired": 0, 2023-10-11T16:43:40.135 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_inodes": 0, 2023-10-11T16:43:40.136 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_base_inodes": 0, 2023-10-11T16:43:40.136 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dirfrag_rstats": 0, 2023-10-11T16:43:40.136 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_file_inodes": 0, 2023-10-11T16:43:40.136 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_inode_file_caps": 0, 2023-10-11T16:43:40.137 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_revoke": 0, 2023-10-11T16:43:40.137 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_grant": 0, 2023-10-11T16:43:40.137 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_trunc": 0, 2023-10-11T16:43:40.138 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-11T16:43:40.138 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-11T16:43:40.138 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps": 0, 2023-10-11T16:43:40.138 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps_dirty": 0, 2023-10-11T16:43:40.139 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_cap_release": 0, 2023-10-11T16:43:40.139 INFO:teuthology.orchestra.run.smithi150.stdout: "process_request_cap_release": 0 2023-10-11T16:43:40.139 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.139 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache": { 2023-10-11T16:43:40.140 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update": 0, 2023-10-11T16:43:40.140 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update_receipt": 0, 2023-10-11T16:43:40.140 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_try_discover": 0, 2023-10-11T16:43:40.141 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_send_discover": 0, 2023-10-11T16:43:40.141 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_handle_discover": 0, 2023-10-11T16:43:40.141 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays": 0, 2023-10-11T16:43:40.141 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_delayed": 0, 2023-10-11T16:43:40.142 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_enqueuing": 0, 2023-10-11T16:43:40.142 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_created": 0, 2023-10-11T16:43:40.142 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_enqueued": 0, 2023-10-11T16:43:40.142 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_reintegrated": 0, 2023-10-11T16:43:40.143 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_migrated": 0, 2023-10-11T16:43:40.143 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_processing": 0, 2023-10-11T16:43:40.143 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_enqueued": 0, 2023-10-11T16:43:40.144 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_prioritized": 0, 2023-10-11T16:43:40.144 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_started": 0, 2023-10-11T16:43:40.144 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_completed": 0, 2023-10-11T16:43:40.144 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_enqueue_scrub": 0, 2023-10-11T16:43:40.145 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_exportdir": 0, 2023-10-11T16:43:40.145 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_flush": 0, 2023-10-11T16:43:40.145 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragmentdir": 0, 2023-10-11T16:43:40.145 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragstats": 0, 2023-10-11T16:43:40.146 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_inodestats": 0 2023-10-11T16:43:40.146 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.146 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log": { 2023-10-11T16:43:40.147 INFO:teuthology.orchestra.run.smithi150.stdout: "evadd": 0, 2023-10-11T16:43:40.147 INFO:teuthology.orchestra.run.smithi150.stdout: "evex": 0, 2023-10-11T16:43:40.147 INFO:teuthology.orchestra.run.smithi150.stdout: "evtrm": 0, 2023-10-11T16:43:40.147 INFO:teuthology.orchestra.run.smithi150.stdout: "ev": 0, 2023-10-11T16:43:40.148 INFO:teuthology.orchestra.run.smithi150.stdout: "evexg": 0, 2023-10-11T16:43:40.148 INFO:teuthology.orchestra.run.smithi150.stdout: "evexd": 0, 2023-10-11T16:43:40.148 INFO:teuthology.orchestra.run.smithi150.stdout: "segadd": 0, 2023-10-11T16:43:40.148 INFO:teuthology.orchestra.run.smithi150.stdout: "segex": 0, 2023-10-11T16:43:40.149 INFO:teuthology.orchestra.run.smithi150.stdout: "segtrm": 0, 2023-10-11T16:43:40.149 INFO:teuthology.orchestra.run.smithi150.stdout: "seg": 130, 2023-10-11T16:43:40.149 INFO:teuthology.orchestra.run.smithi150.stdout: "segexg": 0, 2023-10-11T16:43:40.150 INFO:teuthology.orchestra.run.smithi150.stdout: "segexd": 0, 2023-10-11T16:43:40.150 INFO:teuthology.orchestra.run.smithi150.stdout: "expos": 8396621, 2023-10-11T16:43:40.150 INFO:teuthology.orchestra.run.smithi150.stdout: "wrpos": 0, 2023-10-11T16:43:40.150 INFO:teuthology.orchestra.run.smithi150.stdout: "rdpos": 404793852, 2023-10-11T16:43:40.151 INFO:teuthology.orchestra.run.smithi150.stdout: "jlat": { 2023-10-11T16:43:40.151 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.151 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.151 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.152 INFO:teuthology.orchestra.run.smithi150.stdout: "replayed": 92801 2023-10-11T16:43:40.152 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.153 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_mem": { 2023-10-11T16:43:40.153 INFO:teuthology.orchestra.run.smithi150.stdout: "ino": 38611, 2023-10-11T16:43:40.153 INFO:teuthology.orchestra.run.smithi150.stdout: "ino+": 72657, 2023-10-11T16:43:40.153 INFO:teuthology.orchestra.run.smithi150.stdout: "ino-": 34046, 2023-10-11T16:43:40.154 INFO:teuthology.orchestra.run.smithi150.stdout: "dir": 1561, 2023-10-11T16:43:40.154 INFO:teuthology.orchestra.run.smithi150.stdout: "dir+": 2056, 2023-10-11T16:43:40.154 INFO:teuthology.orchestra.run.smithi150.stdout: "dir-": 495, 2023-10-11T16:43:40.155 INFO:teuthology.orchestra.run.smithi150.stdout: "dn": 86320, 2023-10-11T16:43:40.155 INFO:teuthology.orchestra.run.smithi150.stdout: "dn+": 109038, 2023-10-11T16:43:40.155 INFO:teuthology.orchestra.run.smithi150.stdout: "dn-": 22718, 2023-10-11T16:43:40.155 INFO:teuthology.orchestra.run.smithi150.stdout: "cap": 0, 2023-10-11T16:43:40.156 INFO:teuthology.orchestra.run.smithi150.stdout: "cap+": 0, 2023-10-11T16:43:40.156 INFO:teuthology.orchestra.run.smithi150.stdout: "cap-": 0, 2023-10-11T16:43:40.156 INFO:teuthology.orchestra.run.smithi150.stdout: "rss": 230488, 2023-10-11T16:43:40.156 INFO:teuthology.orchestra.run.smithi150.stdout: "heap": 207156 2023-10-11T16:43:40.157 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.157 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_server": { 2023-10-11T16:43:40.157 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_client_request": 0, 2023-10-11T16:43:40.158 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_server_request": 0, 2023-10-11T16:43:40.158 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_request": 0, 2023-10-11T16:43:40.158 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_session": 0, 2023-10-11T16:43:40.158 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_peer_request": 0, 2023-10-11T16:43:40.159 INFO:teuthology.orchestra.run.smithi150.stdout: "req_create_latency": { 2023-10-11T16:43:40.159 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.159 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.160 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.160 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.160 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getattr_latency": { 2023-10-11T16:43:40.160 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.161 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.161 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.161 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.162 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getfilelock_latency": { 2023-10-11T16:43:40.162 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.162 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.162 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.163 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.163 INFO:teuthology.orchestra.run.smithi150.stdout: "req_link_latency": { 2023-10-11T16:43:40.163 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.163 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.164 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.164 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.164 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookup_latency": { 2023-10-11T16:43:40.164 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.165 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.165 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.165 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.166 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookuphash_latency": { 2023-10-11T16:43:40.166 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.166 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.166 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.167 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.167 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupino_latency": { 2023-10-11T16:43:40.167 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.167 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.168 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.168 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.168 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupname_latency": { 2023-10-11T16:43:40.169 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.169 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.169 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.169 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.170 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupparent_latency": { 2023-10-11T16:43:40.170 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.170 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.171 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.171 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.171 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupsnap_latency": { 2023-10-11T16:43:40.171 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.172 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.172 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.172 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.173 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lssnap_latency": { 2023-10-11T16:43:40.173 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.173 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.173 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.174 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.174 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mkdir_latency": { 2023-10-11T16:43:40.174 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.174 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.175 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.175 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.175 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mknod_latency": { 2023-10-11T16:43:40.176 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.176 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.176 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.177 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.177 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mksnap_latency": { 2023-10-11T16:43:40.177 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.177 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.178 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.178 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.178 INFO:teuthology.orchestra.run.smithi150.stdout: "req_open_latency": { 2023-10-11T16:43:40.178 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.179 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.179 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.179 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.180 INFO:teuthology.orchestra.run.smithi150.stdout: "req_readdir_latency": { 2023-10-11T16:43:40.180 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.180 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.181 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.181 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.181 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rename_latency": { 2023-10-11T16:43:40.181 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.182 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.182 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.182 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.183 INFO:teuthology.orchestra.run.smithi150.stdout: "req_renamesnap_latency": { 2023-10-11T16:43:40.183 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.183 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.183 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.184 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.184 INFO:teuthology.orchestra.run.smithi150.stdout: "req_snapdiff_latency": { 2023-10-11T16:43:40.184 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.184 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.185 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.185 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.185 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmdir_latency": { 2023-10-11T16:43:40.186 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.186 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.186 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.186 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.187 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmsnap_latency": { 2023-10-11T16:43:40.187 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.187 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.188 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.188 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.188 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmxattr_latency": { 2023-10-11T16:43:40.189 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.189 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.189 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.189 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.190 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setattr_latency": { 2023-10-11T16:43:40.190 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.190 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.190 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.191 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.191 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setdirlayout_latency": { 2023-10-11T16:43:40.191 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.192 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.192 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.192 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.192 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setfilelock_latency": { 2023-10-11T16:43:40.193 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.193 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.193 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.193 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.194 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setlayout_latency": { 2023-10-11T16:43:40.194 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.194 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.195 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.195 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.195 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setxattr_latency": { 2023-10-11T16:43:40.195 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.196 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.196 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.196 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.197 INFO:teuthology.orchestra.run.smithi150.stdout: "req_symlink_latency": { 2023-10-11T16:43:40.197 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.197 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.197 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.198 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.198 INFO:teuthology.orchestra.run.smithi150.stdout: "req_unlink_latency": { 2023-10-11T16:43:40.198 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.198 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.199 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.199 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.199 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_revoke_eviction": 0, 2023-10-11T16:43:40.200 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_acquisition_throttle": 0, 2023-10-11T16:43:40.200 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getvxattr_latency": { 2023-10-11T16:43:40.200 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.200 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.201 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.201 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.201 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.201 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_sessions": { 2023-10-11T16:43:40.202 INFO:teuthology.orchestra.run.smithi150.stdout: "session_count": 2, 2023-10-11T16:43:40.202 INFO:teuthology.orchestra.run.smithi150.stdout: "session_add": 5, 2023-10-11T16:43:40.202 INFO:teuthology.orchestra.run.smithi150.stdout: "session_remove": 3, 2023-10-11T16:43:40.203 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_open": 4, 2023-10-11T16:43:40.203 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_stale": 0, 2023-10-11T16:43:40.203 INFO:teuthology.orchestra.run.smithi150.stdout: "total_load": 0, 2023-10-11T16:43:40.203 INFO:teuthology.orchestra.run.smithi150.stdout: "average_load": 0, 2023-10-11T16:43:40.204 INFO:teuthology.orchestra.run.smithi150.stdout: "avg_session_uptime": 4510 2023-10-11T16:43:40.204 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.204 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2023-10-11T16:43:40.205 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:40.205 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:40.205 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:40.205 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:40.206 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:40.206 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:40.206 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:40.206 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:40.207 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:40.207 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:40.207 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:40.208 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:40.208 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:40.208 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:40.208 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:40.209 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:40.209 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:40.209 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:40.209 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:40.210 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:40.210 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:40.210 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:40.211 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:40.211 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:40.211 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:40.211 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:40.212 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:40.212 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:40.212 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:40.212 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:40.213 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:40.213 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2023-10-11T16:43:40.213 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:40.214 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:40.214 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:40.214 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:40.214 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 1552671, 2023-10-11T16:43:40.215 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 38703, 2023-10-11T16:43:40.215 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:40.215 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:40.215 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2023-10-11T16:43:40.216 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2023-10-11T16:43:40.216 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:40.217 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:40.218 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:40.218 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:40.218 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 10152, 2023-10-11T16:43:40.219 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 198, 2023-10-11T16:43:40.219 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:40.219 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:40.219 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:40.220 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2023-10-11T16:43:40.220 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 149727916, 2023-10-11T16:43:40.220 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 1219650, 2023-10-11T16:43:40.221 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:40.221 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2023-10-11T16:43:40.221 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:40.221 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2023-10-11T16:43:40.222 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.222 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter": { 2023-10-11T16:43:40.222 INFO:teuthology.orchestra.run.smithi150.stdout: "op_active": 0, 2023-10-11T16:43:40.223 INFO:teuthology.orchestra.run.smithi150.stdout: "op_laggy": 0, 2023-10-11T16:43:40.223 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send": 2991, 2023-10-11T16:43:40.223 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send_bytes": 21, 2023-10-11T16:43:40.224 INFO:teuthology.orchestra.run.smithi150.stdout: "op_resend": 3, 2023-10-11T16:43:40.224 INFO:teuthology.orchestra.run.smithi150.stdout: "op_reply": 2991, 2023-10-11T16:43:40.224 INFO:teuthology.orchestra.run.smithi150.stdout: "op_latency": { 2023-10-11T16:43:40.224 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 2991, 2023-10-11T16:43:40.225 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 7.222859133, 2023-10-11T16:43:40.225 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002414864 2023-10-11T16:43:40.225 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.226 INFO:teuthology.orchestra.run.smithi150.stdout: "op_inflight": 0, 2023-10-11T16:43:40.226 INFO:teuthology.orchestra.run.smithi150.stdout: "oplen_avg": { 2023-10-11T16:43:40.226 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 2991, 2023-10-11T16:43:40.227 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 2992 2023-10-11T16:43:40.227 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.227 INFO:teuthology.orchestra.run.smithi150.stdout: "op": 2991, 2023-10-11T16:43:40.227 INFO:teuthology.orchestra.run.smithi150.stdout: "op_r": 2991, 2023-10-11T16:43:40.228 INFO:teuthology.orchestra.run.smithi150.stdout: "op_w": 0, 2023-10-11T16:43:40.228 INFO:teuthology.orchestra.run.smithi150.stdout: "op_rmw": 0, 2023-10-11T16:43:40.228 INFO:teuthology.orchestra.run.smithi150.stdout: "op_pg": 0, 2023-10-11T16:43:40.229 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_stat": 1903, 2023-10-11T16:43:40.229 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_create": 0, 2023-10-11T16:43:40.229 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_read": 1086, 2023-10-11T16:43:40.230 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_write": 0, 2023-10-11T16:43:40.230 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writefull": 0, 2023-10-11T16:43:40.230 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writesame": 0, 2023-10-11T16:43:40.230 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_append": 0, 2023-10-11T16:43:40.231 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_zero": 0, 2023-10-11T16:43:40.231 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_truncate": 0, 2023-10-11T16:43:40.231 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_delete": 0, 2023-10-11T16:43:40.232 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_mapext": 0, 2023-10-11T16:43:40.232 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_sparse_read": 0, 2023-10-11T16:43:40.232 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_clonerange": 0, 2023-10-11T16:43:40.233 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_getxattr": 1, 2023-10-11T16:43:40.233 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_setxattr": 0, 2023-10-11T16:43:40.233 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_cmpxattr": 0, 2023-10-11T16:43:40.233 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_rmxattr": 0, 2023-10-11T16:43:40.234 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_resetxattrs": 0, 2023-10-11T16:43:40.234 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_call": 0, 2023-10-11T16:43:40.234 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_watch": 0, 2023-10-11T16:43:40.234 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_notify": 0, 2023-10-11T16:43:40.235 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_src_cmpxattr": 0, 2023-10-11T16:43:40.235 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls": 0, 2023-10-11T16:43:40.235 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls_filter": 0, 2023-10-11T16:43:40.236 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_other": 0, 2023-10-11T16:43:40.236 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_active": 0, 2023-10-11T16:43:40.236 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_send": 0, 2023-10-11T16:43:40.236 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_resend": 0, 2023-10-11T16:43:40.237 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_ping": 0, 2023-10-11T16:43:40.237 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_active": 0, 2023-10-11T16:43:40.237 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_send": 0, 2023-10-11T16:43:40.237 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_resend": 0, 2023-10-11T16:43:40.238 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_active": 0, 2023-10-11T16:43:40.238 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_send": 0, 2023-10-11T16:43:40.238 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_resend": 0, 2023-10-11T16:43:40.239 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_active": 0, 2023-10-11T16:43:40.239 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_send": 0, 2023-10-11T16:43:40.239 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_resend": 0, 2023-10-11T16:43:40.239 INFO:teuthology.orchestra.run.smithi150.stdout: "command_active": 0, 2023-10-11T16:43:40.240 INFO:teuthology.orchestra.run.smithi150.stdout: "command_send": 0, 2023-10-11T16:43:40.240 INFO:teuthology.orchestra.run.smithi150.stdout: "command_resend": 0, 2023-10-11T16:43:40.240 INFO:teuthology.orchestra.run.smithi150.stdout: "map_epoch": 102, 2023-10-11T16:43:40.240 INFO:teuthology.orchestra.run.smithi150.stdout: "map_full": 0, 2023-10-11T16:43:40.241 INFO:teuthology.orchestra.run.smithi150.stdout: "map_inc": 15, 2023-10-11T16:43:40.241 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_sessions": 11, 2023-10-11T16:43:40.241 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_open": 11, 2023-10-11T16:43:40.242 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_close": 0, 2023-10-11T16:43:40.242 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_laggy": 0, 2023-10-11T16:43:40.242 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_wr": 0, 2023-10-11T16:43:40.242 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_rd": 2, 2023-10-11T16:43:40.243 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_del": 0 2023-10-11T16:43:40.243 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.243 INFO:teuthology.orchestra.run.smithi150.stdout: "oft": { 2023-10-11T16:43:40.244 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_objs": 0, 2023-10-11T16:43:40.244 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_kv_pairs": 0, 2023-10-11T16:43:40.244 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_updates": 0, 2023-10-11T16:43:40.244 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_removes": 0 2023-10-11T16:43:40.245 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.245 INFO:teuthology.orchestra.run.smithi150.stdout: "osd-slow-ops": { 2023-10-11T16:43:40.245 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_ops_count": 0 2023-10-11T16:43:40.245 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.246 INFO:teuthology.orchestra.run.smithi150.stdout: "purge_queue": { 2023-10-11T16:43:40.246 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops": 0, 2023-10-11T16:43:40.246 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops_high_water": 0, 2023-10-11T16:43:40.247 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing": 0, 2023-10-11T16:43:40.247 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_high_water": 0, 2023-10-11T16:43:40.247 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executed": 0, 2023-10-11T16:43:40.248 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_item_in_journal": 0 2023-10-11T16:43:40.248 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.248 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:40.248 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:40.249 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2023-10-11T16:43:40.249 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:40.249 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 3261, 2023-10-11T16:43:40.250 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 403405691, 2023-10-11T16:43:40.250 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:40.250 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 3261, 2023-10-11T16:43:40.250 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:40.251 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:40.251 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 3261, 2023-10-11T16:43:40.251 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 403405691, 2023-10-11T16:43:40.252 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:40.252 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.252 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.252 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.253 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.253 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.253 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_bytes": { 2023-10-11T16:43:40.254 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:40.254 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2023-10-11T16:43:40.254 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:40.254 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2023-10-11T16:43:40.255 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2023-10-11T16:43:40.255 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:40.255 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:40.256 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 2991, 2023-10-11T16:43:40.256 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 401370724, 2023-10-11T16:43:40.256 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 2991, 2023-10-11T16:43:40.257 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 401370724, 2023-10-11T16:43:40.257 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:40.257 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.257 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.258 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.258 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.258 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.259 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_ops": { 2023-10-11T16:43:40.259 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:40.259 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 1024, 2023-10-11T16:43:40.259 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:40.260 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2023-10-11T16:43:40.260 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2023-10-11T16:43:40.260 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:40.263 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:40.263 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 2991, 2023-10-11T16:43:40.263 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 2991, 2023-10-11T16:43:40.264 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 2991, 2023-10-11T16:43:40.264 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 2991, 2023-10-11T16:43:40.264 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:40.264 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.265 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.265 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.265 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.265 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.266 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle": { 2023-10-11T16:43:40.266 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:40.266 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2023-10-11T16:43:40.267 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:40.267 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2023-10-11T16:43:40.267 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2023-10-11T16:43:40.267 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:40.268 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:40.268 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:40.268 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:40.268 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 0, 2023-10-11T16:43:40.269 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 0, 2023-10-11T16:43:40.269 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:40.269 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.270 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.270 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.270 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.270 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:40.271 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle-0x55d0e0a06460": { 2023-10-11T16:43:40.271 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:40.271 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2023-10-11T16:43:40.272 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:40.272 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2023-10-11T16:43:40.272 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2023-10-11T16:43:40.272 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:40.273 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:40.273 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:40.273 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:40.274 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 0, 2023-10-11T16:43:40.274 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 0, 2023-10-11T16:43:40.274 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:40.274 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:40.275 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:40.275 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:40.275 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.276 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:40.276 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-10-11T16:43:40.489 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2023-10-11T16:43:40.490 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2023-10-11T16:43:40.490 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2023-10-11T16:43:40.490 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2023-10-11T16:43:40.491 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2023-10-11T16:43:40.491 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2023-10-11T16:43:40.491 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2023-10-11T16:43:40.492 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2023-10-11T16:43:40.492 DEBUG:tasks.check_counter:Getting stats from i 2023-10-11T16:43:40.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.i perf dump 2023-10-11T16:43:41.134 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:43:41.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:41 smithi111 ceph-mon[98527]: pgmap v344: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:43:41.647 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:41 smithi150 ceph-mon[132217]: pgmap v344: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:43:41.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:41 smithi130 ceph-mon[131859]: pgmap v344: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-11T16:43:42.373 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-10-11T16:43:42.373 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:42.374 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 25390, 2023-10-11T16:43:42.374 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 24536, 2023-10-11T16:43:42.374 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 45338876, 2023-10-11T16:43:42.374 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 206476392, 2023-10-11T16:43:42.375 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 29, 2023-10-11T16:43:42.375 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 4, 2023-10-11T16:43:42.375 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 2.144557373, 2023-10-11T16:43:42.375 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.879470776, 2023-10-11T16:43:42.376 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.568084237, 2023-10-11T16:43:42.376 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.685393965, 2023-10-11T16:43:42.376 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:42.377 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 24535, 2023-10-11T16:43:42.377 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 4.148251602, 2023-10-11T16:43:42.377 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000169074 2023-10-11T16:43:42.377 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.378 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:42.378 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 15965, 2023-10-11T16:43:42.378 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.002862902, 2023-10-11T16:43:42.379 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000179 2023-10-11T16:43:42.379 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.379 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 3904, 2023-10-11T16:43:42.379 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 1616 2023-10-11T16:43:42.380 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.380 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:42.380 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 118052, 2023-10-11T16:43:42.380 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 211616, 2023-10-11T16:43:42.381 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 44829591, 2023-10-11T16:43:42.381 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 196578843, 2023-10-11T16:43:42.381 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 14, 2023-10-11T16:43:42.382 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 8, 2023-10-11T16:43:42.382 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 7.176873354, 2023-10-11T16:43:42.382 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 3.150863093, 2023-10-11T16:43:42.382 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 23.480285864, 2023-10-11T16:43:42.383 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.625355993, 2023-10-11T16:43:42.383 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:42.383 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 211613, 2023-10-11T16:43:42.383 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 17.860812051, 2023-10-11T16:43:42.384 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000084403 2023-10-11T16:43:42.384 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.384 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:42.385 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 246715, 2023-10-11T16:43:42.385 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.043934082, 2023-10-11T16:43:42.385 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000178 2023-10-11T16:43:42.385 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.386 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1532352, 2023-10-11T16:43:42.386 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 465696 2023-10-11T16:43:42.386 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.386 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:42.387 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 171235, 2023-10-11T16:43:42.387 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 158932, 2023-10-11T16:43:42.387 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 62498212, 2023-10-11T16:43:42.388 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 126956561, 2023-10-11T16:43:42.388 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 16, 2023-10-11T16:43:42.388 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 5, 2023-10-11T16:43:42.388 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 8.676929525, 2023-10-11T16:43:42.389 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 3.872645034, 2023-10-11T16:43:42.389 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 5.229865343, 2023-10-11T16:43:42.389 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.701692788, 2023-10-11T16:43:42.389 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:42.390 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 158931, 2023-10-11T16:43:42.390 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 12.082161786, 2023-10-11T16:43:42.391 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000076021 2023-10-11T16:43:42.392 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.392 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:42.392 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 280109, 2023-10-11T16:43:42.393 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.038226944, 2023-10-11T16:43:42.393 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000136 2023-10-11T16:43:42.393 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.393 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 5472, 2023-10-11T16:43:42.394 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 141856 2023-10-11T16:43:42.394 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.394 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2023-10-11T16:43:42.394 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 1, 2023-10-11T16:43:42.395 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2023-10-11T16:43:42.395 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.395 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-MDSRank": { 2023-10-11T16:43:42.396 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2023-10-11T16:43:42.396 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2023-10-11T16:43:42.396 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 10772, 2023-10-11T16:43:42.396 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 87.272297887, 2023-10-11T16:43:42.397 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.008101772 2023-10-11T16:43:42.397 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.397 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.397 INFO:teuthology.orchestra.run.smithi150.stdout: "finisher-PurgeQueue": { 2023-10-11T16:43:42.398 INFO:teuthology.orchestra.run.smithi150.stdout: "queue_len": 0, 2023-10-11T16:43:42.398 INFO:teuthology.orchestra.run.smithi150.stdout: "complete_latency": { 2023-10-11T16:43:42.398 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 29665, 2023-10-11T16:43:42.399 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 13.706746754, 2023-10-11T16:43:42.399 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000462051 2023-10-11T16:43:42.399 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.399 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.400 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": { 2023-10-11T16:43:42.400 INFO:teuthology.orchestra.run.smithi150.stdout: "request": 73182, 2023-10-11T16:43:42.400 INFO:teuthology.orchestra.run.smithi150.stdout: "reply": 63669, 2023-10-11T16:43:42.401 INFO:teuthology.orchestra.run.smithi150.stdout: "reply_latency": { 2023-10-11T16:43:42.401 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 63669, 2023-10-11T16:43:42.401 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 1583.917331537, 2023-10-11T16:43:42.401 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.024877370 2023-10-11T16:43:42.402 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.402 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_reply": 0, 2023-10-11T16:43:42.402 INFO:teuthology.orchestra.run.smithi150.stdout: "forward": 9513, 2023-10-11T16:43:42.403 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_complete": 961, 2023-10-11T16:43:42.403 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_fetch_keys": 2, 2023-10-11T16:43:42.403 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_commit": 1257, 2023-10-11T16:43:42.404 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_split": 154, 2023-10-11T16:43:42.404 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_merge": 0, 2023-10-11T16:43:42.404 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes": 69181, 2023-10-11T16:43:42.404 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_top": 2325, 2023-10-11T16:43:42.405 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_bottom": 4744, 2023-10-11T16:43:42.405 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pin_tail": 62112, 2023-10-11T16:43:42.405 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_pinned": 65860, 2023-10-11T16:43:42.405 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_expired": 20482, 2023-10-11T16:43:42.406 INFO:teuthology.orchestra.run.smithi150.stdout: "inodes_with_caps": 6, 2023-10-11T16:43:42.406 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": 6, 2023-10-11T16:43:42.407 INFO:teuthology.orchestra.run.smithi150.stdout: "subtrees": 4, 2023-10-11T16:43:42.407 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse": 92122, 2023-10-11T16:43:42.407 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_hit": 82199, 2023-10-11T16:43:42.407 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_forward": 7612, 2023-10-11T16:43:42.408 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_discover": 5, 2023-10-11T16:43:42.408 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_dir_fetch": 16, 2023-10-11T16:43:42.408 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_remote_ino": 0, 2023-10-11T16:43:42.408 INFO:teuthology.orchestra.run.smithi150.stdout: "traverse_lock": 0, 2023-10-11T16:43:42.409 INFO:teuthology.orchestra.run.smithi150.stdout: "load_cent": 0, 2023-10-11T16:43:42.409 INFO:teuthology.orchestra.run.smithi150.stdout: "q": 0, 2023-10-11T16:43:42.409 INFO:teuthology.orchestra.run.smithi150.stdout: "exported": 399, 2023-10-11T16:43:42.410 INFO:teuthology.orchestra.run.smithi150.stdout: "exported_inodes": 22709, 2023-10-11T16:43:42.410 INFO:teuthology.orchestra.run.smithi150.stdout: "imported": 982, 2023-10-11T16:43:42.410 INFO:teuthology.orchestra.run.smithi150.stdout: "imported_inodes": 0, 2023-10-11T16:43:42.410 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_dir_fetch": 0, 2023-10-11T16:43:42.411 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_backtrace_fetch": 12919, 2023-10-11T16:43:42.411 INFO:teuthology.orchestra.run.smithi150.stdout: "openino_peer_discover": 0, 2023-10-11T16:43:42.411 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rfiles": 4, 2023-10-11T16:43:42.412 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rbytes": 246, 2023-10-11T16:43:42.412 INFO:teuthology.orchestra.run.smithi150.stdout: "root_rsnaps": 6, 2023-10-11T16:43:42.412 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_fetch": 12919, 2023-10-11T16:43:42.412 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_set_tag": 0, 2023-10-11T16:43:42.413 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_backtrace_repaired": 0, 2023-10-11T16:43:42.413 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_inotable_repaired": 0, 2023-10-11T16:43:42.413 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_inodes": 78, 2023-10-11T16:43:42.414 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dir_base_inodes": 0, 2023-10-11T16:43:42.414 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_dirfrag_rstats": 78, 2023-10-11T16:43:42.414 INFO:teuthology.orchestra.run.smithi150.stdout: "scrub_file_inodes": 12841, 2023-10-11T16:43:42.414 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_inode_file_caps": 12, 2023-10-11T16:43:42.415 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_revoke": 164, 2023-10-11T16:43:42.415 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_grant": 42968, 2023-10-11T16:43:42.415 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_trunc": 1, 2023-10-11T16:43:42.416 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-11T16:43:42.416 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-11T16:43:42.416 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps": 28810, 2023-10-11T16:43:42.416 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_caps_dirty": 28605, 2023-10-11T16:43:42.417 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_cap_release": 334, 2023-10-11T16:43:42.417 INFO:teuthology.orchestra.run.smithi150.stdout: "process_request_cap_release": 38261 2023-10-11T16:43:42.417 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.417 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_cache": { 2023-10-11T16:43:42.418 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update": 1751, 2023-10-11T16:43:42.418 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_update_receipt": 2041, 2023-10-11T16:43:42.418 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_try_discover": 1022, 2023-10-11T16:43:42.419 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_send_discover": 1271, 2023-10-11T16:43:42.419 INFO:teuthology.orchestra.run.smithi150.stdout: "dir_handle_discover": 2303, 2023-10-11T16:43:42.419 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays": 8, 2023-10-11T16:43:42.419 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_delayed": 0, 2023-10-11T16:43:42.420 INFO:teuthology.orchestra.run.smithi150.stdout: "num_strays_enqueuing": 0, 2023-10-11T16:43:42.420 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_created": 17703, 2023-10-11T16:43:42.420 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_enqueued": 17695, 2023-10-11T16:43:42.420 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_reintegrated": 0, 2023-10-11T16:43:42.421 INFO:teuthology.orchestra.run.smithi150.stdout: "strays_migrated": 0, 2023-10-11T16:43:42.421 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_processing": 0, 2023-10-11T16:43:42.421 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_enqueued": 0, 2023-10-11T16:43:42.422 INFO:teuthology.orchestra.run.smithi150.stdout: "num_recovering_prioritized": 0, 2023-10-11T16:43:42.422 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_started": 0, 2023-10-11T16:43:42.422 INFO:teuthology.orchestra.run.smithi150.stdout: "recovery_completed": 0, 2023-10-11T16:43:42.423 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_enqueue_scrub": 0, 2023-10-11T16:43:42.423 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_exportdir": 446, 2023-10-11T16:43:42.423 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_flush": 0, 2023-10-11T16:43:42.423 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragmentdir": 156, 2023-10-11T16:43:42.424 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_fragstats": 0, 2023-10-11T16:43:42.424 INFO:teuthology.orchestra.run.smithi150.stdout: "ireq_inodestats": 0 2023-10-11T16:43:42.424 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.424 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_log": { 2023-10-11T16:43:42.425 INFO:teuthology.orchestra.run.smithi150.stdout: "evadd": 92801, 2023-10-11T16:43:42.425 INFO:teuthology.orchestra.run.smithi150.stdout: "evex": 1741, 2023-10-11T16:43:42.425 INFO:teuthology.orchestra.run.smithi150.stdout: "evtrm": 1741, 2023-10-11T16:43:42.425 INFO:teuthology.orchestra.run.smithi150.stdout: "ev": 91060, 2023-10-11T16:43:42.426 INFO:teuthology.orchestra.run.smithi150.stdout: "evexg": 0, 2023-10-11T16:43:42.426 INFO:teuthology.orchestra.run.smithi150.stdout: "evexd": 717, 2023-10-11T16:43:42.426 INFO:teuthology.orchestra.run.smithi150.stdout: "segadd": 130, 2023-10-11T16:43:42.427 INFO:teuthology.orchestra.run.smithi150.stdout: "segex": 2, 2023-10-11T16:43:42.427 INFO:teuthology.orchestra.run.smithi150.stdout: "segtrm": 2, 2023-10-11T16:43:42.427 INFO:teuthology.orchestra.run.smithi150.stdout: "seg": 129, 2023-10-11T16:43:42.427 INFO:teuthology.orchestra.run.smithi150.stdout: "segexg": 0, 2023-10-11T16:43:42.428 INFO:teuthology.orchestra.run.smithi150.stdout: "segexd": 1, 2023-10-11T16:43:42.428 INFO:teuthology.orchestra.run.smithi150.stdout: "expos": 8396621, 2023-10-11T16:43:42.428 INFO:teuthology.orchestra.run.smithi150.stdout: "wrpos": 404794426, 2023-10-11T16:43:42.428 INFO:teuthology.orchestra.run.smithi150.stdout: "rdpos": 0, 2023-10-11T16:43:42.429 INFO:teuthology.orchestra.run.smithi150.stdout: "jlat": { 2023-10-11T16:43:42.429 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 2390, 2023-10-11T16:43:42.429 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 238.381231328, 2023-10-11T16:43:42.429 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.099741100 2023-10-11T16:43:42.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.430 INFO:teuthology.orchestra.run.smithi150.stdout: "replayed": 0 2023-10-11T16:43:42.430 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.431 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_mem": { 2023-10-11T16:43:42.431 INFO:teuthology.orchestra.run.smithi150.stdout: "ino": 39233, 2023-10-11T16:43:42.431 INFO:teuthology.orchestra.run.smithi150.stdout: "ino+": 76124, 2023-10-11T16:43:42.432 INFO:teuthology.orchestra.run.smithi150.stdout: "ino-": 36891, 2023-10-11T16:43:42.432 INFO:teuthology.orchestra.run.smithi150.stdout: "dir": 1564, 2023-10-11T16:43:42.432 INFO:teuthology.orchestra.run.smithi150.stdout: "dir+": 3166, 2023-10-11T16:43:42.432 INFO:teuthology.orchestra.run.smithi150.stdout: "dir-": 1602, 2023-10-11T16:43:42.433 INFO:teuthology.orchestra.run.smithi150.stdout: "dn": 69181, 2023-10-11T16:43:42.433 INFO:teuthology.orchestra.run.smithi150.stdout: "dn+": 107367, 2023-10-11T16:43:42.433 INFO:teuthology.orchestra.run.smithi150.stdout: "dn-": 38186, 2023-10-11T16:43:42.433 INFO:teuthology.orchestra.run.smithi150.stdout: "cap": 6, 2023-10-11T16:43:42.434 INFO:teuthology.orchestra.run.smithi150.stdout: "cap+": 51021, 2023-10-11T16:43:42.434 INFO:teuthology.orchestra.run.smithi150.stdout: "cap-": 51015, 2023-10-11T16:43:42.434 INFO:teuthology.orchestra.run.smithi150.stdout: "rss": 341820, 2023-10-11T16:43:42.435 INFO:teuthology.orchestra.run.smithi150.stdout: "heap": 198964 2023-10-11T16:43:42.435 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.435 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_server": { 2023-10-11T16:43:42.435 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_client_request": 74222, 2023-10-11T16:43:42.436 INFO:teuthology.orchestra.run.smithi150.stdout: "dispatch_server_request": 237, 2023-10-11T16:43:42.436 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_request": 73182, 2023-10-11T16:43:42.436 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_client_session": 114, 2023-10-11T16:43:42.436 INFO:teuthology.orchestra.run.smithi150.stdout: "handle_peer_request": 482, 2023-10-11T16:43:42.437 INFO:teuthology.orchestra.run.smithi150.stdout: "req_create_latency": { 2023-10-11T16:43:42.437 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 25163, 2023-10-11T16:43:42.437 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 1461.147008570, 2023-10-11T16:43:42.438 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.058067281 2023-10-11T16:43:42.438 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.438 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getattr_latency": { 2023-10-11T16:43:42.438 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 125, 2023-10-11T16:43:42.439 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.626176170, 2023-10-11T16:43:42.439 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.005009409 2023-10-11T16:43:42.439 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.439 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getfilelock_latency": { 2023-10-11T16:43:42.440 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.440 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.440 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.441 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.441 INFO:teuthology.orchestra.run.smithi150.stdout: "req_link_latency": { 2023-10-11T16:43:42.441 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.441 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.442 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.442 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.442 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookup_latency": { 2023-10-11T16:43:42.442 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 17582, 2023-10-11T16:43:42.443 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 40.907015274, 2023-10-11T16:43:42.443 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002326641 2023-10-11T16:43:42.443 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.444 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookuphash_latency": { 2023-10-11T16:43:42.444 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.444 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.444 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.445 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.445 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupino_latency": { 2023-10-11T16:43:42.445 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.445 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.446 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.446 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.446 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupname_latency": { 2023-10-11T16:43:42.446 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.447 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.447 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.447 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.448 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupparent_latency": { 2023-10-11T16:43:42.448 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.448 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.448 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.449 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.449 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lookupsnap_latency": { 2023-10-11T16:43:42.449 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.449 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.450 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.450 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.450 INFO:teuthology.orchestra.run.smithi150.stdout: "req_lssnap_latency": { 2023-10-11T16:43:42.451 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.451 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.451 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.451 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.452 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mkdir_latency": { 2023-10-11T16:43:42.452 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 119, 2023-10-11T16:43:42.452 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.113408262, 2023-10-11T16:43:42.452 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000953010 2023-10-11T16:43:42.453 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.453 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mknod_latency": { 2023-10-11T16:43:42.453 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.454 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.454 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.454 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.454 INFO:teuthology.orchestra.run.smithi150.stdout: "req_mksnap_latency": { 2023-10-11T16:43:42.455 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.455 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.455 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.455 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.456 INFO:teuthology.orchestra.run.smithi150.stdout: "req_open_latency": { 2023-10-11T16:43:42.456 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.456 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.456 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.457 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.457 INFO:teuthology.orchestra.run.smithi150.stdout: "req_readdir_latency": { 2023-10-11T16:43:42.457 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 2950, 2023-10-11T16:43:42.458 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 6.283363752, 2023-10-11T16:43:42.458 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.002129953 2023-10-11T16:43:42.458 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.458 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rename_latency": { 2023-10-11T16:43:42.459 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.459 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.459 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.460 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.460 INFO:teuthology.orchestra.run.smithi150.stdout: "req_renamesnap_latency": { 2023-10-11T16:43:42.460 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.460 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.461 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.461 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.461 INFO:teuthology.orchestra.run.smithi150.stdout: "req_snapdiff_latency": { 2023-10-11T16:43:42.461 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.462 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.462 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.462 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.463 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmdir_latency": { 2023-10-11T16:43:42.463 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 114, 2023-10-11T16:43:42.463 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.429363474, 2023-10-11T16:43:42.463 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.003766346 2023-10-11T16:43:42.464 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.464 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmsnap_latency": { 2023-10-11T16:43:42.464 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.464 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.465 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.465 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.465 INFO:teuthology.orchestra.run.smithi150.stdout: "req_rmxattr_latency": { 2023-10-11T16:43:42.466 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.466 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.466 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.466 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.467 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setattr_latency": { 2023-10-11T16:43:42.467 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 3, 2023-10-11T16:43:42.467 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.013625423, 2023-10-11T16:43:42.468 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.004541807 2023-10-11T16:43:42.468 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.468 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setdirlayout_latency": { 2023-10-11T16:43:42.468 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.469 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.469 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.469 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.469 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setfilelock_latency": { 2023-10-11T16:43:42.470 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 22, 2023-10-11T16:43:42.470 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 10.019724037, 2023-10-11T16:43:42.470 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.455442001 2023-10-11T16:43:42.471 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.471 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setlayout_latency": { 2023-10-11T16:43:42.471 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.471 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.472 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.472 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.472 INFO:teuthology.orchestra.run.smithi150.stdout: "req_setxattr_latency": { 2023-10-11T16:43:42.473 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 2, 2023-10-11T16:43:42.473 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.002206835, 2023-10-11T16:43:42.473 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.001103417 2023-10-11T16:43:42.473 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.474 INFO:teuthology.orchestra.run.smithi150.stdout: "req_symlink_latency": { 2023-10-11T16:43:42.474 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.474 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.475 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.475 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.475 INFO:teuthology.orchestra.run.smithi150.stdout: "req_unlink_latency": { 2023-10-11T16:43:42.475 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 17589, 2023-10-11T16:43:42.476 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 64.375439740, 2023-10-11T16:43:42.476 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.003659982 2023-10-11T16:43:42.476 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.476 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_revoke_eviction": 0, 2023-10-11T16:43:42.477 INFO:teuthology.orchestra.run.smithi150.stdout: "cap_acquisition_throttle": 0, 2023-10-11T16:43:42.477 INFO:teuthology.orchestra.run.smithi150.stdout: "req_getvxattr_latency": { 2023-10-11T16:43:42.477 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.477 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.478 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.478 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.478 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.479 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_sessions": { 2023-10-11T16:43:42.479 INFO:teuthology.orchestra.run.smithi150.stdout: "session_count": 2, 2023-10-11T16:43:42.479 INFO:teuthology.orchestra.run.smithi150.stdout: "session_add": 5, 2023-10-11T16:43:42.480 INFO:teuthology.orchestra.run.smithi150.stdout: "session_remove": 3, 2023-10-11T16:43:42.480 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_open": 2, 2023-10-11T16:43:42.480 INFO:teuthology.orchestra.run.smithi150.stdout: "sessions_stale": 0, 2023-10-11T16:43:42.480 INFO:teuthology.orchestra.run.smithi150.stdout: "total_load": 15685, 2023-10-11T16:43:42.481 INFO:teuthology.orchestra.run.smithi150.stdout: "average_load": 7842, 2023-10-11T16:43:42.481 INFO:teuthology.orchestra.run.smithi150.stdout: "avg_session_uptime": 663 2023-10-11T16:43:42.481 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.481 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2023-10-11T16:43:42.482 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:42.482 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:42.482 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:42.483 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:42.483 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:42.483 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:42.484 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:42.484 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:42.484 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:42.484 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:42.485 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:42.485 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:42.485 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:42.485 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:42.486 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:42.486 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:42.486 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:42.487 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:42.487 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:42.487 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:42.487 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:42.488 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:42.488 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:42.488 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:42.488 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:42.489 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:42.489 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:42.489 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:42.490 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:42.490 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:42.490 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:42.490 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2023-10-11T16:43:42.491 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:42.491 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:42.491 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:42.492 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:42.492 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 1613987, 2023-10-11T16:43:42.492 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 36653, 2023-10-11T16:43:42.492 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 880, 2023-10-11T16:43:42.493 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 10, 2023-10-11T16:43:42.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2023-10-11T16:43:42.493 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2023-10-11T16:43:42.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:42.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:42.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:42.494 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:42.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 10152, 2023-10-11T16:43:42.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 198, 2023-10-11T16:43:42.495 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:42.496 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:42.496 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:42.496 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2023-10-11T16:43:42.496 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 140180620, 2023-10-11T16:43:42.497 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 1323342, 2023-10-11T16:43:42.497 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:42.497 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2023-10-11T16:43:42.498 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:42.498 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2023-10-11T16:43:42.498 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.498 INFO:teuthology.orchestra.run.smithi150.stdout: "objecter": { 2023-10-11T16:43:42.499 INFO:teuthology.orchestra.run.smithi150.stdout: "op_active": 0, 2023-10-11T16:43:42.499 INFO:teuthology.orchestra.run.smithi150.stdout: "op_laggy": 0, 2023-10-11T16:43:42.499 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send": 53860, 2023-10-11T16:43:42.499 INFO:teuthology.orchestra.run.smithi150.stdout: "op_send_bytes": 417898562, 2023-10-11T16:43:42.500 INFO:teuthology.orchestra.run.smithi150.stdout: "op_resend": 31, 2023-10-11T16:43:42.500 INFO:teuthology.orchestra.run.smithi150.stdout: "op_reply": 53860, 2023-10-11T16:43:42.500 INFO:teuthology.orchestra.run.smithi150.stdout: "op_latency": { 2023-10-11T16:43:42.501 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 53860, 2023-10-11T16:43:42.501 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 477.483682177, 2023-10-11T16:43:42.501 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.008865274 2023-10-11T16:43:42.501 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.502 INFO:teuthology.orchestra.run.smithi150.stdout: "op_inflight": 0, 2023-10-11T16:43:42.502 INFO:teuthology.orchestra.run.smithi150.stdout: "oplen_avg": { 2023-10-11T16:43:42.502 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 53860, 2023-10-11T16:43:42.503 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 56254 2023-10-11T16:43:42.503 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.503 INFO:teuthology.orchestra.run.smithi150.stdout: "op": 53860, 2023-10-11T16:43:42.503 INFO:teuthology.orchestra.run.smithi150.stdout: "op_r": 14155, 2023-10-11T16:43:42.504 INFO:teuthology.orchestra.run.smithi150.stdout: "op_w": 39705, 2023-10-11T16:43:42.504 INFO:teuthology.orchestra.run.smithi150.stdout: "op_rmw": 0, 2023-10-11T16:43:42.504 INFO:teuthology.orchestra.run.smithi150.stdout: "op_pg": 0, 2023-10-11T16:43:42.504 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_stat": 1, 2023-10-11T16:43:42.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_create": 24, 2023-10-11T16:43:42.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_read": 273, 2023-10-11T16:43:42.505 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_write": 2663, 2023-10-11T16:43:42.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writefull": 138, 2023-10-11T16:43:42.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_writesame": 0, 2023-10-11T16:43:42.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_append": 0, 2023-10-11T16:43:42.506 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_zero": 0, 2023-10-11T16:43:42.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_truncate": 5, 2023-10-11T16:43:42.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_delete": 35541, 2023-10-11T16:43:42.507 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_mapext": 0, 2023-10-11T16:43:42.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_sparse_read": 0, 2023-10-11T16:43:42.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_clonerange": 0, 2023-10-11T16:43:42.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_getxattr": 12925, 2023-10-11T16:43:42.508 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_setxattr": 47, 2023-10-11T16:43:42.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_cmpxattr": 0, 2023-10-11T16:43:42.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_rmxattr": 0, 2023-10-11T16:43:42.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_resetxattrs": 0, 2023-10-11T16:43:42.509 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_call": 0, 2023-10-11T16:43:42.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_watch": 0, 2023-10-11T16:43:42.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_notify": 0, 2023-10-11T16:43:42.510 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_src_cmpxattr": 0, 2023-10-11T16:43:42.511 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls": 0, 2023-10-11T16:43:42.511 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_pgls_filter": 0, 2023-10-11T16:43:42.511 INFO:teuthology.orchestra.run.smithi150.stdout: "osdop_other": 1, 2023-10-11T16:43:42.511 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_active": 0, 2023-10-11T16:43:42.512 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_send": 0, 2023-10-11T16:43:42.512 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_resend": 0, 2023-10-11T16:43:42.512 INFO:teuthology.orchestra.run.smithi150.stdout: "linger_ping": 0, 2023-10-11T16:43:42.512 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_active": 0, 2023-10-11T16:43:42.513 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_send": 0, 2023-10-11T16:43:42.513 INFO:teuthology.orchestra.run.smithi150.stdout: "poolop_resend": 0, 2023-10-11T16:43:42.513 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_active": 0, 2023-10-11T16:43:42.513 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_send": 0, 2023-10-11T16:43:42.514 INFO:teuthology.orchestra.run.smithi150.stdout: "poolstat_resend": 0, 2023-10-11T16:43:42.514 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_active": 0, 2023-10-11T16:43:42.514 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_send": 0, 2023-10-11T16:43:42.514 INFO:teuthology.orchestra.run.smithi150.stdout: "statfs_resend": 0, 2023-10-11T16:43:42.515 INFO:teuthology.orchestra.run.smithi150.stdout: "command_active": 0, 2023-10-11T16:43:42.515 INFO:teuthology.orchestra.run.smithi150.stdout: "command_send": 0, 2023-10-11T16:43:42.515 INFO:teuthology.orchestra.run.smithi150.stdout: "command_resend": 0, 2023-10-11T16:43:42.516 INFO:teuthology.orchestra.run.smithi150.stdout: "map_epoch": 102, 2023-10-11T16:43:42.516 INFO:teuthology.orchestra.run.smithi150.stdout: "map_full": 0, 2023-10-11T16:43:42.516 INFO:teuthology.orchestra.run.smithi150.stdout: "map_inc": 15, 2023-10-11T16:43:42.516 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_sessions": 12, 2023-10-11T16:43:42.517 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_open": 12, 2023-10-11T16:43:42.517 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_session_close": 0, 2023-10-11T16:43:42.517 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_laggy": 0, 2023-10-11T16:43:42.517 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_wr": 2651, 2023-10-11T16:43:42.518 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_rd": 1926, 2023-10-11T16:43:42.518 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_del": 59 2023-10-11T16:43:42.518 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.519 INFO:teuthology.orchestra.run.smithi150.stdout: "oft": { 2023-10-11T16:43:42.519 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_objs": 1, 2023-10-11T16:43:42.519 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_kv_pairs": 0, 2023-10-11T16:43:42.519 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_updates": 0, 2023-10-11T16:43:42.520 INFO:teuthology.orchestra.run.smithi150.stdout: "omap_total_removes": 0 2023-10-11T16:43:42.520 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.520 INFO:teuthology.orchestra.run.smithi150.stdout: "osd-slow-ops": { 2023-10-11T16:43:42.520 INFO:teuthology.orchestra.run.smithi150.stdout: "slow_ops_count": 74026 2023-10-11T16:43:42.521 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.521 INFO:teuthology.orchestra.run.smithi150.stdout: "purge_queue": { 2023-10-11T16:43:42.521 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops": 0, 2023-10-11T16:43:42.521 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_ops_high_water": 16, 2023-10-11T16:43:42.522 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing": 0, 2023-10-11T16:43:42.522 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executing_high_water": 13, 2023-10-11T16:43:42.522 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_executed": 17695, 2023-10-11T16:43:42.523 INFO:teuthology.orchestra.run.smithi150.stdout: "pq_item_in_journal": 0 2023-10-11T16:43:42.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.523 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:42.523 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:42.524 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2023-10-11T16:43:42.524 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:42.524 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 314673, 2023-10-11T16:43:42.525 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 124325669, 2023-10-11T16:43:42.525 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:42.525 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 314673, 2023-10-11T16:43:42.525 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:42.526 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:42.526 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 314673, 2023-10-11T16:43:42.526 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 124325669, 2023-10-11T16:43:42.526 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:42.527 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.527 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.527 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.529 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.530 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_bytes": { 2023-10-11T16:43:42.530 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:42.530 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2023-10-11T16:43:42.531 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:42.531 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2023-10-11T16:43:42.531 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2023-10-11T16:43:42.532 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:42.532 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:42.532 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 53860, 2023-10-11T16:43:42.532 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 420057717, 2023-10-11T16:43:42.533 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 53860, 2023-10-11T16:43:42.533 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 420057717, 2023-10-11T16:43:42.533 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:42.533 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.534 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.534 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.534 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.535 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.535 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-objecter_ops": { 2023-10-11T16:43:42.535 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:42.535 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 1024, 2023-10-11T16:43:42.536 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:42.536 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 0, 2023-10-11T16:43:42.536 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 0, 2023-10-11T16:43:42.536 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:42.537 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 0, 2023-10-11T16:43:42.537 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 53860, 2023-10-11T16:43:42.537 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 53860, 2023-10-11T16:43:42.537 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 53860, 2023-10-11T16:43:42.538 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 53860, 2023-10-11T16:43:42.538 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:42.538 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.539 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.539 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.539 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.539 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.540 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle": { 2023-10-11T16:43:42.540 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:42.540 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2023-10-11T16:43:42.540 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:42.541 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 17695, 2023-10-11T16:43:42.541 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 2174539, 2023-10-11T16:43:42.541 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:42.542 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 17695, 2023-10-11T16:43:42.542 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:42.542 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:42.542 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 273, 2023-10-11T16:43:42.543 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 2174539, 2023-10-11T16:43:42.543 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:42.543 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.543 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.544 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.544 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.544 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:42.544 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-write_buf_throttle-0x558032b0c0a0": { 2023-10-11T16:43:42.545 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:42.545 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 3758096384, 2023-10-11T16:43:42.545 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:42.546 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 92801, 2023-10-11T16:43:42.546 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 400600122, 2023-10-11T16:43:42.546 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:42.546 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 92801, 2023-10-11T16:43:42.547 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:42.547 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:42.547 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 2390, 2023-10-11T16:43:42.547 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 400600122, 2023-10-11T16:43:42.548 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:42.548 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:42.548 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:42.549 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:42.549 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.549 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:42.549 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-10-11T16:43:42.787 INFO:tasks.check_counter:Daemon mds.i mds.exported=399 2023-10-11T16:43:42.788 INFO:tasks.check_counter:Daemon mds.i mds.imported=982 2023-10-11T16:43:42.788 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=1751 2023-10-11T16:43:42.788 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=2041 2023-10-11T16:43:42.789 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2023-10-11T16:43:42.789 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2023-10-11T16:43:42.789 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2023-10-11T16:43:42.790 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=154 2023-10-11T16:43:42.790 DEBUG:tasks.check_counter:Getting stats from l 2023-10-11T16:43:42.791 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 -- ceph daemon mds.l perf dump 2023-10-11T16:43:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:Inferring config /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-11T16:43:43.618 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:43 smithi150 ceph-mon[132217]: pgmap v345: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:43.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:43 smithi111 ceph-mon[98527]: pgmap v345: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:43.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:43 smithi130 ceph-mon[131859]: pgmap v345: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-11T16:43:44.739 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2023-10-11T16:43:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-0": { 2023-10-11T16:43:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 0, 2023-10-11T16:43:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 2, 2023-10-11T16:43:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 0, 2023-10-11T16:43:44.740 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 0, 2023-10-11T16:43:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 4, 2023-10-11T16:43:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 0, 2023-10-11T16:43:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.001243218, 2023-10-11T16:43:44.741 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.000000000, 2023-10-11T16:43:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.000257915, 2023-10-11T16:43:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:44.742 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:44.743 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:44.744 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-11T16:43:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 0 2023-10-11T16:43:44.745 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-1": { 2023-10-11T16:43:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 237, 2023-10-11T16:43:44.746 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 227, 2023-10-11T16:43:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 1425856, 2023-10-11T16:43:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 129376, 2023-10-11T16:43:44.747 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 2, 2023-10-11T16:43:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 1, 2023-10-11T16:43:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.044665542, 2023-10-11T16:43:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.013798658, 2023-10-11T16:43:44.748 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.031837792, 2023-10-11T16:43:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.006547577, 2023-10-11T16:43:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:44.749 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 226, 2023-10-11T16:43:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.023265638, 2023-10-11T16:43:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000102945 2023-10-11T16:43:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.750 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:44.751 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 1425856, 2023-10-11T16:43:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 129376 2023-10-11T16:43:44.752 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "AsyncMessenger::Worker-2": { 2023-10-11T16:43:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_messages": 6, 2023-10-11T16:43:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages": 176, 2023-10-11T16:43:44.753 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_bytes": 3312, 2023-10-11T16:43:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_bytes": 182800, 2023-10-11T16:43:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_created_connections": 5, 2023-10-11T16:43:44.754 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_active_connections": 3, 2023-10-11T16:43:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_total_time": 0.010883951, 2023-10-11T16:43:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_send_time": 0.006857399, 2023-10-11T16:43:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_recv_time": 0.001944876, 2023-10-11T16:43:44.755 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-11T16:43:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_messages_queue_lat": { 2023-10-11T16:43:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 174, 2023-10-11T16:43:44.756 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.028891007, 2023-10-11T16:43:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000166040 2023-10-11T16:43:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_handle_ack_lat": { 2023-10-11T16:43:44.757 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:44.758 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.759 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-10-11T16:43:44.759 INFO:teuthology.orchestra.run.smithi150.stdout: "msgr_send_encrypted_bytes": 182800 2023-10-11T16:43:44.759 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.759 INFO:teuthology.orchestra.run.smithi150.stdout: "cct": { 2023-10-11T16:43:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "total_workers": 0, 2023-10-11T16:43:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "unhealthy_workers": 0 2023-10-11T16:43:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.760 INFO:teuthology.orchestra.run.smithi150.stdout: "mempool": { 2023-10-11T16:43:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_bytes": 0, 2023-10-11T16:43:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "bloom_filter_items": 0, 2023-10-11T16:43:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_bytes": 0, 2023-10-11T16:43:44.761 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_alloc_items": 0, 2023-10-11T16:43:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_bytes": 0, 2023-10-11T16:43:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_data_items": 0, 2023-10-11T16:43:44.762 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-11T16:43:44.765 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_onode_items": 0, 2023-10-11T16:43:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-11T16:43:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_meta_items": 0, 2023-10-11T16:43:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_bytes": 0, 2023-10-11T16:43:44.766 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_other_items": 0, 2023-10-11T16:43:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-11T16:43:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_cache_buffer_items": 0, 2023-10-11T16:43:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_bytes": 0, 2023-10-11T16:43:44.767 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_extent_items": 0, 2023-10-11T16:43:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_bytes": 0, 2023-10-11T16:43:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_blob_items": 0, 2023-10-11T16:43:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-11T16:43:44.768 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_shared_blob_items": 0, 2023-10-11T16:43:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-11T16:43:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_inline_bl_items": 0, 2023-10-11T16:43:44.769 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_bytes": 0, 2023-10-11T16:43:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_fsck_items": 0, 2023-10-11T16:43:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_bytes": 0, 2023-10-11T16:43:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_txc_items": 0, 2023-10-11T16:43:44.770 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-11T16:43:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_deferred_items": 0, 2023-10-11T16:43:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_bytes": 0, 2023-10-11T16:43:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "bluestore_writing_items": 0, 2023-10-11T16:43:44.771 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_bytes": 0, 2023-10-11T16:43:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_items": 0, 2023-10-11T16:43:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_bytes": 0, 2023-10-11T16:43:44.772 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_reader_items": 0, 2023-10-11T16:43:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_bytes": 0, 2023-10-11T16:43:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "bluefs_file_writer_items": 0, 2023-10-11T16:43:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_bytes": 148754, 2023-10-11T16:43:44.773 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_anon_items": 31, 2023-10-11T16:43:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_bytes": 0, 2023-10-11T16:43:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "buffer_meta_items": 0, 2023-10-11T16:43:44.774 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_bytes": 0, 2023-10-11T16:43:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_items": 0, 2023-10-11T16:43:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_bytes": 0, 2023-10-11T16:43:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_mapbl_items": 0, 2023-10-11T16:43:44.775 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_bytes": 0, 2023-10-11T16:43:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_pglog_items": 0, 2023-10-11T16:43:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_bytes": 0, 2023-10-11T16:43:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_items": 0, 2023-10-11T16:43:44.776 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_bytes": 0, 2023-10-11T16:43:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_mapping_items": 0, 2023-10-11T16:43:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_bytes": 0, 2023-10-11T16:43:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "pgmap_items": 0, 2023-10-11T16:43:44.777 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_bytes": 928, 2023-10-11T16:43:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_co_items": 2, 2023-10-11T16:43:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_bytes": 0, 2023-10-11T16:43:44.778 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_1_items": 0, 2023-10-11T16:43:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_bytes": 0, 2023-10-11T16:43:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "unittest_2_items": 0 2023-10-11T16:43:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2023-10-11T16:43:44.779 INFO:teuthology.orchestra.run.smithi150.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-11T16:43:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "val": 0, 2023-10-11T16:43:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "max": 104857600, 2023-10-11T16:43:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: "get_started": 0, 2023-10-11T16:43:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: "get": 239, 2023-10-11T16:43:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: "get_sum": 1392399, 2023-10-11T16:43:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_fail": 0, 2023-10-11T16:43:44.781 INFO:teuthology.orchestra.run.smithi150.stdout: "get_or_fail_success": 239, 2023-10-11T16:43:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "take": 0, 2023-10-11T16:43:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "take_sum": 0, 2023-10-11T16:43:44.782 INFO:teuthology.orchestra.run.smithi150.stdout: "put": 239, 2023-10-11T16:43:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "put_sum": 1392399, 2023-10-11T16:43:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "wait": { 2023-10-11T16:43:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "avgcount": 0, 2023-10-11T16:43:44.783 INFO:teuthology.orchestra.run.smithi150.stdout: "sum": 0.000000000, 2023-10-11T16:43:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: "avgtime": 0.000000000 2023-10-11T16:43:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:44.784 INFO:teuthology.orchestra.run.smithi150.stdout: } 2023-10-11T16:43:44.785 INFO:teuthology.orchestra.run.smithi150.stdout:} 2023-10-11T16:43:45.091 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2023-10-11T16:43:45.091 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2023-10-11T16:43:45.092 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2023-10-11T16:43:45.092 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2023-10-11T16:43:45.092 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2023-10-11T16:43:45.093 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2023-10-11T16:43:45.093 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2023-10-11T16:43:45.093 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2023-10-11T16:43:45.094 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-11T16:43:45.107 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-11T16:43:45.107 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:43:45.108 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-11T16:43:45.176 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2023-10-11T16:43:45.177 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-11T16:43:45.262 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:43:45.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 -- 172.21.15.111:0/1786897173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8c100f00 msgr2=0x7f1e8c104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:45.510 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1786897173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8c100f00 0x7f1e8c104ec0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1e740099c0 tx=0x7f1e74009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 -- 172.21.15.111:0/1786897173 shutdown_connections 2023-10-11T16:43:45.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1786897173 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c1055a0 0x7f1e8c10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1786897173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8c100560 0x7f1e8c100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1786897173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8c100f00 0x7f1e8c104ec0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 -- 172.21.15.111:0/1786897173 >> 172.21.15.111:0/1786897173 conn(0x7f1e8c077080 msgr2=0x7f1e8c077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:45.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 -- 172.21.15.111:0/1786897173 shutdown_connections 2023-10-11T16:43:45.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.508+0000 7f1e90dff700 1 -- 172.21.15.111:0/1786897173 wait complete. 2023-10-11T16:43:45.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.509+0000 7f1e90dff700 1 Processor -- start 2023-10-11T16:43:45.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.509+0000 7f1e90dff700 1 -- start start 2023-10-11T16:43:45.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e90dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c100560 0x7f1e8c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:45.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e90dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8c100f00 0x7f1e8c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:45.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e90dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8c1055a0 0x7f1e8c19bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:45.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e90dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19c5e0 con 0x7f1e8c100560 2023-10-11T16:43:45.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e90dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19c750 con 0x7f1e8c100f00 2023-10-11T16:43:45.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e90dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19c910 con 0x7f1e8c1055a0 2023-10-11T16:43:45.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c100560 0x7f1e8c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:45.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c100560 0x7f1e8c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41148/0 (socket says 172.21.15.111:41148) 2023-10-11T16:43:45.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 -- 172.21.15.111:0/1862390299 learned_addr learned my addr 172.21.15.111:0/1862390299 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:45.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8ad9d700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8c1055a0 0x7f1e8c19bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:45.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e83fff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8c100f00 0x7f1e8c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:45.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 -- 172.21.15.111:0/1862390299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8c1055a0 msgr2=0x7f1e8c19bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:45.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8c1055a0 0x7f1e8c19bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 -- 172.21.15.111:0/1862390299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8c100f00 msgr2=0x7f1e8c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:45.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8c100f00 0x7f1e8c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.520 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 -- 172.21.15.111:0/1862390299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e740095f0 con 0x7f1e8c100560 2023-10-11T16:43:45.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e83fff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8c100f00 0x7f1e8c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:43:45.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.510+0000 7f1e8a59c700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c100560 0x7f1e8c1986f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00ef10 tx=0x7f1e7c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:45.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.511+0000 7f1e837fe700 1 -- 172.21.15.111:0/1862390299 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c00ca70 con 0x7f1e8c100560 2023-10-11T16:43:45.521 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.511+0000 7f1e837fe700 1 -- 172.21.15.111:0/1862390299 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e7c00f040 con 0x7f1e8c100560 2023-10-11T16:43:45.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.511+0000 7f1e837fe700 1 -- 172.21.15.111:0/1862390299 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c014480 con 0x7f1e8c100560 2023-10-11T16:43:45.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.511+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c110370 con 0x7f1e8c100560 2023-10-11T16:43:45.522 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.511+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c1101e0 con 0x7f1e8c100560 2023-10-11T16:43:45.523 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.512+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c110500 con 0x7f1e8c100560 2023-10-11T16:43:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.514+0000 7f1e837fe700 1 -- 172.21.15.111:0/1862390299 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f1e7c007500 con 0x7f1e8c100560 2023-10-11T16:43:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.515+0000 7f1e837fe700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1e780a12c0 0x7f1e780a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.515+0000 7f1e837fe700 1 -- 172.21.15.111:0/1862390299 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f1e7c0cd500 con 0x7f1e8c100560 2023-10-11T16:43:45.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.515+0000 7f1e83fff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1e780a12c0 0x7f1e780a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:45.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.516+0000 7f1e83fff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1e780a12c0 0x7f1e780a3770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1e740098e0 tx=0x7f1e7401a040 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:45.525 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.519+0000 7f1e837fe700 1 -- 172.21.15.111:0/1862390299 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1e7c0964a0 con 0x7f1e8c100560 2023-10-11T16:43:45.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:45 smithi111 ceph-mon[98527]: pgmap v346: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2023-10-11T16:43:45.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:45 smithi111 ceph-mon[98527]: osdmap e103: 12 total, 12 up, 12 in 2023-10-11T16:43:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:45 smithi150 ceph-mon[132217]: pgmap v346: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2023-10-11T16:43:45.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:45 smithi150 ceph-mon[132217]: osdmap e103: 12 total, 12 up, 12 in 2023-10-11T16:43:45.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.720+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e8c199b10 con 0x7f1e8c100560 2023-10-11T16:43:45.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.722+0000 7f1e837fe700 1 -- 172.21.15.111:0/1862390299 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25438 (secure 0 0 0) 0x7f1e7c099ac0 con 0x7f1e8c100560 2023-10-11T16:43:45.724 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:43:45.724 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":103,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:43:45.329616+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/2838247405":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/803529843":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/3816963652":"2023-10-12T16:32:37.683058+0000","172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/2838247405":"2023-10-12T16:32:37.683058+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1856545817":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/444576470":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:43:45.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1e780a12c0 msgr2=0x7f1e780a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:45.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1e780a12c0 0x7f1e780a3770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1e740098e0 tx=0x7f1e7401a040 comp rx=0 tx=0).stop 2023-10-11T16:43:45.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c100560 msgr2=0x7f1e8c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c100560 0x7f1e8c1986f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00ef10 tx=0x7f1e7c00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 shutdown_connections 2023-10-11T16:43:45.727 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f1e780a12c0 0x7f1e780a3770 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e8c1055a0 0x7f1e8c19bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e8c100560 0x7f1e8c1986f0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 --2- 172.21.15.111:0/1862390299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e8c100f00 0x7f1e8c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.728 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 >> 172.21.15.111:0/1862390299 conn(0x7f1e8c077080 msgr2=0x7f1e8c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 shutdown_connections 2023-10-11T16:43:45.729 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.724+0000 7f1e90dff700 1 -- 172.21.15.111:0/1862390299 wait complete. 2023-10-11T16:43:45.745 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-11T16:25:56.558461+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-11T16:30:10.034441+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-11T16:30:11.080913+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-11T16:30:14.592036+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-11T16:43:45.746 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-11T16:43:45.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:45 smithi130 ceph-mon[131859]: pgmap v346: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2023-10-11T16:43:45.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:45 smithi130 ceph-mon[131859]: osdmap e103: 12 total, 12 up, 12 in 2023-10-11T16:43:45.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 -- 172.21.15.111:0/4061524184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcc0101660 msgr2=0x7fdcc0104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:45.951 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/4061524184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcc0101660 0x7fdcc0104ef0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40099c0 tx=0x7fdcb4009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 -- 172.21.15.111:0/4061524184 shutdown_connections 2023-10-11T16:43:45.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/4061524184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc0105620 0x7fdcc010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.952 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/4061524184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcc0100cc0 0x7fdcc0101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/4061524184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcc0101660 0x7fdcc0104ef0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 -- 172.21.15.111:0/4061524184 >> 172.21.15.111:0/4061524184 conn(0x7fdcc0077560 msgr2=0x7fdcc0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:45.953 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 -- 172.21.15.111:0/4061524184 shutdown_connections 2023-10-11T16:43:45.954 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.949+0000 7fdcc5dff700 1 -- 172.21.15.111:0/4061524184 wait complete. 2023-10-11T16:43:45.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.950+0000 7fdcc5dff700 1 Processor -- start 2023-10-11T16:43:45.956 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.950+0000 7fdcc5dff700 1 -- start start 2023-10-11T16:43:45.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcc5dff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcc0100cc0 0x7fdcc01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:45.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcc5dff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc0101660 0x7fdcc0198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:45.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcc5dff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcc0105620 0x7fdcc019bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:45.957 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcc5dff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc019c5e0 con 0x7fdcc0101660 2023-10-11T16:43:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcc5dff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc019c750 con 0x7fdcc0100cc0 2023-10-11T16:43:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcc5dff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc019c910 con 0x7fdcc0105620 2023-10-11T16:43:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc0101660 0x7fdcc0198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:45.958 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc0101660 0x7fdcc0198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41168/0 (socket says 172.21.15.111:41168) 2023-10-11T16:43:45.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbeffd700 1 -- 172.21.15.111:0/2883170472 learned_addr learned my addr 172.21.15.111:0/2883170472 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:45.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbffff700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcc0105620 0x7fdcc019bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:45.959 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbf7fe700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcc0100cc0 0x7fdcc01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:45.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbeffd700 1 -- 172.21.15.111:0/2883170472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcc0105620 msgr2=0x7fdcc019bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:45.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbeffd700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcc0105620 0x7fdcc019bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbeffd700 1 -- 172.21.15.111:0/2883170472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcc0100cc0 msgr2=0x7fdcc01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:45.960 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbeffd700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcc0100cc0 0x7fdcc01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:45.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.951+0000 7fdcbeffd700 1 -- 172.21.15.111:0/2883170472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb40095f0 con 0x7fdcc0101660 2023-10-11T16:43:45.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.952+0000 7fdcbeffd700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc0101660 0x7fdcc0198c30 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40098e0 tx=0x7fdcb400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:45.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.952+0000 7fdcbcff9700 1 -- 172.21.15.111:0/2883170472 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb401d070 con 0x7fdcc0101660 2023-10-11T16:43:45.961 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.952+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc0110370 con 0x7fdcc0101660 2023-10-11T16:43:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.952+0000 7fdcbcff9700 1 -- 172.21.15.111:0/2883170472 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcb4022470 con 0x7fdcc0101660 2023-10-11T16:43:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.952+0000 7fdcbcff9700 1 -- 172.21.15.111:0/2883170472 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4017650 con 0x7fdcc0101660 2023-10-11T16:43:45.962 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.952+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc01101e0 con 0x7fdcc0101660 2023-10-11T16:43:45.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.953+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc0110500 con 0x7fdcc0101660 2023-10-11T16:43:45.963 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.955+0000 7fdcbcff9700 1 -- 172.21.15.111:0/2883170472 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fdcb4022850 con 0x7fdcc0101660 2023-10-11T16:43:45.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.957+0000 7fdcbcff9700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a12c0 0x7fdca80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:45.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.957+0000 7fdcbcff9700 1 -- 172.21.15.111:0/2883170472 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fdcb40cdfb0 con 0x7fdcc0101660 2023-10-11T16:43:45.964 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.957+0000 7fdcbf7fe700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a12c0 0x7fdca80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:45.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.958+0000 7fdcbf7fe700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a12c0 0x7fdca80a3770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009c60 tx=0x7fdcb0009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:45.965 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:45.961+0000 7fdcbcff9700 1 -- 172.21.15.111:0/2883170472 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdcb4096fe0 con 0x7fdcc0101660 2023-10-11T16:43:46.163 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.161+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdcc01a5790 con 0x7fdcc0101660 2023-10-11T16:43:46.164 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.162+0000 7fdcbcff9700 1 -- 172.21.15.111:0/2883170472 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdcb409a600 con 0x7fdcc0101660 2023-10-11T16:43:46.164 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2023-10-11T16:43:46.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a12c0 msgr2=0x7fdca80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a12c0 0x7fdca80a3770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009c60 tx=0x7fdcb0009260 comp rx=0 tx=0).stop 2023-10-11T16:43:46.166 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc0101660 msgr2=0x7fdcc0198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc0101660 0x7fdcc0198c30 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40098e0 tx=0x7fdcb400f690 comp rx=0 tx=0).stop 2023-10-11T16:43:46.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 shutdown_connections 2023-10-11T16:43:46.167 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdca80a12c0 0x7fdca80a3770 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcc0105620 0x7fdcc019bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdcc0100cc0 0x7fdcc01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 --2- 172.21.15.111:0/2883170472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc0101660 0x7fdcc0198c30 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.168 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 >> 172.21.15.111:0/2883170472 conn(0x7fdcc0077560 msgr2=0x7fdcc00fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:46.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 shutdown_connections 2023-10-11T16:43:46.169 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.164+0000 7fdcc5dff700 1 -- 172.21.15.111:0/2883170472 wait complete. 2023-10-11T16:43:46.185 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-11T16:43:46.390 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.388+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/923402334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b54074b00 msgr2=0x7f2b54073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.388+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/923402334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b54074b00 0x7f2b54073260 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2b44009990 tx=0x7f2b44009ca0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.391 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.389+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/923402334 shutdown_connections 2023-10-11T16:43:46.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.389+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/923402334 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b5410cec0 0x7f2b5410f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.389+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/923402334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b540746a0 0x7f2b54072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.389+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/923402334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b54074b00 0x7f2b54073260 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.392 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.389+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/923402334 >> 172.21.15.111:0/923402334 conn(0x7f2b54077590 msgr2=0x7f2b54077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:46.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.389+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/923402334 shutdown_connections 2023-10-11T16:43:46.393 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.389+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/923402334 wait complete. 2023-10-11T16:43:46.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5c7ff700 1 Processor -- start 2023-10-11T16:43:46.395 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5c7ff700 1 -- start start 2023-10-11T16:43:46.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5c7ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b540746a0 0x7f2b541986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:46.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5c7ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b54074b00 0x7f2b54198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:46.396 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5c7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b5410cec0 0x7f2b5419bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:46.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5c7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5419c5b0 con 0x7f2b5410cec0 2023-10-11T16:43:46.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5c7ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5419c720 con 0x7f2b54074b00 2023-10-11T16:43:46.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5c7ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5419c8e0 con 0x7f2b540746a0 2023-10-11T16:43:46.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.390+0000 7f2b5a59b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b540746a0 0x7f2b541986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:46.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5ad9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b5410cec0 0x7f2b5419bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:46.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b59d9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b54074b00 0x7f2b54198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:46.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5a59b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b540746a0 0x7f2b541986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.111:37378/0 (socket says 172.21.15.111:37378) 2023-10-11T16:43:46.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5a59b700 1 -- 172.21.15.111:0/1259509993 learned_addr learned my addr 172.21.15.111:0/1259509993 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:46.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5ad9c700 1 -- 172.21.15.111:0/1259509993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b540746a0 msgr2=0x7f2b541986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5ad9c700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b540746a0 0x7f2b541986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5ad9c700 1 -- 172.21.15.111:0/1259509993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b54074b00 msgr2=0x7f2b54198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5ad9c700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b54074b00 0x7f2b54198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5ad9c700 1 -- 172.21.15.111:0/1259509993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b440095f0 con 0x7f2b5410cec0 2023-10-11T16:43:46.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5ad9c700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b5410cec0 0x7f2b5419bfb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00ef90 tx=0x7f2b4c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:46.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b4b7fe700 1 -- 172.21.15.111:0/1259509993 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c018070 con 0x7f2b5410cec0 2023-10-11T16:43:46.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b4b7fe700 1 -- 172.21.15.111:0/1259509993 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b4c00f040 con 0x7f2b5410cec0 2023-10-11T16:43:46.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b4b7fe700 1 -- 172.21.15.111:0/1259509993 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c00cc30 con 0x7f2b5410cec0 2023-10-11T16:43:46.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.391+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b54112620 con 0x7f2b5410cec0 2023-10-11T16:43:46.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.392+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b54112490 con 0x7f2b5410cec0 2023-10-11T16:43:46.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.392+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b541127b0 con 0x7f2b5410cec0 2023-10-11T16:43:46.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.394+0000 7f2b4b7fe700 1 -- 172.21.15.111:0/1259509993 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2b4c003a00 con 0x7f2b5410cec0 2023-10-11T16:43:46.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.397+0000 7f2b4b7fe700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b400a1340 0x7f2b400a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:46.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.397+0000 7f2b4b7fe700 1 -- 172.21.15.111:0/1259509993 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f2b4c0cd590 con 0x7f2b5410cec0 2023-10-11T16:43:46.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.397+0000 7f2b5a59b700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b400a1340 0x7f2b400a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:46.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.397+0000 7f2b5a59b700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b400a1340 0x7f2b400a37f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b50009ad0 tx=0x7f2b50009290 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:46.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.399+0000 7f2b4b7fe700 1 -- 172.21.15.111:0/1259509993 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2b4c0965c0 con 0x7f2b5410cec0 2023-10-11T16:43:46.597 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.596+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2b54199a10 con 0x7f2b5410cec0 2023-10-11T16:43:46.598 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.596+0000 7f2b4b7fe700 1 -- 172.21.15.111:0/1259509993 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2b4c099be0 con 0x7f2b5410cec0 2023-10-11T16:43:46.599 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:43:46.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b400a1340 msgr2=0x7f2b400a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.600 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b400a1340 0x7f2b400a37f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2b50009ad0 tx=0x7f2b50009290 comp rx=0 tx=0).stop 2023-10-11T16:43:46.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b5410cec0 msgr2=0x7f2b5419bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b5410cec0 0x7f2b5419bfb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00ef90 tx=0x7f2b4c00c3a0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.601 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 shutdown_connections 2023-10-11T16:43:46.602 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2b400a1340 0x7f2b400a37f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b5410cec0 0x7f2b5419bfb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b540746a0 0x7f2b541986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.603 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 --2- 172.21.15.111:0/1259509993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2b54074b00 0x7f2b54198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 >> 172.21.15.111:0/1259509993 conn(0x7f2b54077590 msgr2=0x7f2b54104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:46.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 shutdown_connections 2023-10-11T16:43:46.604 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.599+0000 7f2b5c7ff700 1 -- 172.21.15.111:0/1259509993 wait complete. 2023-10-11T16:43:46.617 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:46 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1862390299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:43:46.617 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:46 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2883170472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:43:46.619 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-11T16:43:46.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:46 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1862390299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:43:46.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:46 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2883170472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:43:46.820 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 -- 172.21.15.111:0/1807761433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0100cc0 msgr2=0x7fb2b0101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/1807761433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0100cc0 0x7fb2b0101090 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40099c0 tx=0x7fb2a4009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.821 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 -- 172.21.15.111:0/1807761433 shutdown_connections 2023-10-11T16:43:46.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/1807761433 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2b0105620 0x7fb2b010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/1807761433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0100cc0 0x7fb2b0101090 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/1807761433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b0101660 0x7fb2b0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.822 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 -- 172.21.15.111:0/1807761433 >> 172.21.15.111:0/1807761433 conn(0x7fb2b0077560 msgr2=0x7fb2b0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:46.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 -- 172.21.15.111:0/1807761433 shutdown_connections 2023-10-11T16:43:46.823 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.819+0000 7fb2b71ff700 1 -- 172.21.15.111:0/1807761433 wait complete. 2023-10-11T16:43:46.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.820+0000 7fb2b71ff700 1 Processor -- start 2023-10-11T16:43:46.825 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b71ff700 1 -- start start 2023-10-11T16:43:46.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b71ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b0100cc0 0x7fb2b01965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:46.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b71ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2b0101660 0x7fb2b0196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:46.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b71ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0105620 0x7fb2b019af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:46.826 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b71ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b019b540 con 0x7fb2b0105620 2023-10-11T16:43:46.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b71ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b019b6b0 con 0x7fb2b0101660 2023-10-11T16:43:46.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b71ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b019b870 con 0x7fb2b0100cc0 2023-10-11T16:43:46.827 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b579c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0105620 0x7fb2b019af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:46.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b579c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0105620 0x7fb2b019af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41206/0 (socket says 172.21.15.111:41206) 2023-10-11T16:43:46.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.821+0000 7fb2b579c700 1 -- 172.21.15.111:0/2600282261 learned_addr learned my addr 172.21.15.111:0/2600282261 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:46.828 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2affff700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2b0101660 0x7fb2b0196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:46.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b4f9b700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b0100cc0 0x7fb2b01965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:46.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b579c700 1 -- 172.21.15.111:0/2600282261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b0100cc0 msgr2=0x7fb2b01965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b579c700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b0100cc0 0x7fb2b01965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.829 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b579c700 1 -- 172.21.15.111:0/2600282261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2b0101660 msgr2=0x7fb2b0196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:46.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b579c700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2b0101660 0x7fb2b0196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:46.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b579c700 1 -- 172.21.15.111:0/2600282261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a40095f0 con 0x7fb2b0105620 2023-10-11T16:43:46.830 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b579c700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0105620 0x7fb2b019af00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8007ee0 tx=0x7fb2a800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:46.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2adffb700 1 -- 172.21.15.111:0/2600282261 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8017070 con 0x7fb2b0105620 2023-10-11T16:43:46.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2adffb700 1 -- 172.21.15.111:0/2600282261 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2a800f040 con 0x7fb2b0105620 2023-10-11T16:43:46.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b01103d0 con 0x7fb2b0105620 2023-10-11T16:43:46.831 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2adffb700 1 -- 172.21.15.111:0/2600282261 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8005580 con 0x7fb2b0105620 2023-10-11T16:43:46.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.822+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b0110240 con 0x7fb2b0105620 2023-10-11T16:43:46.832 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.823+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b0110560 con 0x7fb2b0105620 2023-10-11T16:43:46.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:46 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1862390299' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:43:46.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:46 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2883170472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-11T16:43:46.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.825+0000 7fb2adffb700 1 -- 172.21.15.111:0/2600282261 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fb2a800db20 con 0x7fb2b0105620 2023-10-11T16:43:46.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.828+0000 7fb2adffb700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb2a00a1340 0x7fb2a00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:46.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.828+0000 7fb2adffb700 1 -- 172.21.15.111:0/2600282261 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fb2a80cce10 con 0x7fb2b0105620 2023-10-11T16:43:46.835 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.828+0000 7fb2b4f9b700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb2a00a1340 0x7fb2a00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:46.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.829+0000 7fb2b4f9b700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb2a00a1340 0x7fb2a00a37f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009990 tx=0x7fb2a4005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:46.836 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:46.830+0000 7fb2adffb700 1 -- 172.21.15.111:0/2600282261 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb2a8095db0 con 0x7fb2b0105620 2023-10-11T16:43:47.029 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.028+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb2b01978b0 con 0x7fb2b0105620 2023-10-11T16:43:47.030 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.028+0000 7fb2adffb700 1 -- 172.21.15.111:0/2600282261 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb2a80993d0 con 0x7fb2b0105620 2023-10-11T16:43:47.031 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 64 2023-10-11T16:43:47.032 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb2a00a1340 msgr2=0x7fb2a00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb2a00a1340 0x7fb2a00a37f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009990 tx=0x7fb2a4005fb0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0105620 msgr2=0x7fb2b019af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0105620 0x7fb2b019af00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8007ee0 tx=0x7fb2a800d3b0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.033 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 shutdown_connections 2023-10-11T16:43:47.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fb2a00a1340 0x7fb2a00a37f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2b0105620 0x7fb2b019af00 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.034 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b0100cc0 0x7fb2b01965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 --2- 172.21.15.111:0/2600282261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2b0101660 0x7fb2b0196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 >> 172.21.15.111:0/2600282261 conn(0x7fb2b0077560 msgr2=0x7fb2b00fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:47.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 shutdown_connections 2023-10-11T16:43:47.035 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.031+0000 7fb2b71ff700 1 -- 172.21.15.111:0/2600282261 wait complete. 2023-10-11T16:43:47.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-11T16:43:47.255 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.253+0000 7fddf8fff700 1 -- 172.21.15.111:0/3982223009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 msgr2=0x7fddf4101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.253+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3982223009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 0x7fddf4101090 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdde40099c0 tx=0x7fdde4009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.254+0000 7fddf8fff700 1 -- 172.21.15.111:0/3982223009 shutdown_connections 2023-10-11T16:43:47.256 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.254+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3982223009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf4105620 0x7fddf410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.254+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3982223009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 0x7fddf4101090 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.254+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3982223009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4101660 0x7fddf4104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.254+0000 7fddf8fff700 1 -- 172.21.15.111:0/3982223009 >> 172.21.15.111:0/3982223009 conn(0x7fddf4077560 msgr2=0x7fddf4077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:47.257 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.254+0000 7fddf8fff700 1 -- 172.21.15.111:0/3982223009 shutdown_connections 2023-10-11T16:43:47.258 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.254+0000 7fddf8fff700 1 -- 172.21.15.111:0/3982223009 wait complete. 2023-10-11T16:43:47.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.255+0000 7fddf8fff700 1 Processor -- start 2023-10-11T16:43:47.260 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.255+0000 7fddf8fff700 1 -- start start 2023-10-11T16:43:47.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf8fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 0x7fddf41964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:47.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf8fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf4101660 0x7fddf4196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:47.261 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf8fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4105620 0x7fddf419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:47.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf8fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419b4a0 con 0x7fddf4100cc0 2023-10-11T16:43:47.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf8fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419b610 con 0x7fddf4105620 2023-10-11T16:43:47.262 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf8fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419b7d0 con 0x7fddf4101660 2023-10-11T16:43:47.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 0x7fddf41964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:47.263 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 0x7fddf41964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41212/0 (socket says 172.21.15.111:41212) 2023-10-11T16:43:47.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf259c700 1 -- 172.21.15.111:0/3127297034 learned_addr learned my addr 172.21.15.111:0/3127297034 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:47.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf1d9b700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf4101660 0x7fddf4196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:47.264 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf2d9d700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4105620 0x7fddf419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:47.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf259c700 1 -- 172.21.15.111:0/3127297034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf4101660 msgr2=0x7fddf4196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf259c700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf4101660 0x7fddf4196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.265 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf259c700 1 -- 172.21.15.111:0/3127297034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4105620 msgr2=0x7fddf419ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.256+0000 7fddf259c700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4105620 0x7fddf419ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.266 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.257+0000 7fddf259c700 1 -- 172.21.15.111:0/3127297034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec009c40 con 0x7fddf4100cc0 2023-10-11T16:43:47.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.257+0000 7fddf259c700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 0x7fddf41964c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdde400b580 tx=0x7fdde400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:47.267 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.257+0000 7fdde37fe700 1 -- 172.21.15.111:0/3127297034 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde401d070 con 0x7fddf4100cc0 2023-10-11T16:43:47.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.257+0000 7fdde37fe700 1 -- 172.21.15.111:0/3127297034 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdde4005050 con 0x7fddf4100cc0 2023-10-11T16:43:47.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.257+0000 7fdde37fe700 1 -- 172.21.15.111:0/3127297034 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4004a40 con 0x7fddf4100cc0 2023-10-11T16:43:47.268 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.257+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde40095f0 con 0x7fddf4100cc0 2023-10-11T16:43:47.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.257+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf41103d0 con 0x7fddf4100cc0 2023-10-11T16:43:47.269 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.258+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf4110240 con 0x7fddf4100cc0 2023-10-11T16:43:47.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.260+0000 7fdde37fe700 1 -- 172.21.15.111:0/3127297034 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fdde40051c0 con 0x7fddf4100cc0 2023-10-11T16:43:47.270 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.261+0000 7fdde37fe700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdddc0a12c0 0x7fdddc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:47.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.262+0000 7fdde37fe700 1 -- 172.21.15.111:0/3127297034 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fdde40ce410 con 0x7fddf4100cc0 2023-10-11T16:43:47.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.262+0000 7fddf1d9b700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdddc0a12c0 0x7fdddc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:47.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.262+0000 7fddf1d9b700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdddc0a12c0 0x7fdddc0a3770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fddf4197ad0 tx=0x7fdde8009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:47.271 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.264+0000 7fdde37fe700 1 -- 172.21.15.111:0/3127297034 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdde4097440 con 0x7fddf4100cc0 2023-10-11T16:43:47.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.465+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fddf4110560 con 0x7fddf4100cc0 2023-10-11T16:43:47.467 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.465+0000 7fdde37fe700 1 -- 172.21.15.111:0/3127297034 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdde409aa60 con 0x7fddf4100cc0 2023-10-11T16:43:47.467 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2023-10-11T16:43:47.469 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.467+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdddc0a12c0 msgr2=0x7fdddc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.467+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdddc0a12c0 0x7fdddc0a3770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fddf4197ad0 tx=0x7fdde8009260 comp rx=0 tx=0).stop 2023-10-11T16:43:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.467+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 msgr2=0x7fddf41964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.470 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.467+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 0x7fddf41964c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdde400b580 tx=0x7fdde400bfd0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.468+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 shutdown_connections 2023-10-11T16:43:47.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.468+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fdddc0a12c0 0x7fdddc0a3770 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.471 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.468+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4105620 0x7fddf419ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.468+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddf4100cc0 0x7fddf41964c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.468+0000 7fddf8fff700 1 --2- 172.21.15.111:0/3127297034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddf4101660 0x7fddf4196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.468+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 >> 172.21.15.111:0/3127297034 conn(0x7fddf4077560 msgr2=0x7fddf40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:47.472 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.468+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 shutdown_connections 2023-10-11T16:43:47.473 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.468+0000 7fddf8fff700 1 -- 172.21.15.111:0/3127297034 wait complete. 2023-10-11T16:43:47.488 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-11T16:43:47.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:47 smithi111 ceph-mon[98527]: pgmap v348: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:47.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:47 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1259509993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:43:47.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:47 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2600282261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:43:47.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 -- 172.21.15.111:0/876003379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5841009e0 msgr2=0x7fc584100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/876003379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5841009e0 0x7fc584100e50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc574009990 tx=0x7fc574009ca0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.689 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 -- 172.21.15.111:0/876003379 shutdown_connections 2023-10-11T16:43:47.690 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/876003379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc584101390 0x7fc58410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/876003379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc584106330 0x7fc584106700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/876003379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5841009e0 0x7fc584100e50 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 -- 172.21.15.111:0/876003379 >> 172.21.15.111:0/876003379 conn(0x7fc584077560 msgr2=0x7fc584077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:47.691 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 -- 172.21.15.111:0/876003379 shutdown_connections 2023-10-11T16:43:47.692 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.687+0000 7fc58c5ff700 1 -- 172.21.15.111:0/876003379 wait complete. 2023-10-11T16:43:47.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.688+0000 7fc58c5ff700 1 Processor -- start 2023-10-11T16:43:47.694 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.688+0000 7fc58c5ff700 1 -- start start 2023-10-11T16:43:47.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58c5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5841009e0 0x7fc5841986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:47.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58c5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc584101390 0x7fc584198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:47.695 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58c5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc584106330 0x7fc58419bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:47.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58c5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58419c5a0 con 0x7fc584101390 2023-10-11T16:43:47.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58c5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58419c710 con 0x7fc5841009e0 2023-10-11T16:43:47.696 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58c5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58419c8d0 con 0x7fc584106330 2023-10-11T16:43:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58a39b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5841009e0 0x7fc5841986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58a39b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5841009e0 0x7fc5841986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50318/0 (socket says 172.21.15.111:50318) 2023-10-11T16:43:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58a39b700 1 -- 172.21.15.111:0/1310745890 learned_addr learned my addr 172.21.15.111:0/1310745890 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc589b9a700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc584101390 0x7fc584198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:47.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58ab9c700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc584106330 0x7fc58419bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:47.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58a39b700 1 -- 172.21.15.111:0/1310745890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc584106330 msgr2=0x7fc58419bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.698 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58a39b700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc584106330 0x7fc58419bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58a39b700 1 -- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc584101390 msgr2=0x7fc584198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58a39b700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc584101390 0x7fc584198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc58a39b700 1 -- 172.21.15.111:0/1310745890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5740095f0 con 0x7fc5841009e0 2023-10-11T16:43:47.699 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.689+0000 7fc589b9a700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc584101390 0x7fc584198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:43:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.690+0000 7fc58a39b700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5841009e0 0x7fc5841986b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc58000e9a0 tx=0x7fc58000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.690+0000 7fc57b7fe700 1 -- 172.21.15.111:0/1310745890 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc580009790 con 0x7fc5841009e0 2023-10-11T16:43:47.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.690+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc584110360 con 0x7fc5841009e0 2023-10-11T16:43:47.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.690+0000 7fc57b7fe700 1 -- 172.21.15.111:0/1310745890 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc580010430 con 0x7fc5841009e0 2023-10-11T16:43:47.701 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.690+0000 7fc57b7fe700 1 -- 172.21.15.111:0/1310745890 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc580018680 con 0x7fc5841009e0 2023-10-11T16:43:47.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.690+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5841101d0 con 0x7fc5841009e0 2023-10-11T16:43:47.702 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.691+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5841104f0 con 0x7fc5841009e0 2023-10-11T16:43:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:47 smithi150 ceph-mon[132217]: pgmap v348: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:47 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1259509993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:43:47.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:47 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2600282261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:43:47.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.693+0000 7fc57b7fe700 1 -- 172.21.15.111:0/1310745890 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fc5800105a0 con 0x7fc5841009e0 2023-10-11T16:43:47.704 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.694+0000 7fc57b7fe700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5700a12c0 0x7fc5700a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:47.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.695+0000 7fc57b7fe700 1 -- 172.21.15.111:0/1310745890 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fc580014070 con 0x7fc5841009e0 2023-10-11T16:43:47.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.695+0000 7fc589b9a700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5700a12c0 0x7fc5700a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:47.705 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.695+0000 7fc589b9a700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5700a12c0 0x7fc5700a3770 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc584199cc0 tx=0x7fc574005fd0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:47.706 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.697+0000 7fc57b7fe700 1 -- 172.21.15.111:0/1310745890 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc5800961f0 con 0x7fc5841009e0 2023-10-11T16:43:47.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:47 smithi130 ceph-mon[131859]: pgmap v348: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:47.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:47 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1259509993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-11T16:43:47.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:47 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2600282261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-11T16:43:47.921 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.920+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc584199b00 con 0x7fc5841009e0 2023-10-11T16:43:47.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.921+0000 7fc57b7fe700 1 -- 172.21.15.111:0/1310745890 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v103) v1 ==== 81+0+958 (secure 0 0 0) 0x7fc584199b00 con 0x7fc5841009e0 2023-10-11T16:43:47.923 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/2838247405 2023-10-12T16:32:37.683058+0000 2023-10-11T16:43:47.923 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/803529843 2023-10-12T16:32:37.683058+0000 2023-10-11T16:43:47.923 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3816963652 2023-10-12T16:32:37.683058+0000 2023-10-11T16:43:47.924 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/4086392563 2023-10-12T16:23:39.970669+0000 2023-10-11T16:43:47.924 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/2838247405 2023-10-12T16:32:37.683058+0000 2023-10-11T16:43:47.924 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/4086392563 2023-10-12T16:23:39.970669+0000 2023-10-11T16:43:47.924 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3251673175 2023-10-12T16:23:39.970669+0000 2023-10-11T16:43:47.925 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1856545817 2023-10-12T16:32:37.683058+0000 2023-10-11T16:43:47.925 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3319300422 2023-10-12T16:23:39.970669+0000 2023-10-11T16:43:47.925 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3405550474 2023-10-12T16:23:13.996718+0000 2023-10-11T16:43:47.926 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/444576470 2023-10-12T16:32:37.683058+0000 2023-10-11T16:43:47.926 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/4129094163 2023-10-12T16:23:39.970669+0000 2023-10-11T16:43:47.926 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1272467376 2023-10-12T16:23:13.996718+0000 2023-10-11T16:43:47.926 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/699479694 2023-10-12T16:23:13.996718+0000 2023-10-11T16:43:47.927 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/699479694 2023-10-12T16:23:13.996718+0000 2023-10-11T16:43:47.927 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3715184603 2023-10-12T16:23:13.996718+0000 2023-10-11T16:43:47.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.923+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5700a12c0 msgr2=0x7fc5700a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.923+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5700a12c0 0x7fc5700a3770 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc584199cc0 tx=0x7fc574005fd0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.923+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5841009e0 msgr2=0x7fc5841986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:47.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.923+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5841009e0 0x7fc5841986b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc58000e9a0 tx=0x7fc58000ecb0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.924+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 shutdown_connections 2023-10-11T16:43:47.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.924+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fc5700a12c0 0x7fc5700a3770 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.924+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc584106330 0x7fc58419bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.924+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc5841009e0 0x7fc5841986b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.924+0000 7fc58c5ff700 1 --2- 172.21.15.111:0/1310745890 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc584101390 0x7fc584198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:47.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.924+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 >> 172.21.15.111:0/1310745890 conn(0x7fc584077560 msgr2=0x7fc5840ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:47.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.924+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 shutdown_connections 2023-10-11T16:43:47.932 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:47.924+0000 7fc58c5ff700 1 -- 172.21.15.111:0/1310745890 wait complete. 2023-10-11T16:43:47.932 INFO:teuthology.orchestra.run.smithi111.stderr:listed 16 entries 2023-10-11T16:43:47.944 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-11T16:43:47.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:43:47.945 DEBUG:teuthology.orchestra.run.smithi111:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-11T16:43:48.066 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:43:48.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:43:48.067 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-11T16:43:48.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:43:48.169 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2023-10-11T16:43:48.297 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-11T16:43:48.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:43:48.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-11T16:43:48.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-11T16:43:48.425 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2023-10-11T16:43:48.546 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-10-11T16:43:48.557 INFO:tasks.cephadm:Teardown begin 2023-10-11T16:43:48.558 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-11T16:43:48.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:43:48.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:48 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3127297034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:43:48.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:48 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1310745890' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:43:48.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:48 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3127297034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:43:48.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:48 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1310745890' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:43:48.763 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 -- 172.21.15.111:0/2867091616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 msgr2=0x7efedc1008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:48.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 --2- 172.21.15.111:0/2867091616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 0x7efedc1008e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7efecc0099c0 tx=0x7efecc009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:48.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 -- 172.21.15.111:0/2867091616 shutdown_connections 2023-10-11T16:43:48.764 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 --2- 172.21.15.111:0/2867091616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc105550 0x7efedc10d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:48.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 --2- 172.21.15.111:0/2867091616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 0x7efedc1008e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:48.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 --2- 172.21.15.111:0/2867091616 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efedc100eb0 0x7efedc104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:48.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 -- 172.21.15.111:0/2867091616 >> 172.21.15.111:0/2867091616 conn(0x7efedc077050 msgr2=0x7efedc077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:48.765 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 -- 172.21.15.111:0/2867091616 shutdown_connections 2023-10-11T16:43:48.766 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.761+0000 7efee13ff700 1 -- 172.21.15.111:0/2867091616 wait complete. 2023-10-11T16:43:48.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.762+0000 7efee13ff700 1 Processor -- start 2023-10-11T16:43:48.768 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.762+0000 7efee13ff700 1 -- start start 2023-10-11T16:43:48.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efee13ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 0x7efedc198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:48.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efee13ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efedc100eb0 0x7efedc198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:48.769 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efee13ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc105550 0x7efedc19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:48.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efee13ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc19d600 con 0x7efedc100510 2023-10-11T16:43:48.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efee13ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc19d770 con 0x7efedc100eb0 2023-10-11T16:43:48.770 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efee13ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc19d930 con 0x7efedc105550 2023-10-11T16:43:48.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efedaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 0x7efedc198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:48.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efeda7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efedc100eb0 0x7efedc198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:48.771 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efeda7fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efedc100eb0 0x7efedc198b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50342/0 (socket says 172.21.15.111:50342) 2023-10-11T16:43:48.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efeda7fc700 1 -- 172.21.15.111:0/2535476666 learned_addr learned my addr 172.21.15.111:0/2535476666 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:48.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efedb7fe700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc105550 0x7efedc19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:48.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efeda7fc700 1 -- 172.21.15.111:0/2535476666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc105550 msgr2=0x7efedc19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:48.772 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efeda7fc700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc105550 0x7efedc19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:48.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efeda7fc700 1 -- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 msgr2=0x7efedc198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:48.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efeda7fc700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 0x7efedc198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:48.773 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efeda7fc700 1 -- 172.21.15.111:0/2535476666 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc0095f0 con 0x7efedc100eb0 2023-10-11T16:43:48.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.763+0000 7efedaffd700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 0x7efedc198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:43:48.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.764+0000 7efeda7fc700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efedc100eb0 0x7efedc198b60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efed000e940 tx=0x7efed000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:48.774 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.764+0000 7efec3fff700 1 -- 172.21.15.111:0/2535476666 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0009730 con 0x7efedc100eb0 2023-10-11T16:43:48.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.764+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efedc110360 con 0x7efedc100eb0 2023-10-11T16:43:48.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.764+0000 7efec3fff700 1 -- 172.21.15.111:0/2535476666 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efed0010430 con 0x7efedc100eb0 2023-10-11T16:43:48.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.764+0000 7efec3fff700 1 -- 172.21.15.111:0/2535476666 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed000cea0 con 0x7efedc100eb0 2023-10-11T16:43:48.775 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.764+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efedc1101d0 con 0x7efedc100eb0 2023-10-11T16:43:48.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.765+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efedc1104f0 con 0x7efedc100eb0 2023-10-11T16:43:48.776 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.767+0000 7efec3fff700 1 -- 172.21.15.111:0/2535476666 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7efed0010810 con 0x7efedc100eb0 2023-10-11T16:43:48.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.768+0000 7efec3fff700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efec40a12c0 0x7efec40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:48.777 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.768+0000 7efec3fff700 1 -- 172.21.15.111:0/2535476666 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7efed0014070 con 0x7efedc100eb0 2023-10-11T16:43:48.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.768+0000 7efedaffd700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efec40a12c0 0x7efec40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:48.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.769+0000 7efedaffd700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efec40a12c0 0x7efec40a3770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efecc009990 tx=0x7efecc005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:48.778 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:48.770+0000 7efec3fff700 1 -- 172.21.15.111:0/2535476666 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efed0096220 con 0x7efedc100eb0 2023-10-11T16:43:48.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:48 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3127297034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-11T16:43:48.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:48 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1310745890' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-11T16:43:49.014 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.013+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efedc067a60 con 0x7efedc100eb0 2023-10-11T16:43:49.015 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.014+0000 7efec3fff700 1 -- 172.21.15.111:0/2535476666 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+10820 (secure 0 0 0) 0x7efed0099840 con 0x7efedc100eb0 2023-10-11T16:43:49.016 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:43:49.016 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:41:48.484142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:43:49.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efec40a12c0 msgr2=0x7efec40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.018 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efec40a12c0 0x7efec40a3770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efecc009990 tx=0x7efecc005fb0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efedc100eb0 msgr2=0x7efedc198b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efedc100eb0 0x7efedc198b60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efed000e940 tx=0x7efed000ec50 comp rx=0 tx=0).stop 2023-10-11T16:43:49.019 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 shutdown_connections 2023-10-11T16:43:49.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7efec40a12c0 0x7efec40a3770 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc105550 0x7efedc19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.020 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efedc100510 0x7efedc198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 --2- 172.21.15.111:0/2535476666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efedc100eb0 0x7efedc198b60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 >> 172.21.15.111:0/2535476666 conn(0x7efedc077050 msgr2=0x7efedc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:49.021 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.016+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 shutdown_connections 2023-10-11T16:43:49.022 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.017+0000 7efee13ff700 1 -- 172.21.15.111:0/2535476666 wait complete. 2023-10-11T16:43:49.022 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 39 2023-10-11T16:43:49.037 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-11T16:43:49.237 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 -- 172.21.15.111:0/1314162576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100dc0 msgr2=0x7f0684104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 --2- 172.21.15.111:0/1314162576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100dc0 0x7f0684104d80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f06780099c0 tx=0x7f0678009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 -- 172.21.15.111:0/1314162576 shutdown_connections 2023-10-11T16:43:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 --2- 172.21.15.111:0/1314162576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0684105460 0x7f068410d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 --2- 172.21.15.111:0/1314162576 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0684100420 0x7f06841007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 --2- 172.21.15.111:0/1314162576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100dc0 0x7f0684104d80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 -- 172.21.15.111:0/1314162576 >> 172.21.15.111:0/1314162576 conn(0x7f06840fbc70 msgr2=0x7f06840fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:49.239 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 -- 172.21.15.111:0/1314162576 shutdown_connections 2023-10-11T16:43:49.240 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.235+0000 7f068a9ff700 1 -- 172.21.15.111:0/1314162576 wait complete. 2023-10-11T16:43:49.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.236+0000 7f068a9ff700 1 Processor -- start 2023-10-11T16:43:49.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f068a9ff700 1 -- start start 2023-10-11T16:43:49.242 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f068a9ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100420 0x7f0684198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f068a9ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0684100dc0 0x7f0684198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f068a9ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0684105460 0x7f068419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:49.243 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f068a9ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068419d570 con 0x7f0684100420 2023-10-11T16:43:49.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f068a9ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068419d6e0 con 0x7f0684105460 2023-10-11T16:43:49.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f068a9ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068419d8a0 con 0x7f0684100dc0 2023-10-11T16:43:49.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f0683fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100420 0x7f0684198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:49.244 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f0683fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100420 0x7f0684198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41280/0 (socket says 172.21.15.111:41280) 2023-10-11T16:43:49.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f0683fff700 1 -- 172.21.15.111:0/2456558519 learned_addr learned my addr 172.21.15.111:0/2456558519 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:49.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f0688f9c700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0684105460 0x7f068419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:49.245 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.237+0000 7f06837fe700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0684100dc0 0x7f0684198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:49.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f0683fff700 1 -- 172.21.15.111:0/2456558519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0684100dc0 msgr2=0x7f0684198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f0683fff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0684100dc0 0x7f0684198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.246 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f0683fff700 1 -- 172.21.15.111:0/2456558519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0684105460 msgr2=0x7f068419cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f0683fff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0684105460 0x7f068419cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f0683fff700 1 -- 172.21.15.111:0/2456558519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780095f0 con 0x7f0684100420 2023-10-11T16:43:49.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f0683fff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100420 0x7f0684198590 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f067400d980 tx=0x7f067400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:49.247 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f06817fa700 1 -- 172.21.15.111:0/2456558519 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674004220 con 0x7f0684100420 2023-10-11T16:43:49.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0684110230 con 0x7f0684100420 2023-10-11T16:43:49.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f06817fa700 1 -- 172.21.15.111:0/2456558519 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0674007640 con 0x7f0684100420 2023-10-11T16:43:49.248 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f06817fa700 1 -- 172.21.15.111:0/2456558519 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674005690 con 0x7f0684100420 2023-10-11T16:43:49.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.238+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06841100a0 con 0x7f0684100420 2023-10-11T16:43:49.249 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.239+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06841103c0 con 0x7f0684100420 2023-10-11T16:43:49.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.241+0000 7f06817fa700 1 -- 172.21.15.111:0/2456558519 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f0674004750 con 0x7f0684100420 2023-10-11T16:43:49.250 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.242+0000 7f06817fa700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f066c0a12c0 0x7f066c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:49.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.242+0000 7f06817fa700 1 -- 172.21.15.111:0/2456558519 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7f06740ccf60 con 0x7f0684100420 2023-10-11T16:43:49.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.243+0000 7f06837fe700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f066c0a12c0 0x7f066c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:49.251 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.243+0000 7f06837fe700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f066c0a12c0 0x7f066c0a3770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f067800b580 tx=0x7f0678000bc0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:49.252 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.246+0000 7f06817fa700 1 -- 172.21.15.111:0/2456558519 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0674095f00 con 0x7f0684100420 2023-10-11T16:43:49.488 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.486+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06840679f0 con 0x7f0684100420 2023-10-11T16:43:49.489 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.487+0000 7f06817fa700 1 -- 172.21.15.111:0/2456558519 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+10820 (secure 0 0 0) 0x7f0674016020 con 0x7f0684100420 2023-10-11T16:43:49.489 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:43:49.490 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14424,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1910869239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1910869239},{"type":"v1","addr":"172.21.15.111:6835","nonce":1910869239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14436,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6837/3168828680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6836","nonce":3168828680},{"type":"v1","addr":"172.21.15.111:6837","nonce":3168828680}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14484,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6839/2081616561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6838","nonce":2081616561},{"type":"v1","addr":"172.21.15.111:6839","nonce":2081616561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14496,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6841/3150990523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6840","nonce":3150990523},{"type":"v1","addr":"172.21.15.111:6841","nonce":3150990523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/447441834","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":447441834},{"type":"v1","addr":"172.21.15.130:6833","nonce":447441834}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2297521930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2297521930},{"type":"v1","addr":"172.21.15.150:6833","nonce":2297521930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-11T16:30:11.633745+0000","modified":"2023-10-11T16:41:48.484142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6835/89303870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":89303870},{"type":"v1","addr":"172.21.15.130:6835","nonce":89303870}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.150:6835/2357736565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2357736565},{"type":"v1","addr":"172.21.15.150:6835","nonce":2357736565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6837/3927432367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":3927432367},{"type":"v1","addr":"172.21.15.130:6837","nonce":3927432367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6837/612139379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":612139379},{"type":"v1","addr":"172.21.15.150:6837","nonce":612139379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6839/3698951370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6838","nonce":3698951370},{"type":"v1","addr":"172.21.15.150:6839","nonce":3698951370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2448702777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2448702777},{"type":"v1","addr":"172.21.15.130:6839","nonce":2448702777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-11T16:43:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f066c0a12c0 msgr2=0x7f066c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.491 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f066c0a12c0 0x7f066c0a3770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f067800b580 tx=0x7f0678000bc0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100420 msgr2=0x7f0684198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.492 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100420 0x7f0684198590 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f067400d980 tx=0x7f067400dc90 comp rx=0 tx=0).stop 2023-10-11T16:43:49.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 shutdown_connections 2023-10-11T16:43:49.494 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f066c0a12c0 0x7f066c0a3770 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0684105460 0x7f068419cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0684100420 0x7f0684198590 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 --2- 172.21.15.111:0/2456558519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0684100dc0 0x7f0684198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.495 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.489+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 >> 172.21.15.111:0/2456558519 conn(0x7f06840fbc70 msgr2=0x7f06840fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.490+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 shutdown_connections 2023-10-11T16:43:49.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.490+0000 7f068a9ff700 1 -- 172.21.15.111:0/2456558519 wait complete. 2023-10-11T16:43:49.498 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 39 2023-10-11T16:43:49.510 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T16:43:49.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:49 smithi111 ceph-mon[98527]: pgmap v349: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:49.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:49 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2535476666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:43:49.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:49 smithi150 ceph-mon[132217]: pgmap v349: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:49.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:49 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2535476666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:43:49.710 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 -- 172.21.15.111:0/3409749862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 msgr2=0x7fe014104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 --2- 172.21.15.111:0/3409749862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 0x7fe014104e70 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0099c0 tx=0x7fdffc009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.711 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 -- 172.21.15.111:0/3409749862 shutdown_connections 2023-10-11T16:43:49.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 --2- 172.21.15.111:0/3409749862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe014105550 0x7fe01410d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 --2- 172.21.15.111:0/3409749862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe014100510 0x7fe0141008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 --2- 172.21.15.111:0/3409749862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 0x7fe014104e70 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.712 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 -- 172.21.15.111:0/3409749862 >> 172.21.15.111:0/3409749862 conn(0x7fe014077050 msgr2=0x7fe014077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:49.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 -- 172.21.15.111:0/3409749862 shutdown_connections 2023-10-11T16:43:49.713 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.709+0000 7fe01a1ff700 1 -- 172.21.15.111:0/3409749862 wait complete. 2023-10-11T16:43:49.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.710+0000 7fe01a1ff700 1 Processor -- start 2023-10-11T16:43:49.715 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.710+0000 7fe01a1ff700 1 -- start start 2023-10-11T16:43:49.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.710+0000 7fe01a1ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe014100510 0x7fe014072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:49.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.710+0000 7fe01a1ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 0x7fe014073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:49.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.710+0000 7fe01a1ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe014105550 0x7fe014073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:49.716 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.710+0000 7fe01a1ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0141a47d0 con 0x7fe014100eb0 2023-10-11T16:43:49.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.710+0000 7fe01a1ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0141a4940 con 0x7fe014100510 2023-10-11T16:43:49.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.710+0000 7fe01a1ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0141a4b00 con 0x7fe014105550 2023-10-11T16:43:49.717 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 0x7fe014073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:49.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe0137fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe014100510 0x7fe014072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:49.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 0x7fe014073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41292/0 (socket says 172.21.15.111:41292) 2023-10-11T16:43:49.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe0137fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe014100510 0x7fe014072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50360/0 (socket says 172.21.15.111:50360) 2023-10-11T16:43:49.718 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 -- 172.21.15.111:0/1700254786 learned_addr learned my addr 172.21.15.111:0/1700254786 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:49.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe013fff700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe014105550 0x7fe014073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:49.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 -- 172.21.15.111:0/1700254786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe014105550 msgr2=0x7fe014073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.719 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe014105550 0x7fe014073870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 -- 172.21.15.111:0/1700254786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe014100510 msgr2=0x7fe014072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe014100510 0x7fe014072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 -- 172.21.15.111:0/1700254786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc0095f0 con 0x7fe014100eb0 2023-10-11T16:43:49.720 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.711+0000 7fe00bfff700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 0x7fe014073330 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0098e0 tx=0x7fdffc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:49.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.712+0000 7fe0117fa700 1 -- 172.21.15.111:0/1700254786 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc01d070 con 0x7fe014100eb0 2023-10-11T16:43:49.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.712+0000 7fe0117fa700 1 -- 172.21.15.111:0/1700254786 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdffc005050 con 0x7fe014100eb0 2023-10-11T16:43:49.721 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.712+0000 7fe01a1ff700 1 -- 172.21.15.111:0/1700254786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe014110320 con 0x7fe014100eb0 2023-10-11T16:43:49.722 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.712+0000 7fe0117fa700 1 -- 172.21.15.111:0/1700254786 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc004970 con 0x7fe014100eb0 2023-10-11T16:43:49.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.712+0000 7fe01a1ff700 1 -- 172.21.15.111:0/1700254786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe014110190 con 0x7fe014100eb0 2023-10-11T16:43:49.723 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.713+0000 7fe00a7fc700 1 -- 172.21.15.111:0/1700254786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0141104b0 con 0x7fe014100eb0 2023-10-11T16:43:49.724 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.715+0000 7fe0117fa700 1 -- 172.21.15.111:0/1700254786 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fdffc005470 con 0x7fe014100eb0 2023-10-11T16:43:49.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.716+0000 7fe0117fa700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe0000a1340 0x7fe0000a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:49.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.716+0000 7fe0117fa700 1 -- 172.21.15.111:0/1700254786 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fdffc0ce100 con 0x7fe014100eb0 2023-10-11T16:43:49.725 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.716+0000 7fe0137fe700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe0000a1340 0x7fe0000a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:49.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.717+0000 7fe0137fe700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe0000a1340 0x7fe0000a37f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe004009c60 tx=0x7fe004009260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:49.726 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.718+0000 7fe0117fa700 1 -- 172.21.15.111:0/1700254786 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdffc097130 con 0x7fe014100eb0 2023-10-11T16:43:49.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:49 smithi130 ceph-mon[131859]: pgmap v349: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-11T16:43:49.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:49 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2535476666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:43:49.922 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.921+0000 7fe00a7fc700 1 -- 172.21.15.111:0/1700254786 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe014074320 con 0x7fe014100eb0 2023-10-11T16:43:49.924 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.923+0000 7fe0117fa700 1 -- 172.21.15.111:0/1700254786 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25438 (secure 0 0 0) 0x7fdffc09a750 con 0x7fe014100eb0 2023-10-11T16:43:49.925 INFO:teuthology.orchestra.run.smithi111.stdout: 2023-10-11T16:43:49.925 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":103,"fsid":"495f265e-6852-11ee-8db6-212e2dc638e7","created":"2023-10-11T16:22:44.513312+0000","modified":"2023-10-11T16:43:45.329616+0000","last_up_change":"2023-10-11T16:29:23.669052+0000","last_in_change":"2023-10-11T16:29:06.128144+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T16:25:56.558461+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-11T16:30:10.034441+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-11T16:30:11.080913+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-11T16:30:14.592036+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4f58cc5-3bfa-4fdb-a127-1926aadd513d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6803","nonce":937472257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6805","nonce":937472257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6809","nonce":937472257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":937472257},{"type":"v1","addr":"172.21.15.111:6807","nonce":937472257}]},"public_addr":"172.21.15.111:6803/937472257","cluster_addr":"172.21.15.111:6805/937472257","heartbeat_back_addr":"172.21.15.111:6809/937472257","heartbeat_front_addr":"172.21.15.111:6807/937472257","state":["exists","up"]},{"osd":1,"uuid":"6c29c935-2362-4f82-a918-4fb7c7ce51f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6801","nonce":1526012165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6803","nonce":1526012165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6807","nonce":1526012165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":1526012165},{"type":"v1","addr":"172.21.15.130:6805","nonce":1526012165}]},"public_addr":"172.21.15.130:6801/1526012165","cluster_addr":"172.21.15.130:6803/1526012165","heartbeat_back_addr":"172.21.15.130:6807/1526012165","heartbeat_front_addr":"172.21.15.130:6805/1526012165","state":["exists","up"]},{"osd":2,"uuid":"614135a6-6a84-40f7-8b31-58459c019526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6801","nonce":1993692554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6803","nonce":1993692554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6807","nonce":1993692554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1993692554},{"type":"v1","addr":"172.21.15.150:6805","nonce":1993692554}]},"public_addr":"172.21.15.150:6801/1993692554","cluster_addr":"172.21.15.150:6803/1993692554","heartbeat_back_addr":"172.21.15.150:6807/1993692554","heartbeat_front_addr":"172.21.15.150:6805/1993692554","state":["exists","up"]},{"osd":3,"uuid":"3f2ce623-b8ba-448b-b9b3-f0970bc8be1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6811","nonce":4229058554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6813","nonce":4229058554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6817","nonce":4229058554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":4229058554},{"type":"v1","addr":"172.21.15.111:6815","nonce":4229058554}]},"public_addr":"172.21.15.111:6811/4229058554","cluster_addr":"172.21.15.111:6813/4229058554","heartbeat_back_addr":"172.21.15.111:6817/4229058554","heartbeat_front_addr":"172.21.15.111:6815/4229058554","state":["exists","up"]},{"osd":4,"uuid":"0c35c66a-dcd9-48ed-9b46-d62ea7add46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6809","nonce":1737228093}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6811","nonce":1737228093}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6815","nonce":1737228093}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":1737228093},{"type":"v1","addr":"172.21.15.130:6813","nonce":1737228093}]},"public_addr":"172.21.15.130:6809/1737228093","cluster_addr":"172.21.15.130:6811/1737228093","heartbeat_back_addr":"172.21.15.130:6815/1737228093","heartbeat_front_addr":"172.21.15.130:6813/1737228093","state":["exists","up"]},{"osd":5,"uuid":"9cbf73bf-6335-4f23-946e-b1b50c55d94f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6809","nonce":2672847721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6811","nonce":2672847721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6815","nonce":2672847721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2672847721},{"type":"v1","addr":"172.21.15.150:6813","nonce":2672847721}]},"public_addr":"172.21.15.150:6809/2672847721","cluster_addr":"172.21.15.150:6811/2672847721","heartbeat_back_addr":"172.21.15.150:6815/2672847721","heartbeat_front_addr":"172.21.15.150:6813/2672847721","state":["exists","up"]},{"osd":6,"uuid":"40e6958d-a7e1-4cd8-abcb-dd770747d086","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6819","nonce":2248469058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6821","nonce":2248469058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6825","nonce":2248469058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2248469058},{"type":"v1","addr":"172.21.15.111:6823","nonce":2248469058}]},"public_addr":"172.21.15.111:6819/2248469058","cluster_addr":"172.21.15.111:6821/2248469058","heartbeat_back_addr":"172.21.15.111:6825/2248469058","heartbeat_front_addr":"172.21.15.111:6823/2248469058","state":["exists","up"]},{"osd":7,"uuid":"3e9ae755-b404-4fbd-ad30-08dee80f0a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6817","nonce":2699897389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6819","nonce":2699897389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6823","nonce":2699897389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2699897389},{"type":"v1","addr":"172.21.15.130:6821","nonce":2699897389}]},"public_addr":"172.21.15.130:6817/2699897389","cluster_addr":"172.21.15.130:6819/2699897389","heartbeat_back_addr":"172.21.15.130:6823/2699897389","heartbeat_front_addr":"172.21.15.130:6821/2699897389","state":["exists","up"]},{"osd":8,"uuid":"1bfb9857-3e28-481a-8a1e-9f3b9b1ea176","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6817","nonce":410436517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6819","nonce":410436517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6823","nonce":410436517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":410436517},{"type":"v1","addr":"172.21.15.150:6821","nonce":410436517}]},"public_addr":"172.21.15.150:6817/410436517","cluster_addr":"172.21.15.150:6819/410436517","heartbeat_back_addr":"172.21.15.150:6823/410436517","heartbeat_front_addr":"172.21.15.150:6821/410436517","state":["exists","up"]},{"osd":9,"uuid":"48901f71-e122-4d76-b262-e0f1d8f18ede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6827","nonce":3788746669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6829","nonce":3788746669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6833","nonce":3788746669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":3788746669},{"type":"v1","addr":"172.21.15.111:6831","nonce":3788746669}]},"public_addr":"172.21.15.111:6827/3788746669","cluster_addr":"172.21.15.111:6829/3788746669","heartbeat_back_addr":"172.21.15.111:6833/3788746669","heartbeat_front_addr":"172.21.15.111:6831/3788746669","state":["exists","up"]},{"osd":10,"uuid":"13ec3960-3990-4c4e-a67f-cafbd49e83eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6825","nonce":2844086684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6827","nonce":2844086684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6831","nonce":2844086684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2844086684},{"type":"v1","addr":"172.21.15.130:6829","nonce":2844086684}]},"public_addr":"172.21.15.130:6825/2844086684","cluster_addr":"172.21.15.130:6827/2844086684","heartbeat_back_addr":"172.21.15.130:6831/2844086684","heartbeat_front_addr":"172.21.15.130:6829/2844086684","state":["exists","up"]},{"osd":11,"uuid":"7af842cf-42d3-4cd4-ab14-a1f0a20ea7fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6825","nonce":2209861208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6827","nonce":2209861208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6831","nonce":2209861208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":2209861208},{"type":"v1","addr":"172.21.15.150:6829","nonce":2209861208}]},"public_addr":"172.21.15.150:6825/2209861208","cluster_addr":"172.21.15.150:6827/2209861208","heartbeat_back_addr":"172.21.15.150:6831/2209861208","heartbeat_front_addr":"172.21.15.150:6829/2209861208","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:11.056743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:32.926240+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:25:54.400018+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:21.073667+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:26:42.707990+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:03.835997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:29.979629+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:27:52.350255+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:13.432762+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:28:38.942530+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:00.850528+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-11T16:29:21.736595+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.a","mappings":[{"from":1,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8},{"from":5,"to":6}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/2838247405":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/803529843":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/3816963652":"2023-10-12T16:32:37.683058+0000","172.21.15.111:6801/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:6800/2838247405":"2023-10-12T16:32:37.683058+0000","172.21.15.111:6800/4086392563":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3251673175":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1856545817":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/3319300422":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/3405550474":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/444576470":"2023-10-12T16:32:37.683058+0000","172.21.15.111:0/4129094163":"2023-10-12T16:23:39.970669+0000","172.21.15.111:0/1272467376":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6800/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:6801/699479694":"2023-10-12T16:23:13.996718+0000","172.21.15.111:0/3715184603":"2023-10-12T16:23:13.996718+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T16:43:49.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 -- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe0000a1340 msgr2=0x7fe0000a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.927 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe0000a1340 0x7fe0000a37f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe004009c60 tx=0x7fe004009260 comp rx=0 tx=0).stop 2023-10-11T16:43:49.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 -- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 msgr2=0x7fe014073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:49.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 0x7fe014073330 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0098e0 tx=0x7fdffc00f690 comp rx=0 tx=0).stop 2023-10-11T16:43:49.928 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 -- 172.21.15.111:0/1700254786 shutdown_connections 2023-10-11T16:43:49.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fe0000a1340 0x7fe0000a37f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe014105550 0x7fe014073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.929 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe014100510 0x7fe014072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 --2- 172.21.15.111:0/1700254786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe014100eb0 0x7fe014073330 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:49.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 -- 172.21.15.111:0/1700254786 >> 172.21.15.111:0/1700254786 conn(0x7fe014077050 msgr2=0x7fe0140fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:49.930 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 -- 172.21.15.111:0/1700254786 shutdown_connections 2023-10-11T16:43:49.931 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:49.925+0000 7fe00a7fc700 1 -- 172.21.15.111:0/1700254786 wait complete. 2023-10-11T16:43:49.946 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-11T16:43:50.146 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/2634884169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 msgr2=0x7fa4c8101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:50.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/2634884169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 0x7fa4c8101020 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80099c0 tx=0x7fa4b8009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:50.147 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/2634884169 shutdown_connections 2023-10-11T16:43:50.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/2634884169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c81055b0 0x7fa4c810d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:50.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/2634884169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 0x7fa4c8101020 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:50.148 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/2634884169 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4c81015f0 0x7fa4c8104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:50.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/2634884169 >> 172.21.15.111:0/2634884169 conn(0x7fa4c8077550 msgr2=0x7fa4c8077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:50.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/2634884169 shutdown_connections 2023-10-11T16:43:50.149 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.145+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/2634884169 wait complete. 2023-10-11T16:43:50.151 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.146+0000 7fa4cd5ff700 1 Processor -- start 2023-10-11T16:43:50.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.146+0000 7fa4cd5ff700 1 -- start start 2023-10-11T16:43:50.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.146+0000 7fa4cd5ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 0x7fa4c8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:50.152 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.146+0000 7fa4cd5ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4c81015f0 0x7fa4c8073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:50.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.146+0000 7fa4cd5ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c81055b0 0x7fa4c81a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:50.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.146+0000 7fa4cd5ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c8073b60 con 0x7fa4c8100c50 2023-10-11T16:43:50.153 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.146+0000 7fa4cd5ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c81a4a90 con 0x7fa4c81015f0 2023-10-11T16:43:50.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.146+0000 7fa4cd5ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c81a4bd0 con 0x7fa4c81055b0 2023-10-11T16:43:50.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 0x7fa4c8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:50.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 0x7fa4c8072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41314/0 (socket says 172.21.15.111:41314) 2023-10-11T16:43:50.154 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 -- 172.21.15.111:0/278652225 learned_addr learned my addr 172.21.15.111:0/278652225 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:50.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4bffff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4c81015f0 0x7fa4c8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:50.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c77fe700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c81055b0 0x7fa4c81a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:50.155 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 -- 172.21.15.111:0/278652225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c81055b0 msgr2=0x7fa4c81a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:50.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c81055b0 0x7fa4c81a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:50.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 -- 172.21.15.111:0/278652225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4c81015f0 msgr2=0x7fa4c8073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:50.156 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4c81015f0 0x7fa4c8073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 -- 172.21.15.111:0/278652225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b80095f0 con 0x7fa4c8100c50 2023-10-11T16:43:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c77fe700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c81055b0 0x7fa4c81a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-11T16:43:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c6ffd700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 0x7fa4c8072df0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009990 tx=0x7fa4b8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:50.157 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c4ff9700 1 -- 172.21.15.111:0/278652225 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b801d070 con 0x7fa4c8100c50 2023-10-11T16:43:50.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.147+0000 7fa4c4ff9700 1 -- 172.21.15.111:0/278652225 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4b8003f90 con 0x7fa4c8100c50 2023-10-11T16:43:50.158 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.148+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c8110330 con 0x7fa4c8100c50 2023-10-11T16:43:50.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.148+0000 7fa4c4ff9700 1 -- 172.21.15.111:0/278652225 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8022cb0 con 0x7fa4c8100c50 2023-10-11T16:43:50.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.148+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c81101a0 con 0x7fa4c8100c50 2023-10-11T16:43:50.159 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.149+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c81104c0 con 0x7fa4c8100c50 2023-10-11T16:43:50.160 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.151+0000 7fa4c4ff9700 1 -- 172.21.15.111:0/278652225 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fa4b8022470 con 0x7fa4c8100c50 2023-10-11T16:43:50.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.153+0000 7fa4c4ff9700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa4b40a1340 0x7fa4b40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:50.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.153+0000 7fa4c4ff9700 1 -- 172.21.15.111:0/278652225 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9556+0+0 (secure 0 0 0) 0x7fa4b80ce5b0 con 0x7fa4c8100c50 2023-10-11T16:43:50.161 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.153+0000 7fa4bffff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa4b40a1340 0x7fa4b40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:50.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.154+0000 7fa4bffff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa4b40a1340 0x7fa4b40a37f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80745e0 tx=0x7fa4b000b410 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:50.162 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.157+0000 7fa4c4ff9700 1 -- 172.21.15.111:0/278652225 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa4b80975e0 con 0x7fa4c8100c50 2023-10-11T16:43:50.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:50.402+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa4c81a54e0 con 0x7fa4c8100c50 2023-10-11T16:43:50.635 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:50 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2456558519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:43:50.636 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:50 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1700254786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:43:50.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:50 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2456558519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:43:50.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:50 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1700254786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:43:50.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:50 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2456558519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-11T16:43:50.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:50 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1700254786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-11T16:43:51.394 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.392+0000 7fa4c4ff9700 1 -- 172.21.15.111:0/278652225 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v40) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa4b809ac00 con 0x7fa4c8100c50 2023-10-11T16:43:51.397 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa4b40a1340 msgr2=0x7fa4b40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa4b40a1340 0x7fa4b40a37f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80745e0 tx=0x7fa4b000b410 comp rx=0 tx=0).stop 2023-10-11T16:43:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 msgr2=0x7fa4c8072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:51.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 0x7fa4c8072df0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009990 tx=0x7fa4b8005740 comp rx=0 tx=0).stop 2023-10-11T16:43:51.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 shutdown_connections 2023-10-11T16:43:51.399 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fa4b40a1340 0x7fa4b40a37f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4c81055b0 0x7fa4c81a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4c8100c50 0x7fa4c8072df0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 --2- 172.21.15.111:0/278652225 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa4c81015f0 0x7fa4c8073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 >> 172.21.15.111:0/278652225 conn(0x7fa4c8077550 msgr2=0x7fa4c80fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 shutdown_connections 2023-10-11T16:43:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.397+0000 7fa4cd5ff700 1 -- 172.21.15.111:0/278652225 wait complete. 2023-10-11T16:43:51.401 INFO:teuthology.orchestra.run.smithi111.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-11T16:43:51.420 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-11T16:43:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 -- 172.21.15.111:0/342156400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 msgr2=0x7f0ad4100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/342156400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 0x7f0ad4100ec0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80099c0 tx=0x7f0ac8009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 -- 172.21.15.111:0/342156400 shutdown_connections 2023-10-11T16:43:51.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/342156400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad4101400 0x7f0ad410d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/342156400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad41063f0 0x7f0ad41067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/342156400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 0x7f0ad4100ec0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 -- 172.21.15.111:0/342156400 >> 172.21.15.111:0/342156400 conn(0x7f0ad4077590 msgr2=0x7f0ad4077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:51.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 -- 172.21.15.111:0/342156400 shutdown_connections 2023-10-11T16:43:51.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.628+0000 7f0ad9fff700 1 -- 172.21.15.111:0/342156400 wait complete. 2023-10-11T16:43:51.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.629+0000 7f0ad9fff700 1 Processor -- start 2023-10-11T16:43:51.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.629+0000 7f0ad9fff700 1 -- start start 2023-10-11T16:43:51.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.629+0000 7f0ad9fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 0x7f0ad41963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:51.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.629+0000 7f0ad9fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad4101400 0x7f0ad4196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:51.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.629+0000 7f0ad9fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad41063f0 0x7f0ad419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:51.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.629+0000 7f0ad9fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad419b3a0 con 0x7f0ad4101400 2023-10-11T16:43:51.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.629+0000 7f0ad9fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad419b510 con 0x7f0ad4100a50 2023-10-11T16:43:51.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.629+0000 7f0ad9fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad419b6d0 con 0x7f0ad41063f0 2023-10-11T16:43:51.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 0x7f0ad41963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:51.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad4101400 0x7f0ad4196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:51.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 0x7f0ad41963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.111:50384/0 (socket says 172.21.15.111:50384) 2023-10-11T16:43:51.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad4101400 0x7f0ad4196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41324/0 (socket says 172.21.15.111:41324) 2023-10-11T16:43:51.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 -- 172.21.15.111:0/2233862919 learned_addr learned my addr 172.21.15.111:0/2233862919 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:51.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad3fff700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad41063f0 0x7f0ad419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:51.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 -- 172.21.15.111:0/2233862919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad41063f0 msgr2=0x7f0ad419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:51.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad41063f0 0x7f0ad419ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 -- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad4101400 msgr2=0x7f0ad4196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:51.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad4101400 0x7f0ad4196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:51.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 -- 172.21.15.111:0/2233862919 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac80095f0 con 0x7f0ad4100a50 2023-10-11T16:43:51.640 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad2ffd700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad4101400 0x7f0ad4196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-11T16:43:51.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.630+0000 7f0ad37fe700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 0x7f0ad41963c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400e9a0 tx=0x7f0ac400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:51.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.631+0000 7f0ad0ff9700 1 -- 172.21.15.111:0/2233862919 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4009790 con 0x7f0ad4100a50 2023-10-11T16:43:51.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.631+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4110410 con 0x7f0ad4100a50 2023-10-11T16:43:51.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.631+0000 7f0ad0ff9700 1 -- 172.21.15.111:0/2233862919 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ac4010430 con 0x7f0ad4100a50 2023-10-11T16:43:51.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.631+0000 7f0ad0ff9700 1 -- 172.21.15.111:0/2233862919 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4018680 con 0x7f0ad4100a50 2023-10-11T16:43:51.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.631+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad4110280 con 0x7f0ad4100a50 2023-10-11T16:43:51.642 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.632+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad41105a0 con 0x7f0ad4100a50 2023-10-11T16:43:51.643 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.633+0000 7f0ad0ff9700 1 -- 172.21.15.111:0/2233862919 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f0ac40105a0 con 0x7f0ad4100a50 2023-10-11T16:43:51.643 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:51 smithi111 ceph-mon[98527]: pgmap v350: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2023-10-11T16:43:51.643 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:51 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/278652225' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-10-11T16:43:51.644 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:51 smithi111 ceph-mon[98527]: osdmap e104: 12 total, 12 up, 12 in 2023-10-11T16:43:51.644 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:51 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-a[98503]: 2023-10-11T16:43:51.380+0000 7f8503163700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-11T16:43:51.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.635+0000 7f0ad0ff9700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0abc0a12c0 0x7f0abc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:51.645 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.635+0000 7f0ad0ff9700 1 -- 172.21.15.111:0/2233862919 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9726+0+0 (secure 0 0 0) 0x7f0ac4014070 con 0x7f0ad4100a50 2023-10-11T16:43:51.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.635+0000 7f0ad2ffd700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0abc0a12c0 0x7f0abc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:51.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.635+0000 7f0ad2ffd700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0abc0a12c0 0x7f0abc0a3770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009990 tx=0x7f0ac8005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:51.646 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.639+0000 7f0ad0ff9700 1 -- 172.21.15.111:0/2233862919 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0ac4051730 con 0x7f0ad4100a50 2023-10-11T16:43:51.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-mon[132217]: pgmap v350: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2023-10-11T16:43:51.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/278652225' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-10-11T16:43:51.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-mon[132217]: osdmap e104: 12 total, 12 up, 12 in 2023-10-11T16:43:51.704 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-c[158181]: ignoring --setuser ceph since I am not root 2023-10-11T16:43:51.704 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-c[158181]: ignoring --setgroup ceph since I am not root 2023-10-11T16:43:51.704 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-mds[158205]: -- 172.21.15.150:0/4115439359 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564d9af5d520 con 0x564d9af32c00 2023-10-11T16:43:51.705 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-c[158181]: starting mds.c at 2023-10-11T16:43:51.705 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-f[157839]: ignoring --setuser ceph since I am not root 2023-10-11T16:43:51.706 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-f[157839]: ignoring --setgroup ceph since I am not root 2023-10-11T16:43:51.706 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-mds[157863]: -- 172.21.15.150:0/3441242418 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555625d25520 con 0x555625cfac00 2023-10-11T16:43:51.706 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-f[157839]: starting mds.f at 2023-10-11T16:43:51.707 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-i[158523]: ignoring --setuser ceph since I am not root 2023-10-11T16:43:51.707 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-i[158523]: ignoring --setgroup ceph since I am not root 2023-10-11T16:43:51.708 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-mds[158547]: -- 172.21.15.150:0/2418707818 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55717e167520 con 0x55717e13cc00 2023-10-11T16:43:51.708 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:43:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-i[158523]: starting mds.i at 2023-10-11T16:43:51.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-mon[131859]: pgmap v350: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2023-10-11T16:43:51.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/278652225' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-10-11T16:43:51.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-mon[131859]: osdmap e104: 12 total, 12 up, 12 in 2023-10-11T16:43:51.833 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b[157084]: ignoring --setuser ceph since I am not root 2023-10-11T16:43:51.833 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b[157084]: ignoring --setgroup ceph since I am not root 2023-10-11T16:43:51.833 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-mds[157108]: -- 172.21.15.130:0/3088712522 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56401416d520 con 0x564014143000 2023-10-11T16:43:51.834 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b[157084]: starting mds.b at 2023-10-11T16:43:51.834 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-e[156401]: ignoring --setuser ceph since I am not root 2023-10-11T16:43:51.834 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-e[156401]: ignoring --setgroup ceph since I am not root 2023-10-11T16:43:51.835 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-mds[156425]: -- 172.21.15.130:0/1433618934 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5648927bd520 con 0x564892792800 2023-10-11T16:43:51.835 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-e[156401]: starting mds.e at 2023-10-11T16:43:51.835 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-e[156401]: 2023-10-11T16:43:51.456+0000 7f9540a8e700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6838/0: (98) Address already in use 2023-10-11T16:43:51.836 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-h[156742]: ignoring --setuser ceph since I am not root 2023-10-11T16:43:51.836 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-h[156742]: ignoring --setgroup ceph since I am not root 2023-10-11T16:43:51.836 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-mds[156766]: -- 172.21.15.130:0/142781270 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5594ceaf5520 con 0x5594ceacac00 2023-10-11T16:43:51.837 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:43:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-h[156742]: starting mds.h at 2023-10-11T16:43:51.884 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:51.883+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0ad41a34c0 con 0x7f0ad4100a50 2023-10-11T16:43:52.398 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.396+0000 7f0ad0ff9700 1 -- 172.21.15.111:0/2233862919 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0ac4096260 con 0x7f0ad4100a50 2023-10-11T16:43:52.400 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0abc0a12c0 msgr2=0x7f0abc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:52.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0abc0a12c0 0x7f0abc0a3770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009990 tx=0x7f0ac8005fb0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 msgr2=0x7f0ad41963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:52.401 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 0x7f0ad41963c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400e9a0 tx=0x7f0ac400ecb0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 shutdown_connections 2023-10-11T16:43:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f0abc0a12c0 0x7f0abc0a3770 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ad41063f0 0x7f0ad419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.402 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ad4100a50 0x7f0ad41963c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 --2- 172.21.15.111:0/2233862919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad4101400 0x7f0ad4196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 >> 172.21.15.111:0/2233862919 conn(0x7f0ad4077590 msgr2=0x7f0ad40ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:52.403 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 shutdown_connections 2023-10-11T16:43:52.404 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.399+0000 7f0ad9fff700 1 -- 172.21.15.111:0/2233862919 wait complete. 2023-10-11T16:43:52.420 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-11T16:43:52.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 -- 172.21.15.111:0/1077421145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c1063f0 msgr2=0x7f2a3c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:52.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 --2- 172.21.15.111:0/1077421145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c1063f0 0x7f2a3c1067c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009990 tx=0x7f2a38009ca0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.625 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 -- 172.21.15.111:0/1077421145 shutdown_connections 2023-10-11T16:43:52.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 --2- 172.21.15.111:0/1077421145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a3c101400 0x7f2a3c10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 --2- 172.21.15.111:0/1077421145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c1063f0 0x7f2a3c1067c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.626 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 --2- 172.21.15.111:0/1077421145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a3c100a50 0x7f2a3c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 -- 172.21.15.111:0/1077421145 >> 172.21.15.111:0/1077421145 conn(0x7f2a3c077590 msgr2=0x7f2a3c077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:52.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 -- 172.21.15.111:0/1077421145 shutdown_connections 2023-10-11T16:43:52.627 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.623+0000 7f2a43fff700 1 -- 172.21.15.111:0/1077421145 wait complete. 2023-10-11T16:43:52.629 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.624+0000 7f2a43fff700 1 Processor -- start 2023-10-11T16:43:52.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.624+0000 7f2a43fff700 1 -- start start 2023-10-11T16:43:52.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.624+0000 7f2a43fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a3c100a50 0x7f2a3c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:52.630 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.624+0000 7f2a43fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c101400 0x7f2a3c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:52.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.624+0000 7f2a43fff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a3c1063f0 0x7f2a3c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:52.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.624+0000 7f2a43fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c19d660 con 0x7f2a3c101400 2023-10-11T16:43:52.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.624+0000 7f2a43fff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c19d7d0 con 0x7f2a3c1063f0 2023-10-11T16:43:52.631 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.624+0000 7f2a43fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c19d990 con 0x7f2a3c100a50 2023-10-11T16:43:52.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c101400 0x7f2a3c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:52.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c101400 0x7f2a3c198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41346/0 (socket says 172.21.15.111:41346) 2023-10-11T16:43:52.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 -- 172.21.15.111:0/3975879993 learned_addr learned my addr 172.21.15.111:0/3975879993 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:52.632 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a41d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a3c100a50 0x7f2a3c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:52.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4259c700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a3c1063f0 0x7f2a3c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:52.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 -- 172.21.15.111:0/3975879993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a3c100a50 msgr2=0x7f2a3c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:52.633 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a3c100a50 0x7f2a3c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 -- 172.21.15.111:0/3975879993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a3c1063f0 msgr2=0x7f2a3c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:52.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a3c1063f0 0x7f2a3c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:52.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 -- 172.21.15.111:0/3975879993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a380095f0 con 0x7f2a3c101400 2023-10-11T16:43:52.634 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a4159a700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c101400 0x7f2a3c198bc0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00e9a0 tx=0x7f2a2c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:52.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a32ffd700 1 -- 172.21.15.111:0/3975879993 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c009790 con 0x7f2a3c101400 2023-10-11T16:43:52.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a32ffd700 1 -- 172.21.15.111:0/3975879993 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a2c010430 con 0x7f2a3c101400 2023-10-11T16:43:52.635 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c1103d0 con 0x7f2a3c101400 2023-10-11T16:43:52.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.626+0000 7f2a32ffd700 1 -- 172.21.15.111:0/3975879993 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c018680 con 0x7f2a3c101400 2023-10-11T16:43:52.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.627+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c110240 con 0x7f2a3c101400 2023-10-11T16:43:52.636 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.628+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3c110560 con 0x7f2a3c101400 2023-10-11T16:43:52.637 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.630+0000 7f2a32ffd700 1 -- 172.21.15.111:0/3975879993 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f2a2c0105a0 con 0x7f2a3c101400 2023-10-11T16:43:52.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.631+0000 7f2a32ffd700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2a280a1340 0x7f2a280a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:52.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.631+0000 7f2a32ffd700 1 -- 172.21.15.111:0/3975879993 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9726+0+0 (secure 0 0 0) 0x7f2a2c014070 con 0x7f2a3c101400 2023-10-11T16:43:52.638 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.631+0000 7f2a41d9b700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2a280a1340 0x7f2a280a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:52.639 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.632+0000 7f2a41d9b700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2a280a1340 0x7f2a280a37f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2a38000c00 tx=0x7f2a3800b260 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:52.639 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:52 smithi111 ceph-mon[98527]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-10-11T16:43:52.640 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:52 smithi111 ceph-mon[98527]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-11T16:43:52.640 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:52 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/278652225' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-10-11T16:43:52.640 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:52 smithi111 ceph-mon[98527]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-10-11T16:43:52.640 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:52 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/2233862919' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:43:52.641 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:52 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:43:52.641 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.635+0000 7f2a32ffd700 1 -- 172.21.15.111:0/3975879993 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2a2c051730 con 0x7f2a3c101400 2023-10-11T16:43:52.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:52 smithi150 ceph-mon[132217]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-10-11T16:43:52.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:52 smithi150 ceph-mon[132217]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-11T16:43:52.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:52 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/278652225' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-10-11T16:43:52.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:52 smithi150 ceph-mon[132217]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-10-11T16:43:52.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:52 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/2233862919' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:43:52.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:52 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:43:52.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:52 smithi130 ceph-mon[131859]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-10-11T16:43:52.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:52 smithi130 ceph-mon[131859]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-11T16:43:52.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:52 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/278652225' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-10-11T16:43:52.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:52 smithi130 ceph-mon[131859]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-10-11T16:43:52.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:52 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/2233862919' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:43:52.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:52 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-11T16:43:52.834 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:52.834+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2a3c1999d0 con 0x7f2a3c101400 2023-10-11T16:43:53.496 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.496+0000 7f2a32ffd700 1 -- 172.21.15.111:0/3975879993 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2a2c099d30 con 0x7f2a3c101400 2023-10-11T16:43:53.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2a280a1340 msgr2=0x7f2a280a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:53.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2a280a1340 0x7f2a280a37f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2a38000c00 tx=0x7f2a3800b260 comp rx=0 tx=0).stop 2023-10-11T16:43:53.502 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c101400 msgr2=0x7f2a3c198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:53.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c101400 0x7f2a3c198bc0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00e9a0 tx=0x7f2a2c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 shutdown_connections 2023-10-11T16:43:53.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f2a280a1340 0x7f2a280a37f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.503 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2a3c1063f0 0x7f2a3c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a3c100a50 0x7f2a3c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 --2- 172.21.15.111:0/3975879993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a3c101400 0x7f2a3c198bc0 secure :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00e9a0 tx=0x7f2a2c00ecb0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.504 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.501+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 >> 172.21.15.111:0/3975879993 conn(0x7f2a3c077590 msgr2=0x7f2a3c10b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.502+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 shutdown_connections 2023-10-11T16:43:53.505 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.502+0000 7f2a43fff700 1 -- 172.21.15.111:0/3975879993 wait complete. 2023-10-11T16:43:53.506 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_metadata' removed 2023-10-11T16:43:53.533 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-11T16:43:53.702 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: pgmap v352: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-10-11T16:43:53.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-10-11T16:43:53.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-10-11T16:43:53.703 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: Cluster is now healthy 2023-10-11T16:43:53.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.130:6836/1549018895,v1:172.21.15.130:6838/1549018895] up:boot 2023-10-11T16:43:53.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.130:6834/665098293,v1:172.21.15.130:6835/665098293] up:boot 2023-10-11T16:43:53.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6834/1822611943,v1:172.21.15.150:6835/1822611943] up:boot 2023-10-11T16:43:53.704 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.130:6837/468820457,v1:172.21.15.130:6839/468820457] up:boot 2023-10-11T16:43:53.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6838/3788273158,v1:172.21.15.150:6839/3788273158] up:boot 2023-10-11T16:43:53.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: mds.? [v2:172.21.15.150:6836/4111185774,v1:172.21.15.150:6837/4111185774] up:boot 2023-10-11T16:43:53.705 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-10-11T16:43:53.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: fsmap 12 up:standby 2023-10-11T16:43:53.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:43:53.706 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:43:53.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:43:53.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:43:53.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:43:53.707 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:43:53.708 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:53 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3975879993' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:53.741 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.740+0000 7f51f67ff700 1 -- 172.21.15.111:0/38854227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0100cc0 msgr2=0x7f51f0101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:53.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.740+0000 7f51f67ff700 1 --2- 172.21.15.111:0/38854227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0100cc0 0x7f51f0101090 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f51e00099c0 tx=0x7f51e0009cd0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.742 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.741+0000 7f51f67ff700 1 -- 172.21.15.111:0/38854227 shutdown_connections 2023-10-11T16:43:53.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.741+0000 7f51f67ff700 1 --2- 172.21.15.111:0/38854227 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51f0105620 0x7f51f010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.741+0000 7f51f67ff700 1 --2- 172.21.15.111:0/38854227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0100cc0 0x7f51f0101090 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.743 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.741+0000 7f51f67ff700 1 --2- 172.21.15.111:0/38854227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f0101660 0x7f51f0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.741+0000 7f51f67ff700 1 -- 172.21.15.111:0/38854227 >> 172.21.15.111:0/38854227 conn(0x7f51f0077560 msgr2=0x7f51f0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:53.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.741+0000 7f51f67ff700 1 -- 172.21.15.111:0/38854227 shutdown_connections 2023-10-11T16:43:53.744 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.741+0000 7f51f67ff700 1 -- 172.21.15.111:0/38854227 wait complete. 2023-10-11T16:43:53.746 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f67ff700 1 Processor -- start 2023-10-11T16:43:53.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f67ff700 1 -- start start 2023-10-11T16:43:53.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f67ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51f0100cc0 0x7f51f0198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:53.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f67ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f0101660 0x7f51f0198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:53.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f67ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0105620 0x7f51f019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:53.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f67ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f019d650 con 0x7f51f0105620 2023-10-11T16:43:53.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f67ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f019d7c0 con 0x7f51f0101660 2023-10-11T16:43:53.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f67ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f019d980 con 0x7f51f0100cc0 2023-10-11T16:43:53.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f4d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0105620 0x7f51f019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:53.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f4d9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0105620 0x7f51f019d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41364/0 (socket says 172.21.15.111:41364) 2023-10-11T16:43:53.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51f4d9c700 1 -- 172.21.15.111:0/1907190475 learned_addr learned my addr 172.21.15.111:0/1907190475 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:53.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.742+0000 7f51effff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51f0100cc0 0x7f51f0198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:53.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51ef7fe700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f0101660 0x7f51f0198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:53.750 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51f4d9c700 1 -- 172.21.15.111:0/1907190475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51f0100cc0 msgr2=0x7f51f0198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51f4d9c700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51f0100cc0 0x7f51f0198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51f4d9c700 1 -- 172.21.15.111:0/1907190475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f0101660 msgr2=0x7f51f0198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51f4d9c700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f0101660 0x7f51f0198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:53.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51f4d9c700 1 -- 172.21.15.111:0/1907190475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e00095f0 con 0x7f51f0105620 2023-10-11T16:43:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51f4d9c700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0105620 0x7f51f019d010 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f51e800ef30 tx=0x7f51e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51ed7fa700 1 -- 172.21.15.111:0/1907190475 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e8018070 con 0x7f51f0105620 2023-10-11T16:43:53.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51ed7fa700 1 -- 172.21.15.111:0/1907190475 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51e800f040 con 0x7f51f0105620 2023-10-11T16:43:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f01103d0 con 0x7f51f0105620 2023-10-11T16:43:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51ed7fa700 1 -- 172.21.15.111:0/1907190475 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e800cd00 con 0x7f51f0105620 2023-10-11T16:43:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.743+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f0110240 con 0x7f51f0105620 2023-10-11T16:43:53.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.745+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f0110560 con 0x7f51f0105620 2023-10-11T16:43:53.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.746+0000 7f51ed7fa700 1 -- 172.21.15.111:0/1907190475 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7f51e8003a00 con 0x7f51f0105620 2023-10-11T16:43:53.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.747+0000 7f51ed7fa700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51d80a12c0 0x7f51d80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:53.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.748+0000 7f51ed7fa700 1 -- 172.21.15.111:0/1907190475 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9143+0+0 (secure 0 0 0) 0x7f51e80cca60 con 0x7f51f0105620 2023-10-11T16:43:53.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.748+0000 7f51effff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51d80a12c0 0x7f51d80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:53.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.748+0000 7f51effff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51d80a12c0 0x7f51d80a3770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009990 tx=0x7f51e0005fb0 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:53.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.751+0000 7f51ed7fa700 1 -- 172.21.15.111:0/1907190475 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f51e8095c30 con 0x7f51f0105620 2023-10-11T16:43:53.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: pgmap v352: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-10-11T16:43:53.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-10-11T16:43:53.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-10-11T16:43:53.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: Cluster is now healthy 2023-10-11T16:43:53.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.130:6836/1549018895,v1:172.21.15.130:6838/1549018895] up:boot 2023-10-11T16:43:53.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.130:6834/665098293,v1:172.21.15.130:6835/665098293] up:boot 2023-10-11T16:43:53.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6834/1822611943,v1:172.21.15.150:6835/1822611943] up:boot 2023-10-11T16:43:53.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.130:6837/468820457,v1:172.21.15.130:6839/468820457] up:boot 2023-10-11T16:43:53.835 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6838/3788273158,v1:172.21.15.150:6839/3788273158] up:boot 2023-10-11T16:43:53.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: mds.? [v2:172.21.15.150:6836/4111185774,v1:172.21.15.150:6837/4111185774] up:boot 2023-10-11T16:43:53.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-10-11T16:43:53.836 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: fsmap 12 up:standby 2023-10-11T16:43:53.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:43:53.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:43:53.837 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:43:53.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:43:53.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:43:53.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:43:53.838 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:53 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3975879993' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:53.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: pgmap v352: 129 pgs: 129 active+clean; 663 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-10-11T16:43:53.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-10-11T16:43:53.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-10-11T16:43:53.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: Cluster is now healthy 2023-10-11T16:43:53.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.130:6836/1549018895,v1:172.21.15.130:6838/1549018895] up:boot 2023-10-11T16:43:53.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.130:6834/665098293,v1:172.21.15.130:6835/665098293] up:boot 2023-10-11T16:43:53.887 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6834/1822611943,v1:172.21.15.150:6835/1822611943] up:boot 2023-10-11T16:43:53.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.130:6837/468820457,v1:172.21.15.130:6839/468820457] up:boot 2023-10-11T16:43:53.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6838/3788273158,v1:172.21.15.150:6839/3788273158] up:boot 2023-10-11T16:43:53.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: mds.? [v2:172.21.15.150:6836/4111185774,v1:172.21.15.150:6837/4111185774] up:boot 2023-10-11T16:43:53.888 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-10-11T16:43:53.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: fsmap 12 up:standby 2023-10-11T16:43:53.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-11T16:43:53.889 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-11T16:43:53.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-11T16:43:53.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-11T16:43:53.890 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-11T16:43:53.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: from='mgr.25544 172.21.15.111:0/3825776887' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-11T16:43:53.891 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:53 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3975879993' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:53.970 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:53.970+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f51f0067b00 con 0x7f51f0105620 2023-10-11T16:43:54.508 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.508+0000 7f51ed7fa700 1 -- 172.21.15.111:0/1907190475 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) v1 ==== 169+0+0 (secure 0 0 0) 0x7f51e8099250 con 0x7f51f0105620 2023-10-11T16:43:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.512+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51d80a12c0 msgr2=0x7f51d80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:54.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.512+0000 7f51f67ff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51d80a12c0 0x7f51d80a3770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009990 tx=0x7f51e0005fb0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.512+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0105620 msgr2=0x7f51f019d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:54.513 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.512+0000 7f51f67ff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0105620 0x7f51f019d010 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f51e800ef30 tx=0x7f51e800c3a0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.514 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.513+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 shutdown_connections 2023-10-11T16:43:54.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.513+0000 7f51f67ff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7f51d80a12c0 0x7f51d80a3770 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.513+0000 7f51f67ff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51f0105620 0x7f51f019d010 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.513+0000 7f51f67ff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51f0100cc0 0x7f51f0198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.513+0000 7f51f67ff700 1 --2- 172.21.15.111:0/1907190475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51f0101660 0x7f51f0198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.513+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 >> 172.21.15.111:0/1907190475 conn(0x7f51f0077560 msgr2=0x7f51f00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.513+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 shutdown_connections 2023-10-11T16:43:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.514+0000 7f51f67ff700 1 -- 172.21.15.111:0/1907190475 wait complete. 2023-10-11T16:43:54.517 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data' removed 2023-10-11T16:43:54.540 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-10-11T16:43:54.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 -- 172.21.15.111:0/528056553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0074400 msgr2=0x7fabd00747d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:54.747 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 --2- 172.21.15.111:0/528056553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0074400 0x7fabd00747d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009990 tx=0x7fabcc009ca0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 -- 172.21.15.111:0/528056553 shutdown_connections 2023-10-11T16:43:54.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 --2- 172.21.15.111:0/528056553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd00736d0 0x7fabd01136a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.748 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 --2- 172.21.15.111:0/528056553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0074400 0x7fabd00747d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 --2- 172.21.15.111:0/528056553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabd0072d20 0x7fabd0073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 -- 172.21.15.111:0/528056553 >> 172.21.15.111:0/528056553 conn(0x7fabd0077560 msgr2=0x7fabd0077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:54.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 -- 172.21.15.111:0/528056553 shutdown_connections 2023-10-11T16:43:54.749 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.746+0000 7fabd75ff700 1 -- 172.21.15.111:0/528056553 wait complete. 2023-10-11T16:43:54.751 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.747+0000 7fabd75ff700 1 Processor -- start 2023-10-11T16:43:54.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd75ff700 1 -- start start 2023-10-11T16:43:54.752 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd75ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0072d20 0x7fabd01a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:54.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd75ff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd00736d0 0x7fabd01a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:54.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd75ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabd0074400 0x7fabd019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:54.753 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd75ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd019cfe0 con 0x7fabd0072d20 2023-10-11T16:43:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd75ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd019d150 con 0x7fabd0074400 2023-10-11T16:43:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd75ff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd019d310 con 0x7fabd00736d0 2023-10-11T16:43:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd539b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0072d20 0x7fabd01a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:54.754 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd539b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0072d20 0x7fabd01a2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41394/0 (socket says 172.21.15.111:41394) 2023-10-11T16:43:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd539b700 1 -- 172.21.15.111:0/732079339 learned_addr learned my addr 172.21.15.111:0/732079339 (peer_addr_for_me v2:172.21.15.111:0/0) 2023-10-11T16:43:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd4b9a700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd00736d0 0x7fabd01a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:54.755 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.748+0000 7fabd5b9c700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabd0074400 0x7fabd019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabd539b700 1 -- 172.21.15.111:0/732079339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd00736d0 msgr2=0x7fabd01a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabd539b700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd00736d0 0x7fabd01a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabd539b700 1 -- 172.21.15.111:0/732079339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabd0074400 msgr2=0x7fabd019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:54.756 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabd539b700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabd0074400 0x7fabd019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabd539b700 1 -- 172.21.15.111:0/732079339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0095f0 con 0x7fabd0072d20 2023-10-11T16:43:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabd539b700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0072d20 0x7fabd01a2900 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fabcc005910 tx=0x7fabcc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:54.757 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabc67fc700 1 -- 172.21.15.111:0/732079339 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc01d070 con 0x7fabd0072d20 2023-10-11T16:43:54.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabc67fc700 1 -- 172.21.15.111:0/732079339 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabcc022470 con 0x7fabd0072d20 2023-10-11T16:43:54.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabc67fc700 1 -- 172.21.15.111:0/732079339 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc017650 con 0x7fabd0072d20 2023-10-11T16:43:54.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd0116970 con 0x7fabd0072d20 2023-10-11T16:43:54.758 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.749+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd01167e0 con 0x7fabd0072d20 2023-10-11T16:43:54.759 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.750+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd0116b00 con 0x7fabd0072d20 2023-10-11T16:43:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.752+0000 7fabc67fc700 1 -- 172.21.15.111:0/732079339 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 135847+0+0 (secure 0 0 0) 0x7fabcc00fad0 con 0x7fabd0072d20 2023-10-11T16:43:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.753+0000 7fabc67fc700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fabbc0a1340 0x7fabbc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-11T16:43:54.760 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.753+0000 7fabc67fc700 1 -- 172.21.15.111:0/732079339 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8510+0+0 (secure 0 0 0) 0x7fabcc0ce160 con 0x7fabd0072d20 2023-10-11T16:43:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.754+0000 7fabd4b9a700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fabbc0a1340 0x7fabbc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-11T16:43:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.754+0000 7fabd4b9a700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fabbc0a1340 0x7fabbc0a37f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fabd010efd0 tx=0x7fabc0007400 comp rx=0 tx=0).ready entity=mgr.25544 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-11T16:43:54.761 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.757+0000 7fabc67fc700 1 -- 172.21.15.111:0/732079339 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fabcc0975b0 con 0x7fabd0072d20 2023-10-11T16:43:54.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:54 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/3975879993' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-10-11T16:43:54.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:54 smithi130 ceph-mon[131859]: osdmap e105: 12 total, 12 up, 12 in 2023-10-11T16:43:54.834 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:54 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1907190475' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:54.885 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:54 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/3975879993' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-10-11T16:43:54.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:54 smithi111 ceph-mon[98527]: osdmap e105: 12 total, 12 up, 12 in 2023-10-11T16:43:54.886 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:54 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1907190475' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:54.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:54 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/3975879993' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-10-11T16:43:54.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:54 smithi150 ceph-mon[132217]: osdmap e105: 12 total, 12 up, 12 in 2023-10-11T16:43:54.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:54 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1907190475' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:54.977 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:54.977+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fabd019deb0 con 0x7fabd0072d20 2023-10-11T16:43:55.511 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.510+0000 7fabc67fc700 1 -- 172.21.15.111:0/732079339 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v107) v1 ==== 178+0+0 (secure 0 0 0) 0x7fabcc09abd0 con 0x7fabd0072d20 2023-10-11T16:43:55.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.514+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fabbc0a1340 msgr2=0x7fabbc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:55.515 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.514+0000 7fabd75ff700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fabbc0a1340 0x7fabbc0a37f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fabd010efd0 tx=0x7fabc0007400 comp rx=0 tx=0).stop 2023-10-11T16:43:55.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.515+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0072d20 msgr2=0x7fabd01a2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-11T16:43:55.516 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.515+0000 7fabd75ff700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0072d20 0x7fabd01a2900 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fabcc005910 tx=0x7fabcc005740 comp rx=0 tx=0).stop 2023-10-11T16:43:55.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.515+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 shutdown_connections 2023-10-11T16:43:55.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.515+0000 7fabd75ff700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:6800/1233945144,v1:172.21.15.111:6801/1233945144] conn(0x7fabbc0a1340 0x7fabbc0a37f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:55.517 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.515+0000 7fabd75ff700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabd0074400 0x7fabd019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:55.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.515+0000 7fabd75ff700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd0072d20 0x7fabd01a2900 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:55.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.515+0000 7fabd75ff700 1 --2- 172.21.15.111:0/732079339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabd00736d0 0x7fabd01a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-11T16:43:55.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.515+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 >> 172.21.15.111:0/732079339 conn(0x7fabd0077560 msgr2=0x7fabd0112890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-11T16:43:55.518 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.516+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 shutdown_connections 2023-10-11T16:43:55.519 INFO:teuthology.orchestra.run.smithi111.stderr:2023-10-11T16:43:55.516+0000 7fabd75ff700 1 -- 172.21.15.111:0/732079339 wait complete. 2023-10-11T16:43:55.519 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'cephfs_data_ec' removed 2023-10-11T16:43:55.548 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:43:55.622 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:43:55.659 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:43:55.695 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-10-11T16:43:55.695 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-11T16:43:55.758 DEBUG:teuthology.orchestra.run.smithi130:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-11T16:43:55.777 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-11T16:43:55.795 INFO:tasks.cephadm:Stopping all daemons... 2023-10-11T16:43:55.796 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-10-11T16:43:55.796 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.a 2023-10-11T16:43:55.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:55 smithi130 ceph-mon[131859]: pgmap v354: 97 pgs: 97 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:43:55.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:55 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/1907190475' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-10-11T16:43:55.832 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:55 smithi130 ceph-mon[131859]: osdmap e106: 12 total, 12 up, 12 in 2023-10-11T16:43:55.833 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:55 smithi130 ceph-mon[131859]: from='client.? 172.21.15.111:0/732079339' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:55.866 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:55 smithi111 ceph-mon[98527]: pgmap v354: 97 pgs: 97 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:43:55.866 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:55 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/1907190475' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-10-11T16:43:55.869 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:55 smithi111 ceph-mon[98527]: osdmap e106: 12 total, 12 up, 12 in 2023-10-11T16:43:55.869 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:55 smithi111 ceph-mon[98527]: from='client.? 172.21.15.111:0/732079339' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:55.952 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:55 smithi150 ceph-mon[132217]: pgmap v354: 97 pgs: 97 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail 2023-10-11T16:43:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:55 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/1907190475' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-10-11T16:43:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:55 smithi150 ceph-mon[132217]: osdmap e106: 12 total, 12 up, 12 in 2023-10-11T16:43:55.953 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:55 smithi150 ceph-mon[132217]: from='client.? 172.21.15.111:0/732079339' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-11T16:43:56.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:55 smithi111 systemd[1]: Stopping Ceph mon.a for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:43:56.814 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-a[98503]: 2023-10-11T16:43:56.387+0000 7f8506169700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:43:56.814 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-a[98503]: 2023-10-11T16:43:56.387+0000 7f8506169700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-10-11T16:43:57.135 INFO:journalctl@ceph.mon.a.smithi111.stdout:Oct 11 16:43:56 smithi111 bash[264286]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-a 2023-10-11T16:43:57.387 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.a.service' 2023-10-11T16:43:57.492 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:43:57.493 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-10-11T16:43:57.493 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-10-11T16:43:57.494 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.b 2023-10-11T16:43:57.831 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:57 smithi130 systemd[1]: Stopping Ceph mon.b for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:43:58.159 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:57 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-b[131834]: 2023-10-11T16:43:57.849+0000 7fbdc5aa2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:43:58.160 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:57 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-b[131834]: 2023-10-11T16:43:57.849+0000 7fbdc5aa2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-10-11T16:43:58.407 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.b.service' 2023-10-11T16:43:58.433 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:58 smithi130 bash[163055]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-b 2023-10-11T16:43:58.433 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:58 smithi130 systemd[1]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.b.service: Succeeded. 2023-10-11T16:43:58.434 INFO:journalctl@ceph.mon.b.smithi130.stdout:Oct 11 16:43:58 smithi130 systemd[1]: Stopped Ceph mon.b for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:43:58.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:43:58.489 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-10-11T16:43:58.490 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-10-11T16:43:58.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.c 2023-10-11T16:43:58.870 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:58 smithi150 systemd[1]: Stopping Ceph mon.c for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:43:59.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:58 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-c[132190]: 2023-10-11T16:43:58.868+0000 7f006f65a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:43:59.174 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:58 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-c[132190]: 2023-10-11T16:43:58.868+0000 7f006f65a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-10-11T16:43:59.430 INFO:journalctl@ceph.mon.c.smithi150.stdout:Oct 11 16:43:59 smithi150 bash[164406]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mon-c 2023-10-11T16:43:59.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mon.c.service' 2023-10-11T16:43:59.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:43:59.689 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-10-11T16:43:59.690 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-10-11T16:43:59.690 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.x 2023-10-11T16:44:00.135 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:43:59 smithi111 systemd[1]: Stopping Ceph mgr.x for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:00.577 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Oct 11 16:44:00 smithi111 bash[264518]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-x 2023-10-11T16:44:00.984 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.x.service' 2023-10-11T16:44:01.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:01.074 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-10-11T16:44:01.074 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-10-11T16:44:01.074 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.y 2023-10-11T16:44:01.581 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:44:01 smithi130 systemd[1]: Stopping Ceph mgr.y for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:03.959 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:44:03 smithi130 bash[163197]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-y 2023-10-11T16:44:03.959 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Oct 11 16:44:03 smithi130 systemd[1]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-10-11T16:44:03.966 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.y.service' 2023-10-11T16:44:04.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:04.003 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-10-11T16:44:04.003 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-10-11T16:44:04.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.z 2023-10-11T16:44:04.452 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:44:04 smithi150 systemd[1]: Stopping Ceph mgr.z for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:06.090 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:44:05 smithi150 bash[164547]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mgr-z 2023-10-11T16:44:06.091 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:44:05 smithi150 systemd[1]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-10-11T16:44:06.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.z.service' 2023-10-11T16:44:06.118 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:44:06 smithi150 systemd[1]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mgr.z.service: Failed with result 'exit-code'. 2023-10-11T16:44:06.119 INFO:journalctl@ceph.mgr.z.smithi150.stdout:Oct 11 16:44:06 smithi150 systemd[1]: Stopped Ceph mgr.z for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:44:06.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:06.132 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-10-11T16:44:06.132 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-10-11T16:44:06.133 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.0 2023-10-11T16:44:06.635 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:44:06 smithi111 systemd[1]: Stopping Ceph osd.0 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:07.135 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:44:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-0[110751]: 2023-10-11T16:44:06.643+0000 7fd575bf3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:44:07.136 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:44:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-0[110751]: 2023-10-11T16:44:06.643+0000 7fd575bf3700 -1 osd.0 107 *** Got signal Terminated *** 2023-10-11T16:44:07.136 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:44:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-0[110751]: 2023-10-11T16:44:06.643+0000 7fd575bf3700 -1 osd.0 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:44:12.636 INFO:journalctl@ceph.osd.0.smithi111.stdout:Oct 11 16:44:12 smithi111 bash[264662]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-0 2023-10-11T16:44:15.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.0.service' 2023-10-11T16:44:15.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:15.782 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-10-11T16:44:15.783 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-10-11T16:44:15.783 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.1 2023-10-11T16:44:16.082 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:44:15 smithi130 systemd[1]: Stopping Ceph osd.1 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:16.581 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:44:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-1[136889]: 2023-10-11T16:44:16.140+0000 7f1fa0168700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:44:16.582 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:44:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-1[136889]: 2023-10-11T16:44:16.140+0000 7f1fa0168700 -1 osd.1 107 *** Got signal Terminated *** 2023-10-11T16:44:16.582 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:44:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-1[136889]: 2023-10-11T16:44:16.140+0000 7f1fa0168700 -1 osd.1 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:44:22.082 INFO:journalctl@ceph.osd.1.smithi130.stdout:Oct 11 16:44:21 smithi130 bash[163344]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-1 2023-10-11T16:44:23.184 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.1.service' 2023-10-11T16:44:23.225 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:23.226 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-10-11T16:44:23.226 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-10-11T16:44:23.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.2 2023-10-11T16:44:23.603 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:44:23 smithi150 systemd[1]: Stopping Ceph osd.2 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:23.952 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:44:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-2[138166]: 2023-10-11T16:44:23.602+0000 7f4872862700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:44:23.952 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:44:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-2[138166]: 2023-10-11T16:44:23.602+0000 7f4872862700 -1 osd.2 107 *** Got signal Terminated *** 2023-10-11T16:44:23.953 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:44:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-2[138166]: 2023-10-11T16:44:23.602+0000 7f4872862700 -1 osd.2 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:44:29.452 INFO:journalctl@ceph.osd.2.smithi150.stdout:Oct 11 16:44:29 smithi150 bash[164691]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-2 2023-10-11T16:44:30.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.2.service' 2023-10-11T16:44:30.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:30.799 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-10-11T16:44:30.799 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-10-11T16:44:30.800 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.3 2023-10-11T16:44:31.135 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:44:30 smithi111 systemd[1]: Stopping Ceph osd.3 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:31.635 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:44:31 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3[116284]: 2023-10-11T16:44:31.314+0000 7fd1e4143700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:44:31.636 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:44:31 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3[116284]: 2023-10-11T16:44:31.314+0000 7fd1e4143700 -1 osd.3 107 *** Got signal Terminated *** 2023-10-11T16:44:31.636 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:44:31 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3[116284]: 2023-10-11T16:44:31.314+0000 7fd1e4143700 -1 osd.3 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:44:33.720 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:33 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:33.466+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:34.135 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:44:33 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3[116284]: 2023-10-11T16:44:33.719+0000 7fd1e0739700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.552917+0000 front 2023-10-11T16:44:11.552659+0000 (oldest deadline 2023-10-11T16:44:33.252603+0000) 2023-10-11T16:44:34.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:34 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:34.050+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:34.452 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:33.954+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:34.635 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:34.493+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:34.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:34.319+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:35.135 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:44:34 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3[116284]: 2023-10-11T16:44:34.738+0000 7fd1e0739700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.552917+0000 front 2023-10-11T16:44:11.552659+0000 (oldest deadline 2023-10-11T16:44:33.252603+0000) 2023-10-11T16:44:35.202 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:34.923+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:35.331 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:35 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:35.068+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:35.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:35 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:35.317+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:35.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:35 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:35.463+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:36.135 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:44:35 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3[116284]: 2023-10-11T16:44:35.746+0000 7fd1e0739700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.552917+0000 front 2023-10-11T16:44:11.552659+0000 (oldest deadline 2023-10-11T16:44:33.252603+0000) 2023-10-11T16:44:36.202 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:35.944+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:36.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:36 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:36.100+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:36.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:36.273+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:36.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:36 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:36.444+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:37.202 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:36.933+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:37.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:36.995+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:37.455 INFO:journalctl@ceph.osd.3.smithi111.stdout:Oct 11 16:44:37 smithi111 bash[264949]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-3 2023-10-11T16:44:37.455 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:37.301+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:37.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:37 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:37.095+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:37.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:37 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:37.453+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:38.202 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:37.941+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:38.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:38.031+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:38.331 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:38 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:38.049+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:38.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:38.300+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:38.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:38 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:38.421+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:39.370 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.3.service' 2023-10-11T16:44:39.452 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:38.959+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:39.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:39 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:39.020+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:39.463 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:39.464 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-10-11T16:44:39.464 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-10-11T16:44:39.465 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.4 2023-10-11T16:44:39.496 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:39 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:39.096+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:39.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:39 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:39.315+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:39.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:39 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:39.448+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:39.821 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:44:39 smithi130 systemd[1]: Stopping Ceph osd.4 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:40.081 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:44:39 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4[142394]: 2023-10-11T16:44:39.819+0000 7f7749e78700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:44:40.082 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:44:39 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4[142394]: 2023-10-11T16:44:39.819+0000 7f7749e78700 -1 osd.4 107 *** Got signal Terminated *** 2023-10-11T16:44:40.082 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:44:39 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4[142394]: 2023-10-11T16:44:39.819+0000 7f7749e78700 -1 osd.4 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:44:40.452 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:39 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:39.951+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:40.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:40 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:40.029+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:40.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:40 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:40.124+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:40.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:40 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:40.316+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:40.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:40 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:40.422+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:41.202 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:40 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:40.929+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:41.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:41 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:41.014+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:41.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:41 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:41.142+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:41.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:41 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:41.286+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:41.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:41 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:41.460+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:42.202 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:41 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:41.933+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:42.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:42 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:42.062+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:42.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:42 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:42.096+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:42.635 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:42 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:42.458+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:42.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:42 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:42.244+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:42.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:42 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:42.244+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:43.331 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:44:43 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4[142394]: 2023-10-11T16:44:43.099+0000 7f774646e700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.079422+0000 front 2023-10-11T16:44:21.079450+0000 (oldest deadline 2023-10-11T16:44:42.779568+0000) 2023-10-11T16:44:43.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:42 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:42.877+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:43.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:43 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:43.066+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:43.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:43 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:43.066+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:43.452 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:42 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:42.976+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:43.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:43.101+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:43.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:43.034+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:43.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:43.034+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:43.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:43 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:43.231+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:43.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:43 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:43.231+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:43.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:43 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:43.418+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:44.202 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:43 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:43.949+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:44.203 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:44 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:44.108+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:44.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:44 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:44.057+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:44.204 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:44 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:44.057+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:44.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:43 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:43.906+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:44.332 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:44:44 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4[142394]: 2023-10-11T16:44:44.121+0000 7f774646e700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.079422+0000 front 2023-10-11T16:44:21.079450+0000 (oldest deadline 2023-10-11T16:44:42.779568+0000) 2023-10-11T16:44:44.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:44 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:44.030+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:44.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:44 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:44.030+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:44.635 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:44 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:44.446+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:44.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:44 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:44.223+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:44.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:44 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:44.223+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:45.315 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:44 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:44.946+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:45.315 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:45.007+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:45.315 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:45.007+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:45.452 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:44 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:44.986+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:45.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:45 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:45.123+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:45.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:45 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:45.030+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:45.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:45 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:45.030+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:45.581 INFO:journalctl@ceph.osd.4.smithi130.stdout:Oct 11 16:44:45 smithi130 bash[163614]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-4 2023-10-11T16:44:45.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:45 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:45.264+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:45.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:45 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:45.264+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:45.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:45 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:45.436+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:45.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:45 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:45.436+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:46.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:45.948+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:46.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:45.993+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:46.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:45 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:45.993+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:46.452 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:45 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:45.972+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:46.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:46.148+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:46.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:46.048+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:46.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:46.048+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:46.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:46 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:46.284+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:46.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:46 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:46.284+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:46.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:46 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:46.406+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:46.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:46 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:46.406+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:46.868 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.4.service' 2023-10-11T16:44:46.949 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:46.950 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-10-11T16:44:46.951 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-10-11T16:44:46.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.5 2023-10-11T16:44:47.313 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:46.972+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:47.314 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:46 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:46.972+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:19.630306+0000 front 2023-10-11T16:44:19.630516+0000 (oldest deadline 2023-10-11T16:44:46.030864+0000) 2023-10-11T16:44:47.315 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:46 smithi150 systemd[1]: Stopping Ceph osd.5 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:47.315 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:47.311+0000 7f2f6def5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:44:47.316 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:47.311+0000 7f2f6def5700 -1 osd.5 107 *** Got signal Terminated *** 2023-10-11T16:44:47.317 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:47.187+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:47.317 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:47.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:47.318 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:47.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:47.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:46 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:46.995+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:47.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:46 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:46.953+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:47.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:46 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:46.953+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:47.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:47 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:47.285+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:47.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:47 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:47.285+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:47.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:47 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:47.418+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:47.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:47 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:47.418+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:47.702 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:47.311+0000 7f2f6def5700 -1 osd.5 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:44:48.202 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:48.179+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:48.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:48.011+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:48.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:48.011+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:48.204 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:47.950+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:48.204 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:47 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:47.950+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:19.630306+0000 front 2023-10-11T16:44:19.630516+0000 (oldest deadline 2023-10-11T16:44:46.030864+0000) 2023-10-11T16:44:48.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:47 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:47.981+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:48.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:47 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:47.977+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:48.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:47 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:47.977+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:48.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:48 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:48.253+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:48.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:48 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:48.253+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:48.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:48 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:48.461+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:48.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:48 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:48.461+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:49.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:48 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:48.986+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:49.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:49 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:49.027+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:49.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:49 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:49.027+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:49.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:49 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:49.019+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:49.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:49 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:49.019+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:49.453 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:48.973+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:49.454 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:48 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:48.973+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:19.630306+0000 front 2023-10-11T16:44:19.630516+0000 (oldest deadline 2023-10-11T16:44:46.030864+0000) 2023-10-11T16:44:49.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:49 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:49.177+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:49.455 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:49 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:49.177+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:49.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:49 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:49.293+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:49.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:49 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:49.293+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:49.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:49 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:49.495+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:49.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:49 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:49.495+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:49.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:49 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:49.495+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:50.202 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:50 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:50.062+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:50.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:50 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:50.062+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:50.203 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:49 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:49.939+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:50.204 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:49 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:49.939+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:19.630306+0000 front 2023-10-11T16:44:19.630516+0000 (oldest deadline 2023-10-11T16:44:46.030864+0000) 2023-10-11T16:44:50.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:50 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:50.027+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:50.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:50 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:50.052+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:50.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:50 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:50.052+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:50.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:50 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:50.303+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:50.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:50 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:50.303+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:50.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:50 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:50.448+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:50.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:50 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:50.448+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:50.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:50 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:50.448+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:50.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:50 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:50.224+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:50.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:50 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:50.224+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:51.202 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:51.056+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:51.202 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:51.056+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:51.203 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:50 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:50.944+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:51.203 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:50 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:50.944+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:19.630306+0000 front 2023-10-11T16:44:19.630516+0000 (oldest deadline 2023-10-11T16:44:46.030864+0000) 2023-10-11T16:44:51.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:51.035+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:51.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:51.035+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:51.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:51.075+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:51.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:51.075+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:51.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:51 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:51.257+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:51.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:51 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:51.257+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:51.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:51 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:51.257+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:51.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:51 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:51.400+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:51.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:51 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:51.400+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:51.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:51 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:51.400+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:51.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:51.250+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:51.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:51.250+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:52.202 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:52 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:52.029+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:52.202 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:52 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:52.029+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:52.203 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:51.932+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.328571+0000 front 2023-10-11T16:44:10.328636+0000 (oldest deadline 2023-10-11T16:44:33.828397+0000) 2023-10-11T16:44:52.203 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:51.932+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:19.630306+0000 front 2023-10-11T16:44:19.630516+0000 (oldest deadline 2023-10-11T16:44:46.030864+0000) 2023-10-11T16:44:52.203 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:51 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5[143602]: 2023-10-11T16:44:51.932+0000 7f2f68ce8700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.031275+0000 front 2023-10-11T16:44:26.031250+0000 (oldest deadline 2023-10-11T16:44:51.931487+0000) 2023-10-11T16:44:52.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:51.991+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:52.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:51 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:51.991+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:52.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:52 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:52.041+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:52.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:52 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:52.041+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:52.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:52.303+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:52.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:52.303+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:52.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:52.303+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:52.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:52.442+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:52.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:52.442+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:52.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:52.442+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:52.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:52 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:52.204+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:52.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:52 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:52.204+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:53.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:53 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:53.023+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:53.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:53 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:53.023+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:53.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:53 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:53.041+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:53.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:53 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:53.041+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:53.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:53 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:53.041+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:44:53.452 INFO:journalctl@ceph.osd.5.smithi150.stdout:Oct 11 16:44:53 smithi150 bash[164980]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-5 2023-10-11T16:44:53.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:53 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:53.241+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:53.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:53 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:53.241+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:53.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:53 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:53.068+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:53.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:53 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:53.068+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:53.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:53 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:53.068+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:44:53.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:53.319+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:53.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:53.319+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:53.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:53.319+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:53.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:53.457+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:53.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:53.457+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:53.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:53 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:53.457+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:54.333 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:54.071+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:54.334 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:54.071+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:54.334 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:54.008+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:54.334 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:54.008+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:54.335 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:54.008+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:44:54.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:54 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:54.254+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:54.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:54 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:54.254+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:54.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:54 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:54.116+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:54.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:54 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:54.116+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:54.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:54 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:54.116+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:44:54.586 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.5.service' 2023-10-11T16:44:54.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:44:54.630 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-10-11T16:44:54.631 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-10-11T16:44:54.631 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.6 2023-10-11T16:44:54.637 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:54 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:54.349+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:54.638 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:54 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:54.349+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:54.638 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:54 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:54.349+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:54.639 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:54 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:54.410+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:54.639 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:54 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:54.410+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:54.639 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:54 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:54.410+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:55.135 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:54 smithi111 systemd[1]: Stopping Ceph osd.6 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:44:55.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:55 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:55.026+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:55.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:55 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:55.026+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:55.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:54.959+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:55.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:54.959+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:55.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:54 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:54.959+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:44:55.433 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:55.145+0000 7f744243a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:44:55.433 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:55.145+0000 7f744243a700 -1 osd.6 107 *** Got signal Terminated *** 2023-10-11T16:44:55.433 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:55.145+0000 7f744243a700 -1 osd.6 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:44:55.434 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:55.341+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:55.434 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:55.341+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:55.434 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:55.341+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:55.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:55 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:55.228+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:55.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:55 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:55.228+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:55.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:55 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:55.127+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:55.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:55 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:55.127+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:55.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:55 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:55.127+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:44:55.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:55.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:55.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:55.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:55.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:55 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:55.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:56.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:55 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:55.986+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:56.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:55 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:55.986+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:56.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:55 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:55.939+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:56.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:55 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:55.939+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:56.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:55 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:55.939+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:44:56.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:56 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:56.214+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:56.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:56 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:56.214+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:56.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:56 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:56.086+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:56.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:56 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:56.086+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:56.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:56 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:56.086+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:44:56.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:56.352+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:56.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:56.352+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:56.637 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:56.352+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:56.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:56.404+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:56.638 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:56.404+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:56.638 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:56 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:56.404+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:57.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:56 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:56.958+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:57.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:56 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:56.958+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:57.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:56 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:56.946+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:57.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:56 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:56.946+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:57.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:56 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:56.946+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:44:57.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:57 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:57.173+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:57.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:57 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:57.173+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:57.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:57 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:57.117+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:57.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:57 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:57.117+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:57.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:57 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:57.117+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:44:57.885 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:57 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:57.386+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:57.886 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:57 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:57.386+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:57.886 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:57 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:57.386+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:57.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:57 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:57.438+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:57.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:57 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:57.439+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:57.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:57 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:57.439+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:58.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:57 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:57.966+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:58.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:57 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:57.966+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:58.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:57 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:57.966+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:44:58.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:57 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:57.967+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:58.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:57 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:57.967+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:58.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:57 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:57.967+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:44:58.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:58 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:58.211+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:58.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:58 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:58.211+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:58.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:58 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:58.094+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:58.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:58 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:58.094+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:58.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:58 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:58.094+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:44:58.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:58.356+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:58.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:58.356+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:58.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:58.356+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:58.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:58.429+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:58.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:58.429+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:58.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:58 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:58.429+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:59.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:58 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:58.933+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:44:59.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:58 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:58.933+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:44:59.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:58 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:58.933+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:44:59.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:58 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:58.978+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:44:59.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:58 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:58.978+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:44:59.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:58 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:58.978+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:44:59.334 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:44:58 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:44:58.978+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:44:59.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:59 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:59.204+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:44:59.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:44:59 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:44:59.204+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:44:59.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:59 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:59.052+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:44:59.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:59 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:59.052+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:44:59.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:44:59 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:44:59.052+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:44:59.635 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:59 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:59.348+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:11.341330+0000 front 2023-10-11T16:44:11.341424+0000 (oldest deadline 2023-10-11T16:44:34.241510+0000) 2023-10-11T16:44:59.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:59 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:59.348+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.143193+0000 front 2023-10-11T16:44:17.143316+0000 (oldest deadline 2023-10-11T16:44:41.843306+0000) 2023-10-11T16:44:59.636 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:44:59 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6[121758]: 2023-10-11T16:44:59.348+0000 7f743d22d700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:25.845228+0000 front 2023-10-11T16:44:25.844951+0000 (oldest deadline 2023-10-11T16:44:51.144877+0000) 2023-10-11T16:44:59.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:59 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:59.410+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:44:59.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:59 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:59.410+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:44:59.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:59 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:59.410+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:44:59.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:44:59 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:44:59.410+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:00.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:59 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:59.929+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:00.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:59 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:59.929+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:00.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:44:59 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:44:59.929+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:00.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:00.007+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:00.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:00.007+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:00.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:00.007+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:00.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:00.007+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:00.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:00 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:00.243+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:00.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:00 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:00.243+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:00.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:00 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:00.102+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:00.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:00 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:00.102+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:00.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:00 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:00.102+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:00.793 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:00.429+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:00.793 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:00.429+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:00.794 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:00.429+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:00.794 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:00 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:00.429+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:01.135 INFO:journalctl@ceph.osd.6.smithi111.stdout:Oct 11 16:45:00 smithi111 bash[265231]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-6 2023-10-11T16:45:01.331 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:00.961+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:01.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:00.961+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:01.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:00.961+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:01.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:00.961+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:01.333 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:00.929+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:01.333 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:00.929+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:01.333 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:00.929+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:01.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:01 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:01.258+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:01.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:01 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:01.258+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:01.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:01 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:01.258+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:01.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:01 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:01.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:01.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:01 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:01.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:01.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:01 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:01.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:01.880 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:01 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:01.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:01.881 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:01 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:01.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:01.881 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:01 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:01.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:01.882 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:01 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:01.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:02.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:01.951+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:02.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:01.951+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:02.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:01.951+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:02.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:01.996+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:02.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:01.996+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:02.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:01.996+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:02.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:01 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:01.996+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:02.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:02 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:02.298+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:02.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:02 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:02.298+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:02.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:02 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:02.298+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:02.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:02 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:02.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:02.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:02 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:02.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:02.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:02 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:02.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:02.635 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:02 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:02.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:02.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:02 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:02.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:02.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:02 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:02.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:02.637 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:02 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:02.432+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:02.785 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.6.service' 2023-10-11T16:45:02.874 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:02.875 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-10-11T16:45:02.876 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-10-11T16:45:02.876 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.7 2023-10-11T16:45:03.234 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:02 smithi130 systemd[1]: Stopping Ceph osd.7 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:03.234 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:02 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:02.934+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:03.235 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:02 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:02.934+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:03.235 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:02 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:02.934+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:03.235 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:02 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:02.975+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:03.236 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:02 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:02.975+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:03.236 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:02 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:02.975+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:03.236 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:02 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:02.975+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:03.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:03 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:03.081+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:03.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:03 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:03.081+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:03.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:03 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:03.081+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:03.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:03 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:03.271+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:03.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:03 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:03.271+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:03.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:03 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:03.271+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:03.582 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:03 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:03.233+0000 7f4f27788700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:03.582 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:03 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:03.233+0000 7f4f27788700 -1 osd.7 107 *** Got signal Terminated *** 2023-10-11T16:45:03.583 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:03 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:03.233+0000 7f4f27788700 -1 osd.7 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:45:03.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:03 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:03.436+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:03.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:03 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:03.436+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:03.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:03 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:03.436+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:03.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:03 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:03.436+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:04.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:03 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:03.971+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:04.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:03 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:03.971+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:04.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:03 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:03.971+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:04.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:03.999+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:04.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:03.999+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:04.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:03.999+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:04.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:03.999+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:04.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:04 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:04.293+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:04.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:04 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:04.293+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:04.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:04 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:04.293+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:04.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:04 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:04.046+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:04.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:04 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:04.046+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:04.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:04 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:04.046+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:04.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:04 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:04.483+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:04.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:04 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:04.483+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:04.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:04 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:04.483+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:04.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:04 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:04.483+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:05.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:04.989+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:05.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:04.989+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:05.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:04.989+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:05.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:05 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:05.031+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:05.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:05 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:05.031+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:05.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:05 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:05.031+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:05.334 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:05 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:05.031+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:05.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:05 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:05.009+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:05.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:05 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:05.009+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:05.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:05 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:05.009+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:05.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:05 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:05.253+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:05.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:05 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:05.253+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:05.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:05 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:05.253+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:05.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:05 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:05.503+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:05.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:05 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:05.503+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:05.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:05 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:05.503+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:05.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:05 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:05.503+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:06.299 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:06.028+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:06.299 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:06.028+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:06.300 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:06.028+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:06.300 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:06.028+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:06.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:06.022+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:06.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:06.022+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:06.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:06.022+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:06.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:06.047+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:06.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:06.047+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:06.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:06.047+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:06.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:06.047+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:06.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:06.298+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:06.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:06.298+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:06.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:06.298+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:06.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:06.517+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:06.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:06.517+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:06.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:06.517+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:06.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:06.517+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:06.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:06 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:06.517+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:07.321 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:06.988+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:07.321 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:06.988+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:07.321 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:06.988+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:07.322 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:06 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:06.988+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:07.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:06.987+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:07.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:06.987+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:07.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:06 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:06.987+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:07.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:07 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:07.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:07.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:07 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:07.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:07.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:07 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:07.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:07.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:07 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:07.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:07.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:07 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:07.320+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:07.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:07 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:07.320+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:07.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:07 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:07.320+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:07.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:07 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:07.320+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:07.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:07 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:07.524+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:07.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:07 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:07.524+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:07.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:07 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:07.524+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:07.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:07 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:07.524+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:07.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:07 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:07.524+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:08.331 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:07 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:07.972+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:20.410717+0000 front 2023-10-11T16:44:20.410746+0000 (oldest deadline 2023-10-11T16:44:42.710819+0000) 2023-10-11T16:45:08.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:07 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:07.972+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:22.711262+0000 front 2023-10-11T16:44:22.711221+0000 (oldest deadline 2023-10-11T16:44:50.912075+0000) 2023-10-11T16:45:08.332 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:07 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7[147790]: 2023-10-11T16:45:07.972+0000 7f4f2257b700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:35.514151+0000 front 2023-10-11T16:44:35.514007+0000 (oldest deadline 2023-10-11T16:44:57.814018+0000) 2023-10-11T16:45:08.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:08 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:08.055+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:08.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:08 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:08.055+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:08.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:08 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:08.055+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:08.334 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:08 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:08.055+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:08.334 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:08 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:08.055+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:08.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:08 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:08.270+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:08.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:08 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:08.270+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:08.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:08 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:08.270+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:08.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:08 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:08.270+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:08.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:08 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:08.024+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:08.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:08 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:08.024+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:08.457 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:08 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:08.024+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:08.458 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:08 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:08.024+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:08.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:08 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:08.536+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:08.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:08 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:08.536+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:08.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:08 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:08.536+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:08.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:08 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:08.536+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:08.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:08 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:08.536+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:09.113 INFO:journalctl@ceph.osd.7.smithi130.stdout:Oct 11 16:45:09 smithi130 bash[163878]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-7 2023-10-11T16:45:09.113 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:09 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:09.088+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:09.114 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:09 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:09.088+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:09.114 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:09 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:09.088+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:09.114 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:09 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:09.088+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:09.114 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:09 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:09.088+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:09.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:09 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:09.257+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:09.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:09 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:09.257+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:09.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:09 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:09.257+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:09.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:09 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:09.257+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:09.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:09 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:09.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:09.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:09 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:09.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:09.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:09 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:09.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:09.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:09 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:09.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:09.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:09 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:09.579+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:09.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:09 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:09.579+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:09.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:09 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:09.579+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:09.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:09 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:09.579+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:09.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:09 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:09.579+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:10.280 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:10.008+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:10.280 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:10.008+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:10.281 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:10.008+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:10.281 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:10.008+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:10.331 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:10 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:10.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:10.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:10 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:10.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:10.332 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:10 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:10.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:10.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:10 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:10.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:10.333 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:10 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:10.086+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:10.376 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.7.service' 2023-10-11T16:45:10.416 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:10.417 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-10-11T16:45:10.418 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-10-11T16:45:10.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.8 2023-10-11T16:45:10.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:10.279+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:10.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:10.279+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:10.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:10.279+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:10.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:10.279+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:10.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:10 smithi150 systemd[1]: Stopping Ceph osd.8 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:10.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:10.563+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:10.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:10.563+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:10.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:10.563+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:10.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:10.563+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:10.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:10 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:10.563+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:11.202 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:10.776+0000 7f2a337f5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:11.202 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:10.776+0000 7f2a337f5700 -1 osd.8 107 *** Got signal Terminated *** 2023-10-11T16:45:11.203 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:10 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:10.776+0000 7f2a337f5700 -1 osd.8 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:45:11.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:11.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:11.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:11.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:11.204 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:11.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:11.204 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:11.026+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:11.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:11 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:11.118+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:11.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:11 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:11.118+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:11.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:11 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:11.118+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:11.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:11 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:11.118+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:11.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:11 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:11.118+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:11.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:11.258+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:11.702 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:11.258+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:11.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:11.258+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:11.703 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:11.258+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:11.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:11 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:11.565+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:11.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:11 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:11.565+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:11.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:11 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:11.565+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:11.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:11 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:11.565+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:11.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:11 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:11.565+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:12.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:11.998+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:12.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:11.998+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:12.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:11.998+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:12.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:11 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:11.998+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:12.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:12.221+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:12.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:12.221+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:12.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:12.221+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:12.455 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:12.221+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:12.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:12 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:12.138+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:12.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:12 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:12.138+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:12.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:12 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:12.138+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:12.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:12 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:12.138+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:12.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:12 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:12.138+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:12.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:12.597+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:12.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:12.597+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:12.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:12.597+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:12.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:12.597+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:12.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:12 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:12.597+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:13.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:12.983+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:13.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:12.983+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:13.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:12.983+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:13.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:12.983+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:13.455 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:13.182+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:13.455 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:13.182+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:13.456 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:13.182+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:13.456 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:13.182+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:13.457 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:13.182+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:50.077350+0000 front 2023-10-11T16:44:50.077369+0000 (oldest deadline 2023-10-11T16:45:12.977526+0000) 2023-10-11T16:45:13.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:13 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:13.136+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:13.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:13 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:13.136+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:13.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:13 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:13.136+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:13.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:13 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:13.136+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:13.585 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:13 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:13.136+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:13.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:13 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:13.555+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:13.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:13 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:13.555+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:13.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:13 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:13.555+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:13.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:13 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:13.555+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:13.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:13 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:13.555+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:14.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:14 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:14.200+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:14.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:14 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:14.200+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:14.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:14 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:14.200+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:14.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:14 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:14.200+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:14.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:14 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:14.200+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:50.077350+0000 front 2023-10-11T16:44:50.077369+0000 (oldest deadline 2023-10-11T16:45:12.977526+0000) 2023-10-11T16:45:14.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:13.991+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:14.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:13.991+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:14.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:13.991+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:14.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:13.991+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:14.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:13 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:13.991+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:14.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:14 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:14.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:14.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:14 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:14.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:14.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:14 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:14.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:14.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:14 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:14.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:14.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:14 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:14.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:14.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:14 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:14.598+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:14.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:14 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:14.598+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:14.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:14 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:14.598+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:14.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:14 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:14.598+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:14.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:14 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:14.598+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:14.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:14 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:14.598+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:15.452 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:15.155+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.770021+0000 front 2023-10-11T16:44:17.770072+0000 (oldest deadline 2023-10-11T16:44:43.070102+0000) 2023-10-11T16:45:15.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:15.155+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:27.472839+0000 front 2023-10-11T16:44:27.472893+0000 (oldest deadline 2023-10-11T16:44:49.173020+0000) 2023-10-11T16:45:15.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:15.155+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.474395+0000 front 2023-10-11T16:44:34.474213+0000 (oldest deadline 2023-10-11T16:45:00.274566+0000) 2023-10-11T16:45:15.453 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:15.155+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.375747+0000 front 2023-10-11T16:44:44.375760+0000 (oldest deadline 2023-10-11T16:45:07.275842+0000) 2023-10-11T16:45:15.454 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8[149143]: 2023-10-11T16:45:15.155+0000 7f2a2bde3700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:50.077350+0000 front 2023-10-11T16:44:50.077369+0000 (oldest deadline 2023-10-11T16:45:12.977526+0000) 2023-10-11T16:45:15.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:15.005+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:15.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:15.005+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:15.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:15.005+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:15.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:15.005+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:15.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:15 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:15.005+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:15.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:15 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:15.113+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:15.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:15 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:15.113+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:15.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:15 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:15.113+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:15.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:15 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:15.113+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:15.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:15 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:15.113+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:15.885 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:15 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:15.615+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:15.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:15 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:15.615+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:15.886 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:15 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:15.615+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:15.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:15 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:15.615+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:15.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:15 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:15.615+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:15.887 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:15 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:15.615+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:16.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:16 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:16.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:16.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:16 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:16.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:16.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:16 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:16.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:16.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:16 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:16.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:16.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:16 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:16.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:16.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:16.083+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:16.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:16.083+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:16.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:16.083+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:16.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:16.083+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:16.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:16.083+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:16.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:16 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:16.083+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:16.952 INFO:journalctl@ceph.osd.8.smithi150.stdout:Oct 11 16:45:16 smithi150 bash[165255]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-8 2023-10-11T16:45:17.135 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:16 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:16.657+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:17.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:16 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:16.657+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:17.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:16 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:16.657+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:17.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:16 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:16.657+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:17.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:16 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:16.657+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:17.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:16 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:16.657+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:17.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:17 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:17.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:17.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:17 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:17.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:17.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:17 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:17.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:17.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:17 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:17.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:17.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:17 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:17.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:17.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:17 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:17.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:17.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:17 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:17.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:17.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:17 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:17.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:17.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:17 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:17.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:17.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:17 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:17.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:17.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:17 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:17.132+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:17.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.8.service' 2023-10-11T16:45:17.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:17.862 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-10-11T16:45:17.862 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-10-11T16:45:17.863 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.9 2023-10-11T16:45:17.948 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:17 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:17.694+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:17.948 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:17 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:17.694+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:17.948 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:17 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:17.694+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:17.949 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:17 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:17.694+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:17.949 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:17 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:17.694+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:17.949 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:17 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:17.694+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:18.202 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:18 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:18.034+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:18.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:18 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:18.034+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:18.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:18 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:18.034+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:18.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:18 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:18.034+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:18.203 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:18 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:18.034+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:18.375 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:17 smithi111 systemd[1]: Stopping Ceph osd.9 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:18.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:18 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:18.168+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:18.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:18 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:18.168+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:18.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:18 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:18.168+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:18.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:18 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:18.168+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:18.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:18 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:18.168+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:18.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:18 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:18.168+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:18.635 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.373+0000 7fb76a247700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:18.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.373+0000 7fb76a247700 -1 osd.9 107 *** Got signal Terminated *** 2023-10-11T16:45:18.636 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.373+0000 7fb76a247700 -1 osd.9 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:45:19.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.678+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:19.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.678+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:19.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.679+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:19.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.679+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:19.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.679+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:19.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:18 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:18.679+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:19.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:19 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:19.042+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:19.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:19 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:19.042+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:19.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:19 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:19.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:19.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:19 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:19.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:19.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:19 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:19.043+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:19.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:19 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:19.188+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:19.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:19 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:19.188+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:19.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:19 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:19.188+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:19.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:19 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:19.188+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:19.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:19 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:19.188+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:19.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:19 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:19.188+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:20.135 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:19.641+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:20.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:19.641+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:20.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:19.641+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:20.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:19.641+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:20.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:19.641+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:20.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:19 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:19.642+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:20.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:20 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:20.025+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:20.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:20 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:20.025+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:20.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:20 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:20.025+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:20.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:20 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:20.025+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:20.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:20 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:20.025+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:20.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:20 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:20.207+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:20.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:20 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:20.207+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:20.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:20 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:20.207+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:20.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:20 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:20.207+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:20.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:20 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:20.207+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:20.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:20 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:20.207+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:21.135 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:20 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:20.683+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:21.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:20 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:20.683+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:21.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:20 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:20.683+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:21.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:20 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:20.683+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:21.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:20 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:20.683+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:21.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:20 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:20.683+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:21.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:21 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:21.056+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:21.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:21 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:21.056+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:21.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:21 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:21.056+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:21.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:21 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:21.056+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:21.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:21 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:21.056+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:21.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:21 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:21.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:21.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:21 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:21.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:21.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:21 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:21.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:21.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:21 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:21.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:21.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:21 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:21.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:21.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:21 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:21.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:22.135 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:21 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:21.638+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:22.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:21 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:21.638+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:22.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:21 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:21.638+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:22.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:21 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:21.638+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:22.138 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:21 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:21.638+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:22.138 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:21 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:21.638+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:22.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:22 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:22.095+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:22.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:22 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:22.095+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:22.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:22 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:22.095+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:22.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:22 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:22.095+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:22.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:22 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:22.095+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:22.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:22 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:22.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:22.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:22 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:22.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:22.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:22 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:22.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:22.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:22 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:22.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:22.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:22 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:22.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:22.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:22 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:22.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:22.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:22 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:22.192+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:23.135 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:22 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:22.653+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:08.998928+0000 front 2023-10-11T16:44:08.998859+0000 (oldest deadline 2023-10-11T16:44:32.498987+0000) 2023-10-11T16:45:23.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:22 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:22.653+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.101782+0000 front 2023-10-11T16:44:21.101732+0000 (oldest deadline 2023-10-11T16:44:44.601875+0000) 2023-10-11T16:45:23.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:22 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:22.653+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:24.602455+0000 front 2023-10-11T16:44:24.602484+0000 (oldest deadline 2023-10-11T16:44:49.302471+0000) 2023-10-11T16:45:23.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:22 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:22.653+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:34.003546+0000 front 2023-10-11T16:44:34.003592+0000 (oldest deadline 2023-10-11T16:44:58.604217+0000) 2023-10-11T16:45:23.136 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:22 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:22.653+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:42.105366+0000 front 2023-10-11T16:44:42.105540+0000 (oldest deadline 2023-10-11T16:45:05.505945+0000) 2023-10-11T16:45:23.137 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:22 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9[127229]: 2023-10-11T16:45:22.653+0000 7fb76603c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.507615+0000 front 2023-10-11T16:44:49.507454+0000 (oldest deadline 2023-10-11T16:45:14.207410+0000) 2023-10-11T16:45:23.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:23.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:23.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:23.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:23.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:23.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:23.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:23.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:23.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:23.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:23.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:23 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:23.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:23.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:23 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:23.203+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:23.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:23 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:23.203+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:23.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:23 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:23.203+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:23.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:23 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:23.203+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:23.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:23 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:23.203+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:23.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:23 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:23.203+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:23.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:23 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:23.203+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:24.385 INFO:journalctl@ceph.osd.9.smithi111.stdout:Oct 11 16:45:24 smithi111 bash[265502]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-9 2023-10-11T16:45:24.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:24 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:24.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:24.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:24 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:24.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:24.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:24 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:24.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:24.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:24 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:24.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:24.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:24 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:24.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:24.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:24 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:24.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:24.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:24 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:24.211+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:24.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:24 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:24.211+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:24.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:24 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:24.211+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:24.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:24 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:24.211+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:24.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:24 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:24.211+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:24.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:24 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:24.211+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:24.584 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:24 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:24.211+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:25.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:25 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:25.111+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:25.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:25 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:25.111+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:25.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:25 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:25.111+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:25.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:25 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:25.111+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:25.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:25 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:25.111+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:25.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:25 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:25.111+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:25.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:25 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:25.226+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:25.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:25 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:25.226+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:25.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:25 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:25.226+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:25.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:25 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:25.226+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:25.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:25 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:25.226+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:25.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:25 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:25.226+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:25.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:25 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:25.226+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:26.097 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.9.service' 2023-10-11T16:45:26.186 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:26.187 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-10-11T16:45:26.188 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-10-11T16:45:26.188 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.10 2023-10-11T16:45:26.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:26 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:26.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:26.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:26 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:26.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:26.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:26 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:26.084+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:26.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:26 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:26.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:26.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:26 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:26.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:26.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:26 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:26.085+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:26.547 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.212+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:26.548 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.212+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:26.548 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.212+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:26.548 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.212+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:26.549 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.212+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:26.549 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.212+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:26.549 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.212+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:26.549 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 systemd[1]: Stopping Ceph osd.10 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:26.832 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.545+0000 7facc5127700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:26.832 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.545+0000 7facc5127700 -1 osd.10 107 *** Got signal Terminated *** 2023-10-11T16:45:26.832 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:26 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:26.545+0000 7facc5127700 -1 osd.10 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:45:27.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:27 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:27.055+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:27.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:27 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:27.055+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:27.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:27 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:27.055+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:27.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:27 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:27.055+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:27.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:27 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:27.055+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:27.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:27 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:27.055+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:27.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:27 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:27.235+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:27.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:27 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:27.235+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:27.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:27 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:27.235+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:27.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:27 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:27.235+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:27.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:27 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:27.235+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:27.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:27 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:27.235+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:27.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:27 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:27.235+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:28.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:28 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:28.070+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:28.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:28 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:28.070+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:28.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:28 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:28.070+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:28.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:28 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:28.070+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:28.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:28 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:28.070+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:28.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:28 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:28.070+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:28.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:28 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:28.223+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:28.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:28 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:28.223+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:28.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:28 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:28.223+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:28.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:28 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:28.223+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:28.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:28 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:28.223+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:28.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:28 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:28.223+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:28.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:28 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:28.223+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:29.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:29.089+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:29.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:29.089+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:29.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:29.089+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:29.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:29.089+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:29.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:29.089+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:29.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:29 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:29.089+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:29.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:29.254+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:29.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:29.254+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:29.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:29.254+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:29.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:29.254+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:29.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:29.254+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:29.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:29.254+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:29.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:29 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:29.254+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:30.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:30.049+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:30.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:30.049+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:30.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:30.049+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:30.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:30.049+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:30.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:30.049+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:30.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:30.049+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:30.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:30 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:30.049+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:30.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:30.267+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:30.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:30.267+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:30.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:30.267+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:30.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:30.267+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:30.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:30.267+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:30.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:30.267+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:30.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:30 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:30.267+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:31.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:31.031+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:31.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:31.031+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:31.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:31.031+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:31.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:31.031+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:31.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:31.031+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:31.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:31.031+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:31.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:31 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:31.031+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:31.581 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:31.289+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.046563+0000 front 2023-10-11T16:44:10.046473+0000 (oldest deadline 2023-10-11T16:44:33.546438+0000) 2023-10-11T16:45:31.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:31.289+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:17.647451+0000 front 2023-10-11T16:44:17.647471+0000 (oldest deadline 2023-10-11T16:44:42.947611+0000) 2023-10-11T16:45:31.582 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:31.289+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:28.550345+0000 front 2023-10-11T16:44:28.550481+0000 (oldest deadline 2023-10-11T16:44:52.650358+0000) 2023-10-11T16:45:31.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:31.289+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6814 osd.3 since back 2023-10-11T16:44:32.651019+0000 front 2023-10-11T16:44:32.650956+0000 (oldest deadline 2023-10-11T16:44:58.551073+0000) 2023-10-11T16:45:31.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:31.289+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:44.452239+0000 front 2023-10-11T16:44:44.452216+0000 (oldest deadline 2023-10-11T16:45:07.952380+0000) 2023-10-11T16:45:31.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:31.289+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:51.954030+0000 front 2023-10-11T16:44:51.953860+0000 (oldest deadline 2023-10-11T16:45:15.453720+0000) 2023-10-11T16:45:31.583 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:31 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10[153305]: 2023-10-11T16:45:31.289+0000 7facbef18700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:55.454248+0000 front 2023-10-11T16:44:55.454185+0000 (oldest deadline 2023-10-11T16:45:21.354173+0000) 2023-10-11T16:45:32.227 INFO:journalctl@ceph.osd.10.smithi130.stdout:Oct 11 16:45:32 smithi130 bash[164158]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-10 2023-10-11T16:45:32.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:32.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:32.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:32.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:32.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:32.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:32.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:32.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:32.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:32.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:32.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:32.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:32.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:32 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:32.053+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:33.387 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.10.service' 2023-10-11T16:45:33.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:33.427 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-10-11T16:45:33.427 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-10-11T16:45:33.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.11 2023-10-11T16:45:33.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.103+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:33.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.103+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:33.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.103+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:33.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.103+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:33.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.103+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:33.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.103+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:33.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.103+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:33.791 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 systemd[1]: Stopping Ceph osd.11 for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:34.105 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.789+0000 7fa60057d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:34.106 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.789+0000 7fa60057d700 -1 osd.11 107 *** Got signal Terminated *** 2023-10-11T16:45:34.106 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:33 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:33.789+0000 7fa60057d700 -1 osd.11 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-11T16:45:34.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:34.104+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:34.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:34.104+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:34.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:34.104+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:34.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:34.104+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:34.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:34.104+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:34.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:34.104+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:34.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:34 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:34.104+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:35.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:35.144+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:35.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:35.144+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:35.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:35.144+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:35.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:35.144+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:35.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:35.144+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:35.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:35.144+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:35.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:35 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:35.144+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:36.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:36.139+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:36.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:36.139+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:36.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:36.139+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:36.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:36.139+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:36.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:36.139+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:36.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:36.139+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:36.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:36 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:36.139+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:37.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:37.174+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:37.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:37.174+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:37.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:37.174+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:37.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:37.174+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:37.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:37.174+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:37.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:37.174+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:37.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:37.174+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:37.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:37 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:37.174+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6820 osd.8 since back 2023-10-11T16:45:14.050293+0000 front 2023-10-11T16:45:14.050323+0000 (oldest deadline 2023-10-11T16:45:36.350315+0000) 2023-10-11T16:45:38.452 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:38.182+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2023-10-11T16:44:10.639279+0000 front 2023-10-11T16:44:10.639293+0000 (oldest deadline 2023-10-11T16:44:36.538999+0000) 2023-10-11T16:45:38.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:38.182+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-10-11T16:44:21.140801+0000 front 2023-10-11T16:44:21.140782+0000 (oldest deadline 2023-10-11T16:44:42.840836+0000) 2023-10-11T16:45:38.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:38.182+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6804 osd.2 since back 2023-10-11T16:44:26.941858+0000 front 2023-10-11T16:44:26.941897+0000 (oldest deadline 2023-10-11T16:44:52.842031+0000) 2023-10-11T16:45:38.453 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:38.182+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-10-11T16:44:40.944743+0000 front 2023-10-11T16:44:40.944825+0000 (oldest deadline 2023-10-11T16:45:05.644822+0000) 2023-10-11T16:45:38.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:38.182+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6812 osd.5 since back 2023-10-11T16:44:49.745751+0000 front 2023-10-11T16:44:49.745827+0000 (oldest deadline 2023-10-11T16:45:13.845934+0000) 2023-10-11T16:45:38.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:38.182+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.111:6822 osd.6 since back 2023-10-11T16:44:59.548164+0000 front 2023-10-11T16:44:59.548146+0000 (oldest deadline 2023-10-11T16:45:23.047926+0000) 2023-10-11T16:45:38.454 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:38.182+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-10-11T16:45:07.049446+0000 front 2023-10-11T16:45:07.049542+0000 (oldest deadline 2023-10-11T16:45:29.349366+0000) 2023-10-11T16:45:38.455 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:38 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11[154663]: 2023-10-11T16:45:38.182+0000 7fa5f936c700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.150:6820 osd.8 since back 2023-10-11T16:45:14.050293+0000 front 2023-10-11T16:45:14.050323+0000 (oldest deadline 2023-10-11T16:45:36.350315+0000) 2023-10-11T16:45:39.702 INFO:journalctl@ceph.osd.11.smithi150.stdout:Oct 11 16:45:39 smithi150 bash[165517]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-osd-11 2023-10-11T16:45:40.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@osd.11.service' 2023-10-11T16:45:40.814 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:40.815 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-10-11T16:45:40.816 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-10-11T16:45:40.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.a 2023-10-11T16:45:41.135 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:45:40 smithi111 systemd[1]: Stopping Ceph mds.a for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:41.635 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:45:41 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-a[132687]: 2023-10-11T16:45:41.320+0000 7f1fa555b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:41.635 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:45:41 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-a[132687]: 2023-10-11T16:45:41.320+0000 7f1fa555b700 -1 mds.a *** got signal Terminated *** 2023-10-11T16:45:43.886 INFO:journalctl@ceph.mds.a.smithi111.stdout:Oct 11 16:45:43 smithi111 bash[265778]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-a 2023-10-11T16:45:43.967 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.a.service' 2023-10-11T16:45:44.054 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:44.054 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-10-11T16:45:44.055 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-10-11T16:45:44.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.d 2023-10-11T16:45:44.385 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:45:44 smithi111 systemd[1]: Stopping Ceph mds.d for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:44.885 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:45:44 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-d[131678]: 2023-10-11T16:45:44.594+0000 7f67a955e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:44.886 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:45:44 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-d[131678]: 2023-10-11T16:45:44.594+0000 7f67a955e700 -1 mds.d *** got signal Terminated *** 2023-10-11T16:45:46.885 INFO:journalctl@ceph.mds.d.smithi111.stdout:Oct 11 16:45:46 smithi111 bash[265927]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-d 2023-10-11T16:45:46.944 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.d.service' 2023-10-11T16:45:47.030 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:47.031 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-10-11T16:45:47.031 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-10-11T16:45:47.032 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.g 2023-10-11T16:45:47.386 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:45:47 smithi111 systemd[1]: Stopping Ceph mds.g for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:47.885 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:45:47 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-g[132292]: 2023-10-11T16:45:47.519+0000 7f0b50fe0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:47.886 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:45:47 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-g[132292]: 2023-10-11T16:45:47.519+0000 7f0b50fe0700 -1 mds.g *** got signal Terminated *** 2023-10-11T16:45:52.135 INFO:journalctl@ceph.mds.g.smithi111.stdout:Oct 11 16:45:51 smithi111 bash[266073]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-g 2023-10-11T16:45:52.270 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.g.service' 2023-10-11T16:45:52.353 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:52.353 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-10-11T16:45:52.354 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-10-11T16:45:52.354 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.j 2023-10-11T16:45:52.635 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:45:52 smithi111 systemd[1]: Stopping Ceph mds.j for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:53.135 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:45:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-j[131288]: 2023-10-11T16:45:52.844+0000 7fa750261700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:53.136 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:45:52 smithi111 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-j[131288]: 2023-10-11T16:45:52.844+0000 7fa750261700 -1 mds.j *** got signal Terminated *** 2023-10-11T16:45:55.635 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:45:55 smithi111 bash[266215]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-j 2023-10-11T16:45:55.924 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.j.service' 2023-10-11T16:45:55.985 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:45:55 smithi111 systemd[1]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.j.service: Succeeded. 2023-10-11T16:45:55.985 INFO:journalctl@ceph.mds.j.smithi111.stdout:Oct 11 16:45:55 smithi111 systemd[1]: Stopped Ceph mds.j for 495f265e-6852-11ee-8db6-212e2dc638e7. 2023-10-11T16:45:56.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:45:56.007 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-10-11T16:45:56.008 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-10-11T16:45:56.008 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.b 2023-10-11T16:45:56.331 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:45:56 smithi130 systemd[1]: Stopping Ceph mds.b for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:45:56.831 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:45:56 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b[157084]: 2023-10-11T16:45:56.366+0000 7f5ea545e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:45:56.832 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:45:56 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b[157084]: 2023-10-11T16:45:56.366+0000 7f5ea545e700 -1 mds.b *** got signal Terminated *** 2023-10-11T16:45:59.971 INFO:journalctl@ceph.mds.b.smithi130.stdout:Oct 11 16:45:59 smithi130 bash[164422]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-b 2023-10-11T16:45:59.976 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.b.service' 2023-10-11T16:46:00.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:46:00.014 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-10-11T16:46:00.014 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-10-11T16:46:00.015 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.e 2023-10-11T16:46:00.331 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:46:00 smithi130 systemd[1]: Stopping Ceph mds.e for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:46:00.831 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:46:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-e[156401]: 2023-10-11T16:46:00.382+0000 7f953ea8a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:46:00.832 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:46:00 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-e[156401]: 2023-10-11T16:46:00.382+0000 7f953ea8a700 -1 mds.e *** got signal Terminated *** 2023-10-11T16:46:04.010 INFO:journalctl@ceph.mds.e.smithi130.stdout:Oct 11 16:46:03 smithi130 bash[164566]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-e 2023-10-11T16:46:04.016 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.e.service' 2023-10-11T16:46:04.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:46:04.052 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-10-11T16:46:04.052 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-10-11T16:46:04.053 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.h 2023-10-11T16:46:04.331 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:46:04 smithi130 systemd[1]: Stopping Ceph mds.h for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:46:04.831 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:46:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-h[156742]: 2023-10-11T16:46:04.431+0000 7f14fbc62700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:46:04.832 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:46:04 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-h[156742]: 2023-10-11T16:46:04.431+0000 7f14fbc62700 -1 mds.h *** got signal Terminated *** 2023-10-11T16:46:07.800 INFO:journalctl@ceph.mds.h.smithi130.stdout:Oct 11 16:46:07 smithi130 bash[164713]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-h 2023-10-11T16:46:07.989 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.h.service' 2023-10-11T16:46:08.023 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:46:08.024 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-10-11T16:46:08.024 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-10-11T16:46:08.025 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.k 2023-10-11T16:46:08.331 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:46:08 smithi130 systemd[1]: Stopping Ceph mds.k for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:46:08.831 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:46:08 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-k[156062]: 2023-10-11T16:46:08.398+0000 7f394ed08700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:46:08.832 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:46:08 smithi130 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-k[156062]: 2023-10-11T16:46:08.398+0000 7f394ed08700 -1 mds.k *** got signal Terminated *** 2023-10-11T16:46:12.563 INFO:journalctl@ceph.mds.k.smithi130.stdout:Oct 11 16:46:12 smithi130 bash[164851]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-k 2023-10-11T16:46:12.569 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.k.service' 2023-10-11T16:46:12.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:46:12.603 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-10-11T16:46:12.604 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-10-11T16:46:12.605 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.c 2023-10-11T16:46:12.952 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:46:12 smithi150 systemd[1]: Stopping Ceph mds.c for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:46:13.452 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:46:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-c[158181]: 2023-10-11T16:46:12.968+0000 7f878cb1c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:46:13.452 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:46:12 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-c[158181]: 2023-10-11T16:46:12.968+0000 7f878cb1c700 -1 mds.c *** got signal Terminated *** 2023-10-11T16:46:15.829 INFO:journalctl@ceph.mds.c.smithi150.stdout:Oct 11 16:46:15 smithi150 bash[165802]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-c 2023-10-11T16:46:16.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.c.service' 2023-10-11T16:46:16.053 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:46:16.053 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-10-11T16:46:16.054 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-10-11T16:46:16.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.f 2023-10-11T16:46:16.452 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:46:16 smithi150 systemd[1]: Stopping Ceph mds.f for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:46:16.952 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:46:16 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-f[157839]: 2023-10-11T16:46:16.476+0000 7f09efeef700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:46:16.952 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:46:16 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-f[157839]: 2023-10-11T16:46:16.476+0000 7f09efeef700 -1 mds.f *** got signal Terminated *** 2023-10-11T16:46:19.952 INFO:journalctl@ceph.mds.f.smithi150.stdout:Oct 11 16:46:19 smithi150 bash[165939]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-f 2023-10-11T16:46:20.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.f.service' 2023-10-11T16:46:20.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:46:20.116 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-10-11T16:46:20.116 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-10-11T16:46:20.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.i 2023-10-11T16:46:20.454 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:46:20 smithi150 systemd[1]: Stopping Ceph mds.i for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:46:20.952 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:46:20 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-i[158523]: 2023-10-11T16:46:20.514+0000 7f506573a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:46:20.952 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:46:20 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-i[158523]: 2023-10-11T16:46:20.514+0000 7f506573a700 -1 mds.i *** got signal Terminated *** 2023-10-11T16:46:23.938 INFO:journalctl@ceph.mds.i.smithi150.stdout:Oct 11 16:46:23 smithi150 bash[166087]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-i 2023-10-11T16:46:23.945 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.i.service' 2023-10-11T16:46:23.980 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:46:23.981 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-10-11T16:46:23.981 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-10-11T16:46:23.982 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.l 2023-10-11T16:46:24.202 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:46:24 smithi150 systemd[1]: Stopping Ceph mds.l for 495f265e-6852-11ee-8db6-212e2dc638e7... 2023-10-11T16:46:24.702 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:46:24 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-l[157491]: 2023-10-11T16:46:24.364+0000 7f14e2bac700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-11T16:46:24.702 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:46:24 smithi150 ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-l[157491]: 2023-10-11T16:46:24.364+0000 7f14e2bac700 -1 mds.l *** got signal Terminated *** 2023-10-11T16:46:26.320 INFO:journalctl@ceph.mds.l.smithi150.stdout:Oct 11 16:46:26 smithi150 bash[166227]: ceph-495f265e-6852-11ee-8db6-212e2dc638e7-mds-l 2023-10-11T16:46:26.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-495f265e-6852-11ee-8db6-212e2dc638e7@mds.l.service' 2023-10-11T16:46:26.547 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-11T16:46:26.547 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-10-11T16:46:26.548 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 --force --keep-logs 2023-10-11T16:46:30.872 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 --force --keep-logs 2023-10-11T16:46:32.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 --force --keep-logs 2023-10-11T16:46:34.795 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:46:34.876 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:46:34.913 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-11T16:46:34.943 INFO:tasks.cephadm:Archiving crash dumps... 2023-10-11T16:46:34.945 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi111/crash 2023-10-11T16:46:34.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash -- . 2023-10-11T16:46:35.030 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-11T16:46:35.030 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2023-10-11T16:46:35.032 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi130/crash 2023-10-11T16:46:35.033 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash -- . 2023-10-11T16:46:35.098 INFO:teuthology.orchestra.run.smithi130.stderr:tar: /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-11T16:46:35.098 INFO:teuthology.orchestra.run.smithi130.stderr:tar: Error is not recoverable: exiting now 2023-10-11T16:46:35.100 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi150/crash 2023-10-11T16:46:35.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash -- . 2023-10-11T16:46:35.135 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-11T16:46:35.135 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2023-10-11T16:46:35.136 INFO:tasks.cephadm:Checking cluster log for badness... 2023-10-11T16:46:35.137 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-10-11T16:46:35.213 INFO:teuthology.orchestra.run.smithi111.stderr:grep: /var/log/ceph/495f265e-6852-11ee-8db6-212e2dc638e7/ceph.log: No such file or directory 2023-10-11T16:46:35.219 INFO:tasks.cephadm:Compressing logs... 2023-10-11T16:46:35.220 DEBUG:teuthology.orchestra.run.smithi111:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:46:35.225 DEBUG:teuthology.orchestra.run.smithi130:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:46:35.228 DEBUG:teuthology.orchestra.run.smithi150:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:46:35.256 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-10-11T16:46:35.288 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-10-11T16:46:35.301 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-10-11T16:51:56.675 INFO:tasks.cephadm:Archiving logs... 2023-10-11T16:51:56.676 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi111/log 2023-10-11T16:51:56.678 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-11T16:52:11.916 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi130/log 2023-10-11T16:52:11.917 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-11T16:52:35.369 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi150/log 2023-10-11T16:52:35.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-11T16:53:11.819 INFO:tasks.cephadm:Removing cluster... 2023-10-11T16:53:11.820 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 --force 2023-10-11T16:53:13.386 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 --force 2023-10-11T16:53:14.432 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 495f265e-6852-11ee-8db6-212e2dc638e7 --force 2023-10-11T16:53:15.570 INFO:tasks.cephadm:Removing cephadm ... 2023-10-11T16:53:15.571 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-11T16:53:15.620 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-11T16:53:15.641 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-11T16:53:15.659 INFO:tasks.cephadm:Teardown complete 2023-10-11T16:53:15.660 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-11T16:53:15.673 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-11T16:53:15.673 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-11T16:53:15.678 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-11T16:53:15.684 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-11T16:53:15.825 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-11T16:53:15.838 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-11T16:53:15.839 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T16:53:15.844 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T16:53:15.847 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T16:53:15.863 INFO:teuthology.orchestra.run.smithi150.stderr:bash: ntpq: command not found 2023-10-11T16:53:15.864 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2023-10-11T16:53:15.869 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T16:53:15.869 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2023-10-11T16:53:15.870 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 -1612us[-1612us] +/- 72ms 2023-10-11T16:53:15.870 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -841us[ -855us] +/- 56ms 2023-10-11T16:53:15.870 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +2595us[+2581us] +/- 72ms 2023-10-11T16:53:15.870 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:53:15.882 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2023-10-11T16:53:15.898 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T16:53:15.898 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-10-11T16:53:15.899 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 0 -1595us[-1595us] +/- 72ms 2023-10-11T16:53:15.899 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -814us[ -825us] +/- 56ms 2023-10-11T16:53:15.899 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +2603us[+2591us] +/- 72ms 2023-10-11T16:53:15.899 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:53:15.905 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T16:53:15.905 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2023-10-11T16:53:15.906 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 -1587us[-1587us] +/- 72ms 2023-10-11T16:53:15.906 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -806us[ -791us] +/- 56ms 2023-10-11T16:53:15.906 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +2599us[+2614us] +/- 72ms 2023-10-11T16:53:15.907 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T16:53:15.908 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-11T16:53:15.921 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-11T16:53:15.921 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-11T16:53:15.955 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-11T16:53:16.218 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-11T16:53:16.322 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-11T16:53:16.431 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T16:53:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:53:16.508 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T16:53:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:53:16.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T16:53:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:53:16.577 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-11T16:53:16.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-11T16:53:16.630 INFO:teuthology.task.internal:Duration was 2827.988062 seconds 2023-10-11T16:53:16.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-11T16:53:16.681 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-11T16:53:16.682 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-11T16:53:16.687 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-11T16:53:16.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-11T16:53:16.736 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T16:53:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T16:53:16.848 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T16:53:17.242 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-11T16:53:17.242 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2023-10-11T16:53:17.243 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-11T16:53:17.336 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2023-10-11T16:53:17.337 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-11T16:53:17.362 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2023-10-11T16:53:17.362 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-11T16:53:17.394 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-11T16:53:17.395 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:53:17.400 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:53:17.405 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T16:53:17.554 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-11T16:53:17.555 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-11T16:53:17.559 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-11T16:53:17.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-11T16:53:17.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-11T16:53:17.747 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-11T16:53:17.748 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-11T16:53:17.826 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-11T16:53:17.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-11T16:53:17.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-11T16:53:17.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:53:17.899 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:53:17.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:53:17.925 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2023-10-11T16:53:17.948 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2023-10-11T16:53:17.984 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2023-10-11T16:53:18.021 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:53:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:53:18.064 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:53:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:53:18.080 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-11T16:53:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T16:53:18.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-11T16:53:18.110 INFO:teuthology.task.internal:Transferring archived files... 2023-10-11T16:53:18.110 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi111 2023-10-11T16:53:18.111 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-11T16:53:18.195 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi130 2023-10-11T16:53:18.196 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-11T16:53:18.235 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971/remote/smithi150 2023-10-11T16:53:18.236 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-11T16:53:18.277 INFO:teuthology.task.internal:Removing archive directory... 2023-10-11T16:53:18.278 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-11T16:53:18.283 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-11T16:53:18.286 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-11T16:53:18.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-11T16:53:18.342 INFO:teuthology.task.internal:Not uploading archives. 2023-10-11T16:53:18.343 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-11T16:53:18.407 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-11T16:53:18.408 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-11T16:53:18.413 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-11T16:53:18.415 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-11T16:53:18.430 INFO:teuthology.orchestra.run.smithi130.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 11 16:53 /home/ubuntu/cephtest 2023-10-11T16:53:18.432 INFO:teuthology.orchestra.run.smithi150.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 11 16:53 /home/ubuntu/cephtest 2023-10-11T16:53:18.452 INFO:teuthology.orchestra.run.smithi111.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 11 16:53 /home/ubuntu/cephtest 2023-10-11T16:53:18.456 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-11T16:53:18.469 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-11T16:53:18.569 INFO:teuthology.nuke:Checking targets against current locks 2023-10-11T16:53:18.598 DEBUG:teuthology.nuke:shortname: smithi111 2023-10-11T16:53:18.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T16:53:18.623 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 15:56:09.277490', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T16:53:18.629 DEBUG:teuthology.nuke:shortname: smithi130 2023-10-11T16:53:18.630 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T16:53:18.659 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 15:56:09.279473', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T16:53:18.660 DEBUG:teuthology.nuke:shortname: smithi150 2023-10-11T16:53:18.661 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T16:53:18.691 INFO:teuthology.orchestra.console.smithi111:Power off 2023-10-11T16:53:18.692 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-11T16:53:18.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-11_15:53:20-fs:workload-wip-choffman-62793-distro-default-smithi/7421971', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 15:56:09.281207', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-11T16:53:18.716 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2023-10-11T16:53:18.716 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:18.736 INFO:teuthology.orchestra.console.smithi130:Power off 2023-10-11T16:53:18.737 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-11T16:53:18.750 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-10-11T16:53:18.759 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2023-10-11T16:53:18.760 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:18.775 INFO:teuthology.orchestra.console.smithi150:Power off 2023-10-11T16:53:18.776 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-11T16:53:18.785 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2023-10-11T16:53:18.799 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2023-10-11T16:53:18.800 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:18.823 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-11T16:53:22.752 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:22.787 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:22.825 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:22.863 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2023-10-11T16:53:22.898 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2023-10-11T16:53:22.936 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-11T16:53:26.866 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:26.899 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:26.937 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:26.976 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2023-10-11T16:53:27.010 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2023-10-11T16:53:27.049 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-10-11T16:53:27.078 INFO:teuthology.orchestra.console.smithi111:Power off completed 2023-10-11T16:53:31.014 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:31.050 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-11T16:53:31.126 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2023-10-11T16:53:31.162 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2023-10-11T16:53:31.227 INFO:teuthology.orchestra.console.smithi130:Power off completed 2023-10-11T16:53:31.264 INFO:teuthology.orchestra.console.smithi150:Power off completed 2023-10-11T16:53:31.371 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} duration: 2827.9880623817444 failure_reason: 'Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi111 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f7a9d091654926539d989dd4c83dfee517abc9e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh''' flavor: default owner: scheduled_choffman@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4dc6a431bc2c451d9a5061e5357d01a8 status: fail success: false 2023-10-11T16:53:31.372 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-11T16:53:31.456 INFO:teuthology.run:FAIL